Apparatus and method for recommending treatment guide information for vascular disease on basis of medical image

Abstract

A vascular therapy recommendation apparatus according to an embodiment may comprise: a processor that selects a target vascular image including a target blood vessel from among medical images obtained by photographing the target blood vessel in a plurality of directions, on the basis of a result of applying a first machine learning model to the medical images, selects a representative frame image including the target blood vessel from among a plurality of frame images included in the selected target vascular image, on the basis of a result of applying a second machine learning model to the plurality of frame images, generates input data including time series information by stacking the selected representative frame image, a previous frame of the representative frame image, and a subsequent frame of the representative frame image in order of photographing time, and recommends treatment guide information for the target blood vessel among pieces of candidate treatment guide information by applying a third machine learning model to the stacked input data; and a display for outputting the treatment guide information.

Description

Device and method for recommending treatment guide information for vascular diseases based on medical images

[0001] Hereinafter, a technology for providing treatment guidance information for a patient based on analysis of medical images is disclosed.

[0002] The coronary arteries, which supply blood to the myocardium, are composed of three blood vessels: the left anterior descending artery, the left circumflex artery, and the right coronary artery. Coronary artery disease is a condition in which one of the three blood vessels described above becomes narrower or occluded compared to the normal coronary artery. Patients with coronary artery disease may experience severe chest pain, and in severe cases, it can even lead to sudden death. Furthermore, patients with coronary artery disease can have a wide spectrum of diseases, including stable angina, non-ST segment elevation acute coronary syndrome (NSTE-ACS), and ST-segment elevation myocardial infarction. Coronary angiography (CAG) is the most direct method for examining the coronary arteries of patients with the various diseases described above, and is a test method for determining whether the disease is single-vessel or multi-vessel. Treatment methods for CAD can be broadly divided into medical treatment (e.g., drug therapy), percutaneous coronary intervention (PCI), and coronary artery bypass grafting (CABG). These treatment methods can be determined based on the extent and severity of the vascular disease as diagnosed through medical imaging.For example, CABG has lower mortality rates, lower rates of major adverse cardiovascular events (MACE), and better revascularization rates than PCI in the treatment of multivessel coronary artery disease. However, based on advancements in treatment technologies (e.g., the development of drugs and drug-eluting stents), PCI has become a partial replacement for CABG. Because PCI can partially replace CABG, choosing a treatment for coronary artery disease revascularization has become more difficult. The existing method for determining the treatment of coronary artery disease was the SYNTAX (SYNergy between PCI with TAXUS and cardiac surgery) technique, which was based on a complexity score of coronary artery disease calculated based on coronary angiography (CAG) length occlusion, degree of calcification, etc., and the higher the score, the more favorable CABG was. The SYNTAX technique fails to reflect the individual characteristics of each patient, and there are some differences among scoring specialists, which reduces objectivity in determining treatment methods. Therefore, a technology is needed that analyzes medical images using machine learning models and, based on these images, considers more variables (e.g., patient report results, blood sampling, etc.) to provide treatment guidance for patients.

[0003] In one embodiment, a vascular treatment recommendation device may include a processor that selects a target blood vessel image including the target blood vessel from among medical images based on a result of applying a first machine learning model to medical images captured from a plurality of directions of the target blood vessel, selects a representative frame image including the target blood vessel from among the plurality of frame images based on a result of applying a second machine learning model to a plurality of frame images included in the selected target blood vessel image, stacks the selected representative frame image, a previous frame of the representative frame image, and a subsequent frame of the representative frame in the order of capturing time, thereby generating input data including time series information, and applies a third machine learning model to the stacked input data, thereby recommending treatment guide information for the target blood vessel from among candidate treatment guide information; and a display that outputs the treatment guide information.

[0004] The processor applies the first machine learning model to the medical images to calculate a first probability value indicating a probability that a blood vessel in a first frame image included in the medical images corresponds to the target blood vessel, calculates a second probability value indicating a probability that a blood vessel in a second frame image included in the medical images corresponds to the target blood vessel, and compares an average value of the first probability value and the second probability value with a predetermined threshold value, thereby selecting the medical image as the target blood vessel image.

[0005] The processor may select the corresponding medical image as the target blood vessel image when the average value of the first probability value and the second probability value is greater than or equal to the predetermined threshold value.

[0006] The processor, when the medical images include different target blood vessels, applies the first machine learning model to the medical images to calculate a third probability value indicating a probability that a blood vessel in a first frame image included in the corresponding medical images corresponds to the different target blood vessel, calculates a fourth probability value indicating a probability that a blood vessel in a second frame image included in the corresponding medical images corresponds to the different target blood vessel, and compares a larger value among an average value of the first probability value and the second probability value, and an average value of the third probability value and the fourth probability value, with the predetermined threshold value, thereby selecting the corresponding medical image as the target blood vessel image.

[0007] The processor may pre-train the first machine learning model including the Resnet-50 model based on the ImageNet dataset, and perform supervised learning on the pre-trained first machine learning model based on a training data set including frame images labeled with the target blood vessels.

[0008] The processor may apply a second machine learning model to a plurality of frame images included in the selected target blood vessel image, thereby calculating probability values ​​indicating a probability that a blood vessel in each of the plurality of frame images corresponds to the target blood vessel, and select the representative frame image based on a comparison result of the calculated probability values.

[0009] The above processor can select a frame image corresponding to the largest value among the calculated probability values ​​as the representative frame image.

[0010] The processor may supervise the second machine learning model based on a first training data set including frame images labeled with the representative frame images, and may further train the second machine learning model based on a second training data set including frame images pseudo-labeled with the representative frame images for a predetermined ratio of frames among a plurality of frames included in the target blood vessel image.

[0011] The processor may further train the second machine learning model based on the second training data set, and then generate a third training data set including more pseudo-labeled frame images than the second training data set by increasing the predetermined ratio, and train the second machine learning model based on the third training data set.

[0012] The above processor can generate the input data by stacking the selected representative frame image, the first frame image and the last frame image included in the target blood vessel image in chronological order.

[0013] The processor can generate a training data set based on the stacked input data and identification information about the target patient having the target blood vessel, and train the third machine learning model based on the training data set.

[0014] FIG. 1 illustrates an electronic device for recommending a vascular treatment method, according to one embodiment.

[0015] FIG. 2 schematically illustrates, as an example, a process in which a vascular treatment recommendation device recommends treatment guide information for a patient with multivessel coronary artery disease.

[0016] FIG. 3 is a drawing specifically explaining how a vascular treatment recommendation device according to one embodiment selects a target vascular image.

[0017] FIG. 4 illustrates a vascular treatment recommendation device training a machine learning model to select a target image from among a plurality of medical images, according to one embodiment.

[0018] FIG. 5 is a diagram illustrating a device for recommending a vascular treatment method according to one embodiment of the present invention selecting a representative frame image from a target vascular image.

[0019] FIG. 6 illustrates a vascular treatment recommendation device according to one embodiment training a machine learning model to select a representative frame image from among multiple frame images included in one medical image.

[0020] FIG. 7 is a drawing specifically explaining how a vascular treatment recommendation device according to one embodiment generates input data and applies a machine learning model to the input data to recommend treatment guide information.

[0021] FIG. 8 illustrates a vascular treatment recommendation device according to one embodiment training a machine learning model to recommend treatment guide information through medical images.

[0022] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives within the technical concepts described in the embodiments.

[0023] Although terms such as "first" or "second" may be used to describe various components, these terms should be interpreted solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0024] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0025] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0026] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0027] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0028]

[0029] FIG. 1 illustrates an electronic device for recommending a vascular treatment method, according to one embodiment.

[0030] The vascular treatment recommendation device (100) can recommend treatment guide information for a patient based on medical images of the patient. For example, the vascular treatment recommendation device (100) can recommend at least one of percutaneous coronary intervention (hereinafter, 'PCI') treatment guide information or coronary artery bypass grafting (hereinafter, 'CABG') treatment guide information for the multivessel coronary artery disease of the patient based on CAG image data (130) of the patient with multivessel coronary artery disease. The treatment guide information provided by the vascular treatment recommendation device (100) based on the medical images of the patient is not limited thereto, but the following description focuses on treatment guide information for multivessel coronary artery disease.

[0031] The vascular treatment recommendation device (100) may include a memory (110), a processor (120), and a display (not shown).

[0032] The memory (110) can store medical image data. For example, the memory (110) can store CAG image data (130). In addition, the memory (110) can store a machine learning model and instructions for operating the machine learning model. For example, the memory (110) can store a ResNet-50 model, and further, can store instructions for operating the ResNet-50 model. For example, the memory (110) can store an instruction for performing a convolution operation for operating the ResNet-50 model, and the memory (110) can store an instruction for performing a residual operation. For reference, the CAG image data (130) can represent an X-ray image of a patient's coronary artery taken through a C-arm. For example, the CAG image data (130) may include image data taken with a C-arm for patients who underwent coronary artery disease examination during a specific period (e.g., from January 2003 to December 2018). For example, the CAG image data (130) may include image data extracted from a file in the Digital Imaging and Communications in Medicine (DICOM) standard. DICOM may represent a standard specification for storing and transmitting data related to images generated in medical electronic devices.

[0033] The processor (120) can analyze medical image data and, based on the analysis results of the medical image data, recommend treatment guide information for treating the patient. For example, the processor (120) can generate analysis results for an X-ray image of a target patient.

[0034] Specifically, the processor (120) may select a target blood vessel image (145) including a target blood vessel among the medical images based on a result of applying the first machine learning model (140) to medical images of the target blood vessel captured from multiple directions. For example, the processor (120) may select a target blood vessel image (145) including a target blood vessel from the CAG image data (130) based on a result of applying the first machine learning model (140) to the CAG image data (130). The processor (120) may input the CAG image data (130) itself into the first machine learning model (140), or may preprocess the CAG image data (130) and input it into the first machine learning model (140) in the form of first input data (135). In the following, the description focuses on the case where the processor (120) preprocesses CAG image data (130) into first input data (135) and inputs it into the first machine learning model (140).

[0035] For example, the processor (120) can load CAG image data (130) from a DICOM standard file stored in the memory (110). The processor (120) can resize each frame image included in the CAG image data (130) from a size of 512×512 to a size of 224×224. As will be described below, when the first machine learning model (140) is based on a model of the ResNet-50 structure, the input size needs to be resized to a size of 224×224. Therefore, the processor (120) resizes each frame image included in the CAG image data (130) from a size of 512×512 to a size of 224×224, thereby enabling the internal architecture of the first machine learning model (140) that is fine-tuned for images of the same dimension to effectively process the resized input. The processor (120) can convert the size of each frame image included in the CAG image data (130) (e.g., from a size of 512×512 to a size of 224×224) and then convert the CAG image data (130) into a tensor type. By converting the CAG image data (130) into a tensor type, the processor (120) can perform parallel operations through vectorization of input data. The processor (120) can preprocess the CAG image data (130) by normalizing the CAG image data (130) converted into a tensor type. For example, the processor (120) can normalize the converted CAG image data (130) based on the mean and standard deviation of a data set related to .

[0036] The processor (120) can select a target blood vessel image (145) from among the CAG image data (130) by applying a first machine learning model (140) to the first input data (135) of which the CAG image data (130) is preprocessed. The first machine learning model (140) can correspond to a model that receives a plurality of medical images and selects a desired medical image from among them. For example, the first machine learning model (140) can correspond to a model that selects an image in which a target blood vessel is the clearest from among medical images (e.g., CAG image data (130)) captured over a certain period of time with a C-arm. In other words, the first machine learning model (140) may be a model that selects the clearest image of the left coronary artery (hereinafter, 'LCA') or the right coronary artery (hereinafter, 'RCA') among a plurality of medical images, and the processor (120) may select the target blood vessel image (145) as the clearest image of the target blood vessel selected by the first machine learning model (140). The method by which the processor (120) selects the target blood vessel image (145) based on the first machine learning model (140) is described in detail in FIG. 3 below.

[0037] For reference, the first to third machine learning models (140, 150, 160) may include a model having a structure in which three FC layers (Fully Connected Layers) are combined with a model having a ResNet-50 structure. The ResNet-50 structure may be a structure including multiple convolutional layers and residual blocks on which convolution operations are performed. The model having the ResNet-50 structure may be configured with a convolution neural network (CNN) layer and may perform the role of analyzing an image. The FC layer combined with the ResNet-50 model may perform the role of mapping high-dimensional features for the input image extracted from ResNet-5 to labels. In other words, the FC layer may perform an operation of flattening a two-dimensional array image into a one-dimensional vector (or feature). Since the first to third machine learning models (140, 150, 160) use the same structure, the amount of computing resources can be reduced when the vascular treatment recommendation device (100) loads pre-learned weights.

[0038] The processor (120) may select a representative frame image including a target blood vessel from among the plurality of frame images based on a result of applying the second machine learning model (150) to the plurality of frame images included in the selected target blood vessel image (145). The target blood vessel image (145) may include a plurality of frame images. For example, the target blood vessel image (145) may be a set of images periodically captured at predetermined intervals. For example, the target blood vessel image (145) may include frame images captured every second. The processor (120) may apply the second machine learning model (150) to the target blood vessel image (145) to select a representative frame image in which the target blood vessel appears most clearly. For example, the processor (120) may select a frame image in which the LCA or RCA is most clear from among the frame images included in the target blood vessel image (145). A specific method for selecting a target frame image from a target blood vessel image (145) based on a second machine learning model (150) by a processor (120) is described in detail in FIG. 5 below.

[0039] The processor (120) can generate input data including time series information by stacking the selected representative frame image, the previous frame of the representative frame image, and the subsequent frame of the representative frame in the order of shooting time. For example, the processor (120) can generate second input data (155) by stacking the selected representative frame image, the first frame image, and the last frame image included in the target blood vessel image (145) in the order of time. For reference, the CAG images included in the CAG image data (130) may have different lengths for each CAG image. For example, the CAG images included in the CAG image data (130) may have different shooting times, and thus, the number of frame images included in the CAG images may be different from each other. For example, assuming that a CAG image corresponds to a set of images captured at a 1-second cycle, a first CAG image captured for 10 seconds may include 10 frame images, and a second CAG image captured for 20 seconds may include 20 frame images. In other words, the first CAG image and the second CAG image may have different image lengths (or the number of frame images included). The process of padding CAG image data (130) of different image lengths to match the same image length may require more energy and computation time. In addition, using all frame images included in the CAG image data (130) may well represent the changes in the captured frame images over time, but may require more time and energy for data processing. Therefore, the processor (120) may generate the second input data (155) by stacking the previous frame of the representative frame image and the subsequent frame of the representative frame image in the shooting time order on the selected representative frame image.The second input data (155) may include not only a representative frame in which the target blood vessel is clearly shown, but also data obtained by stacking a previous frame taken at a time before the representative frame was taken and a subsequent frame taken at a time after the representative frame was taken, and may include information about the target blood vessel that changes over time.

[0040] The processor (120) can recommend treatment guide information (165) for a target blood vessel among candidate treatment guide information by applying the third machine learning model (160) to the stacked input data. For example, the processor (120) can recommend treatment guide information (165) for RCA or LCA by applying the third machine learning model (160) to the second input data (155). Specifically, the processor (120) can recommend treatment guide information (165) on whether to perform PCI treatment or CABG treatment for RCA or LCA by applying the third machine learning model (160) to the second input data (155). For reference, CABG treatment has been preferentially utilized because it shows a superior therapeutic effect than PCI in multivessel coronary artery disease. However, in the past several years, the development of drugs and drug-eluting stents has enabled PCI to replace CABG to some extent in multivessel coronary artery disease. CABG is an open heart surgery, which means it has a longer recovery period than PCI and can cause more severe postoperative pain. Furthermore, CABG typically requires general anesthesia, which carries a risk of complications during and after the procedure. In contrast, PCI utilizes a catheter, making it relatively less invasive than CABG. Furthermore, unlike CABG, which requires opening the chest cavity to directly access the heart, PCI involves inserting a catheter through a small hole made through the skin, allowing for a quicker return to daily life compared to CABG. Therefore, accurately recommending treatment guidance information (165) (e.g., either PCI or CABG) based on the interpretation of target blood vessels identified from medical images via the processor (120) can shorten the patient's recovery period, etc.

[0041] Although not illustrated in FIG. 1, the vascular treatment recommendation device (100) may include a display. The display may output treatment guide information recommended by the processor (120).

[0042]

[0043] FIG. 2 schematically illustrates, as an example, a process in which a vascular treatment recommendation device recommends treatment guide information for a patient with multivessel coronary artery disease.

[0044] According to one embodiment, a vascular treatment recommendation device may select a target blood vessel image by inputting medical images (201) captured from multiple directions of a target blood vessel of a target patient (e.g., Patient) into a first machine learning model (210) and filtering the input medical images. For example, assume that the medical images (201) include a first image captured from a first direction of a target blood vessel and a second image captured from a second direction. At this time, if the target blood vessel is captured more clearly in the first image than in the second image, the vascular treatment recommendation device may filter the medical images (201) based on the first machine learning model (210) and select the first image as a target blood vessel image. In other words, the vascular treatment recommendation device may input medical images (201) captured from various angles of a target blood vessel (e.g., RCA or LCA) into a target blood vessel classification model (e.g., the first machine learning model (210)), and select a target blood vessel image by removing the remaining images except for the blood vessel image in which the target blood vessel is captured most clearly in the medical images.

[0045] According to one embodiment, a vascular treatment recommendation device may select a representative frame image (220) by applying a second machine learning model (220) to a selected target blood vessel image. The vascular treatment recommendation device may calculate, for each of a plurality of frame images included in the target blood vessel image, a probability that a blood vessel included in the corresponding frame image is a target blood vessel using the second machine learning model (220). For example, if the target blood vessel image includes 10 frame images, the vascular treatment recommendation device may input the 10 frame images into the second machine learning model (220) and calculate 10 probability values ​​that indicate that a blood vessel appearing in each of the frame images corresponds to a target blood vessel. Based on the 10 calculated probability values, the vascular treatment recommendation device may select a frame image having a probability value greater than or equal to a predetermined threshold value as a representative frame image.

[0046] As an example, the vascular treatment recommendation device can generate input data for a third machine learning model (230) by stacking the selected representative frame image with the first frame image and the last frame image of the target blood vessel image in chronological order. The vascular treatment recommendation device can recommend treatment guide information (240) for the target blood vessel among candidate treatment guide information by inputting the generated input data into the third machine learning model (230). As another example, the vascular treatment recommendation device can recommend treatment guide information (240) by inputting the generated input data and identification information (225) (e.g., clinical characteristic, age, BMI, gender, etc.) for the target patient together into the third machine learning model (230). The vascular treatment recommendation device can recommend more personalized treatment guide information (240) for the target patient by inputting the identification information (225) for the target patient together into the third machine learning model (230). Treatment guide information (240) may include, for example, multivessel coronary artery disease treatments such as PCI or CABG.

[0047] As an example, the vascular treatment recommendation device may determine the final treatment guide information through hard-voting on the recommended treatment guide information (240). For example, the vascular treatment recommendation device may recommend treatment guide information (240) by independently repeating the process described above for medical images (201) taken from multiple directions of a target blood vessel of a target patient (e.g., Patient). If the vascular treatment recommendation device recommends treatment guide information (240) by repeating the process 10 times in total, for example, the 10 pieces of treatment guide information (240) may not all be the same. The vascular treatment recommendation device may determine the treatment guide information that is recommended the most among the 10 pieces of treatment guide information as the final treatment guide information.

[0048]

[0049] FIG. 3 is a drawing specifically explaining how a vascular treatment recommendation device according to one embodiment selects a target vascular image.

[0050] As an example, a vascular treatment recommendation device may select a target blood vessel image including a target blood vessel among medical images based on a result of applying a first machine learning model to medical images captured from multiple directions of the target blood vessel. Since some of the medical images captured from multiple directions do not properly show the structure of the target blood vessel, it is necessary to select an image in which the structure of the target blood vessel is clearly captured. FIG. 3 illustrates that, in order to select a target blood vessel image, the vascular treatment recommendation device inputs one medical image (301) into the first machine learning model. The medical image (301) may include a plurality of frame images. As described in FIG. 1, the plurality of frame images may be images captured at preset time intervals. For example, the plurality of frame images may correspond to frame images captured every second. The vascular treatment recommendation device may input the medical image (301) into the first machine learning model and calculate a probability value that a blood vessel included in each of the plurality of frame images corresponds to the target blood vessel. For example, the vascular treatment recommendation device can calculate a first probability value (320) indicating a probability that a blood vessel in a first frame image (302) included in a corresponding medical image (301) corresponds to a target blood vessel. In addition, the vascular treatment recommendation device can calculate a second probability value (321) indicating a probability that a blood vessel in a second frame image (303) included in the corresponding medical image (301) corresponds to a target blood vessel. The vascular treatment recommendation device can calculate an average value (322) of the first probability value (320) and the second probability value (321). In step (350), the vascular treatment recommendation device can determine whether to select the corresponding medical image (301) as a target blood vessel image by comparing the average value (322) with a predetermined threshold value. For example, the vascular treatment recommendation device can determine whether to select the corresponding medical image (301) as a target blood vessel image when the average value (322) of the first probability value (320) and the second probability value (321) is equal to the predetermined threshold value of 0.If it is 95 or higher, the medical image (301) can be selected as the target blood vessel image.

[0051] Additionally, there may be a case where the medical image (301) includes a plurality of target blood vessels. Specifically, there may be a case where a plurality of frame images of the medical image (301) include a target blood vessel (e.g., LCA) and another target blood vessel (e.g., RCA). The vascular treatment recommendation device may, in addition to calculating the first probability value (320) and the second probability value (321) for the target blood vessel, calculate a probability value for another target blood vessel. Specifically, the vascular treatment recommendation device may apply a first machine learning model to the medical image (301) to calculate a third probability value (330) indicating a probability that a blood vessel in a first frame image (302) included in the corresponding medical image (301) corresponds to another target blood vessel. The vascular treatment recommendation device may further calculate a fourth probability value (331) indicating a probability that a blood vessel in a second frame image (303) included in the corresponding medical image (301) corresponds to another target blood vessel. The blood vessel treatment recommendation device can calculate an average value (332) of the third probability value (330) and the fourth probability value (331). The vascular treatment recommendation device can select the corresponding medical image (301) as the target blood vessel image by comparing a larger value among the average value (322) associated with the target blood vessel and the average value (332) associated with other target blood vessels with a predetermined threshold value (e.g., 0.95). In other words, the vascular treatment recommendation device can select the target blood vessel image by inputting the medical image (301) into a first machine learning model, calculating an average value (322) of probability values ​​(320, 321) that the blood vessel included in each of a plurality of frame images corresponds to the target blood vessel, and using the corresponding medical image (301) if the calculated average value (322) is equal to or greater than the predetermined threshold value (e.g., 0.95), and not using the corresponding medical image (301) if it is less than the predetermined threshold value.

[0052]

[0053] FIG. 4 illustrates a vascular treatment recommendation device training a machine learning model to select a target image from among a plurality of medical images, according to one embodiment.

[0054] As an example, the vascular treatment recommendation device may train the first machine learning model (402) using a supervised learning method. The vascular treatment recommendation device may pre-train the first machine learning model (402) including the ResNet-50 model based on the ImageNet dataset. For reference, the ImageNet dataset may be a dataset stored in ImageNet, an image database including a large amount of annotations. The ImageNet dataset may include images corresponding to semantically grouped words. For example, the ImageNet dataset may include images pre-labeled as coronary artery images. The first machine learning model (402) may be pre-trained to output images including coronary arteries based on the ImageNet dataset including images pre-labeled as coronary artery images. A vascular treatment recommendation device can supervise a pre-trained first machine learning model (402) based on a training data set (401) including frame images labeled with target blood vessels. The training data set (401) can include frame images labeled in advance with target blood vessels (e.g., LCA or RCA) of a target patient as ground truth. The vascular treatment recommendation device can input the training data set (401) into the first machine learning model (402) and measure an error (e.g., loss) between the output blood vessel image (403) and the true value based on a loss function. The vascular treatment recommendation device can propagate the gradient of the loss function for each weight included in the first machine learning model (402) from the output layer of the first machine learning model (402) toward the input layer by backpropagating the measured error to the first machine learning model (402).The vascular treatment recommendation device can update the operating parameters of the first machine learning model (402) in a direction in which the backpropagated error (or gradient of the loss function) decreases. The supervised learning method for the first machine learning model (402) of the vascular treatment recommendation device has been described above, but the supervised learning method is not limited thereto and may vary depending on the design method of the loss function.

[0055]

[0056] FIG. 5 is a diagram illustrating a device for recommending a vascular treatment method according to one embodiment of the present invention selecting a representative frame image from a target vascular image.

[0057] As an example, the vascular treatment recommendation device may select a target vascular image, as described in FIG. 3. In other words, the vascular treatment recommendation device may calculate a probability that a blood vessel included in each of the medical images (e.g., CAG image data) corresponds to a target blood vessel based on a first machine learning model, and may calculate an average of the probabilities. For example, a first average value may be calculated for a first image among the medical images, and a second average value may be calculated for a second image. The vascular treatment recommendation device may select the medical image as a target vascular image if the calculated average value is greater than or equal to a predetermined threshold value. That is, for example, if both the first average value and the second average value are greater than or equal to 0.95, both the first image and the second image may be selected as target vascular images. However, the first image and the second image may have different image lengths. In other words, the number of frame images included in the first image and the second image may be different. Converting the first image and the second image to the same length and processing them may be inefficient in terms of energy and time. Therefore, it is necessary to extract a representative frame image from among the frame images included in the video.

[0058] As an example, the vascular treatment recommendation device may apply a second machine learning model to a plurality of frame images (502) included in a target blood vessel image (501) to produce probability values ​​representing the probability that a blood vessel in each of the plurality of frame images (502) corresponds to a target blood vessel. For reference, the second machine learning model may correspond to a model that produces a probability that an object requiring recognition is included in an input image (or image). For example, the second machine learning model may correspond to a model that produces a probability that an object included in an image input to the second machine learning model corresponds to a coronary artery by adding three FC layers to a ResNet-50 model.

[0059] The vascular treatment recommendation device may select representative frame image candidates (510) based on the comparison results of the calculated probability values. For example, the vascular treatment recommendation device may select one of the frame image candidates (510) corresponding to the largest value among the probability values ​​calculated for each of the plurality of frame images (502) as the representative frame image. As another example, the vascular treatment recommendation device may select a frame image having a probability value greater than or equal to a predetermined probability value among the calculated probability values ​​as the representative frame image candidates (510). For example, if there are a total of five frame images (502), and the probability of blood vessels being clearly visible for each frame included in the plurality of frame images (502) is calculated as 0.1, 0.2, 0.96, 0.97, and 0.3, respectively, by the second machine learning model, the vascular treatment recommendation device may select frame images corresponding to probability values ​​of 0.96 and 0.97, which have a probability of 0.95 or higher, as representative frame image candidates (510), and select only the frame image corresponding to the probability value of 0.97 as one representative frame image among the representative frame candidates (510). For another example, if there are two frame images (502) in total, and the second machine learning model calculates the probability of blood vessels being visible for each frame included in the plurality of frame images (502) to be 0.95 and 0.95, respectively (i.e., if there are two frames with the same probability), the vascular treatment recommendation device may select the second frame image with a higher frame index number as the representative frame image. For another example, if there are three frame images (502) in total, and the second machine learning model calculates the probability of blood vessels being visible for each frame included in the plurality of frame images (502) to be 0.95, 0.95, and 0.If each is calculated equally as 95 (i.e., if there are three frames with the same probability), the vascular treatment recommendation device can select the frame with the middle frame index number as the representative frame image. The representative frame image candidates (510) selected by the vascular treatment recommendation device through the second machine learning model can correspond to a frame image in which the target blood vessel appears most clearly among the multiple frame images (502) included in the target blood vessel image (501). In other words, the vascular treatment recommendation device can select a representative frame image in which the target blood vessel, such as the RCA or LCA, appears most clearly in one blood vessel image.

[0060]

[0061] FIG. 6 illustrates a vascular treatment recommendation device according to one embodiment training a machine learning model to select a representative frame image from among multiple frame images included in one medical image.

[0062] As an example, the vascular treatment recommendation device can train the second machine learning model (610) using a semi-supervised learning method. Semi-supervised learning is a method in which a machine learning model is first supervisedly trained with a labeled training data set, and then falsely labeled the predicted results of the supervised machine learning model and used again as training data. The vascular treatment recommendation device must train the second machine learning model (610) to extract representative frame images (e.g., images in which the target blood vessel is captured most clearly) from the input medical images. However, labeling all medical images requires the knowledge of an expert (e.g., a doctor), which can be costly and time-consuming. Therefore, rather than training the second machine learning model (610) with data that labels all medical images, the vascular treatment recommendation device can train the second machine learning model (610) through a semi-supervised learning method that utilizes both a first training data set (601) that includes labels for some representative frame images and a training data set that does not include labels for representative frame images.

[0063] The vascular treatment recommendation device may first perform supervised learning on a second machine learning model (610) based on a first training data set (601) including frame images labeled as representative frame images among medical images. For example, the vascular treatment recommendation device may perform supervised learning on the second machine learning model (610) for a total of 10 epochs using the first training data set (601). For reference, an epoch may indicate the number of times a training data set is input to a machine learning model. For example, if a machine learning model is trained by inputting a training data set 5 times, it can be considered that the machine learning model is trained for 5 epochs.

[0064] The vascular treatment recommendation device may apply a second machine learning model (610) to a first training data set (601) to output a first prediction result (620). For example, the first prediction result (620) may correspond to a frame image predicted as a representative frame image output by a second machine learning model (610) that has undergone supervised learning for 10 epochs and inputs the first training data set (601). The first prediction result (620) may be used as a training data set that does not include a label for the representative frame image.

[0065] The vascular treatment recommendation device includes a second training data set (e.g., D) including a frame image (630) pseudo-labeled as a representative frame image for a predetermined ratio of frames among a plurality of frames included in a target vascular image. L +D S t), the second machine learning model (640) can be additionally trained. For example, the vascular treatment recommendation device can pseudo-label some frame images among the first prediction result (620) that do not include a label for the representative frame image as a representative frame image. Pseudo-labeling may refer to a technique in which a prediction result is generated through a machine learning model for data that does not have a label for the ground truth, and the model gives a false label as the ground truth for the generated prediction result. For example, the vascular treatment recommendation device can pseudo-label a predetermined ratio of frame images among the frame images included in the first prediction result (620) as true values. Specifically, the vascular treatment recommendation device can pseudo-label (tХk)% of the frame images included in the first prediction result (620) as true values. At this time, t and k may represent integers greater than or equal to 1. Specifically, k is set to 20, and t can be increased by 1 every 5 generations until the training data set (e.g., the first prediction result (620)) that does not include labels for representative frame images is completely used by pseudo-labeling. The vascular treatment recommendation device comprises a second training data set (e.g., D) that includes pseudo-labeled frame images (630) and the first training data set (601). L +D S t ), a second machine learning model (640) can be additionally trained.

[0066] The vascular treatment recommendation device is a second training data set (e.g., D L +D S t) is further trained based on the second machine learning model (640), and then the second training data set (e.g., D) is trained by increasing the predetermined ratio. L +D S t ) a third training dataset containing more pseudo-labeled frame images (e.g., D L +D S t+1 ) can be generated. For example, the vascular treatment recommendation device can train the second machine learning model (610, 640, 670) through a curriculum learning method. In other words, the vascular treatment recommendation device can train the second machine learning model (610, 640, 670) by using easy samples and increasingly difficult samples. For example, the vascular treatment recommendation device can generate a second training data set by pseudo-labeling (tХk)% of the frame images included in the first prediction result (620). For example, when k is 20 and t is 1, the vascular treatment recommendation device can pseudo-label 20 frame images in order of decreasing probability from a frame image having a high probability of corresponding to a representative frame in the first prediction result (620) including 100 frame images. The vascular treatment recommendation device can additionally train a second machine learning model (640) based on a second training data set including 20 pseudo-labeled one-frame images (630).

[0067] The vascular treatment recommendation device can output a second prediction result (650) based on the second machine learning model (640). The vascular treatment recommendation device pseudo-labels ((t+1)Хk)% of frame images included in the second prediction result (650) to form a third training set (e.g., D L +D S t+1) can be generated. In other words, the vascular treatment recommendation device can increase the predetermined ratio in the order of (tХk)%, ((t+1)Хk)%, ((t+2)Хk)% while repeating the training. Therefore, the number of pseudo-labeled frame images (660) in the second prediction result (650) may be greater than the number of pseudo-labeled frame images (630) in the first prediction result (620). The vascular treatment recommendation device may generate a third training data set (e.g., D L +D S t+1 ), a second machine learning model (670) can be trained.

[0068]

[0069] FIG. 7 is a drawing specifically explaining how a vascular treatment recommendation device according to one embodiment generates input data and applies a machine learning model to the input data to recommend treatment guide information.

[0070] As an example, the vascular treatment recommendation device can generate input data (720) including time series information by stacking a representative frame image (710), a previous frame of the representative frame image, and a subsequent frame of the representative frame in the order of shooting time. For example, the vascular treatment recommendation device can determine frame images in which a target blood vessel is clearly revealed among frame images included in a target blood vessel image as the representative frame image (710). For example, the vascular treatment recommendation device can extract representative frame image candidates (510) based on a probability value that a blood vessel appearing in each of the frame images included in the target blood vessel image corresponds to a target blood vessel. The vascular treatment recommendation device can extract frame images in which a probability value that a blood vessel appearing in each of the frame images included in the target blood vessel image corresponds to a target blood vessel is, for example, 0.95 or higher, as the representative frame image candidates (510). The vascular treatment recommendation device can select a frame image having the highest probability value among the representative frame image candidates (510) as the representative frame image (710). The vascular treatment recommendation device can generate input data (720) by stacking the first frame image (701) and the last frame image (702) among the frame images included in the representative frame image (710) and the target blood vessel image in chronological order. The vascular treatment recommendation device can consider time-series information of the target blood vessel that changes over time without inputting all the frame images of the target blood vessel image to the third machine learning model (730) by generating input data (720) stacked in the order of the first frame image (701) - the representative frame image (710) - the last frame image (702). In other words, the vascular treatment recommendation device can generate input data (720) in the form of a three-channel tensor by stacking the first frame image (701), the representative frame image (710), and the last frame image (702).The vascular treatment recommendation device can recommend treatment guide information (740) for a target blood vessel among candidate treatment guide information by applying a third machine learning model (730) to input data (720).

[0071]

[0072] FIG. 8 illustrates a vascular treatment recommendation device according to one embodiment training a machine learning model to recommend treatment guide information through medical images.

[0073] As an example, the vascular treatment recommendation device can generate input data (820) by stacking the first frame image (801), the representative frame image (810), and the last frame image (802) among the multiple frame images included in the target vascular image (e.g., CAG videos). The vascular treatment recommendation device can input the input data (820) in the form of a three-channel tensor in which three frame images (801, 810, 802) are stacked, into a third machine learning model (830). For example, the vascular treatment recommendation device can input the input data (820) into a third machine learning model (830) including a ResNet-50 backbone model. In step (840), the vascular treatment recommendation device can generate combined data by combining (e.g., concat) intermediate output data based on the input data (820) of the ResNet-50 backbone model included in the third machine learning model (830) and identification information (e.g., clinical characteristic) of the target patient. The vascular treatment recommendation device can comprehensively consider not only the medical image of the target patient but also the patient's clinical characteristics (e.g., height, age, weight, gender, etc.) by combining the intermediate output data with the identification information of the target patient. The vascular treatment recommendation device can recommend treatment guide information that takes into account the medical image and the target patient's personal characteristics by inputting the combined data into a classifier layer (e.g., classifier layer) included in a third machine learning model (830). The vascular treatment recommendation device can update the third machine learning model parameters in a direction in which the loss function of the third machine learning model (830) is minimized by backpropagating the error between the recommended treatment guide information and the treatment guide information labeled by an expert (e.g., doctor) to the third machine learning model (830).Although the method of training the third machine learning model (830) has been described using a supervised learning method as an example, it is only an example and is not limited thereto.

[0074]

[0075] In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof.

[0076] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0077] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In the recommended device for vascular treatment, Based on the results of applying the first machine learning model to medical images taken of the target blood vessel from multiple directions, target blood vessel images including the target blood vessel are selected from among the medical images, Based on the result of applying the second machine learning model to a plurality of frame images included in the selected target blood vessel images, a representative frame image including the target blood vessel is selected from among the plurality of frame images. By stacking the selected representative frame image, the previous frame of the representative frame image, and the subsequent frame of the representative frame in shooting time order, input data including time series information is generated, A processor that recommends treatment guide information for the target blood vessel among candidate treatment guide information by applying a third machine learning model to the stacked input data; and A display that outputs the above treatment guide information Recommended vascular treatment devices including:

2. In paragraph 1, The above processor, By applying the first machine learning model to the above medical images, A first probability value is calculated indicating the probability that a blood vessel in a first frame image included in the medical image among the above medical images corresponds to the target blood vessel, A second probability value is calculated indicating the probability that a blood vessel in a second frame image included in the above medical image corresponds to the target blood vessel, By comparing the average value of the first probability value and the second probability value with a predetermined threshold value, the corresponding medical image is selected as the target blood vessel image. Recommended Device for Vascular Therapy.

3. In paragraph 2, The above processor, If the average value of the first probability value and the second probability value is greater than or equal to the predetermined threshold value, the corresponding medical image is selected as the target blood vessel image. Recommended Device for Vascular Therapy.

4. In paragraph 2, The above processor, If the above medical images include other target blood vessels, By applying the first machine learning model to the above medical images, A third probability value is calculated to indicate the probability that a blood vessel in a first frame image included in the medical image among the above medical images corresponds to the other target blood vessel, A fourth probability value is calculated indicating the probability that a blood vessel in a second frame image included in the above medical image corresponds to the other target blood vessel, By comparing the larger value among the average value of the first probability value and the second probability value, and the average value of the third probability value and the fourth probability value with the predetermined threshold value, the corresponding medical image is selected as the target blood vessel image. Recommended Device for Vascular Therapy.

5. In paragraph 1, The above processor, Pre-training the first machine learning model including the Resnet-50 model based on the ImageNet dataset, and supervising learning the pre-trained first machine learning model based on a training data set including frame images labeling the target blood vessels. Recommended Device for Vascular Therapy.

6. In paragraph 1, The above processor, By applying a second machine learning model to a plurality of frame images included in the selected target blood vessel images, probability values ​​representing the probability that a blood vessel in each of the plurality of frame images corresponds to the target blood vessel are calculated, and the representative frame image is selected based on the comparison result of the calculated probability values. Recommended Device for Vascular Therapy.

7. In paragraph 6, The above processor, Selecting a frame image corresponding to the largest value among the above-described probability values ​​as the representative frame image. Recommended Device for Vascular Therapy.

8. In paragraph 1, The above processor, The second machine learning model is supervisedly trained based on a first training data set including a frame image labeled with the representative frame image, and The second machine learning model is additionally trained based on a second training data set including frame images pseudo-labeled as representative frame images for a predetermined ratio of frame images among a plurality of frame images included in the target blood vessel image. Recommended Device for Vascular Therapy.

9. In paragraph 8, The above processor, After further training the second machine learning model based on the second training data set, a third training data set is generated by increasing the predetermined ratio, which includes more pseudo-labeled frame images than the second training data set, and the second machine learning model is trained based on the third training data set. Recommended Device for Vascular Therapy.

10. In paragraph 1, The above processor, By stacking the selected representative frame image, the first frame image and the last frame image included in the target blood vessel image in chronological order, the input data is generated. Recommended Device for Vascular Therapy.

11. In paragraph 1, The above processor, Generating a training data set based on the stacked input data and identification information for the target patient having the target blood vessel, and training the third machine learning model based on the training data set. Recommended Device for Vascular Therapy.

Images