Method and apparatus for predicting coronary artery calcification using mammography

Abstract

A method for predicting an anomaly in coronary artery calcification (CAC) using a mammography image and executed by at least one processor according to exemplary embodiments of the present invention may comprise the steps of: acquiring a mammography image and clinical data of a subject; and determining the presence of a CAC anomaly in the subject on the basis of the mammography image and the clinical data.

Description

Method and device for predicting coronary artery calcification using mammography images

[0001] The present invention relates to a method and device for predicting coronary artery calcification using mammography images. More specifically, the present invention relates to a method and device for predicting coronary artery calcification using images captured by mammography, i.e., mammography.

[0002] Cardiovascular disease (CVD) has been reported to be the leading cause of death worldwide. Furthermore, the prevalence of CVD among middle-aged women is known to be increasing.

[0003] Identifying high-risk groups and implementing appropriate interventions is known to be a key strategy for preventing cardiovascular disease. Accordingly, research is underway on models to predict cardiovascular disease risk (i.e., the likelihood of developing the disease). For example, models for predicting cardiovascular disease risk, such as the Framingham risk score and the 10-year ASCVD risk score, have been developed.

[0004] Meanwhile, coronary artery calcification (CAC) has been identified as a significant indicator for predicting cardiovascular disease risk. For example, the Coronary Artery Calcium Score (CACS), which scores the progression of coronary artery calcification, is a reliable predictor of cardiovascular disease risk. In particular, the CACS has been shown to be an effective tool for predicting cardiovascular risk in asymptomatic adults.

[0005] Coronary artery calcification can be quantified using computed tomography (CT), which is widely used in clinical practice. However, coronary artery CT (CAC CT) has limitations: high radiation exposure, high cost, and limited availability at large hospitals.

[0006] On the other hand, mammography (i.e., mammography) has the advantage of being relatively inexpensive and can be performed in primary care settings. Furthermore, there is growing evidence that breast arterial calcification (BAC) can significantly increase the risk of cardiovascular disease. Thus, information on coronary artery calcification can be used to predict cardiovascular disease risk. Therefore, there is a need to develop methods and models for predicting coronary artery calcification (e.g., abnormality / abnormality of CACS and CAC) using mammography images.

[0007] For example, US 2018-0075628 (prior document 1) discloses a method for automatically detecting malignant signs in mammographic images, KR 10-2023-0108213 (prior document 2) discloses a method for detecting cardiovascular calcification based on an artificial intelligence model, and US 2019-0223809 (prior document 3) discloses a method for mapping breast artery calcification. However, the aforementioned prior documents either do not disclose a method for predicting coronary artery calcification-related information using mammographic images, or even if they do, they only disclose prediction methods with low accuracy, etc.

[0008] An object of the present invention is to provide a method and device for predicting whether coronary artery calcification is abnormal.

[0009] One object of the present invention is to provide an artificial intelligence prediction model and deep learning system capable of predicting whether coronary artery calcification is abnormal.

[0010] A method for predicting coronary artery calcification (CAC) abnormalities using mammography images, executed by at least one processor according to exemplary embodiments of the present invention, may include the steps of: acquiring a mammography image and clinical data of a subject; and determining whether the subject has a CAC abnormality based on the mammography image and the clinical data.

[0011] In one embodiment, the clinical data may include data having a different modality than the mammography image.

[0012] In one embodiment, the clinical data may include at least one of age data and menopause status data of the subject.

[0013] In one embodiment, the step of determining whether there is a CAC abnormality may include: a step of forming fusion data by fusing the mammography image and the clinical data; and a step of determining whether there is a CAC abnormality based on the fusion data.

[0014] In one embodiment, the step of forming the fusion data may include the step of converting the clinical data into matrix data having the same size as the mammography image; and the step of concatenating the matrix data and the mammography image in the channel direction.

[0015] In one embodiment, the mammography image may include a first mammography image captured in a first direction and a second mammography image captured in a second direction different from the first direction.

[0016] The step of forming the above fusion data may include forming first fusion data by fusing the first mammography image and the clinical data, and forming second fusion data by fusing the second mammography image and the clinical data.

[0017] The step of determining whether the CAC is abnormal may include determining whether the CAC is abnormal based on the first fusion data and the second fusion data.

[0018] In one embodiment, the step of determining whether the CAC is abnormal may include: calculating a first CAC abnormality prediction value based on the first fused data, calculating a second CAC abnormality prediction value based on the second fused data; and determining whether the CAC is abnormal by fusing the first CAC abnormality prediction value and the second CAC abnormality prediction value using at least one of an arithmetic mean, a weighted mean, a MAX selection, and a MIN selection.

[0019] In one embodiment, the step of determining whether there is a CAC abnormality based on the fusion data may include the step of extracting feature information from the fusion data; and the step of classifying the CAC abnormality or CAC normal based on the feature information.

[0020] In one embodiment, the step of determining whether there is a CAC abnormality based on the fusion data may be performed by a trained artificial intelligence model.

[0021] The above-mentioned trained artificial intelligence model may include an encoder trained to extract the feature information from the fused data; and a classifier trained to classify CAC abnormality or V CAC normal based on the feature information.

[0022] In one embodiment, the trained artificial intelligence model may be trained according to a training phase.

[0023] The training phase may include: a step of obtaining anchor fusion data corresponding to CAC abnormality, a plurality of positive fusion data corresponding to CAC abnormality other than the anchor fusion data, and a plurality of negative fusion data corresponding to CAC normality; a step of extracting feature information for each of the anchor fusion data, the positive fusion data, and the negative fusion data using an encoder; and a step of adjusting a parameter of the encoder such that, in a feature space, a distance between feature information of the anchor fusion data and feature information of the positive fusion data becomes closer than a distance between feature information of the anchor fusion data and feature information of the negative data.

[0024] In one embodiment, the training phase may further include a step of pre-performing representation learning based on self-supervised learning using inpainting for the encoder.

[0025] A device for predicting abnormalities of coronary artery calcification (CAC) using mammography images according to exemplary embodiments of the present invention may include a communication unit; a memory; and at least one processor connected to the communication unit and the memory.

[0026] The at least one processor may be controlled to acquire a mammography image and clinical data of the subject, and determine whether the subject has a CAC abnormality based on the mammography image and the clinical data.

[0027] According to exemplary embodiments of the present invention, an application program stored in a recording medium may be provided to execute the above-described prediction method when operated by at least one processor.

[0028] According to exemplary embodiments of the present invention, a method and device for predicting abnormalities of coronary artery calcification with improved prediction performance (e.g., classification performance such as sensitivity, specificity, AUROC, accuracy, etc.) can be provided.

[0029] According to exemplary embodiments of the present invention, a coronary artery calcification abnormality prediction model and a deep learning system having improved prediction performance can be provided.

[0030] FIG. 1 illustrates a block diagram of a service system for providing a prediction service for coronary artery calcification abnormalities according to one embodiment of the present invention.

[0031] Figure 2 illustrates a block diagram of a device according to one embodiment of the present invention.

[0032] FIG. 3 schematically illustrates a methodology for predicting coronary artery calcification abnormalities according to one embodiment of the present invention.

[0033] FIG. 4 illustrates an operational flowchart for performing prediction of coronary artery calcification abnormalities according to one embodiment of the present invention.

[0034] FIG. 5 illustrates a block diagram of a coronary artery calcification abnormality prediction model according to one embodiment of the present invention.

[0035] Figures 6a and 6b illustrate a contrastive learning framework according to one embodiment of the present invention.

[0036] Figure 7 illustrates a contrastive learning framework according to another embodiment of the present invention.

[0037] FIG. 8 illustrates an operational flowchart for performing prediction of coronary artery calcification abnormalities according to another embodiment of the present invention.

[0038] FIG. 9 schematically illustrates the framework of a deep learning system for predicting coronary artery calcification abnormalities according to one embodiment of the present invention.

[0039] Figure 10 briefly illustrates the training phase in Example 1 of the present invention.

[0040] Figure 11 briefly illustrates the test phase in Embodiment 1 of the present invention.

[0041] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below, along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms.

[0042] To clearly explain embodiments of the present invention, portions irrelevant to the description may be omitted. Furthermore, when describing embodiments of the present invention, if a detailed description of a related known configuration or function is deemed to obscure the gist or description of the present invention, a detailed description thereof may be omitted.

[0043] In this specification, the term "unit" may refer to a unit that processes at least one function or operation. Some "units" may be implemented in hardware, software, or a combination of hardware and software.

[0044] The division of components in this specification is merely based on the primary function each component is responsible for. In other words, two or more components may be combined into a single component, or a single component may be further subdivided into two or more components with more specific functions. Furthermore, each component may, in addition to its own primary function, additionally perform some or all of the functions performed by other components. Furthermore, some of the primary functions of each component may be exclusively performed by other components.

[0045] In describing components in this specification, terms such as "first," "second," etc. may be used. The aforementioned terms are intended to distinguish one component from another for convenience of description, and unless otherwise specified, the nature, order, etc. of the components are not limited by the aforementioned terms.

[0046] In each of the steps mentioned in this specification, unless the context clearly dictates a specific order, the steps may be performed in a different order than stated. That is, the steps may be performed in the same order as stated, may be performed substantially simultaneously, or may be performed in the opposite order.

[0047] In this specification, "and / or" may mean each of the listed components and any combination of two or more of the listed components. For example, "A, B and / or C" may be used with the same meaning as "at least one of A, B, and C."

[0048] According to exemplary embodiments of the present invention, a method, a prediction device, and a prediction service for predicting abnormalities in coronary artery calcification (CAC) may be provided. Hereinafter, for convenience of explanation, "abnormalities in coronary artery calcification" may be abbreviated as "CAC abnormalities."

[0049] FIG. 1 illustrates a block diagram of a service system for providing a CAC abnormality prediction service according to one embodiment of the present invention.

[0050] Referring to FIG. 1, a service system for providing a CAC abnormality prediction service may include a user device (101) and a server device (102).

[0051] The user device (101) may be a device used by a user to check and manage subject-related information (e.g., the subject's mammography images, clinical data, etc.). The user device (101) may interact with a server device (102) via a network. For example, the user device (101) may provide subject-related information to the server device (102) and receive a prediction result regarding whether the subject has a CAC abnormality. For example, the user device (101) may be a variety of devices, such as an electronic device such as a smartphone, a laptop, a desktop computer, or a tablet PC.

[0052] The server device (102) may be a device of a service provider that provides a CAC abnormality prediction service. The server device (102) may transmit and receive subject-related information and prediction results regarding the presence or absence of a CAC abnormality of the subject to and from the user device (101) via a network. The server device (102) stores a plurality of subject-related information, a trained CAC abnormality prediction model, and the like, and may provide a function for predicting a subject's CAC abnormality in the form of a platform.

[0053] Although FIG. 1 illustrates only one user device (101), multiple devices including the user device (101) can access the server (102) and use the CAC abnormality prediction service. However, the scope of accessible information may vary depending on the authority granted to the user device (101).

[0054] The relationship between the server device (102) and the user device (101) as shown in FIG. 1 can be established when the CAC anomaly prediction service is provided in the form of a platform. According to another embodiment of the present invention, the CAC anomaly prediction service may be provided in the form of a non-network-based program rather than a platform. In this case, unlike as shown in FIG. 1, the CAC anomaly prediction service may be provided by a program or application installed on the user device (101) without the server device (102). In addition, the server device (102) may be the entity that provides the program or application.

[0055] FIG. 2 illustrates the structure of a device according to one embodiment of the present invention. FIG. 2 illustrates an example of the structure of a user device (101) or a server device (102) of FIG. 1. In this specification, the user device (10) and the server device (102) may be collectively referred to as a "device."

[0056] Referring to FIG. 2, the device may include a control unit (201), a communication unit (202), and a storage unit (203).

[0057] The control unit (201) can control the overall functions and operations of the device. The control unit (201) can control components of the device, for example, the communication unit (202), the storage unit (203), and at least one other component. That is, the control unit (201) can provide information or data necessary for the operation of the components of the device, and perform operations based on information or data generated or managed by the components. For example, the control unit (201) can include at least one processor, at least one circuit, etc. For example, the at least one processor can include at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU). The control unit (201) can perform necessary controls so that the device operates according to various embodiments described herein. For example, the control unit (201) can control the operation of the device by executing software, programs, or commands stored in the storage unit (203).

[0058] The communication unit (202) can perform functions for transmitting and receiving signals with other devices. The communication unit (202) performs wired or wireless communication and can process signals according to the control of the control unit (201). For example, the communication unit (202) may include an RF circuit, an antenna, etc. for wireless communication, or a connection terminal, a modem, a driver module, etc. for wired communication. For example, the communication unit (202) may support at least one of various communication protocols, such as cellular communication such as LTE and 5G, short-range wireless communication such as WiFi and Bluetooth, and short-range wired communication such as Ethernet.

[0059] The storage unit (203) can store data used in the device, software for the operation of the device, programs, and commands. In addition, the storage unit (203) can provide stored data under the control of the control unit (201). In addition, the storage unit (203) can store applications, drivers, etc. to be driven by the control unit (201). For example, the storage unit (203) can include RAM such as DRAM, SRAM, etc.; ROM; EEPROM; HDD; SSD; flash storage means, etc.

[0060] Although not shown in FIG. 2, the device may further include at least one of a power supply device, a display device, an input device, and an output device, depending on the type of the device.

[0061] FIG. 3 illustrates a methodology for predicting CAC abnormalities according to one embodiment of the present invention.

[0062] Referring to FIG. 3, CAC abnormality prediction can be performed based on acquired subject-related information (e.g., mammography images, clinical data, etc.). For example, the server device (102) can acquire subject-related data from various sources, including the user device (101), and predict CAC abnormalities by analyzing information included in the acquired subject-related data. For example, the server device (102) can classify the subject into one of a plurality of groups, including CAC abnormalities and CAC normals, based on the subject-related information.

[0063] In one embodiment, the mammography image may include a CC-view image taken in the craniocaudal direction, an MLO-view image taken in the mediolateral oblique direction, etc.

[0064] For example, a CC-view image may include a Right CC (RCC) image and a Left CC (LCC) image. Additionally, an MLO-view image may include a Right MLO (RMLO) image and a Left MLO (LMLO) image. Detailed information about CC-view and MLO-view images is already well known in the relevant industry, so a detailed description will be omitted.

[0065] In one embodiment, clinical data may include data in a modality other than mammography images. That is, the CAC abnormality prediction service according to one embodiment of the present invention may be multi-modal.

[0066] In one embodiment, clinical data may include age data, menopause status data, and the like.

[0067] For example, age data may include categorical age data. In this case, the range of categories may be evenly distributed over a given time period (e.g., 5-year periods, 10-year periods, etc.) or may be differentially distributed. For example, the time period for distribution in ages over 40 may be smaller than the time period for distribution in ages under 40.

[0068] For example, menopause status data may include dichotomous menopause status data. For example, menopause status data may include information on whether menopause has occurred (○ or ×). That is, menopause status data may include information on menopause status ○ and menopause status ×.

[0069] For example, CAC abnormality prediction can be performed using rule-based models, trained artificial intelligence models, etc.

[0070] In one embodiment, mammography images and clinical data may be fused and used.

[0071] When a trained AI model is used to predict CAC abnormalities, fusion may encompass both early fusion (fusing mammography images and clinical data at the input stage to the AI ​​model) and intermediate fusion (extracting feature information from each of the mammography images and clinical data using different feature extractors and fusion of the extracted feature information). However, some embodiments described below may assume early fusion.

[0072] In one embodiment, multiple mammographic images may be used to predict CAC abnormalities. The multiple mammographic images may include images of the same breast of the subject taken from different orientations.

[0073] For example, the plurality of mammography images may include a first mammography image taken from a first direction of the subject's breast and a second mammography image taken from a second direction different from the first direction. In this case, a first CAC abnormality prediction value may be calculated based on the first mammography image and clinical data, a second CAC abnormality prediction value may be calculated based on the second mammography image and clinical data, and then a final CAC abnormality may be determined based on the first CAC abnormality prediction value and the second CAC abnormality prediction value (e.g., by fusion of the first CAC abnormality prediction value and the second CAC abnormality prediction value).

[0074] For example, multiple mammography images may include RCC, LCC, RMLO, and LMLO. In this case, a first CAC abnormality prediction value may be calculated based on the LCC and clinical data, a second CAC abnormality prediction value may be calculated based on the RCC and clinical data, a third CAC abnormality prediction value may be calculated based on the RMLO and clinical data, and a fourth CAC abnormality prediction value may be calculated based on the LMLO and clinical data. Based on the first CAC abnormality prediction value, the second CAC abnormality prediction value, the third CAC abnormality prediction value, and the fourth CAC abnormality prediction value (e.g., by fusing the first CAC abnormality prediction value to the fourth CAC abnormality prediction value), a final CAC abnormality may be determined.

[0075] For example, the Coronary Artery Calcium Score (CACS) can be used as a criterion (i.e., a decision criterion) for classifying CAC abnormalities and normal CACs. For example, a subject can be classified as having CAC abnormalities or normal CACs based on the CACS cut-off value (threshold). In this case, the CACS of a subject classified as having CAC abnormalities according to a CAC abnormality prediction service can be expected to exceed the CACS cut-off value.

[0076] For example, CAC abnormality prediction results can be used as data to determine medical interventions for a subject (e.g., additional testing, clinical treatment direction, etc.). Furthermore, CAC abnormality prediction results can be utilized to predict cardiovascular disease risk. For example, CAC abnormality prediction results can be directly applied to predict cardiovascular disease risk, or they can be utilized as a factor in predicting cardiovascular disease risk.

[0077] FIG. 4 illustrates an operational flowchart for performing CAC anomaly prediction according to one embodiment of the present invention. FIG. 4 may refer to operations performed by a server device (102). However, as described above, the CAC anomaly prediction service may be provided by a program or application installed on a user device (101) without a server device (102), in which case the server device (102) described below may refer to the user device (101).

[0078] Referring to FIG. 4, the server device (102) can acquire mammography images and clinical data of a subject (S401). The mammography images and clinical data can be received by the server device (102) via a network from, for example, a user device (101), a third-party device (not shown), etc.

[0079] The server device (102) can form fusion data by fusion of mammography images and clinical data (S402).

[0080] In one embodiment, the mammography images and clinical data may be preprocessed before fusing the mammography images and clinical data.

[0081] For example, preprocessing such as resizing, normalization, image cropping, and standardization can be performed on mammography images. For example, preprocessing such as vectorization, matrixization, and tensorization can be performed on clinical data. For example, clinical data can be vectorized, matrixized, or tensorized through one-hot encoding, dummy variableization, and category embedding.

[0082] In one embodiment, a method of fusing mammography images and clinical data may include element-wise sum, element-wise product, concatenation, and the like.

[0083] For example, clinical data can be converted into a matrix having the same size as a mammography image (meaning a preprocessed mammography image if the image size has changed due to preprocessing). That is, if a mammography image is an l×m×n matrix or tensor (wherein l represents a channel and is an integer greater than or equal to 0 or 1, m represents a width and is an integer greater than or equal to 1, and n represents a height and is an integer greater than or equal to 1), the clinical data can be converted into a matrix having a size of m×n. The matrix and the mammography image can be fused through element-wise addition, element-wise multiplication, or concatenation. Preferably, the matrix and the mammography image can be concatenated in the channel direction (channel-wase concatenation). In this case, the fused data becomes a tensor of (l+k)×m×n (wherein k represents the number of clinical data to be concatenated).

[0084] The server device (102) can predict CAC abnormalities based on the fused data (S403). For example, as described above, the server device (102) can classify the subject into one of multiple groups, including CAC abnormalities and CAC normals. CAC abnormality prediction can be performed using a rule-based model, a trained artificial intelligence model, or the like.

[0085] FIG. 5 illustrates a block diagram of a trained artificial intelligence model (hereinafter referred to as a CAC anomaly prediction model) according to one embodiment of the present invention. For example, the CAC anomaly prediction according to FIG. 4 can be performed using the CAC anomaly prediction model illustrated in FIG. 5.

[0086] Referring to FIG. 5, the CAC abnormality prediction model (500) may include a trained feature extractor (501). For example, the trained feature extractor (501) may include a trained encoder.

[0087] For example, a trained encoder (501) can receive fused data as input and extract feature information from the fused data. For example, the trained encoder (501) can extract feature information from the fused data through a convolutional layer, a pooling layer, etc. For example, the feature information can include a feature vector, etc.

[0088] The CAC abnormality prediction model (500) may further include a trained classifier (502). For example, the trained classifier (502) may output a CAC abnormality prediction value (i.e., a CAC abnormality prediction probability value) based on extracted feature information using a classification layer (e.g., a fully connected layer, a dense layer, etc.), an activation function (Softmax, Sigmoid, ReLU, etc.), etc. For example, CAC abnormality and CAC normality may be classified based on the CAC abnormality prediction value.

[0089] In conclusion, the CAC abnormality prediction model (500) can receive fusion data as input and output a CAC abnormality prediction probability value as output.

[0090] The CAC anomaly prediction model (500) may be one that has performed contrastive learning. For example, the CAC anomaly prediction model (500) may be one that has performed supervised contrastive learning.

[0091] Figures 6a and 6b illustrate a contrastive learning framework according to one embodiment of the present invention. Figure 7 illustrates a contrastive learning framework according to another embodiment of the present invention.

[0092] Figures 6a and 6b are examples of contrastive learning performed through two stages. Figure 7 is an example of contrastive learning performed through one stage.

[0093] Referring to Fig. 6a, anchor samples, positive samples, and negative samples can be prepared as input data. An anchor sample can refer to a sample that serves as a comparison reference in contrastive learning; a positive sample can refer to a sample that is similar to the anchor sample, i.e., belongs to the same class as the anchor sample; and a negative sample can refer to a sample that is dissimilar to the anchor sample, i.e., belongs to a different class from the anchor sample.

[0094] The anchor sample was set as the fused data corresponding to the CAC abnormality class (i.e., the fused data of the anchor mammography image labeled as CAC abnormality and the clinical data). The positive sample was set as the fused data corresponding to a CAC abnormality class other than the anchor sample (i.e., the fused data of the positive mammography image labeled as CAC abnormality and the clinical data). The negative sample was set as the fused data corresponding to a class other than the anchor sample (i.e., the fused data of the negative mammography image labeled as CAC normal and the clinical data).

[0095] For example, encoders of well-known CNN models (e.g., Resnet, VGGNet, GooleNET (Inception), etc.) can be used as encoders.

[0096] In one embodiment, the encoder may comprise a residual learning based encoder, for example, an encoder of Resnet.

[0097] For example, an anchor sample, a plurality of positive samples, and a plurality of negative samples can be individually input to the same encoder. The encoder can extract feature information from each of the anchor sample, the plurality of positive samples, and the plurality of negative samples. That is, the encoder can receive an anchor sample as input and extract anchor feature information, receive a positive sample as input and extract positive feature information, and receive a negative sample as input and extract negative feature information.

[0098] For example, the parameters of the encoder can be tuned using a contrastive learning loss function (e.g., Contrastive Loss, Triplet Loss, etc.).

[0099] In feature space, the encoder parameters can be adjusted so that anchor features and positive features are mapped closer to each other (i.e., pull), and anchor features and negative features are mapped farther away from each other (i.e., push). In other words, the encoder parameters can be adjusted so that the similarity metric between anchor samples and positive samples increases, and the similarity metric between anchor samples and negative samples decreases. For example, the inner product, Euclidean distance, and cosine similarity can be used to measure similarity.

[0100] A prediction model can be built by adding a classifier to the contrastive-learned encoder in Fig. 6a (for example, using transfer learning).

[0101] Referring to FIG. 6b, classification learning (e.g., learning decision boundary) can be performed based on additional input, CAC anomaly prediction probability for additional input, and label for additional input.

[0102] The parameters of the classifier can be adjusted using a classification learning loss function (e.g., Cross Entropy Loss). At this time, the parameters of the contrastively trained encoder can be frozen and only the parameters of the classifier can be adjusted, or both the parameters of the contrastively trained encoder and the parameters of the classifier can be adjusted.

[0103] Contrastive learning can be performed through two stages as in Fig. 6a and Fig. 6b, but contrastive learning can also be performed through one stage as in Fig. 7.

[0104] Referring to FIG. 7, the encoder can receive anchor samples and extract anchor feature information, receive positive samples and extract positive feature information, and receive negative samples and extract negative feature information. In addition, the classifier can output a CAC anomaly prediction probability value for the anchor sample based on the anchor feature information.

[0105] Using the contrastive learning loss function and the classification learning loss function, the parameters of the encoder and the parameters of the classifier can be adjusted together. Specifically, the contrastive learning loss function is calculated based on anchor feature information, positive feature information, and negative feature information, and the classification learning loss function can be calculated using the CAC abnormality prediction probability value and the label for the anchor sample. At this time, the loss values ​​according to the contrastive learning loss function and the loss values ​​according to the classification learning loss function can be integrated and used through simple summation, weighted summation, etc.

[0106] FIG. 8 illustrates an operational flowchart for performing prediction of CAC abnormalities according to one embodiment of the present invention. FIG. 8 may refer to operations performed by a server device (102). However, as described above, the CAC abnormality prediction service may be provided by a program or application installed on a user device (101) without a server device (102), in which case the server device (102) described below may refer to the user device (101).

[0107] The server device (102) can acquire multiple mammography images and clinical data of the subject. For example, the server device (102) can acquire a first mammography image, a second mammography image, and clinical data (S801).

[0108] For example, multiple mammography images may include images of the same breast of a subject taken from different orientations. For example, a first mammography image may be an image of the subject's breast taken from a first orientation, and a second mammography image may be an image of the subject's breast taken from a second orientation different from the first orientation.

[0109] The server device (102) may form a plurality of fused data sets by fusing clinical data for each of a plurality of mammography images (e.g., individually). For example, the server device (102) may form a first fused data set by fusing a first mammography image and clinical data, and may form a second fused data set by fusing a second mammography image and clinical data (S802).

[0110] The server device (102) can predict a CAC abnormality based on a plurality of fused data. For example, the server device (102) can predict a CAC abnormality based on the first fused data and the second fused data (S803).

[0111] For example, a plurality of mammography images may include RCC, LCC, RMLO, and LMLO. The plurality of fused data may include first fused data that fuses LCC and clinical data; second fused data that fuses RCC and clinical data; third fused data that fuses RMLO and clinical data; and fourth fused data that fuses LMLO and clinical data. The server device (102) may predict CAC abnormalities based on the first fused data, the second fused data, the third fused data, and the fourth fused data.

[0112] FIG. 9 briefly illustrates the framework of a deep learning system for predicting CAC abnormalities (hereinafter, abbreviated as a CAC abnormality prediction deep learning system) according to one embodiment of the present invention.

[0113] Referring to FIG. 9, CAC abnormality prediction can be performed by the CAC abnormality prediction model (500) of FIG. 5.

[0114] Multiple input data (e.g., INPUT-1 and INPUT-2) can be individually input into the CAC abnormality prediction model (500). For example, INPUT-1 and INPUT-2 can be the first fusion data and the second fusion data in FIG. 8, respectively.

[0115] The CAC anomaly prediction model (500) can output output data (e.g., OUTPUT-1 and OUTPUT-2) for each of a plurality of input data (e.g., individually and independently). For example, it can input first fusion data and output a first CAC anomaly prediction value (i.e., a CAC anomaly prediction probability value), and input second fusion data and output a second CAC anomaly prediction value.

[0116] The CAC anomaly prediction deep learning system can fuse OUTPUT-1 (e.g., the first CAC anomaly prediction value) and OUTPUT-2 (e.g., the second CAC anomaly prediction value) to output the final OUTPUT (e.g., a classification decision for CAC anomaly). For example, the fusion of OUTPUT-1 and OUTPUT-2 can be performed using an arithmetic mean, a weighted mean, a MAX selection, a MIN selection, etc.

[0117] For example, the plurality of input data may include first fusion data that fuses LCC and clinical data; second fusion data that fuses RCC and clinical data; third fusion data that fuses RMLO and clinical data; and fourth fusion data that fuses LMLO and clinical data. The first fusion data, the second fusion data, the third fusion data, and the fourth fusion data may be individually input into a CAC abnormality prediction model, and the CAC abnormality prediction model may output a first CAC abnormality prediction value, a second CAC abnormality prediction value, a third CAC abnormality prediction value, and a fourth CAC abnormality prediction value. The CAC abnormality prediction deep learning system may output a classification decision for CAC abnormality by fusion of the first CAC abnormality prediction value, the second CAC abnormality prediction value, the third CAC abnormality prediction value, and the fourth CAC abnormality prediction value.

[0118] Hereinafter, preferred embodiments and comparative examples of the present invention are described. However, the following examples are only preferred embodiments of the present invention, and the present invention is not limited to the following examples.

[0119] Examples

[0120] 1. Study participants

[0121] The study population consisted of female participants aged 18 years or older who underwent health screenings including mammography and coronary artery calcification computed tomography (CAC CT) at the Kangbuk Samsung Hospital General Medical Center between January 2010 and December 2020.

[0122] However, participants who were judged to have measurement errors in the Coronary Artery Calcium Score (CACS); participants who did not have at least one of the four-view mammographic images (Right cranio caudal (RCC), Left cranio caudal (LCC), Right mediolateral oblique (RMLO), and Left mediolateral oblique (LMLO)); participants who did not have information on postmenopausal status; and participants whose mammographic images had quality below a certain level were excluded from the study population. In addition, only participants with a CACS of 0 were included in the study population if the CACS was determined to be 0 at least twice when the CAC was measured.

[0123] As a result, the study population consisted of 6,433 female participants.

[0124] 2. Data Information

[0125] Information about the data contained in the dataset is as follows. The datasets were randomly divided in an 8:2 ratio and used for training and testing the prediction model. The data used in each example is listed in Table 2 below.

[0126] (1) Mammography image

[0127] Mammographic images measured using the Senograph 2000D / DMR / DS system (GE Healthcare) or the Selenia system (Hologic, Marlborough, MA, USA) were used.

[0128] Two CC-view images (i.e., RCC and LCC) taken in the craniocaudal direction for one breast and two MLO-view images (i.e., RMLO and LMLO) taken in the mediolateral oblique direction were used. Hereinafter, RCC, LCC, RMLO, and LMLO may be collectively referred to as four-view mammography images.

[0129] (2) Clinical data 1: Age

[0130] Age data was converted into categorical data and used by evenly distributing it at 10-year intervals (specifically, 30-39 years old: Group 1, 40-49 years old: Group 2, 50-59 years old: Group 3, 60-69 years old: Group 4, and 70 years old or older: Group 5).

[0131] (3) Clinical data 2: Postmenopausal status (PS)

[0132] Menopause status data was converted to dichotomous data and used (specifically, menopause status ○ and menopause status ×).

[0133] (4) Coronary Artery Calcium Score (CACS)

[0134] CACS data produced in the following manner were used.

[0135] Coronary artery calcification (CAC) examinations were performed using a Lightspeed VCT XTe-64 slice multi-detector CT scanner (GE Healthcare, Chicago, IL, USA) under electrocardiogram-gated dose modulation without intravenous contrast agent administration, according to a standard scanning protocol (2.5 mm thickness, 400 ms rotation time, 120 kV tube voltage, and 124 mAS tube current).

[0136] CACS analysis was performed by an expert using a semi-automated methodology and GE Smartscore software (GE Healthcare). The expert verified the reconstructed images using a 512 × 512 matrix in the axial plane and identified areas suspected of having CAC using a standard minimum calcium threshold of 130 Hounsfield units. CACS was calculated using Agatston units.

[0137] The CUT-OFF value (threshold) for CAC normal and abnormal was set differently to 1 (i.e., CACS=0 vs CACS≥1), 10 (i.e., CACS=0 vs CACS≥10), or 100 (i.e., CACS=0 vs CACS≥100) according to the examples. The CUT-OFF values ​​set for each example are shown in Table 2 below.

[0138] 3. CAC anomaly prediction deep learning system

[0139] (1) Data pre-processing

[0140] Mammography images were converted to Portable Network Graphics (PNG) format. The region of interest was cropped to remove the black background and the image size was resized to 512 × 1024. Contrast-limited adaptive histogram equalization was applied to improve image clarity.

[0141] Age data, converted to categorical data, and menopause status data, converted to dichotomous data, were each converted to matrix data with the same size as the mammography images. The matrix data and mammography images were used after being concatenated along the channel direction (i.e., channel-wise concatenation).

[0142] (2) Pretext task and backbone network

[0143] Among the encoders of known convolutional neural network (CNN) models (Resnet18, Resnet101, Inception-v3, and VGG19), the backbone network was selected.

[0144] To improve feature extraction performance from mammography images, we performed pretexting on the encoders of CNN models. This pretexting was performed using self-supervised learning based on image inpainting.

[0145] To select a backbone network, we built an evaluation deep learning system. Specifically, we added a classifier to the encoder that performed the pretext task. This built an evaluation prediction model that inputs mammography images and outputs predicted values ​​for CAC abnormalities and normals (i.e., classifies CAC abnormalities and normals).

[0146] The cut-off value of CACS, which classifies CAC abnormalities and normals, was set to 1 (i.e., CACS=0 vs CACS≥1). The deep learning system for evaluation was designed to use the evaluation prediction model to calculate the predicted value for CAC abnormalities for each four-view mammography image (i.e., RCC, LCC, RMLO, and LMLO), and to average the calculated predicted values ​​to determine the final predicted value.

[0147] The sensitivity, specificity, area under the ROC curve (AUROC), and accuracy of the deep learning system for evaluation were evaluated, and the evaluation results are shown in Table 1. As shown in Table 1, the classification performance was the best when the encoder of Resnet18 was used, so the encoder of Resnet18 was selected as the backbone network.

[0148] BackboneNetworkSensitivitySpecificityAUROCAccuracy(%)Resnet180.7300.5920.70266.1Resnet1010.7300.5860.69965.8Inception-v30.6760.5870.67163.1VGG190.6370.5800.63760.9

[0149] (3) Contrastive learning

[0150] A prediction model was built by adding a classifier that outputs a prediction value for CAC abnormality (i.e., classifies CAC abnormality and CAC normal) to the backbone network (encoder) that performed the pretext task (i.e., pre-trained backbone (encoder)).

[0151] As shown in Figure 10 (however, the illustration of the classifier is omitted), additional contrastive learning was performed for the prediction model.

[0152] The anchor sample was set as the fused data corresponding to the CAC abnormality class (i.e., the fused data of the anchor mammography image labeled as CAC abnormality and the clinical data). The positive sample was set as the fused data corresponding to the CAC abnormality class other than the anchor sample (i.e., the fused data of the positive mammography image labeled as CAC abnormality and the clinical data). The negative sample was set as the fused data corresponding to a class other than the anchor sample (i.e., the fused data of the negative mammography image labeled as CAC normal and the clinical data).

[0153] The encoder of the prediction model receives anchor samples as input and extracts anchor features from the anchor samples, receives positive samples as input and extracts positive features from the positive samples, and receives negative samples as input and extracts negative features from the negative samples. The classifier of the prediction model outputs a CAC abnormality prediction value for the anchor samples based on the anchor feature information.

[0154] The prediction model was trained using contrastive loss and cross-entropy loss (specifically, the sum of the contrastive loss and cross-entropy loss was used as the final training loss). Specifically, the contrastive loss was calculated based on anchor feature information, positive feature information, and negative feature information, and the cross-entropy loss was calculated using the CAC anomaly prediction value (i.e., the CAC anomaly prediction probability value) and the label for the anchor sample.

[0155] In contrastive learning, the parameters of the prediction model can be adjusted so that anchor features and positive features are mapped closer to each other in the feature space, while anchor features and negative features are mapped further away from each other. In other words, in contrastive learning, the model can be trained so that the similarity metric between anchor samples and positive samples increases, while the similarity between anchor samples and negative samples decreases.

[0156] (4) Downstream task

[0157] The trained prediction model (i.e., the Trained prediction model in Fig. 11) was applied to downstream tasks to classify CAC abnormalities and normals based on mammography images and clinical data.

[0158] Specifically, as shown in Fig. 11, the downstream operation of the deep learning system for predicting CAC abnormalities (hereinafter referred to as the CAC abnormality prediction deep learning system) consists of the following two-step process.

[0159] 1) Step 1

[0160] The CAC anomaly prediction deep learning system is designed to use the trained prediction model to produce a CAC anomaly prediction value (i.e., a CAC anomaly prediction probability value; CAC anomaly prediction probability) using the fusion data of LCC and clinical data (first input data), fusion data of RCC and clinical data (second input data), fusion data of RMLO and clinical data (third input data), and fusion data of LMLO and clinical data (fourth input data) as input data, respectively. The RCC, LCC, RMLO, LMLO, and clinical data are data from the same participant.

[0161] That is, the CAC anomaly prediction deep learning system can output a first CAC anomaly prediction value (Probability 1) from the first input data, a second CAC anomaly prediction value (Probability 2) from the second input data, a third CAC anomaly prediction value (Probability 3) from the third input data, and a fourth CAC anomaly prediction value (Probability 4) from the fourth input data.

[0162] 2) Step 2

[0163] The CAC anomaly prediction deep learning system is designed to average the output CAC anomaly prediction values ​​(i.e., the first CAC anomaly prediction value to the fourth CAC anomaly prediction value; Probability 1 to 4) to produce an average prediction value (Average Probability), and output a decision on CAC anomaly (CAC Anomaly Decision) based on the average prediction value. In other words, the CAC anomaly prediction deep learning system can make a final decision on CAC anomaly or CAC normal by averaging each CAC anomaly prediction value.

[0164] Comparative examples

[0165] As shown in Table 2, for the comparative examples, unlike the examples, contrast learning was not performed or clinical data was not used.

[0166] Mammography image Clinical data CACS CUT-OFFVALUE Contrastive Learning Example 1 ○ Age 1 ○ Example 2 ○ PS 1 ○ Example 3 ○ Age + PS 1 ○ Example 4 ○ Age 10 ○ Example 5 ○ PS 10 ○ Example 6 ○ Age + PS 10 ○ Example 7 ○ Age 100 ○ Example 8 ○ PS 100 ○ Example 9 ○ Age + PS 100 ○ Comparative Example 1 ○ - 1 × Comparative Example 2 ○ - 1 ○

[0167] Reference examples

[0168] The Framingham risk score (FRS) was calculated based on the participants' age, sex, total cholesterol (TC), HDL cholesterol, systolic blood pressure, and smoking status (never smoker or current smoker).

[0169] There is no established FRS threshold to define a non-zero CACS. However, a CACS of 0 typically indicates a low risk of cardiovascular disease. Similarly, an FRS <10% indicates a low risk of cardiovascular disease.

[0170] Therefore, with FRS 10% as the CUT-OFF value, CACS 0 and CACS not 0 (i.e., CACS=0 vs CACS≥1) were classified. That is, when FRS is less than 10, it is classified as CAC normal corresponding to CACS being 0, and when FRS is 10 or more, it is classified as CAC abnormal corresponding to CACS not being 0.

[0171] evaluation

[0172] The sensitivity, specificity, AUROC (area under ROC curve) and accuracy of the deep learning system for predicting CAC abnormalities according to the examples and comparative examples were evaluated and are shown in Table 3.

[0173] In addition, the sensitivity, specificity, AUROC, and accuracy of predicting CAC abnormalities using FRS according to reference examples were evaluated and are shown in Table 3.

[0174] SensitivitySpecificityAUROCAccuracy(%)Example 10.7910.6400.76671.5Example 20.8000.6370.77271.8Example 30.7850.6560.77672.1Example 4--0.804-Example 5--0.815-Example 6--0.822-Example 7--0.859-Example 8--0.858-Example 9--0.866-Comparative Example 10.7300.5920.70266.1Comparative Example 20.7640.6520.76170.8Reference Example 0.5640.9090.73674.6

[0175] Referring to Table 3, the examples showed improved sensitivity, specificity, area under the curve (AUROC), and accuracy compared to the comparative examples. In particular, the examples showed improved sensitivity, a relatively important indicator for predicting medical risk.

[0176] Furthermore, comparing the examples with the reference examples, it is believed that the CAC prediction deep learning system according to the examples can also be utilized for predicting cardiovascular disease risk. For example, the CAC abnormality prediction values ​​obtained by the CAC prediction deep learning system can be directly applied to predicting cardiovascular disease risk, or the CAC abnormality prediction values ​​can be utilized as a factor in predicting cardiovascular disease risk.

[0177] References to "one embodiment" of the principles of the present invention and various variations of this expression in this specification mean that a particular feature, structure, characteristic, etc., is included in at least one embodiment of the principles of the present invention in connection with that embodiment. Accordingly, the expression "in one embodiment" and any other variations disclosed throughout this specification are not necessarily all referring to the same embodiment.

[0178] The methods according to the various embodiments of the present invention described above may be implemented as a computer program or mobile application to be executed by combining a computer as hardware and stored on a medium. Alternatively, the steps of the methods or algorithms described in connection with the embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present invention pertains. In addition, the algorithm may be produced in the form of an installation file and provided in the form of an online download, and for this purpose, may be stored on a server accessible through an online software market.

[0179] All embodiments and conditional examples disclosed in this specification are intended to help those skilled in the art understand the principles and concepts of the present invention. Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from the essential characteristics thereof. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is set forth in the claims, not the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Claims

1. A method for predicting abnormalities of coronary artery calcification (CAC) using mammography images executed by at least one processor, Step of acquiring mammography images and clinical data of the subject; and A method for predicting CAC abnormality, comprising the step of determining whether the subject has a CAC abnormality based on the mammography image and the clinical data.

2. In claim 1, A method for predicting CAC abnormalities, wherein the clinical data includes data having a different modality from the mammography image.

3. In claim 1, A method for predicting CAC abnormalities, wherein the clinical data includes at least one of age data and menopause status data of the subject.

4. In claim 1, The steps to determine whether the above CAC is abnormal are: A step of forming fusion data by fusing the mammography image and the clinical data; and A method for predicting CAC abnormality, comprising: a step of determining whether CAC abnormality exists based on the above fusion data.

5. In claim 4, The step of forming the above fusion data is: A step of converting the above clinical data into matrix data having the same size as the above mammography image; and A method for predicting CAC abnormalities, comprising the step of concatenating the matrix data and the mammography image in the channel direction.

6. In claim 4, The above mammography image is, A first mammography image captured in a first direction and a second mammography image captured in a second direction different from the first direction, The step of forming the above fusion data is: comprising forming first fused data by fusing the first mammography image and the clinical data, and forming second fused data by fusing the second mammography image and the clinical data, The steps to determine whether the above CAC is abnormal are: A method for predicting CAC abnormality, comprising determining whether CAC abnormality exists based on the first fusion data and the second fusion data.

7. In claim 6, The steps to determine whether the above CAC is abnormal are: A step of calculating a first CAC abnormality prediction value based on the first fusion data, and calculating a second CAC abnormality prediction value based on the second fusion data; and A method for predicting CAC abnormality, comprising: a step of determining whether there is a CAC abnormality by fusing the first CAC abnormality prediction value and the second CAC abnormality prediction value using at least one of an arithmetic mean, a weighted mean, a MAX selection, and a MIN selection.

8. In claim 4, The step of determining whether CAC is abnormal or not based on the above fusion data is as follows: A step of extracting feature information from the above fusion data; and A method for predicting CAC abnormality, comprising: a step of classifying CAC abnormality or CAC normality based on the above characteristic information.

9. In claim 8, The step of determining whether CAC is abnormal or not based on the above fusion data is performed by a trained artificial intelligence model. The above trained artificial intelligence model, An encoder trained to extract the feature information from the fused data; and A method for predicting CAC abnormality, comprising a classifier trained to classify CAC abnormality or CAC normal based on the above feature information.

10. In claim 9, The above trained artificial intelligence model is trained according to the training phase, The above training phase is, A step of obtaining anchor fusion data corresponding to CAC abnormality, a plurality of positive fusion data corresponding to CAC abnormality other than the anchor fusion data, and a plurality of negative fusion data corresponding to CAC normality; A step of extracting feature information for each of the anchor fusion data, the positive fusion data, and the negative fusion data using an encoder; and A method for predicting CAC abnormality, comprising: a step of adjusting parameters of the encoder so that, in a feature space, a distance between feature information of the anchor fusion data and feature information of the positive fusion data becomes closer than a distance between feature information of the anchor fusion data and feature information of the negative data.

11. In claim 10, A method for predicting CAC anomalies, wherein the training phase further includes a step of performing representation learning based on self-supervised learning using inpainting for the encoder in advance.

12. A device for predicting abnormalities of coronary artery calcification (CAC) using mammography images, Department of Communications; memory; and comprising at least one processor connected to the communication unit and the memory; At least one processor of the above, Obtain mammographic images and clinical data of the subject, A CAC abnormality prediction device that controls whether the subject has a CAC abnormality based on the mammography image and the clinical data.

13. An application program stored on a recording medium that executes the method according to claim 1 when operated by at least one processor.

Images