Diagnostic apparatus, diagnostic method, and program

Abstract

To estimate a generation part of myocardial infarction even from an electrocardiogram image in which a type of induction is not specified.SOLUTION: A diagnostic apparatus for diagnosing whether or not a subject has myocardial infarction includes: an induction estimation unit for estimating a type of induction of each of a plurality of electrocardiogram images by inputting first data based on the plurality of electrocardiogram images of the subject to an induction estimation model; and an infarction estimation unit for estimating whether or not myocardial infarction is generated in a specific part by inputting second data based on one or more electrocardiogram images in which the type of induction is estimated to be of a specific type of the plurality of electrocardiogram images to an infarction estimation model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a diagnostic device, a diagnostic method, and a program. [Background technology]

[0002] In recent years, AI technology has been rapidly put to practical use, and methods have been proposed for automatically analyzing electrocardiogram waveforms using machine learning and deep learning models by treating them as time-series data or images. Patent Document 1 proposes a method for generating data for electrocardiogram analysis that enables efficient deep learning of relationships between multiple types of waveform information and leads in an electrocardiogram. Patent Document 2 proposes a method for generating data representing a two-dimensional image having a waveform image region and a rhythm image region as data for electrocardiogram analysis. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-130772 A [Patent Document 2] Patent Publication No. 2022-054202 Summary of the Invention [Problem to be solved by the invention]

[0004] In existing methods, the type of lead is specified for each waveform of an electrocardiogram image used as input data. However, the type of lead is not necessarily specified for all electrocardiogram images used to diagnose myocardial infarction. An object of some aspects of the present invention is to provide a technique for making it possible to estimate the site of myocardial infarction even from an electrocardiogram image in which the type of lead is not specified. [Means for solving the problem]

[0005] According to some embodiments, there is provided a diagnostic device for diagnosing whether a subject has myocardial infarction, the diagnostic device comprising: a lead estimation means for estimating a type of lead of each of a plurality of electrocardiogram images of the subject by inputting first data based on the plurality of electrocardiogram images into a lead estimation model; and an infarction estimation means for estimating whether myocardial infarction has occurred at a specific location by inputting second data based on one or more electrocardiogram images estimated to be a specific type of lead among the plurality of electrocardiogram images into an infarction estimation model. Effect of the Invention

[0006] According to some embodiments, it is possible to estimate the site of myocardial infarction even from an electrocardiogram image in which the type of lead is not specified. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a diagnostic device according to some embodiments. [Diagram 2] FIG. 4 is a flow diagram illustrating an example of the operation of the diagnostic device according to some embodiments. [Diagram 3] FIG. 4 is a schematic diagram illustrating an example of the operation of the diagnostic device according to some embodiments. [Figure 4] 1 is a schematic diagram illustrating an example guidance estimation model according to some embodiments. [Diagram 5] FIG. 1 is a schematic diagram illustrating an example of an infarction estimation model according to some embodiments. [Figure 6] FIG. 13 is a diagram illustrating an image explaining the estimation basis according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0009] A configuration of a diagnostic device 100 according to some embodiments will be described with reference to Fig. 1. As described below, the diagnostic device 100 may have a function of diagnosing whether a subject has myocardial infarction based on an electrocardiogram image of the subject. Furthermore, the diagnostic device 100 may have a function of diagnosing which part of the heart the subject has myocardial infarction. The electrocardiogram image may be an image showing a waveform measured by an electrocardiograph.

[0010] The diagnostic device 100 may include the components shown in FIG. 1. The processor 101 is a device that controls the overall operation of the diagnostic device 100. The processor 101 functions as, for example, a CPU (Central Processing Unit). The memory 102 is a device that stores programs and temporary data required for the operation of the diagnostic device 100. The memory 102 is, for example, configured by a RAM (Random Access Memory) or a ROM (Read Only Memory). The operation of the diagnostic device 100 may be performed, for example, by the processor 101 executing a program stored in the memory 102. Alternatively, a part or all of the operation of the diagnostic device 100 may be performed by a dedicated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Furthermore, the processor 101 may include a GPU (Graphical Processing Unit).

[0011] The input device 103 is a device for acquiring input from a user of the diagnostic device 100 (e.g., a doctor. Hereinafter, the user of the diagnostic device 100 will be simply referred to as a user). The input device 103 is constituted by, for example, a keyboard and a mouse. The output device 104 is a device for outputting to the user. The output device 104 is constituted by, for example, a display and a speaker.

[0012] The communication device 105 is a device for the diagnostic device 100 to communicate with other devices. The other devices may be computers connected to a network (e.g., the Internet or a local area network). For example, the communication device 105 may be used to obtain electrocardiogram images to be analyzed from other devices (e.g., an electrocardiograph or an image server).

[0013] The storage device 106 is a device that stores data used for the operation of the diagnostic device 100. The storage device 106 is configured by a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or a DVD (digital versatile disk). The storage device 106 may store a trained model used for diagnosing myocardial infarction.

[0014] An example of an operation for diagnosing whether or not a subject has myocardial infarction will be described with reference to Figures 2 to 6. Figure 2 describes an overall flow of the operation, and Figures 3 to 6 describe the details of the operation. The method shown in Figure 2 may be started by a user instructing the diagnostic device 100 to start diagnosis.

[0015] At the start of the operations of Figure 2, the diagnostic device 100 may have available to it a guided estimation model 310 and an infarct estimation model 320 (Figure 3), which may be stored in the storage device 106 or may be receivable from an external device, for example.

[0016] The lead estimation model 310 may be a model used to estimate the type of lead of an electrocardiogram image. The lead estimation model 310 may be generated by machine learning. As shown in FIG. 3, the lead estimation model 310 may include a feature detector 311 and a classifier 312, or may have other configurations. The feature detector 311 detects features from an electrocardiogram image input to the lead estimation model 310. The feature detector 311 may be configured by a neural network. Furthermore, the feature detector 311 may be configured by a convolutional neural. The classifier 312 classifies the type of lead of the electrocardiogram image based on the features detected from the electrocardiogram image. The classifier 312 may be configured by a decision tree using gradient boosting (e.g., XGboost).

[0017] The diagnostic device 100 may acquire the guidance estimation model 310 in any manner. For example, the diagnostic device 100 may generate the guidance estimation model 310 by executing machine learning by itself. Alternatively, the diagnostic device 100 may acquire the guidance estimation model 310 from a device other than the diagnostic device 100, the device having generated the guidance estimation model 310 by executing machine learning by the device.

[0018] A method for generating feature detector 311 will be described with reference to Fig. 4. In the following example, a case will be described in which diagnostic device 100 generates feature detector 311, but as described above, other devices may generate feature detector 311.

[0019] The diagnostic device 100 may train the model 400. As shown in FIG. 4, the model 400 includes an input layer, multiple convolution layers, multiple pooling layers, multiple fully connected layers, and an output layer. For some layers of the model 400, the size of the input data is shown in parentheses. For example, an electrocardiogram image 403 (tensor data) having a size of 3×112×224 is input to the input layer. The first dimension ("3") indicates the number of channels of the electrocardiogram image 403 (for example, a color space represented by RGB), the next dimension ("112") may represent the height of the electrocardiogram image 403, and the last dimension ("224") may represent the width of the electrocardiogram image 403. That is, the electrocardiogram image 403 input to the model 400 has pixels arranged in a 112×224 array, and each pixel has a pixel value for each of three channels. Vector data including 12 elements is output from the output layer. Each of the 12 elements represents the probability that the electrocardiogram image 403 is a particular lead of a 12-lead electrocardiogram.

[0020] The diagnostic device 100 trains the model 400 using electrocardiogram images with correct labels (types of leads). For example, the diagnostic device 100 may use cross-entropy error as a loss function and Adam as an optimization method. After the training is completed, the diagnostic device 100 may use a front portion 401 of the model 400 as the feature detector 311. The portion 401 includes a convolutional neural network.

[0021] The infarction estimation model 320 may be a model used to estimate whether or not myocardial infarction has occurred in a specific region. The infarction estimation model 320 may be generated by machine learning. As shown in FIG. 3, the infarction estimation model 320 may include a plurality of region diagnostic devices 321a to 321m and a comprehensive determiner 323, or may have other configurations. The plurality of region diagnostic devices 321a to 321m are collectively referred to as region diagnostic device 321. The description of the region diagnostic device 321 may be applied to any of the plurality of region diagnostic devices 321a to 321m.

[0022] The part diagnoser 321 estimates whether myocardial infarction has occurred in a specific part of the heart based on one or more electrocardiogram images that are a specific type of lead. The parts where the occurrence of myocardial infarction is estimated by the multiple part diagnosers 321a to 321m may be different from each other. For example, the part diagnoser 321a may estimate the occurrence of myocardial infarction in the anterior wall of the heart. The part diagnoser 321m may estimate the occurrence of myocardial infarction in the inferior wall of the heart. In this way, the infarction estimation model 320 may use an individual model for each of the multiple parts. The part diagnoser 321 may be configured by a neural network. Furthermore, the part diagnoser 321 may be configured by a convolutional neural. The overall determiner 323 integrates the estimation results by the multiple part diagnosers 321a to 321m and outputs them as a diagnosis result.

[0023] The diagnostic device 100 may acquire the infarction estimation model 320 in any manner. For example, the diagnostic device 100 may generate the infarction estimation model 320 by executing machine learning by itself. Alternatively, the diagnostic device 100 may acquire the infarction estimation model 320 from a device other than the diagnostic device 100, the device having generated the infarction estimation model 320 by executing machine learning by the device.

[0024] A method for generating the part diagnostic device 321 will be described with reference to Fig. 5. In the following example, a case will be described in which the diagnostic device 100 generates the part diagnostic device 321, but as described above, another device may generate the part diagnostic device 321.

[0025] The diagnostic device 100 may train a model 500. As shown in FIG. 5, the model 500 includes an input layer, multiple convolution layers, multiple pooling layers, multiple fully connected layers, and an output layer. For some layers of the model 500, the size of the input data is indicated in parentheses. For example, a synthetic electrocardiogram image 502 (tensor data) having a size of 3×112×224 is input to the input layer. The data structure of the synthetic electrocardiogram image 502 may be the same as that of the electrocardiogram image 403, and therefore a duplicated description will be omitted.

[0026] The diagnostic device 100 may generate a composite electrocardiogram image 502 by superimposing electrocardiogram images 501a to 501c of two or more types (three types in the example of FIG. 5) of leads used for occurrence of myocardial infarction in a specific region. Each of the electrocardiogram images 501a to 501c may have the same data structure as the composite electrocardiogram image 502, or the composite electrocardiogram image 502 may be generated by adding or averaging the multiple electrocardiogram images 501a to 501c for each element. By inputting the composite electrocardiogram image 502 generated in this manner to the model 500, it becomes possible to spatially approximate the values ​​of the waveforms measured at the same time in the multiple electrocardiogram images 501a to 501c. As a result, the learning accuracy of the model 500 is improved. Vector data including two elements is output from the output layer. Each of the two elements indicates whether or not myocardial infarction has occurred in a specific region.

[0027] The diagnostic device 100 trains the model 500 using a synthetic electrocardiogram image with a correct answer label (presence or absence of myocardial infarction). For example, the diagnostic device 100 may use cross entropy error as a loss function and Adam as an optimization method. After training is completed, the diagnostic device 100 may use the model 500 as the region diagnostician 321. The model 500 includes a convolutional neural network. The diagnostic device 100 may train the model 500 separately for each of the multiple region diagnosticians 321a to 321m.

[0028] In S201, the diagnostic device 100 acquires an electrocardiogram image set 300 (FIG. 3) of a subject to be diagnosed with myocardial infarction. The electrocardiogram image set 300 of a certain subject may be a collection of a plurality of electrocardiogram images 301a-301n representing electrocardiogram waveforms measured at the same time for this subject. The plurality of electrocardiogram images 301a-301n are collectively referred to as electrocardiogram image 301. The description of the electrocardiogram image 301 may be applied to any of the plurality of electrocardiogram images 301a-301n.

[0029] When there are multiple subjects to be diagnosed with myocardial infarction, the diagnostic device 100 may execute steps S201 to S204 for each subject. The electrocardiogram image set 300 may be specified in any manner. For example, the user may explicitly specify the electrocardiogram image set 300. For example, the user may input the path of a folder in which the electrocardiogram image set 300 is stored into the input device 103 of the diagnostic device 100. Alternatively, the user may specify the electrocardiogram image set 300 using information for specifying the electrocardiogram image set 300. For example, the user may input the ID of the examination in which the electrocardiogram examination was performed into the input device 103 of the diagnostic device 100.

[0030] The electrocardiogram image 301 may be an image generated in any manner. For example, the electrocardiogram image 301 may be an image output from an electrocardiograph. Alternatively, the electrocardiogram image 301 may be an image of a paper on which an electrocardiogram waveform is recorded, which is imaged by a camera or a scanner. Alternatively, the electrocardiogram image 301 may be an image of an electrocardiogram waveform displayed on the display of an electrocardiograph, which is imaged by a camera.

[0031] The waveform shown by the electrocardiogram image 301 may be a waveform generated by any method. For example, the waveform shown by the electrocardiogram image 301 may be any waveform of a standard 12-lead electrocardiogram, or any waveform of a derived 18-lead electrocardiogram. One electrocardiogram image set 300 may include images of waveforms for all leads of an electrocardiogram of a specific method, or may not include images of waveforms for some leads of an electrocardiogram of a specific method. Furthermore, it may be unclear which lead the electrocardiogram image 301 represents the waveform obtained by.

[0032] When a single image including multiple electrocardiogram waveforms is specified, the diagnostic device 100 may generate multiple electrocardiogram images each including one electrocardiogram waveform by dividing the single image.

[0033] In S202, the diagnostic device 100 estimates the lead type of each of the electrocardiogram images 301a-301n by inputting data based on the electrocardiogram images 301a-301n to the lead estimation model 310. For example, the diagnostic device 100 may estimate that the lead type of the electrocardiogram image 301a is lead II by inputting the electrocardiogram image 301a to the lead estimation model 310.

[0034] In S203, the diagnostic device 100 estimates whether myocardial infarction has occurred in a specific region by inputting data based on one or more electrocardiogram images estimated to be a specific type of lead among the multiple electrocardiogram images 301a to 301n to the infarction estimation model 320. In this step, the diagnostic device 100 may estimate whether myocardial infarction has occurred in each of the multiple regions. As described above with reference to FIG. 5, when there are two or more specific types, the diagnostic device 100 may generate input data by superimposing two or more electrocardiogram images estimated to be a specific type of lead.

[0035] For example, the region diagnostic device 321a may estimate the occurrence of myocardial infarction in the anterior wall of the heart by using, as input data, an electrocardiogram image 301a estimated to be a lead II and an electrocardiogram image 301b estimated to be a lead I. The region diagnostic device 321m may estimate the occurrence of myocardial infarction in the inferior wall of the heart by using, as input data, an electrocardiogram image 301a estimated to be a lead II, an electrocardiogram image 301b estimated to be a lead I, and an electrocardiogram image 301n estimated to be a lead aVR. In this way, the type of lead used as input data may differ for each region to be estimated. The type of lead to be used by each of the region diagnostic devices 321a to 321m may be set in advance and stored in the storage device 106. Alternatively, it may be specified by the user at the start of the operation of FIG. 2.

[0036] Depending on the electrocardiogram image set 300, an electrocardiogram image of a specific lead may not be included. For example, assume that the electrocardiogram image set 300 does not include an electrocardiogram image of lead III. In this case, the diagnostic device 100 may not estimate the occurrence of myocardial infarction for a region using an electrocardiogram image of lead III. Furthermore, the diagnostic device 100 may notify the user that the occurrence of myocardial infarction could not be estimated for a specific region because the specific electrocardiogram image (in this example, an electrocardiogram image of lead III) was not included.

[0037] In S204, the diagnostic device 100 displays the estimation result of S203 to the user. For example, if the diagnostic device 100 estimates that myocardial infarction has occurred in a specific part of the heart (e.g., the anterior wall, the inferior wall, etc.), it may display to the user that myocardial infarction has occurred in this specific part. Alternatively, if the diagnostic device 100 estimates that myocardial infarction has not occurred in any part of the heart, it may display to the user that the subject's heart is normal. The diagnostic device 100 may display the estimation result of S203 on the output device 104, or may output it in another manner. For example, the diagnostic device 100 may store the estimation result of S203 in the storage device 106 so that it can be referenced later, or may transmit it to another device.

[0038] In S205, the diagnostic device 100 may generate an image 600 (FIG. 6) showing a portion of the electrocardiogram images of the diagnosis target that is the basis for the estimation in S203, and display the image 600 to the user. For example, the diagnostic device 100 may generate a heat map of the estimation in S202 for each electrocardiogram image 301 in the electrocardiogram image set 300 using a class activation mapping (CAM) technique (e.g., Grad-cam), and overlay the heat map on the electrocardiogram image 301. This allows the user to recognize which portion of the electrocardiogram image the estimation is based on.

[0039] The diagnostic device 100 may generate an image 601 (FIG. 6) showing a portion of one or more electrocardiogram images used to estimate the infarction that is the basis for the estimation of S204, and display the image 601 to the user. For example, the diagnostic device 100 may generate a heat map of the estimation of S203 using a class activation mapping (CAM) technique (e.g., Grad-cam) and overlay this heat map on one or more electrocardiogram images used to estimate a specific region. This allows the user to recognize which portion of the electrocardiogram image the estimation is based on.

[0040] According to the above-described method, it becomes possible to estimate the site of myocardial infarction even from an electrocardiogram image in which the type of lead is not specified. [Explanation of symbols]

[0041] 100 diagnostic device, 310 induction estimation model, 320 infarction estimation model

Claims

1. A diagnostic device for diagnosing whether a subject has a myocardial infarction, comprising: a lead estimation means for estimating a type of each lead of the plurality of electrocardiogram images by inputting first data based on the plurality of electrocardiogram images of the subject to a lead estimation model; and an infarction estimation means for estimating whether or not a myocardial infarction has occurred in a specific location by inputting second data based on one or more electrocardiogram images estimated to be a specific type of lead among the plurality of electrocardiogram images into an infarction estimation model.

2. The diagnostic device according to claim 1 , wherein the infarction estimation means generates the second data by superimposing two or more electrocardiogram images estimated to be the specific type of lead when the specific type is two or more.

3. the infarction estimation means estimates whether or not a myocardial infarction has occurred in each of a plurality of sites; The diagnostic device according to claim 1 , wherein the infarction estimation means uses an individual model for each of the plurality of regions.

4. The diagnostic device according to claim 1 , further comprising a generating unit configured to generate an image showing a portion of the plurality of electrocardiogram images that serves as a basis for estimation by the lead estimation unit.

5. The diagnostic device according to claim 1 , further comprising a generating unit configured to generate an image showing a portion of the one or more electrocardiogram images that serves as a basis for the infarction estimation by the infarction estimation unit.

6. The diagnostic device of claim 1 , wherein the guided estimation model includes a convolutional neural network.

7. The diagnostic device of claim 1 , wherein the infarction estimation model includes a convolutional neural network.

8. A program for causing a computer to function as each of the means included in the diagnostic device according to any one of claims 1 to 7.

9. 1. A diagnostic method for diagnosing whether a subject has a myocardial infarction, comprising: a lead estimation step in which a lead estimation means estimates a type of each lead in the plurality of electrocardiogram images by inputting first data based on the plurality of electrocardiogram images of the subject to a lead estimation model; and an infarction estimation step in which an infarction estimation means estimates whether or not a myocardial infarction has occurred in a specific location by inputting second data based on one or more electrocardiogram images estimated to be a specific type of lead among the plurality of electrocardiogram images into an infarction estimation model.

Images