Diagnostic apparatus, diagnostic method, diagnostic model, and program

Abstract

To estimate a generation part of myocardial infarction even from an electrocardiogram image in which a type of induction is not specified.SOLUTION: A diagnostic apparatus for diagnosing whether or not a subject has myocardial infarction includes: a diagnostic unit for diagnosing whether or not a subject has myocardial infarction by inputting a plurality of electrocardiogram images of the subject to a diagnostic model. The diagnostic model includes an input layer for converting a plurality of electrocardiogram images to a plurality of patches respectively, an encoder for encoding the plurality of patches converted by the input layer to a class token and a plurality of segment tokens, a pooling layer for generating one segment token on the basis of the plurality of segment tokens encoded by the encoder, and an output layer for outputting a result of the diagnosis on whether or not the subject has myocardial infarction on the basis of the class token encoded by the encoder and the one segment token generated by the pooling layer.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a diagnostic device, a diagnostic method, a diagnostic model, and a program. [Background technology]

[0002] In recent years, AI technology has been rapidly put to practical use, and methods have been proposed for automatically analyzing electrocardiogram waveforms using machine learning and deep learning models by treating them as time-series data or images. Patent Document 1 proposes a method for generating data for electrocardiogram analysis that enables efficient deep learning of relationships between multiple types of waveform information and leads in an electrocardiogram. Patent Document 2 proposes a method for generating data representing a two-dimensional image having a waveform image region and a rhythm image region as data for electrocardiogram analysis. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-130772 A [Patent Document 2] Patent Publication No. 2022-054202 Summary of the Invention [Problem to be solved by the invention]

[0004] In existing methods, the type of lead is specified for each waveform of an electrocardiogram image used as input data. However, the type of lead is not necessarily specified for all electrocardiogram images used to diagnose myocardial infarction. An object of some aspects of the present invention is to provide a technique for making it possible to estimate the site of myocardial infarction even from an electrocardiogram image in which the type of lead is not specified. [Means for solving the problem]

[0005] According to some embodiments, there is provided a diagnostic apparatus for diagnosing whether a subject has myocardial infarction, the diagnostic apparatus comprising: a diagnostic means for diagnosing whether the subject has myocardial infarction by inputting a plurality of electrocardiogram images of the subject into a diagnostic model, the diagnostic model including: an input layer for converting each of the plurality of electrocardiogram images into a plurality of patches; an encoder for encoding the plurality of patches converted by the input layer into a class token and a plurality of segment tokens; a pooling layer for generating one segment token based on the plurality of segment tokens encoded by the encoder; and an output layer for outputting a diagnostic result of whether the subject has myocardial infarction based on the class token encoded by the encoder and the one segment token generated by the pooling layer. Effect of the Invention

[0006] According to some embodiments, it is possible to estimate the site of myocardial infarction even from an electrocardiogram image in which the type of lead is not specified. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a diagnostic device according to some embodiments. [Diagram 2] FIG. 4 is a flow diagram illustrating an example of the operation of the diagnostic device according to some embodiments. [Diagram 3] 1 is a schematic diagram illustrating an example guidance estimation model according to some embodiments. [Figure 4] 10A to 10C are schematic diagrams illustrating an example of a patch generation method according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0009] A configuration of a diagnostic device 100 according to some embodiments will be described with reference to Fig. 1. As described below, the diagnostic device 100 may have a function of diagnosing whether a subject has myocardial infarction based on an electrocardiogram image of the subject. Furthermore, the diagnostic device 100 may have a function of diagnosing which part of the heart the subject has myocardial infarction. The electrocardiogram image may be an image showing a waveform measured by an electrocardiograph.

[0010] The diagnostic device 100 may include the components shown in FIG. 1. The processor 101 is a device that controls the overall operation of the diagnostic device 100. The processor 101 functions as, for example, a CPU (Central Processing Unit). The memory 102 is a device that stores programs and temporary data required for the operation of the diagnostic device 100. The memory 102 is, for example, configured by a RAM (Random Access Memory) or a ROM (Read Only Memory). The operation of the diagnostic device 100 may be performed, for example, by the processor 101 executing a program stored in the memory 102. Alternatively, a part or all of the operation of the diagnostic device 100 may be performed by a dedicated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Furthermore, the processor 101 may include a GPU (Graphical Processing Unit).

[0011] The input device 103 is a device for acquiring input from a user of the diagnostic device 100 (e.g., a doctor. Hereinafter, the user of the diagnostic device 100 will be simply referred to as a user). The input device 103 is constituted by, for example, a keyboard and a mouse. The output device 104 is a device for outputting to the user. The output device 104 is constituted by, for example, a display and a speaker.

[0012] The communication device 105 is a device for the diagnostic device 100 to communicate with other devices. The other devices may be computers connected to a network (e.g., the Internet or a local area network). For example, the communication device 105 may be used to obtain electrocardiogram images to be analyzed from other devices (e.g., an electrocardiograph or an image server).

[0013] The storage device 106 is a device that stores data used for the operation of the diagnostic device 100. The storage device 106 is configured by a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or a DVD (digital versatile disk). The storage device 106 may store a trained model used for diagnosing myocardial infarction.

[0014] An example of an operation for diagnosing whether or not a subject has myocardial infarction will be described with reference to Figures 2 to 4. Figure 2 illustrates the overall flow of the operation, and Figures 3 and 4 illustrate the details of the operation.

[0015] In S201, the diagnostic device 100 generates a diagnostic model 300 for diagnosing whether a subject has a myocardial infarction by machine learning, and stores the diagnostic model 300 in the storage device 106 for subsequent processing. An example of the structure of the diagnostic model 300 will be described with reference to FIG. 3.

[0016] The diagnostic model 300 acquires an electrocardiogram image set 310 of a subject as input data, and outputs a diagnosis result of whether or not the subject has myocardial infarction. The electrocardiogram image set 310 of a certain subject may be a collection of a plurality of electrocardiogram images 311a-311n representing electrocardiogram waveforms measured at the same time for the subject. The electrocardiogram image set 310 includes one or more electrocardiogram images 311a-311n. The electrocardiogram images 311a-311n are collectively referred to as electrocardiogram image 311. The following description of the electrocardiogram image 311 applies to any of the electrocardiogram images 311a-311n.

[0017] The electrocardiogram image 311 may be an image generated in any manner. For example, the electrocardiogram image 311 may be an image output from an electrocardiograph. Alternatively, the electrocardiogram image 311 may be an image of a paper on which an electrocardiogram waveform is recorded, which is imaged by a camera or a scanner. Alternatively, the electrocardiogram image 311 may be an image of an electrocardiogram waveform displayed on the display of an electrocardiograph, which is imaged by a camera.

[0018] The diagnostic model 300 is configured to be able to input electrocardiogram images 311 according to the number of waveforms generated by a specific method in one diagnostic process. For example, when the diagnostic model 300 is configured for a standard 12-lead electrocardiogram, the diagnostic model 300 may be able to input up to 12 electrocardiogram images 311 in one diagnostic process. In another example, when the diagnostic model 300 is configured for a derived 18-lead electrocardiogram, the diagnostic model 300 may be able to input up to 18 electrocardiogram images 311 in one diagnostic process.

[0019] The electrocardiogram image set 310 input to the diagnostic model 300 may include all electrocardiogram images 311 of a specific technique (for example, 12 electrocardiogram images 311 corresponding to 12 leads in the case of a standard 12-lead electrocardiogram), or may include only some of the electrocardiogram images 311. When the electrocardiogram image set 310 does not include some of the electrocardiogram images 311, a dummy electrocardiogram image (for example, an electrocardiogram image of only a baseline) may be input to the diagnostic model 300.

[0020] The diagnosis model 300 may include input layers 320a to 320n, an encoder 330, a pooling layer 340, and an output layer 350. The diagnosis model 300 may be a model used in a Vision Transformer (ViT) that has been modified by adding a pooling layer 340 and a linear layer 351 and by causing the encoder 330 to further output a plurality of segment tokens 331a to 331n.

[0021] The number of input layers 320a-320n corresponds to the number of waveforms generated by a particular method. For example, if the diagnostic model 300 is configured for a standard 12-lead electrocardiogram, the number of input layers 320a-320n is 12. The input layers 320a-320n are collectively referred to as the input layer 320. The following description of the input layer 320 applies to any of the input layers 320a-320n.

[0022] A separate electrocardiogram image 311 is input to each input layer 320. For example, an electrocardiogram image 311a is input to the input layer 320a, and an electrocardiogram image 311n is input to the input layer 320n. The type of lead of the electrocardiogram image 311 input to each input layer 320 may be different for each electrocardiogram image set 310. For example, an electrocardiogram image 311 of lead I may be input to the input layer 320a for one electrocardiogram image set 310, and an electrocardiogram image 311 of lead II may be input to the input layer 320a for another electrocardiogram image set 310. Such variation in the input of leads may be absorbed by the encoder 330 and the pooling layer 340.

[0023] The input layer 320 converts the input electrocardiogram image 311 into a plurality of patches. A patch may be an image obtained by extracting a part of the electrocardiogram image 311. Furthermore, the input layer 320 may convert the patch into a vector. The configuration of the input layer 320 may be the same as the input layer of ViT.

[0024] A method for generating a plurality of patches will be described with reference to Fig. 4. In the method of Fig. 4(a), the diagnostic device 100 divides one electrocardiogram image 311 into a plurality of patches 401 without omissions or overlaps. In the method of Fig. 4(b), the diagnostic device 100 divides one electrocardiogram image 311 into a plurality of patches 402 without omissions but overlapping with each other.

[0025] The encoder 330 encodes the multiple patches converted by the input layers 320a to 320n into a class token 322 and multiple segment tokens 331a to 331n. The encoder 330 generates the same number of segment tokens 331a to 331n as the input layers 320a to 320n. The diagnostic model 300 (particularly, the encoder 330) is trained so that information included in the multiple electrocardiogram images 311a to 311n is condensed into the class token 322. The diagnostic model 300 (particularly, the encoder 330) is trained so that information included in the electrocardiogram image 311a is condensed into the segment token 331a. The same applies to other segment tokens (for example, the segment token 331n).

[0026] The encoder 330 may be the same as the encoder of ViT. The encoder of ViT generates an output class token by converting an input class token. Therefore, the class token 322 may also be input to the encoder 330 of the diagnosis model 300. The class token 322 may be a vector of the same dimension as the vector representing the pitch output from the input layer 320. The initial value of the class token 322 may be set randomly.

[0027] Like the class token, the segment token 331a may be generated by converting the segment token 321a input to the encoder 330. The segment token 331a may also be a vector of the same dimension as the vector representing the pitch output from the input layer 320. The initial value of the segment token 321a may be set randomly. The segment token 321a input to the encoder may be processed in the encoder 330 in the same manner as the class token 322 (i.e., in the same manner as the class token of the ViT encoder). As described later, the class token 332 output from the encoder 330 is directly supplied to the output layer 350, and the segment token 331a output from the encoder 330 is supplied to the output layer 350 via the pooling layer 340. Due to this difference, the class token 332 condenses information contained in the multiple electrocardiogram images 311a to 311n into the class token 322, and the segment token 331a condenses information contained in each electrocardiogram image 311a into the class token 322. The above discussion of segment tokens 321a and 331a applies equally to other segment tokens (eg, segment tokens 321n and 331n).

[0028] The pooling layer 340 generates one segment token 341 based on the multiple segment tokens 331a to 331n encoded by the encoder 330. For example, the pooling layer 340 may perform maximum value pooling or average value pooling. Specifically, the pooling layer 340 may generate the segment token 341 by averaging the multiple segment tokens 331a to 331n. The segment token 341 output from the pooling layer 340 represents information obtained by integrating the segment tokens 331a to 331n, which are individually condensed pieces of information included in the multiple electrocardiogram images 311a to 311n.

[0029] The output layer 350 outputs a diagnosis result of whether the subject has myocardial infarction based on the class token 332 encoded by the encoder 330 and one segment token 341 generated by the pooling layer 340. The output layer 350 may include a linear layer 351 and a classifier 352. The linear layer 351 generates one vector by linearly transforming the class token 332 and the segment token 341. The classifier 352 outputs a diagnosis result of whether the subject has myocardial infarction based on this vector. The classifier 352 may be similar to a classifier of ViT (e.g., MLP Head).

[0030] The diagnostic device 100 may train the diagnostic model 300 in the same manner as the ViT model. When binary information on whether or not the subject has myocardial infarction is used as correct answer data for machine learning, the diagnostic model 300 outputs whether or not the subject has myocardial infarction as a diagnosis result. Alternatively, the diagnostic device 100 may train the diagnostic model 300 to estimate whether or not myocardial infarction has occurred in a specific part of the heart. In this case, the part where myocardial infarction has occurred (for example, the anterior wall or inferior wall of the heart) may be used as correct answer data for machine learning.

[0031] In S202, the diagnostic apparatus 100 acquires an electrocardiogram image set 310 of a subject to be diagnosed with myocardial infarction. When there are multiple subjects to be diagnosed with myocardial infarction, the diagnostic apparatus 100 may execute steps S202 to S204 for each subject. The electrocardiogram image set 310 may be specified in any manner. For example, the user may explicitly specify the electrocardiogram image set 310. For example, the user may input the path of a folder in which the electrocardiogram image set 310 is stored into the input device 103 of the diagnostic apparatus 100. Alternatively, the user may specify the electrocardiogram image set 310 using information for specifying the electrocardiogram image set 310. For example, the user may input the ID of the examination in which the electrocardiogram examination was performed into the input device 103 of the diagnostic apparatus 100.

[0032] In S203, the diagnostic device 100 diagnoses whether or not the subject has myocardial infarction by inputting a plurality of electrocardiogram images 311a to 311n into the diagnostic model 300. Furthermore, the diagnostic device 100 may diagnose whether or not myocardial infarction has occurred in a specific region.

[0033] The diagnostic device 100 may input the electrocardiogram image 311 of each lead to any of the input layers 320. For example, the diagnostic device 100 may input the electrocardiogram image 311 of lead I to the input layer 320a, or may input it to another input layer (e.g., the input layer 320n). Therefore, the diagnostic device 100 does not need to specify which lead's waveform each electrocardiogram image 311 included in the electrocardiogram image set 310 represents. As described above, the variation in the input position of the lead can be absorbed by the encoder 330 and the pooling layer 340.

[0034] Also in S203, the electrocardiogram image set 310 input to the diagnostic model 300 may include all electrocardiogram images 311 of a specific method (for example, 12 electrocardiogram images 311 corresponding to 12 leads in the case of a standard 12-lead electrocardiogram), or may include only some of the electrocardiogram images 311. When the electrocardiogram image set 310 does not include some of the electrocardiogram images 311, a dummy electrocardiogram image (for example, an electrocardiogram image of only a baseline) may be input to the diagnostic model 300.

[0035] In S204, the diagnostic device 100 displays the estimation result of S203 to the user. For example, if the diagnostic device 100 estimates that myocardial infarction has occurred in a specific part of the heart (e.g., the anterior wall, the inferior wall, etc.), it may display to the user that myocardial infarction has occurred in this specific part. Alternatively, if the diagnostic device 100 estimates that myocardial infarction has not occurred in any part of the heart, it may display to the user that the subject's heart is normal. The diagnostic device 100 may display the estimation result of S203 on the output device 104, or may output it in another manner. For example, the diagnostic device 100 may store the estimation result of S203 in the storage device 106 so that it can be referenced later, or may transmit it to another device.

[0036] In the above-mentioned method, the same diagnostic device 100 performs the learning of the diagnostic model (S201) and the estimation using the diagnostic model (S202 to S204). Alternatively, the learning of the diagnostic model (S201) and the estimation using the diagnostic model (S202 to S204) may be performed by separate devices. Specifically, the diagnostic device 100 may obtain a diagnostic model generated by a learning device and perform the operations of S202 to S204 using this diagnostic model.

[0037] According to the above-described method, it becomes possible to estimate the site of myocardial infarction even from an electrocardiogram image in which the type of lead is not specified. [Explanation of symbols]

[0038] 100 diagnostic equipment, 300 diagnostic models

Claims

1. A diagnostic device for diagnosing whether a subject has a myocardial infarction, comprising: a diagnostic means for diagnosing whether the subject has myocardial infarction by inputting a plurality of electrocardiogram images of the subject into a diagnostic model; The diagnostic model comprises: an input layer for converting each of the plurality of electrocardiogram images into a plurality of patches; an encoder for encoding the plurality of patches transformed by the input layer into class tokens and a plurality of segment tokens; a pooling layer that generates one segment token based on the plurality of segment tokens encoded by the encoder; and an output layer that outputs a diagnosis result of whether the subject has a myocardial infarction based on the class token encoded by the encoder and the one segment token generated by the pooling layer.

2. The diagnostic device of claim 1 , wherein the pooling layer generates the one segment token by averaging the multiple segment tokens.

3. The diagnostic device of claim 1 , wherein the plurality of patches include overlapping portions.

4. The diagnostic device according to claim 1 , wherein the diagnostic means diagnoses whether or not a myocardial infarction has occurred in a specific area.

5. A program for causing a computer to function as each of the means included in the diagnostic device according to any one of claims 1 to 4.

6. 1. A diagnostic method for diagnosing whether a subject has a myocardial infarction, comprising: a diagnostic step of diagnosing whether the subject has myocardial infarction by inputting a plurality of electrocardiogram images of the subject into a diagnostic model, The diagnostic model comprises: an input layer for converting each of the plurality of electrocardiogram images into a plurality of patches; an encoder for encoding the plurality of patches transformed by the input layer into a class token and a plurality of segment tokens; a pooling layer that generates one segment token based on the plurality of segment tokens encoded by the encoder; and an output layer that outputs a diagnosis result of whether the subject has a myocardial infarction based on the class token encoded by the encoder and the one segment token generated by the pooling layer.

7. A diagnostic model for diagnosing whether a subject has a myocardial infarction, comprising: an input layer for converting each of the plurality of electrocardiogram images into a plurality of patches; an encoder for encoding the plurality of patches transformed by the input layer into a class token and a plurality of segment tokens; a pooling layer that generates one segment token based on the plurality of segment tokens encoded by the encoder; and an output layer that outputs a diagnosis result of whether the subject has a myocardial infarction based on the class token encoded by the encoder and the one segment token generated by the pooling layer.

Images