Artificial intelligence-based support for detecting cardiac amyloidosis

Abstract

A method for supporting the detection of cardiac amyloidosis, Receiving at least one current video obtained from a cardiac ultrasound examination, wherein the current video comprises multiple current images, each having an ultrasound signal and additional information. A processing step comprising the step of processing each of the current images of the current video in order to remove additional information from the current video, An inference step comprising the step of performing inference using a classification model by inputting at least one input video obtained from processed videos into the classification model, wherein the classification model is a convolutional neural network that performs (2+1)D spatiotemporal convolution, A detection support method comprising the following features.

Description

Technical Field

[0001] The present invention relates to the field of detection of cardiac amyloidosis.

Background Art

[0002] Cardiac amyloidosis is a disease in which insoluble proteins accumulate in heart tissue. It is a serious disease with a poor prognosis unless appropriate treatment and management are carried out.

[0003] However, it is particularly difficult to detect, especially in the early stages. This means that it has been actually considered to be very rare and underdiagnosed for a long time.

[0004] Therefore, as new effective treatment methods for the early stages of the disease are currently emerging, it is considered particularly beneficial to develop a diagnostic support system for early detection of this disease.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Known diagnostic support systems use algorithms for detecting cardiac amyloidosis from cardiovascular MRI (or CMR, cardiovascular magnetic resonance) images. However, CMR examinations are costly. This high cost causes a two-fold problem in that it is difficult to obtain a sufficient number of "reference" images for effectively improving the detection algorithm, it hinders "large-scale" screening programs for the population, although such programs are considered necessary to detect the early stages of this disease.

[0006] Other known diagnostic support systems use electrocardiograms of patients in an attempt to detect the onset of amyloidosis early. This examination is less expensive than CMR. However, the detection accuracy is insufficient, which limits the advantages or hinders the use of this system.

[0007] The objective of this invention is to enable the early detection of cardiac amyloidosis in an inexpensive and effective manner. [Means for solving the problem]

[0008] To achieve this objective, a method for assisting in the detection of cardiac amyloidosis in a patient, which is performed in a processing unit, A receiving step comprising receiving at least one current video obtained from a cardiac ultrasound examination performed on a patient, wherein the current video comprises a plurality of current images, each having an ultrasound signal and additional information, A processing step comprising the step of processing each of the current images of the current video in order to remove additional information from the current video in order to generate a processed video from the current video, An inference step comprising the step of performing inference using a classification model, wherein the classification model is a convolutional neural network performing (2+1)D spatiotemporal convolution, by inputting at least one input video obtained from processed videos into a classification model pre-trained with a set of training videos comprising cardiac amyloidosis cases and control cases, and the classification model is a convolutional neural network performing (2+1)D spatiotemporal convolution. We propose a detection support method that includes the following features.

[0009] The classification model used is particularly effective in detecting cardiac amyloidosis. This is because the classification model can capture extremely precise spatiotemporal characteristics and reduces computational costs by separating spatial and temporal convolutions.

[0010] By performing the processing steps and using this classification model, the early stages of cardiac amyloidosis can be detected very effectively.

[0011] Cardiac ultrasound is significantly less expensive than CMR, thus overcoming the aforementioned drawbacks.

[0012] The convolutional neural network also proposes the aforementioned detection support method based on ResNet.

[0013] We also propose the above-described detection support method, wherein the inference stage comprises the step of generating a plurality of image packets from a processed video, each containing a first predetermined number of consecutive images, the image packets forming an input video that is fed into a classification model, and the inference stage comprises the step of generating a plurality of intermediate predictions, each associated with a different image packet.

[0014] We also propose the detection support method described above, in which image packets are defined sequentially, and each image packet is temporally shifted by a second predetermined number of images relative to the previous image packet.

[0015] We also propose the detection support method described above, in which, for the current video, if at least one intermediate prediction obtained for an image packet is a positive prediction that detects the presence of amyloidosis and is associated with a confidence score that exceeds a predetermined threshold, the inference stage generates a final prediction that is a positive prediction.

[0016] The processing step includes obtaining the coordinates of a reference point that defines a useful portion of the current image for each current image of the current video, and the detection support method described above, which includes an ultrasonic signal, is also proposed.

[0017] We also propose the above-mentioned detection support method, which includes a step in the processing stage of cropping the current image in order to retain only the portion of the current image that passes through the reference point and contains useful parts.

[0018] The additional information also proposes the detection assistance method described above, which includes a scale and further comprises a step of applying a bitmask to a useful portion of the current image to remove the scale.

[0019] The additional information has an electrical signal representation, and the processing stage includes a step of detecting pixels corresponding to the electrical signal representation within the current image, and a step of removing the electrical signal representation by applying a median filter to each of the pixels. The above-described detection support method is also proposed.

[0020] The above-described detection support method is also proposed, which further includes a step of superimposing a heat map obtained from the final convolutional layer of the convolutional neural network on the current image.

[0021] An ultrasonic diagnostic apparatus including a processing unit for implementing the above-described detection support method is also proposed.

[0022] A computer program including instructions for causing a processing unit to execute the steps of the above-described detection support method is also proposed.

[0023] A computer-readable storage medium storing the above-described computer program is also proposed.

[0024] A learning method for a classification model of the above-described detection support method, which is executed by a processing unit, includes a receiving step of receiving a set of reference videos obtained from echocardiograms performed on reference patients, processing steps that are performed on each of the reference videos and are the same as the processing steps of the detection support method, a training step of training a classification model using a set of training videos belonging to the set of reference videos, where the set of training videos includes cardiac amyloidosis cases and control cases, A learning method including the above is also proposed.

[0025] The above-described learning method is also proposed, which further includes a preceding step of applying another pre-trained classification model to the reference images to retain only the A4c view among the reference images of the reference videos.

[0026] The present invention is better understood by referring to the following description of specific non-limiting embodiments. [Brief explanation of the drawing]

[0027] Refer to the attached drawings.

[0028] [Figure 1] Figure 1 shows the steps of the training method for the classification model. [Figure 2] Figure 2 shows the processing unit that processes the reference video and trains the classification model. [Figure 3] Figure 3 shows a histogram illustrating the first distribution of reference videos obtained by sonograph. [Figure 4] Figure 4 is similar to Figure 3 but after applying the A4c filter. [Figure 5] Figure 5 shows the processing of the reference image. [Figure 6] Figure 6 schematically shows a 3D convolutional layer and a (2+1)D convolutional layer. [Figure 7] Figure 7 shows a graph containing the loss curve and various metrics obtained during model training and validation. [Figure 8] Figure 8 shows a processing unit that processes the current video and implements detection support methods. [Figure 9] Figure 9 shows the steps of the detection support method. [Figure 10] Figure 10 shows images illustrating hatching, which is a simplified visualization of the ultrasonic signal and thermal map, respectively. [Figure 11] Figure 11 shows a histogram illustrating the second distribution of reference videos obtained by sonograph. [Modes for carrying out the invention]

[0029] The cardiac amyloidosis detection support method according to the present invention is based on the use of a video classification model. The classification model is a neural network, as will be described later.

[0030] The classification model used to diagnose cardiac amyloidosis in patients needs to be trained in advance during or after the examination. This training is supervised learning using videos (hereinafter referred to as "reference videos") acquired during cardiac ultrasound examinations performed on patients (hereinafter referred to as "reference patients").

[0031] First, I will explain the model training method while referring to Figure 1. Here, the term "training method" refers to both the processing of the reference video and the training of the model.

[0032] The data used to train the classification model (see video) was collected at AP-HP Bisha Claude Bernard Hospital in close cooperation with the hospital's IT department: Step E1.

[0033] Therefore, the reference video is an echocardiogram loop (or "echocardiogram film") obtained during an actual examination performed by a physician on the reference patient (consent for the use of the patient's data has been obtained).

[0034] The physician used the hospital's sonograph to generate a reference video. The reference video Vref was sent to a processing unit 1 integrated into a server installed and configured within the hospital. This processing unit 1 is shown in Figure 2.

[0035] The reference video was processed by processing unit 1, and the classification model was trained by processing unit 1. Therefore, the data flow remained confined to the hospital network.

[0036] Processing Unit 1 is an electronic and software unit. Processing Unit 1 comprises one or more processing components, such as any processor or microprocessor, whether general-purpose or dedicated, such as a DSP (Digital Signal Processor), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), microcontroller, or programmable logic circuit such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit).

[0037] In this case, processing unit 1 is equipped with two GPUs 2. For example, both GPUs 2 are RTX® 3090 components.

[0038] The processing unit 1 further comprises memory. At least one of these memories forms a computer-readable storage medium storing at least one computer program, and the computer program comprises instructions that cause the processing unit 1 to execute at least a portion of the steps of a learning method.

[0039] Reference videos are either transferred directly to processing unit 1 by the hospital's sonograph 3 or transferred to processing unit 1 from another server that receives the videos. The data is collected in DICOM format and includes over 10,000 reference videos from eight different sonographs and more than 800 patients.

[0040] In these reference videos, the proportion of positive cardiac amyloidosis cases is approximately 20%.

[0041] To train the classification model using only "raw" ultrasound signals, i.e., without relying on other information, only grayscale reference videos were considered. Using this type of video as input data ensures that subsequent classifications performed on actual patients using the acquired inference model are effective and accurate.

[0042] Therefore, color videos of the ECO-Plural signal were excluded from the set of reference videos by analyzing the difference in pixel intensity between each of the three channels of the RGB image: step E2. Grayscale images are characterized by substantially identical RGB channels.

[0043] Furthermore, to avoid bias in the classification model, it is preferable that the data is distributed approximately uniformly using a sonograph.

[0044] The reference video forms a set of videos divided into three subsets: a set of training videos, a set of verification videos, and a set of test videos: Step E3.

[0045] Each sonograph needs to properly represent the amount of positive and negative amyloidosis data. However, only three of the eight sonographs produced films for both patients and healthy controls.

[0046] Therefore, to avoid machine bias when training the classification model, the training video set and the validation video set will consist only of videos from these three sonographs.

[0047] Furthermore, to ensure that the positive and negative data were distributed fairly evenly, some negative data from these three sonographs were omitted.

[0048] The set of test videos includes the remaining reference videos.

[0049] The distribution of the data after data selection is shown in the histogram in Figure 3. The reference video was generated using eight sonographs 3a, 3b, 3c, 3d, 3e, 3f, 3g, and 3h.

[0050] The training video set for training the classification model includes cases of cardiac amyloidosis and control cases.

[0051] The case of cardiac amyloidosis is a video (echocardiogram film) of a patient with cardiac amyloidosis.

[0052] The control cases are videos (echocardiographic films) of "healthy" patients, i.e., patients who do not have cardiac amyloidosis or certain characteristics that may influence the development of this disease.

[0053] In this case, the training video set included 915 videos corresponding to 62 patients, with 395 cases of cardiac amyloidosis and 520 control cases. The validation video set included 306 videos corresponding to 14 patients, with 131 cases of cardiac amyloidosis and 175 control cases. The test video set included 7268 echocardiographic films corresponding to 722 patients, with 1351 cases of cardiac amyloidosis and 5917 control cases.

[0054] It should be noted here that the collected videos may include images from all kinds of views: parasternal long-axis view, short-axis view, apical two-chamber view, apical three-chamber view, apical four-chamber view, subcostal view, etc. The apical four-chamber (A4c) view provides the most relevant information for analyzing conditions such as amyloidosis.

[0055] Therefore, only images corresponding to the A4c view can be selected from the echocardiogram film.

[0056] However, the view type is not information provided by the DICOM standard.

[0057] Therefore, a pre-trained classification model for classifying echocardiographic views was used to automatically detect views in echocardiographic films: step E4. The distribution of data containing only A4c views is shown in Figure 4. The classification model used is a deep learning model. With the cooperation of cardiologists, the accuracy of the classification model was estimated to be approximately 80%. Therefore, this filter is sufficiently effective.

[0058] The filtering step of this view is not essential for carrying out the present invention.

[0059] Next, processing unit 1 performs the processing stage Ph of the reference video.

[0060] Echocardiographic films typically display ultrasound signals while simultaneously adding other text or graphic information, such as scales or signals indicating electrical activity, around them. Processing step Ph allows for the removal of this information, which may bias learning, leaving only the ultrasound signals.

[0061] Each reference video comprises multiple reference images. Each reference video comprises an ultrasonic signal and additional information.

[0062] Therefore, depending on the processing stage, each reference image of the reference video can be processed to remove additional information from the reference image in order to generate a processed video from the reference video.

[0063] The processing stage Ph first includes a trimming step E5 which is performed in the first module 4a, which is implemented in the processing unit 1.

[0064] Referring to Figure 5, each reference image Iref in each reference video contains an ultrasonic signal 5 included in the useful portion 6 of the image. The useful portion 6 typically has the shape of a corner segment, with its circular portion located at the bottom of the image.

[0065] The coordinates of the reference points that define the useful region of the reference image, i.e., the three vertices S1, S2, and S3 of the corner segment in this case, are input into the DICOM file.

[0066] Therefore, for each reference image in each reference video, the cropping step involves obtaining the coordinates of these vertices S1, S2, S3 and cropping the reference image Iref to retain only the portion of the reference image that passes through the reference points and contains the useful parts. The cropped image Ir is obtained, and its upper and side edges pass through the vertices S1, S2, S3 of the corner segment.

[0067] The additional information includes a scale 7 that represents the ultrasonic signal.

[0068] The processing step then comprises the step of applying a bitmask to the useful portion 6 of the reference image to remove the scale: step E6. This step is performed in the second module 4b, which is implemented in the processing unit 1.

[0069] First, it is necessary to detect the coordinates of the three vertices of the corner segment in the cropped image Ir.

[0070] Vertex S1 is detected by selecting the point located in the center of the upper edge of the cropped image. The other two vertices, S2 and S3, are detected by detecting the maximum intensity of pixels in the two columns on the left and right sides of the cropped image, respectively. Once these vertices are detected, a bitmask is applied.

[0071] The additional information also includes electrical signal representation 9.

[0072] This representation is formed by pixels that have a color close to cyan blue.

[0073] Processing step Ph comprises the steps of detecting pixels corresponding to the electrical signal representation 9 of the reference image (i.e., pixels with a color close to cyan blue) and removing the electrical signal representation 9 by applying a median filter to each pixel: step E7.

[0074] These steps are executed in a third module 4c, which is implemented in processing unit 1.

[0075] A processed, trimmed image (Irt) containing only the ultrasound signal and no additional information is obtained.

[0076] Advantageously, processing stage Ph comprises a step of spatially resampling (in this case subsampling) each of the reference videos to obtain a video in a standardized format, for example, a video in which each image is 112 × 112 pixels in size: step E8. This step is performed in the fourth module 4d, which is implemented in processing unit 1.

[0077] Next, processing stage Ph includes a step of standardizing the input data: step E9. This step is performed in the fifth module 4e, which is implemented in processing unit 1. The standardization step applies a standard intensity X to each pixel of each reference image in each reference video. standard This includes standardizing each reference video by assigning a standard intensity X. standard This is obtained by subtracting the average μ of all pixels in the training video from the aforementioned intensity X, and then dividing the result by the standard deviation σ.

number

[0078] The five modules 4a, 4b, 4c, 4d, and 4e are electronic modules and / or software modules (in this case, software) and are arranged to form a computing module pipeline 10. This pipelined architecture allows for the optimization of the computing resource utilization of the processing unit 1.

[0079] As mentioned above, the detection algorithm solves the video classification problem. The echocardiogram acquired during the examination is a video, or image sequence, also known as a frame. Therefore, in order to predict the detection score, the neural network needs to analyze the image sequence.

[0080] The number of images per video was set to 16, which is a typical FPS (frames per second) of 50 frames per second, equivalent to 0.32 seconds. This duration corresponds to approximately half to one-third of a heartbeat, which is considered appropriate for detecting the presence of cardiac amyloidosis.

[0081] Therefore, each video used for training, verification, and testing comprises 16 images.

[0082] Let's discuss the classification model itself.

[0083] For example, by treating time as a color channel, echocardiogram videos (size W×H×T: width × height × time) could be classified using a "2D CNN" (two-dimensional convolutional neural network) type convolutional neural network. It is also possible to use a 3D CNN network that directly handles all 16 frames as input.

[0084] However, we chose a convolutional neural network that implements (2+1)D spatiotemporal convolution.

[0085] More precisely, the classification model here is an R(2+1)D convolutional neural network based on ResNet (residual neural network).

[0086] ResNet is a CNN-type deep learning model.

[0087] ResNet is characterized by the addition of residual connections between certain layers. These residual connections enable skip connections, particularly addressing the vanishing gradient problem that occurs during training.

[0088] When separating spatial and temporal convolutions, the (2+1)D convolution allows for more effective separation and extraction of features specific to each domain (space and time) rather than mixing information in a single 3D convolution. Furthermore, this separation reduces computational complexity, enabling faster and more efficient training and inference.

[0089] The ResNet used here is an 18-layer network (ResNet-18).

[0090] Referring to Figure 6, the R(2+1)D model is a ResNet model in which a 3D convolution is decomposed into the product of two convolutions in each convolutional layer: a 2D spatial convolution followed by a 1D time convolution.

[0091] The number of 2D filters (kernels) N is selected so that the same number of parameters are obtained as in the 3D convolutional block.

[0092] Such models are described, for example, in Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2018). A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 6450-6459).

[0093] The classification model is trained in the training module 11 of the processing unit 1.

[0094] This provides some of the hyperparameters used in the training step (step E10).

[0095] The model was trained over 50 epochs, meaning that all training videos of the model were played through completely 50 times during training.

[0096] In this case, the batch size, or the number of training videos that pass through the network simultaneously, is equal to 8.

[0097] The learning rate is equal to 0.0001 and is divided into 1 / 10 parts at epochs 15, 30, and 45.

[0098] The model weights were initialized using pre-trained model weights from the Kinetics400 database. The optimization algorithm used was Adam, and the loss function was cross-entropy loss. Three metrics were calculated to monitor the model's performance: accuracy, precision, and recall.

[0099] Figure 7 shows the model's performance during the training and validation phases.

[0100] Curve C1 shows the change in the loss function during the training phase plotted against the number of epochs.

[0101] Curve C2 shows the change in the loss function during the validation phase plotted against the number of epochs.

[0102] Curve C3 shows the change in accuracy during the training phase plotted against the number of epochs. Curve C4 shows the change in accuracy during the validation phase plotted against the number of epochs.

[0103] Curve C5 shows the progression of accuracy during the training phase plotted against the number of epochs.

[0104] Curve C6 shows the progression of accuracy during the validation phase plotted against the number of epochs.

[0105] Curve C7 shows the progression of recall during the training phase plotted against the number of epochs.

[0106] Curve C8 shows the change in recall rate during the validation phase plotted against the number of epochs.

[0107] Next, we will explain how the detection support method is implemented for "new" patients during the examination. The objective is to detect cardiac amyloidosis in these patients. Therefore, the detection support method uses the inference model obtained through each of the steps described above. It should be noted that the detection support method is intended to assist the physician's diagnosis, not to confirm the diagnosis itself.

[0108] Referring to Figure 8, the detection support method is executed in real time in a processing unit 12 integrated into the sonograph 13. The sonograph 13 also includes an ultrasonic probe 14 that transmits and receives ultrasonic signals to generate ultrasonic video.

[0109] It should be noted that the detection support method can also be implemented in the second stage by using images taken during the inspection after the inspection. Furthermore, the detection support method can be implemented using a processing unit not integrated into the sonograph.

[0110] The processing unit 12 is an electronic and software unit. The processing unit 12 comprises one or more processing components 15, for example, any processor or microprocessor, whether general-purpose or dedicated, such as a DSP (Digital Signal Processor), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), microcontroller, or programmable logic circuit such as an FPGA (Field Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit).

[0111] The processing unit 12 further includes memory. At least one of these memories forms a computer-readable storage medium storing at least one computer program, and the computer program includes instructions that cause the processing unit 12 to execute at least some steps of a detection support method.

[0112] Referring to Figure 9, the processing unit 12 receives at least one current video Vc obtained from an echocardiogram performed on the patient: step E20.

[0113] Optionally, input the current video into the classification model to select the A4c view: Step E21 (The classification model used here is not a video classification model, but a model used to distinguish the A4c view from other views).

[0114] Next, processing unit 12 executes processing stage Ph on the current video. The processing stage is the same as described above for the learning method.

[0115] The processing stage Ph first involves processing each current image of the current video in order to remove additional information from the current image and retain only the ultrasonic signal.

[0116] The processing step comprises a cropping step for each of the current images of the current video: step E22.

[0117] The processing unit 12 obtains the coordinates of a reference point that defines a useful portion of the current image, and the useful portion includes an ultrasonic signal.

[0118] The first module 4a crops the current image to retain only the portion of the current image that passes through the reference point and contains the useful parts.

[0119] Next, the processing step includes applying a bitmask to the useful portion of the current image to remove the tick marks: step E23. The bitmask is applied by the second module 4b.

[0120] Next, the processing step comprises the steps of detecting pixels corresponding to electrical signal representations in the current image and removing the electrical signal representations by applying a median filter to each of the pixels: step E24. These steps are performed in the third module 4c.

[0121] Step E25: Resample the current video to obtain a video in a standardized format. This step is performed in module 4d.

[0122] Standardize the current video: Step E26. This step is performed in Module 5, 4e.

[0123] Therefore, the processing step includes passing the current video through the same processing pipeline 10 used in the training phase.

[0124] By applying the processing stage to the current video, it is possible to obtain the processed video from the current video.

[0125] Next, the processing unit 12 performs the inference stage.

[0126] The inference stage is performed by the inference module 16 of the processing unit 12.

[0127] The processing unit 12 performs inference using a pre-trained classification model by inputting at least one input video obtained from the processed videos into the inference model: step E27. This model is the R(2+1)D convolutional neural network described above.

[0128] At the end of the inference phase, processing unit 12 generates a "final" prediction about whether or not cardiac amyloidosis is present in the "new" patient. The final prediction may be positive (amyloidosis present) or negative (amyloidosis not present). The physician can rely on this final prediction to confirm the diagnosis.

[0129] As mentioned above, the model was trained using training videos, each containing 16 images. However, current videos contain more than 16 images.

[0130] We evaluated several inference methods.

[0131] The first method involves inputting all images from a processed video into an inference model. Therefore, the input video is a processed video. In this case, the output of the inference model provides a single predicted value to be used as the final result. This first method for obtaining the final predicted value is referred to as "Method 1".

[0132] The second method involves generating multiple image packets from a processed video, each image packet comprising a first predetermined number of consecutive images, and the image packets form an input video that is fed into a classification model. Thus, the inference stage comprises the step of generating multiple intermediate predictions, each associated with a different image packet.

[0133] The first predetermined number is equal to 16. Therefore, 16 consecutive image packets are input to the inference model.

[0134] Image packets are defined sequentially, and each image packet is temporally shifted by a second predetermined number of images relative to the previous image packet, where the second predetermined number is equal to 1.

[0135] Therefore, the center of each packet is shifted in time by one image.

[0136] In this way, multiple predictions are obtained, each corresponding to one packet. If the processed video contains N images, N-16 predictions are obtained. The final prediction can be calculated based on multiple methods.

[0137] From the N-16 predictions, the processing unit 12 can determine the label that was predicted most frequently among the "amylose" (positive) and "control" (negative) labels. This most common label is assigned as the final prediction. This second method for obtaining the final prediction value is referred to as "Method 2".

[0138] If amyloidosis is predicted at least once, the processing unit 12 can consider the final prediction to be amyloidosis positive. This third method for obtaining the final prediction is referred to as "Method 3".

[0139] If amyloidosis is predicted at least once with a confidence score exceeding a predetermined threshold, the processing unit 12 can consider the final prediction to be amyloidosis positive. The predetermined threshold is, for example, equal to 60%. This fourth method for obtaining the final prediction is referred to as "Method 4".

[0140] As will be explained later, I chose Method 4.

[0141] During the inspection, a visualization technique is used to visually interpret the predictions proposed by the detection assistance method: step E28. The visualization technique involves visualizing the activation function of a specific appropriate layer of the neural network that the neural network uses to analyze the essential features of the image.

[0142] The heatmap is obtained from the final convolutional layer of the neural network used (see above).

[0143] The final convolutional layer was extracted. This layer contains information relevant to the classification task. The gradient of this layer is used to weight the activation map, thereby generating a heatmap that shows the model's regions of interest.

[0144] Here, we will use the GradCam 3D method described in, for example, the reference Gotkowski, K., Gonzalez, C., Bucher, A., & Mukhopadhyay, A. (2020). M3d-CAM: A PyTorch library to generate 3D data attention maps for medical deep learning. arXiv preprint arXiv:2007.00453.

[0145] A heatmap is obtained by overlaying it onto the original ultrasound signal. This heatmap changes dynamically along with the video, thus indicating the position of the ultrasound signal that led to the prediction for each image. Figure 10 shows a sequence of images I1, I2, I3, and I4, each overlaid with hatching representing a simplified corresponding heatmap. For each image, the intensity increases as the spacing between hatches narrows.

[0146] The visualization method allows physicians to visualize the parts of the ultrasound signal that the classification model focused on when making predictions.

[0147] Therefore, in addition to providing physicians with support in the early stages of diagnosis, the detection support method identifies ultrasound locations that are likely to support or refute the diagnosis using heat maps.

[0148] The model also predicts a confidence score and provides it to the doctor.

[0149] We will discuss the evaluation of the performance of classification models.

[0150] The test metrics used were accuracy, recall, AUC (Area of ​​Operation - Recall) (AUC stands for Area Under the Curve), and AUC of the ROC curve (ROC stands for Receiver Operating Characteristics). The ROC curve is also called the sensitivity curve.

[0151] The training video set is imbalanced because the control data is significantly larger than the positive data, and the accuracy does not adequately represent the model's quality. Relatively high accuracy can be obtained if the model consistently predicts normal labels. The evaluation metrics used are even more stable against class imbalance.

[0152] The results of training the classification model are shown below, according to the four evaluation methods described above.

[0153] Table 1 shows the values ​​(%) obtained from various indicators for each of the different methods (Method 1, Method 2, Method 3, and Method 4) used to obtain the final prediction. [Table 1]

[0154] Methods 1 and 2 have a very high rate of false negatives, resulting in very low recall rates.

[0155] Methods 3 and 4 are more likely to convert false negatives to true positives than methods that convert true negatives to false positives, thereby improving the AUC.

[0156] Methods 3 and 4 offer a good balance between accuracy and recall. The best balance was achieved with Method 4.

[0157] These results demonstrate how the classification model detects amyloidosis during ultrasound sequencing. Amyloidosis is not detected globally and consistently throughout the entire sequence, but rather at specific points in the current video when amyloid-like features appear. Looking at the most common predictions, the classification model tends to frequently predict health labels, which means a higher number of false negatives in Methods 1 and 2. However, looking at local predictions, the number of false negatives decreases in Methods 3 and 4, indicating improved accuracy in detecting amyloidosis.

[0158] Ultimately, Method 4 was selected as the optimal method and used in the simulation shown below.

[0159] Tables 2 and 3 have different configurations: Configuration 1: A model trained with a set of training videos corresponding to Figure 1. Configuration 2: A model trained on a set of training videos filtered to select only A4c views (corresponding to Figure 2); Configuration 3: This section presents simulation results using a model trained with a second set of training videos and validated with a second set of validation videos. The second set of training videos and the second set of validation videos further include videos corresponding to cardiac amyloidosis cases.

[0160] The distribution of this second reference video is shown in Figure 11.

[0161] The amyloidosis cases are due to a new sonograph (Sonograph 3h) that lacks control data. Therefore, this new distribution is somewhat biased because the device's distribution was not adhered to. This demonstrates the importance of a uniform distribution of data across sonographs, as mentioned above.

[0162] The last row of each table in Tables 2 and 3 shows the results for Configuration 3. In this case, the model weights were initialized using the weights of the model trained in Configuration 1. This is referred to as the "pre-trained configuration." This final simulation assumes that the model trained on the imbalanced dataset (Configuration 3) can still continue to learn reasonably well by incorporating information from past training. This technique is called "transfer learning."

[0163] The last three columns in Tables 2 and 3 correspond to the test metrics evaluated using the dataset with the A4c filter applied (Figure 2). [Table 2]

[0164] The processing steps performed and the classification model used resulted in approximately 95% accuracy, 87% precision, and 84% recall on the test dataset. These results are significantly better than those obtained using conventional methods. [Table 3]

[0165] Therefore, Configuration 1 provides the best results, corresponding to the optimal balance between accuracy and recall. Note that if data balancing is not performed using sonography (Configuration 3), using transfer learning will introduce machine bias, significantly degrading the model's performance.

[0166] Regarding GradCam observations using heatmaps, the model tends to observe the mitral valve during systole and both the left ventricle and interventricular septum during diastole. During the video sequence, the heatmap is observed to move in conjunction with the heartbeat and identify various parts of the heart (see Figure 10). Dynamic observation of this heartbeat-linked heatmap movement indicates that characteristic findings of the disease differ depending on the time of day in the cardiac cycle.

[0167] Needless to say, the present invention is not limited to the embodiments described, but includes any modifications that fall within the scope of the invention as defined in the claims.

[0168] The convolutional neural network used for detecting cardiac amyloidosis does not necessarily have to be an 18-layer ResNet. It may be a ResNet with a different number of layers or a different network model. For example, a (2+1)D convolutional block can be integrated into a C3D (Convolutional 3D) or I3D (Inflated 3D ConvNet) architecture.

[0169] Naturally, this invention can be implemented using numerical values ​​(especially different hyperparameter values) that are different from those described herein.

[0170] Naturally, the architecture of the processing units can be different.

[0171] The steps of the learning method and the detection support method (especially those in the processing stage) may be executed in a different order.

[0172] Each of the training video set, verification video set, and test video set may contain a different number of videos than the example shown here.

Claims

1. A method for supporting the detection of cardiac amyloidosis in a patient, which is performed by a processing unit (12), A receiving step comprising receiving at least one current video (Vc) obtained from a cardiac ultrasound examination performed on a patient, wherein the current video comprises a plurality of current images, each having an ultrasound signal (5) and additional information (7, 9), A processing step (Ph) comprising the step of processing each of the current images of the current video in order to remove the additional information from the current video in order to generate a processed video from the current video, An inference step (E27) comprising the step of inputting at least one input video obtained from the processed video into a classification model pre-trained with a set of training videos comprising cardiac amyloidosis cases and control cases, thereby performing inference using the classification model, wherein the classification model is a convolutional neural network performing (2+1)D spatiotemporal convolution, A detection support method comprising the following features.

2. The aforementioned convolutional neural network is based on ResNet, and is the detection support method according to claim 1.

3. The detection support method according to claim 1 or 2, wherein the inference step comprises generating a plurality of image packets from the processed video, each containing a first predetermined number of consecutive images, the image packets forming an input video to be input to the classification model, and the inference step comprises generating a plurality of intermediate predictions, each associated with a different image packet.

4. The detection support method according to claim 3, wherein the image packets are defined sequentially, and each of the image packets is temporally shifted by a second predetermined number of images relative to the previous image packet.

5. The detection support method according to any one of claims 3 or 4, wherein, with respect to the current video, if at least one intermediate prediction obtained for an image packet is a positive prediction that detects the presence of amyloidosis and is associated with a confidence score exceeding a predetermined threshold, the inference stage generates a final prediction that is a positive prediction.

6. The detection support method according to any one of claims 1 to 5, wherein the processing step includes obtaining the coordinates of reference points (S1, S2, S3) that define a useful portion (6) of the current image of the current video, and the useful portion includes an ultrasonic signal.

7. The detection support method according to claim 6, wherein the processing step comprises a step (E22) of cropping the current image in order to retain only the portion of the current image that has passed through the reference point and contains the useful portion.

8. The detection support method according to claim 6 or 7, wherein the additional information comprises a scale (7), and the processing step further comprises the step (E23) of applying a bitmask to the useful portion of the current image to remove the scale.

9. The detection support method according to any one of claims 1 to 8, wherein the additional information comprises an electrical signal representation (9), and the processing step comprises the steps of detecting pixels corresponding to the electrical signal representation in the current image (E24), and removing the electrical signal representation by applying a median filter to each of the pixels (E24).

10. The detection support method according to any one of claims 1 to 9, further comprising the step (E28) of superimposing a heatmap obtained from the final convolutional layer of a convolutional neural network onto the current image.

11. A sonograph (13) comprising a processing unit (12) that implements the detection support method described in any one of claims 1 to 10.

12. A computer program comprising instructions to cause a processing unit (12) to execute the steps of the detection support method described in any one of claims 1 to 10.

13. A computer-readable storage medium storing the computer program described in claim 12.

14. A method for learning a classification model of a detection support method according to any one of claims 1 to 10, which is performed in a processing unit (1), A receiving step that receives a set of reference videos (Vref) obtained from cardiac ultrasound examinations performed on a reference patient, A processing step performed for each of the aforementioned reference videos, which is the same as the processing step of the detection support method, Training stage (E10) is a training stage in which the classification model is trained using a set of training videos belonging to the set of reference videos, wherein the set of training videos comprises cardiac amyloidosis cases and control cases. A learning method that includes [the following features].

15. The learning method according to claim 14, further comprising the preceding step of applying another pre-trained classification model to the reference images in order to retain only the A4c view from among the reference images of the reference video.

Images