Assistance in the detection of cardiac amyloidosis using artificial intelligence

Abstract

The invention relates to a method for assisting in the detection of cardiac amyloidosis, the method comprising: - receiving at least one current video obtained by means of echocardiography, the current video comprising current images that each comprise an ultrasound signal and additional information; - a processing phase (Ph) comprising the step of processing each current image of the current video in order to remove the additional information from the current image; - an inference phase (E27) comprising the step of carrying out inference using a previously trained classification model by applying, as input to the classification model, at least one input video obtained from the processed video, the classification model being a convolutional neural network implementing (2+1)D spatiotemporal convolutions.

Description

[0001] AID IN THE DETECTION OF CARDIAC AMYLOIDOSIS BY ARTIFICIAL INTELLIGENCE

[0002] The invention relates to the field of detection of cardiac amyloidosis.

[0003] BACKGROUND OF THE INVENTION

[0004] Cardiac amyloidosis is a disease characterized by the presence of insoluble proteins in heart tissue. It is a severe disease with a poor prognosis in the absence of appropriate treatment and management.

[0005] However, this disease is difficult to detect, especially in its early stages. As a result, it was long considered very rare, when in reality it is largely underdiagnosed.

[0006] It therefore seems particularly interesting to develop a diagnostic support system to detect this disease early, especially since we are now seeing the arrival of new treatments that are effective in the early stages of the disease.

[0007] A diagnostic support system is known that uses a cardiac amyloidosis detection algorithm based on cardiovascular MRI images (or CMR, for Cardiovascular Magnetic Resonance). However, the examination using the CMR technique is a high-cost examination. This high cost is doubly problematic because: it complicates the acquisition of a sufficient number of "reference" images, allowing the detection algorithm to be adjusted effectively; it prevents the organization of "massive" detection in the population, which would seem necessary to detect this disease in its early stages. A diagnostic support system is also known that uses the patient's electrocardiogram to attempt early detection of the onset of amyloidosis. This examination is less expensive than CMR. However, the accuracy of the detection is not satisfactory, which limits the interest of the system - or even prevents its deployment.

[0008] SUBJECT OF THE INVENTION

[0009] The invention aims to enable cardiac amyloidosis to be detected early, inexpensively and effectively.

[0010] SUMMARY OF THE INVENTION

[0011] In order to achieve this goal, a method is proposed for assisting in the detection of cardiac amyloidosis in a patient, implemented in a treatment unit and comprising:

[0012] - a reception step, consisting of receiving at least one current video obtained from an echocardiography performed on the patient, the current video comprising a plurality of current images each comprising an ultrasound signal and additional information;

[0013] - a processing phase, comprising the step of processing each current image of the current video to remove additional information from said current image, so as to produce a processed video from the current video;

[0014] - an inference phase, comprising the step of executing an inference of a classification model, previously trained on a set of training videos comprising cases of cardiac amyloidosis and control cases, by applying as input to the classification model at least one input video obtained from the processed video, the classification model being a convolutional neural network implementing spatiotemporal convolutions of type (2+1) D.

[0015] The classification model used is particularly effective in detecting cardiac amyloidosis, due to its ability to capture very precise spatio-temporal characteristics and to reduce the computational cost by separating spatial and temporal convolution.

[0016] The implementation of the treatment phase and the use of this classification model allow the very effective detection of cardiac amyloidosis at an early stage.

[0017] Echocardiography is also significantly less expensive than CMR, which overcomes the disadvantages mentioned above.

[0018] We further propose a method of assisting detection as previously described, said convolutional neural network being based on a ResNet.

[0019] We further propose a method for assisting detection as previously described, in which the inference phase comprises the step of producing, from the processed video, a plurality of image packets, each comprising a first predetermined number of successive images, the image packets forming input videos applied as input to the classification model, the inference phase thus comprising the step of producing a plurality of intermediate predictions each associated with a distinct image packet.

[0020] We further propose a method of assisting detection as previously described, in which the image packets are defined successively, each image packet being shifted in time, relative to a previous image packet, by a second predetermined number of images.

[0021] We further propose a method for assisting with detection as previously described, in which if, for the current video, at least one intermediate prediction, obtained for a packet of images, is a positive prediction, which therefore detects a presence of amyloidosis, and which is associated with a confidence score greater than a predetermined threshold, the inference phase produces a final prediction which is also a positive prediction.

[0022] We further propose a method for assisting detection as previously described, in which the processing phase comprises, for each current image of the current video, the step of acquiring coordinates of reference points delimiting a useful portion of the current image comprising the ultrasonic signal.

[0023] We further propose a method for assisting detection as previously described, in which the processing phase comprises the step of cropping the current image to retain only a part of the current image passing through the reference points and comprising the useful portion.

[0024] We further propose a method for assisting detection as previously described, in which the additional information comprises graduations, the processing phase further comprising the step of applying a binary mask to the useful portion of the current image to remove said graduations.

[0025] We further propose a method for assisting detection as previously described, in which the additional information comprises an electrical signal representation, the processing phase comprising the steps of detecting in the current image pixels corresponding to the electrical signal representation, and of removing said electrical signal representation by applying a median filter to each of said pixels.

[0026] We further propose a method for assisting detection as previously described, further comprising the step of superimposing on a current image a heat map obtained from the last convolutional layer of the convolutional neural network.

[0027] We further propose an ultrasound scanner comprising a processing unit in which the detection assistance method as previously described is implemented.

[0028] A computer program is further provided comprising instructions which cause a processing unit to execute the steps of the detection assistance method as previously described.

[0029] Further provided is a computer-readable recording medium on which the computer program as previously described is recorded.

[0030] We further propose a method for learning the classification model of the detection assistance method as previously described, implemented in a processing unit and comprising:

[0031] - a reception step, consisting of receiving a set of reference videos obtained from echocardiographies carried out on reference patients;

[0032] - a processing phase, implemented on each reference video, which is the same as that of the detection assistance process;

[0033] - a training phase, consisting of training the classification model using a set of training videos belonging to the reference video set, the training video set comprising cases of cardiac amyloidosis and control cases.

[0034] We further propose a learning method as previously described, further comprising the preliminary step of applying, on reference images of the reference videos, another previously trained classification model, to retain only A4c sections among them.

[0035] The invention will be better understood in light of the following description of a particular, non-limiting embodiment of the invention.

[0036] BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Reference will be made to the attached drawings, including:

[0038] [Fig. 1] Figure 1 represents steps in the classification model training process;

[0039] [Fig. 2] Figure 2 represents a processing unit, in which the processing of the reference videos and the training of the classification model are implemented;

[0040] [Fig. 3] Figure 3 represents histograms illustrating a first distribution of reference videos according to ultrasound scanners;

[0041] [Fig. 4] Figure 4 is a figure similar to Figure 3, after applying the A4c filter;

[0042] [Fig. 5] Figure 5 illustrates the processing of a reference image;

[0043] [Fig. 6] Figure 6 schematically represents a 3D convolution layer and a (2+1) D convolution layer;

[0044] [Fig. 7] Figure 7 represents graphs including loss curves and different metrics, obtained during the training and validation of the model; [Fig. 8] Figure 8 represents a processing unit, in which the processing of current videos and the detection assistance method are implemented;

[0045] [Fig. 9] Figure 9 represents steps of the detection assistance method;

[0046] [Fig. 10] Figure 10 represents images on each of which we see the ultrasound signal and hatching which represents in a simplified manner a heat map; [Fig. 11] Figure 11 represents histograms illustrating a second distribution of the reference videos according to ultrasound scanners.

[0047] DETAILED DESCRIPTION OF THE INVENTION

[0048] The method for assisting in the detection of cardiac amyloidosis according to the invention is based on the use of a video classification model. The classification model is a neural network, which will be described below.

[0049] The use of the classification model to diagnose cardiac amyloidosis in a patient, during or following a medical examination, requires that said model has been previously trained during a learning phase. This is supervised learning, which uses videos called here “reference videos”, obtained during cardiac ultrasound examinations previously carried out on patients called here “reference patients”.

[0050] Referring to Figure 1, we first focus on the model's learning process. By "learning process" we mean both the processing of reference videos and the training of the model.

[0051] The data collection (reference videos) used to train the classification model was carried out within the AP-HP Bichat-Claude-Bernard hospital center, in close collaboration with the hospital's IT department: stage El.

[0052] The reference videos are therefore echocardiographic loops (or “echocardiographic films”) which were obtained during actual examinations carried out on the reference patients by doctors (the patients’ non-opposition to the use of their data was obtained).

[0053] The physicians used ultrasound machines at the hospital to produce the reference videos. The Vref reference videos were transmitted to a processing unit 1 integrated into a server that was installed and configured within the hospital. This processing unit 1 is visible in Figure 2.

[0054] The processing of the reference videos and the training of the classification model were carried out in processing unit 1. Thus, the data only circulates within the hospital network.

[0055] The processing unit 1 is an electronic and software unit. The processing unit 1 comprises one or more processing components, and for example any processor or microprocessor, general or specialized (for example a DSP, for Digital Signal Processor, or a GPU, for Graphics Processing Unit, or an NPU, for Neural Processing Unit), a microcontroller, or a programmable logic circuit such as an FPGA (for Field Programmable Gate Arrays) or an ASIC (for Application Specific Integrated Circuit).

[0056] Here, the processing unit 1 comprises two GPUs 2. The two GPUs 2 are, for example, both RTX™ 3090 components. The processing unit 1 also comprises memories. At least one of these memories forms a computer-readable storage medium, on which at least one computer program is recorded comprising instructions that cause the processing unit 1 to execute at least some of the steps of the learning method.

[0057] The reference videos are transferred to Treatment Unit 1 directly from the hospital's 3 ultrasound scanners, or from another server that has received them. The data is collected in DICOM format and includes more than 10,000 reference videos from eight different ultrasound scanners and more than 800 patients.

[0058] In these reference videos, the proportion of positive cases for cardiac amyloidosis is approximately 20%.

[0059] Only grayscale reference videos were considered, in order to train the classification model exclusively on the "raw" ultrasound signal, and therefore without depending on any other information. This ensures that the classification carried out subsequently on a real patient, and using the obtained inference model, will be efficient and accurate using videos of this type as input data.

[0060] The color echo-Doppler signal videos were therefore excluded from the set of reference videos, by analyzing the difference in pixel intensity between each of the three channels of the RGB image: step E2. Indeed, a grayscale image is characterized by almost identical RGB channels.

[0061] Furthermore, to avoid biasing the classification model, it is preferable to have a roughly homogeneous distribution of data across ultrasound scanners. The reference videos form a set of videos, which is divided into three subsets: a set of training videos, a set of validation videos, and a set of test videos: step E3.

[0062] Each ultrasound scanner must be equally represented in terms of the number of amyloidosis-positive and negative data. However, among the eight ultrasound scanners, only three produced films of sick patients and films of healthy patients.

[0063] Thus, in order to avoid any machine bias when training the classification model, the training video set and the validation video set only include videos from these three ultrasound machines.

[0064] Furthermore, in order to ensure a quasi-homogeneous distribution between positive and negative data, some negative data from these three ultrasound scanners were not selected.

[0065] The test video set, on the other hand, includes the rest of the reference videos.

[0066] The data distribution after data selection is illustrated in the histograms of Figure 3. The reference videos were produced by the eight ultrasound scanners 3a, 3b, 3c, 3d, 3e, 3f, 3g, 3h.

[0067] The training video set, on which the classification model is trained, includes cardiac amyloidosis cases and control cases.

[0068] Cardiac amyloidosis cases are videos (echocardiographic films) of patients with cardiac amyloidosis.

[0069] Control cases (or "control cases") are videos (echocardiographic films) of "healthy" patients: patients not presenting cardiac amyloidosis, and not presenting any particular characteristic which could influence the onset of the disease.

[0070] The training video set here contains 915 videos corresponding to 62 patients, with 395 cardiac amyloidosis cases and 520 control cases. The validation video set contains 306 videos corresponding to 14 patients, with 131 cardiac amyloidosis cases and 175 control cases. The test video set contains 7268 echocardiographic movies corresponding to 722 patients, with 1351 cardiac amyloidosis cases and 5917 control cases.

[0071] It should be noted here that the collected videos can include images from any type of section: parasternal long axis section, short axis section, apical two, three or four chambers, subcostal section, etc. The apical four-chamber section, called A4c, is the one that presents the most relevant information for the analysis of pathologies such as amyloidosis.

[0072] It is therefore possible to select, among the images of the echocardiographic films, only those which correspond to the A4c section.

[0073] However, the type of cut is not information available in the DICOM standard.

[0074] We therefore used a classification model, pre-trained for the classification of echocardiographic slices, to automatically detect the slice of an echocardiographic film: step E4. The distribution of data containing only A4c slices is shown in Figure 4. The classification model used is a deep learning model. With the help of a cardiologist, the accuracy (which is translated here as "exactness") of the classification model was estimated at around 80%. This filter is therefore quite efficient.

[0075] This step of filtering the cuts is not obligatory to implement the invention.

[0076] A Ph processing phase of the reference videos is then implemented in processing unit 1.

[0077] An echocardiographic film generally displays the ultrasound signal by aggregating other textual or graphic information around it, such as a graduation scale or a signal describing electrical activity. The Ph processing phase allows us to get rid of this information, which could bias learning, and therefore to keep only the ultrasound signal.

[0078] Each reference video comprises a plurality of reference images. Each reference image comprises an ultrasound signal and additional information.

[0079] The processing phase therefore makes it possible to process each reference image of the reference video to remove the additional information from said reference image, so as to produce a processed video from the reference video.

[0080] The processing phase Ph firstly comprises a trimming step E5, implemented in a first module 4a implemented in the processing unit 1.

[0081] With reference to Figure 5, each reference image Iref of each reference video therefore comprises an ultrasonic signal 5, which is included in a useful portion 6 of the image. The useful portion 6 generally has the shape of an angular sector whose circular part is positioned at the bottom of the image. The coordinates in the reference image of reference points which delimit the useful portion, that is to say in this case the three vertices SI, S2, S3 of the angular sector, are entered in the DICOM file.

[0082] The cropping step therefore consists, for each reference image of each reference video, in acquiring coordinates of these vertices SI, S2, S3, and in cropping the reference image Iref to keep only a part of the reference image passing through the reference points and comprising the useful portion. We obtain a cropped image Ir whose upper edge and lateral edges pass through the vertices SI, S2, S3 of the angular sector.

[0083] Additional information includes 7 graduations that surround the ultrasonic signal.

[0084] The processing phase then comprises the step of applying a binary mask to the useful portion 6 of the reference image to remove said graduations: step E6. This step is implemented in a second module 4b implemented in the processing unit 1.

[0085] First of all, it is necessary to detect the coordinates in the cropped image Ir of the three vertices of the angular sector .

[0086] The detection of the top vertex SI consists of selecting the point located in the middle of the top edge of the cropped image. The detection of the other two vertices S2, S3 consists of detecting the intensity maxima of the two columns of pixels respectively to the left and right of the cropped image. Once these vertices are detected, the binary mask is applied.

[0087] The additional information also includes an electrical signal representation 9. This representation is formed of pixels having a color close to cyan blue.

[0088] The processing phase Ph comprises the steps of detecting in the reference image pixels corresponding to the representation of electrical signal 9 (and therefore the pixels of color close to cyan blue), and of removing said representation of electrical signal 9 by applying a median filter to each of said pixels: step E7.

[0089] These steps are implemented in a third module 4c implemented in processing unit 1.

[0090] We obtain a processed cropped image Irt comprising only the ultrasound signal and stripped of additional information.

[0091] Advantageously, the processing phase Ph then comprises a step consisting of spatially resampling (this is a sub-sampling) each reference video to obtain videos having a standardized format, for example videos in which each image has a size of 112 x 112 pixels: step E8. This step is implemented in a fourth 4d module implemented in the processing unit 1.

[0092] The processing phase Ph then includes a step of standardizing the input data: step E9. This step is implemented in a fifth module 4e implemented in the processing unit 1. The standardization step consists of standardizing each reference video by associating with each pixel of each reference image of said reference video a standard intensity X standard . The standard intensity X standardis obtained by subtracting from the intensity X of said pixel the average of all the pixels in the training videos, and dividing the result by the standard deviation a:

[0093] The five modules 4a, 4b, 4c, 4d, 4e are electronic and / or software modules (here software), and are arranged to form a pipeline 10 of computing modules. This pipelined architecture makes it possible to optimize the use of the computing resources of the processing unit 1.

[0094] As we have seen, the detection algorithm solves a video classification problem. An echocardiographic loop, obtained during the examination, is a video, that is to say a succession of images also called frames. Thus, the neural network must analyze a succession of images to be able to predict a detection score.

[0095] The number of frames per video was set to 16, which corresponds, for a typical FPS (frames per second) of 50 frames / second, to 0.32 seconds. This duration corresponds approximately to the duration of half or a third of a heartbeat, which seems relevant for detecting the presence of cardiac amyloidosis.

[0096] The videos used for training, validation and testing therefore each include 16 images.

[0097] We now turn to the classification model itself.

[0098] For example, it would have been possible to classify echocardiographic videos (videos of size W x H x T: width x height x time), using convolutional neural networks of the "2D CNN" type (2D for two dimensions, and CNN for Convolutional Neural Network) by considering time as a color channel. It would also have been possible to use a 3D CNN type network, taking all 16 frames as input directly.

[0099] However, we chose to use a convolutional neural network implementing spatiotemporal convolutions of type (2+1) D.

[0100] More precisely, the classification model here is a convolutional neural network of type R(2 + 1)D based on a ResNet (for residual neural network).

[0101] ResNet networks are Deep Learning models, of the CNN type.

[0102] ResNet networks are characterized by the addition of residual links between certain layers. These residual links allow connection jumps to be made which notably solve the problem of the vanishing gradient, or Vanishing Gradient, encountered during training.

[0103] By separating spatial and temporal convolutions, (2+1)D convolution allows for better isolation and extraction of domain-specific (spatial and temporal) features, rather than mixing this information in a single 3D convolution. Also, this decomposition reduces computational complexity, making training and inference faster and more efficient.

[0104] The ResNet network used here is an 18-layer network (ResNet-18).

[0105] Referring to Figure 6, R(2 + 1)D models are therefore ResNet models in which, for each convolution layer, the 3D convolution is factorized into a product of two convolutions: a 2D spatial convolution followed by a 1D temporal convolution.

[0106] The number N of two-dimensional filters (kernels) is chosen to obtain the same number of parameters as a 3D convolution block.

[0107] Such a model is described for example in Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2018) . A doser look at spatiotemporal convolutions for action recognition . In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 6450-6459) .

[0108] The training of the classification model is carried out in a training module 11 of the processing unit 1.

[0109] We provide some hyperparameters used during the training step (step E10).

[0110] The model was trained on 50 epochs, i.e., via 50 complete runs of the entire training set of videos in the model.

[0111] The batch size, that is, the number of training videos passing together through the network, is here equal to 8.

[0112] The learning rate is equal to 0.0001 and is divided by 10 at epochs 15, 30 and 45.

[0113] The model weights were initialized using the pre-trained model weights from the Kinetics 400 database. The optimizer used was Adam, and the loss function was cross-entropy loss. Accuracy, precision, and recall were the three metrics that were calculated to monitor the model's performance.

[0114] Figure 7 shows the performance of the model during the training phase and the validation phase.

[0115] Curve Cl represents the evolution of the loss function as a function of epochs, during the training phase. Curve C2 represents the evolution of the loss function as a function of epochs, during the validation phase.

[0116] Curve C3 represents the evolution of accuracy as a function of epochs, during the training phase. Curve C4 represents the evolution of accuracy as a function of epochs, during the validation phase.

[0117] Curve C5 represents the evolution of accuracy as a function of epochs, during the training phase. Curve C6 represents the evolution of accuracy as a function of epochs, during the validation phase.

[0118] Curve C7 represents the evolution of recall as a function of epochs, during the training phase. Curve C8 represents the evolution of recall as a function of epochs, during the validation phase.

[0119] We are now interested in implementing the detection assistance method on a "new" patient during an examination. We are therefore seeking to detect cardiac amyloidosis in this patient. The detection assistance method therefore uses the inference model that was obtained through the different steps that have just been described. Note that the detection assistance method is intended to help the doctor establish a diagnosis, and not to establish said diagnosis.

[0120] With reference to Figure 8, the detection assistance method is here carried out in real time and is implemented in a processing unit 12 which is here integrated into an ultrasound scanner 13. The ultrasound scanner 13 also comprises an ultrasound probe 14 which emits and receives ultrasound signals to produce ultrasound videos.

[0121] It is noted that the detection assistance method could also be carried out at a later stage, after the examination, using the images captured during said examination. The detection assistance method could also be implemented in a processing unit which is not integrated into an ultrasound scanner.

[0122] Again, the processing unit 12 is an electronic and software unit. The processing unit 12 comprises one or more processing components 15, and for example any processor or microprocessor, general or specialized (for example a DSP, for Digital Signal Processor, or a GPU, for Graphics Processing Unit, or even an NPU, for Neural Processing Unit), a microcontroller, or even a programmable logic circuit such as an FPGA (for Field Programmable Gate Arrays) or an ASIC (for Application Specific Integrated Circuit).

[0123] The processing unit 12 also comprises memories. At least one of these memories forms a computer-readable recording medium, on which is recorded at least one computer program comprising instructions which cause the processing unit 12 to execute at least some of the steps of the detection assistance method. With reference to FIG. 9, the processing unit 12 therefore receives at least one current video Vc obtained from an echocardiography performed on the patient: step E20.

[0124] Optionally, the current video is applied as input to the classification model used to select the A4c cuts: step E21 (the classification model used here is the one to distinguish an A4c cut from other cuts, and not the video classification model).

[0125] The processing unit 12 then implements a processing phase Ph on the current video. The processing phase is the same as that previously described for the learning method.

[0126] The Ph processing phase therefore firstly consists of processing each current image of the current video to remove the additional information from said current image and keep only the ultrasonic signal.

[0127] The processing phase includes, for each current image of the current video, the trimming step: step E22.

[0128] The processing unit 12 acquires coordinates of reference points delimiting a useful portion of the current image comprising the ultrasound signal.

[0129] The current image is cropped by the first module 4a to keep only a part of the current image passing through the reference points and comprising the useful portion.

[0130] The processing phase then comprises the step of applying a binary mask to the useful portion of the current image to remove the graduations: step E23. The binary mask is applied by the second module 4b. Then, the processing phase comprises the steps of detecting in the current image the pixels corresponding to the electrical signal representation, and removing said electrical signal representation by applying a median filter to each of said pixels: step E24. These steps are carried out in the third module 4c.

[0131] The current video is resampled to obtain a video in a standardized format: step E25. This step is carried out in the fourth 4d module.

[0132] The current video is standardized: step E26. This step is implemented in the fifth module 4e.

[0133] The processing phase therefore consists of passing the current video through the same processing pipeline 10 as that used during the training phase.

[0134] The processing phase, applied to the current video, makes it possible to obtain a processed video from the current video.

[0135] Then, the processing unit 12 implements an inference phase.

[0136] The inference phase is implemented in an inference module 16 of the processing unit 12.

[0137] The processing unit 12 executes an inference of the previously trained classification model, by applying as input to the inference model at least one input video obtained from the processed video: step E27. The model is the convolutional neural network R(2+1)D which has just been described.

[0138] The processing unit 12 thus produces, at the output of the inference phase, a “final” prediction, for the “new” patient, of the presence or absence of cardiac amyloidosis. The final prediction can be positive (presence of amyloidosis) or negative (absence of amyloidosis). The doctor can rely on the final prediction to establish his diagnosis.

[0139] As we have seen, the model was trained with training videos each comprising 16 frames. However, the current video has more than 16 frames.

[0140] Several inference approaches have been evaluated.

[0141] A first approach is to apply all the frames of the processed video as input to the inference model. The input video is therefore the processed video. In this case, the output of the inference model provides a single prediction used as the final result. This first method of obtaining the final prediction is called "method 1".

[0142] A second approach consists in producing, from the processed video, a plurality of image packets, each comprising a first predetermined number of successive images, the image packets forming the input videos applied as input to the classification model. The inference phase thus comprises the step of producing a plurality of intermediate predictions each associated with a distinct image packet.

[0143] The first predetermined number is equal to 16. We therefore apply packets of 16 consecutive images as input to the inference model.

[0144] The image packets are defined successively, each image packet being shifted in time, relative to a previous image packet, by a second predetermined number of images. The second predetermined number is here equal to 1.

[0145] The center of each packet is therefore shifted by one image in time. This gives us several predictions, each corresponding to one of the packets. If the processed video contains N images, we obtain N-16 predictions. The calculation of the final prediction can then be carried out using several methods.

[0146] The processing unit 12 can determine, among the N-16 predictions, the label which was predicted predominantly among the labels “amyloidosis” (positive) and “control” (negative). This majority label is assigned to the final prediction. This second method of obtaining the final prediction is called “method 2”.

[0147] If amyloidosis has been predicted at least once, the processing unit 12 may consider the final prediction positive for amyloidosis. This third method of obtaining the final prediction is called “method 3”.

[0148] If amyloidosis has been predicted at least once with a confidence score greater than a predetermined threshold, the processing unit 12 may consider the final prediction positive for amyloidosis. The predetermined threshold is, for example, equal to 60%. This fourth method of obtaining the final prediction is called “method 4”.

[0149] As we will see below, method 4 is chosen.

[0150] During the examination, to provide a visual interpretation of the prediction proposed by the detection assistance method, a visualization method is used: step E28. The visualization method consists of visualizing the activation functions of a certain well-chosen layer of the neural network, where the network analyzes the essential characteristics of the image. The heat map is obtained from the last convolutional layer of the neural network used (see above).

[0151] The last convolutional layer has been extracted. This layer contains the information related to the classification task. The gradients of this layer are used to weight the activation maps, thus producing a heat map showing the regions that have aroused the model's interest.

[0152] Here we use the GradCam 3D method, described for example in the document Gotkowski , K. , Gonzalez, G. , Bucher, A . , & Mukhopadhyay , A. (2020) . M3d-CAM: A PyTorch library to generate 3D data attention maps for medical deep learning . arXiv preprint arXiv: 2007. 00453 .

[0153] We obtain a heat map superimposed on the original ultrasound signal which evolves dynamically with the video, thus showing image by image the location of the ultrasound signal which oriented the prediction. We see in Figure 10 the successive images II, 12, 13, 14, on each of which are superimposed hatchings which represent in a simplified manner the corresponding heat map. For each image, the smaller the spacing between the hatchings, the greater the intensity.

[0154] The visualization process allows the physician to visualize the parts of the ultrasound signal in which the classification model focused its attention when making its predictions.

[0155] In addition to providing the doctor with initial diagnostic assistance, the detection assistance process therefore uses the heat map to identify areas of the ultrasound likely to confirm or refute the diagnosis. A confidence score is also predicted by the model and transmitted to the practitioner.

[0156] We are now interested in evaluating the performance of the classification model.

[0157] The test metrics that were used are: precision, recall, as well as AUC Precision - Recall (AUC for Area Under Curve, which is translated by Surface Under the Curve, Precision - Recall which is translated by Precision-Recall) and AUC of the ROC curve (ROC for Receiver Operating Characteristic). The ROC curve is also called sensitivity curve.

[0158] Since the training video set is unbalanced, containing much more control data than positive data, the accuracy will not be sufficiently representative of the model's quality. Indeed, with a model consistently predicting the healthy label, we would obtain relatively high accuracy. The evaluation metrics used are more robust to class imbalance.

[0159] The results of training the classification model are presented, based on the four evaluation methods just described.

[0160] Table 1 provides the values ​​obtained from the different metrics (in %), for the different methods of obtaining the final prediction: method 1, method 2, method 3 and method 4.

[0161] Table 1

[0162] We note that method 1 and method 2 have too low a recall, which is due to too high a number of false negatives.

[0163] Method 3 and Method 4 convert more false negatives into true positives than true negatives into false positives, which increases the AUCs.

[0164] Method 3 and Method 4 show a good compromise between precision and recall. The best compromise is obtained by Method 4.

[0165] These results give an indication of how the classification model detects amyloidosis during an ultrasound sequence. Amyloidosis detection is done punctually, at certain moments in the current video where certain characteristics of amyloidosis seem to appear, rather than globally and uniformly throughout the sequence. Indeed, if we observe the majority predictions, the classification model tends to predict the healthy label more often, and the number of false negatives is therefore higher for methods 1 and 2. If, on the other hand, we focus on local predictions, we detect amyloidosis better by reducing the number of false negatives with methods 3 and 4.

[0166] Finally, method 4 was chosen as the most optimal method and is used in the simulations presented below.

[0167] Tables 2 and 3 present the results of simulations using different configurations: configuration 1: model trained on the training video set corresponding to Figure 1; configuration 2: model trained on the training video set filtered to select only A4c slices (thus corresponding to Figure 2); configuration 3: model trained with a second training video set, and validated with a second validation video set. The second training video set and the second validation video set contain a larger number of videos corresponding to cases of cardiac amyloidosis.

[0168] This second distribution of reference videos is visible in Figure 11.

[0169] The amyloidosis cases come from a new ultrasound machine (the 3h ultrasound machine) with no control data. This new distribution is therefore somewhat biased because the distribution of the machines is not respected. It serves to validate the importance of the homogeneous distribution of data per ultrasound machine as explained above.

[0170] The last row of Tables 2 and 3 presents the results of configuration 3, this time initializing the model weights with those of the model trained in configuration 1. This is called the "pre-trained configuration." This last simulation assumes that a model trained on an unbalanced data set (configuration 3) can still continue to learn reasonably well by incorporating information from previous training. This technique is called "transfer learning."

[0171] The last three columns of Tables 2 and 3 correspond to the test metrics evaluated on the entire dataset with A4c filter (Figure 2).

[0172] Table 2

[0173] It can be seen that the processing phase carried out, and the classification model used, make it possible to obtain an accuracy of approximately 95% on the test data set, as well as a precision of approximately 87% and a recall of approximately 84%. These results are significantly superior to those obtained by the methods of the prior art.

[0174]

[0175] Table 3

[0176] Configuration 1 therefore gives the best result, corresponding to the best compromise between precision and recall. In addition, we observe that a non-balancing of the data by ultrasound (configuration 3) drastically degrades the performance of the model, because of the machine bias introduced, even when using transfer learning.

[0177] Regarding GradCam observations via heat maps, the model tends to often observe the mitral valve in systole, and the left ventricle as well as the septum in diastole. During a video sequence, the heat map is observed to move and point to different parts of the heart in a coordinated manner with the heartbeats (see Figure 10). The fact that this movement of the heat map is observed dynamically with the beats seems to show that the characteristic signs of the disease differ depending on the moments of the cardiac cycle.

[0178] Of course, the invention is not limited to the embodiment described but encompasses any variant falling within the scope of the invention as defined by the claims.

[0179] The convolutional neural network used to detect cardiac amyloidosis is not necessarily an 18-layer ResNet. It can be a ResNet with a different number of layers, or a different network model. It is thus possible, for example, to integrate the (2+1)D convolution blocks into C3D (for Convolutional 3D) or I3D (for Inflated 3D ConvNet) type architectures.

[0180] The invention can of course be implemented with numerical values ​​different from those cited here (and in particular different values ​​of hyperparameters).

[0181] The architectures of the processing units can of course be different.

[0182] The steps of the learning process and the detection assistance process can be carried out in a different order (and in particular those of the processing phases).

[0183] The training video set, validation video set, and test video set may each include a different number of videos than the examples provided here.

Claims

CLAIMS 1. Method for assisting in the detection of cardiac amyloidosis in a patient, implemented in a treatment unit (12) and comprising: - a reception step, consisting of receiving at least one current video (Vc) obtained from an echocardiography performed on the patient, the current video comprising a plurality of current images each comprising an ultrasound signal (5) and additional information (7, 9); - a processing phase (Ph), comprising the step of processing each current image of the current video to remove the additional information from said current image, so as to produce a processed video from the current video; - an inference phase (E27), comprising the step of executing an inference of a classification model, previously trained on a set of training videos comprising cases of cardiac amyloidosis and control cases, by applying as input to the classification model at least one input video obtained from the processed video, the classification model being a convolutional neural network implementing spatiotemporal convolutions of type (2+1) D.

2. Detection assistance method according to claim 1, said convolutional neural network being based on a ResNet.

3. Detection assistance method according to one of the preceding claims, in which the inference phase comprises the step of producing, from the processed video, a plurality of image packets, each comprising a first predetermined number of successive images, the image packets forming input videos applied as input to the classification model, the inference phase thus comprising the step of producing a plurality of intermediate predictions each associated with a distinct image packet.

4. Detection assistance method according to claim 3, in which the image packets are defined successively, each image packet being shifted in time, relative to a previous image packet, by a second predetermined number of images.

5. Detection assistance method according to one of claims 3 or 4, in which if, for the current video, at least one intermediate prediction, obtained for a packet of images, is a positive prediction, which therefore detects a presence of amyloidosis, and which is associated with a confidence score greater than a predetermined threshold, the inference phase produces a final prediction which is also a positive prediction.

6. Detection assistance method according to one of the preceding claims, in which the processing phase comprises, for each current image of the current video, the step of acquiring coordinates of reference points (S1, S2, S3) delimiting a useful portion (6) of the current image comprising the ultrasonic signal.

7. Detection assistance method according to claim 6, in which the processing phase comprises the step (E22) of cropping the current image to retain only a part of the current image passing through the reference points and including the useful portion.

8. Detection assistance method according to one of claims 6 or 7, in which the additional information comprises graduations (7), the processing phase further comprising the step (E23) of applying a binary mask to the useful portion of the current image to remove said graduations.

9. Detection assistance method according to one of the preceding claims, in which the additional information comprises an electrical signal representation (9), the processing phase comprising the steps (E24) of detecting in the current image pixels corresponding to the electrical signal representation, and of removing said electrical signal representation by applying a median filter to each of said pixels.

10. Detection assistance method according to one of the preceding claims, further comprising the step (E28) of superimposing on a current image a heat map obtained from the last convolutional layer of the convolutional neural network.

11. Ultrasound scanner (13) comprising a processing unit (12) in which the detection assistance method according to one of the preceding claims is implemented.

12. Computer program comprising instructions which cause the processing unit (12) to execute the steps of the detection assistance method according to one of the claims 1 to 10.

13. Computer-readable recording medium, on which the computer program according to claim 12 is recorded.

14. Method for learning the classification model of the detection assistance method according to one of claims 1 to 10, implemented in a processing unit (1) and comprising: - a reception step, consisting of receiving a set of reference videos (Vref) obtained from echocardiographies carried out on reference patients; - a processing phase, implemented on each reference video, which is the same as that of the detection assistance process; - a training phase (E10), consisting of training the classification model using a set of training videos belonging to the set of reference videos, the set of training videos comprising cases of cardiac amyloidosis and control cases.

15. Learning method according to claim 14, further comprising the preliminary step of applying, on reference images of the reference videos, another previously trained classification model, to retain only A4c sections among them.

Images