Artificial intelligence-assisted detection of cardiac amyloidosis

Abstract

A method for aiding the detection of cardiac amyloidosis, comprising: receiving at least one current video obtained from an echocardiogram, the current video containing current frames each comprising an ultrasound signal and additional information; a processing phase (Ph), comprising the step of processing each current frame of the current video to remove the additional information from said current frame; an inference phase (E27), comprising the step of performing an inference on a previously trained classification model, by applying as input to the classification model at least one input video obtained from the processed video, the classification model being a convolutional neural network implementing (2+1)D spatiotemporal convolutions. ABSTRACT FIGURE: Fig. 9

Description

Title of the invention: Aid in the detection of cardiac amyloidosis by artificial intelligence

[0001] The invention relates to the field of detection of cardiac amyloidosis.

[0002] BACKGROUND OF THE INVENTION

[0003] Cardiac amyloidosis is a disease characterized by the presence of insoluble proteins in cardiac tissue. It is a severe disease, with a poor prognosis in the absence of appropriate treatment and management.

[0004] However, this disease is difficult to detect, especially in its early stages. Consequently, it was long considered to be very rare, while in reality it is largely underdiagnosed.

[0005] It therefore seems particularly interesting to develop a diagnostic support system enabling this disease to be detected early, especially since we are now seeing the arrival of new treatments that are effective in the early stages of the disease.

[0006] A diagnostic assistance system is known that uses an algorithm for detecting cardiac amyloidosis from cardiovascular MRI images (or CMR, for Cardio-vascular Magnetic Resonance). However, the examination using the CMR technique is an examination that presents a high cost. This high cost is doubly problematic, because: - it complicates the acquisition of a sufficient number of “reference” images, allowing the detection algorithm to be effectively adjusted; - it prevents the organization of "massive" detection in the population, which would seem necessary to detect this disease in its early stages.

[0007] A diagnostic assistance system is also known which uses the patient's electrocardiogram to attempt to detect the onset of amyloidosis early. This examination is less expensive than CMR. However, the accuracy of the detection is not satisfactory, which limits the interest of the system - or even prevents its deployment.

[0008] SUBJECT OF THE INVENTION

[0009] The object of the invention is to enable cardiac amyloidosis to be detected early, inexpensively and effectively. Summary of the invention

[0010] With a view to achieving this aim, a method is proposed for assisting in the detection of cardiac amyloidosis in a patient, implemented in a treatment unit and comprising: • a reception step, consisting of receiving at least one current video obtained from an echocardiography performed on the patient, the current video comprising a plurality of current images each comprising an ultrasound signal and additional information; • a processing phase, comprising the step of processing each current image of the current video to remove additional information from said current image, so as to produce a processed video from the current video; • an inference phase, comprising the step of executing an inference of a previously trained classification model, by applying as input to the classification model at least one input video obtained from the processed video, the classification model being a convolutional neural network implementing spatiotemporal convolutions of type (2+l)D.

[0011] The classification model used is particularly effective for detecting cardiac amyloidosis, due to its ability to capture very precise spatio-temporal characteristics and to reduce the computational cost by separating the spatial and temporal convolution.

[0012] The implementation of the treatment phase and the use of this classification model make it possible to detect cardiac amyloidosis very effectively, early on.

[0013] Echocardiography examination is also significantly less expensive than CMR, which makes it possible to overcome the disadvantages mentioned above.

[0014] We further propose a method for assisting detection as previously described, said convolutional neural network being based on a ResNet.

[0015] A method for assisting detection is further proposed as previously described, in which the inference phase comprises the step of producing, from the processed video, a plurality of image packets, each comprising a first predetermined number of successive images, the image packets forming input videos applied as input to the classification model, the inference phase thus comprising the step of producing a plurality of intermediate predictions each associated with a distinct image packet.

[0016] A method for aiding detection is further proposed as previously described, in which the image packets are defined successively, each image packet being shifted in time, relative to a previous image packet, by a second predetermined number of images.

[0017] We further propose a method for assisting detection as previously described, in which if, for the current video, at least one intermediate prediction, obtained for a packet of images, is a positive prediction, which therefore detects a presence of amyloidosis, and which is associated with a confidence score above a predetermined threshold, the inference phase produces a final prediction which is also a positive prediction.

[0018] We further propose a method for assisting detection as previously described, in which the processing phase comprises, for each current image of the current video, the step of acquiring coordinates of reference points delimiting a useful portion of the current image comprising the ultrasonic signal.

[0019] We further propose a method for assisting detection as previously described, in which the processing phase comprises the step of cropping the current image to retain only a part of the current image passing through the reference points and comprising the useful portion.

[0020] A detection assistance method is further proposed as previously described, in which the additional information comprises graduations, the processing phase further comprising the step of applying a binary mask to the useful portion of the current image to remove said graduations.

[0021] A method for assisting detection is further proposed as previously described, in which the additional information comprises an electrical signal representation, the processing phase comprising the steps of detecting in the current image pixels corresponding to the electrical signal representation, and of removing said electrical signal representation by applying a median filter to each of said pixels.

[0022] We further propose a method for assisting detection as previously described, further comprising the step of superimposing on a current image a heat map obtained from the last convolutional layer of the convolutional neural network.

[0023] We further propose an ultrasound scanner comprising a processing unit in which the detection assistance method as previously described is implemented.

[0024] A computer program is further provided comprising instructions which cause a processing unit to execute the steps of the detection assistance method as previously described.

[0025] A computer-readable recording medium is further provided, on which the computer program as previously described is recorded.

[0026] We further propose a method for learning the classification model of the detection assistance method as previously described, implemented in a processing unit and comprising: • a reception step, consisting of receiving a set of reference videos obtained from echocardiographies carried out on reference patients; • a processing phase, implemented on each reference video, which is the same as that of the detection assistance process; • a training phase, consisting of training the classification model using a set of training videos belonging to the set of reference videos.

[0027] We further propose a learning method as previously described, further comprising the preliminary step of applying, on reference images of the reference videos, another previously trained classification model, to retain only A4c sections among them.

[0028] The invention will be better understood in light of the following description of a particular non-limiting embodiment of the invention. Brief description of the drawings

[0029] Reference will be made to the attached drawings, among which:

[0030] [Fig-1] [Fig.l] represents steps of the learning process of the model of classification;

[0031] [Fig.2] [Fig.2] represents a processing unit, in which are implemented processing reference videos and training the classification model;

[0032] [Fig.3] [Fig.3] represents histograms illustrating a first distribution of the reference videos according to ultrasound scanners;

[0033] [Fig.4] [Fig.4] is a figure similar to [Fig.3], after application of the A4c filter;

[0034] [Fig.5] [Fig.5] illustrates the processing of a reference image;

[0035] [Fig.6] [Fig.6] schematically represents a 3D convolution layer and a (2+l)D convolution layer;

[0036] [Fig.7] [Fig.7] represents graphs comprising loss and different metrics, obtained during model training and validation;

[0037] [Fig.8] [Fig.8] represents a processing unit, in which are implemented processing of current videos and the detection assistance process;

[0038] [Fig.9] [Fig.9] represents steps of the detection assistance method;

[0039] [Fig. 10] [Fig. 10] represents images on each of which we see the ultrasonic signal and hatching which represents in a simplified manner a heat map;

[0040] [Fig. 11] [Fig. 11] represents histograms illustrating a second distribution of the reference videos according to ultrasound scanners. DETAILED DESCRIPTION OF THE INVENTION

[0041] The method for assisting in the detection of cardiac amyloidosis according to the invention is based on the use of a video classification model. The classification model is a neural network, which will be described below.

[0042] Use of classification model to diagnose cardiac amyloidosis in a patient, during or following a medical examination, requires that said model has been previously trained during a learning phase. This is supervised learning, which uses videos called here "reference videos", obtained during cardiac ultrasound examinations previously carried out on patients called here "reference patients".

[0043] With reference to [Fig.l], we are first interested in the model learning process. By "learning process" we mean here both the processing of the reference videos and the training of the model.

[0044] The collection of data (reference videos) used to train the classification model was carried out within the AP-HP Bichat-Claude-Bemard hospital center, in close collaboration with the hospital's IT department: step El.

[0045] The reference videos are therefore echocardiographic loops (or “echocardiographic films”) which were obtained during actual examinations carried out on the reference patients by doctors (the patients’ non-opposition to the use of their data was obtained).

[0046] The doctors used ultrasound scanners from the said hospital to produce the reference videos. The reference videos Vref were transmitted to a processing unit 1 integrated into a server that was installed and configured within the hospital. This processing unit 1 is visible in [Fig.2].

[0047] The processing of the reference videos and the training of the classification model were carried out in processing unit 1. Thus, the data only circulates within the hospital network.

[0048] The processing unit 1 is an electronic and software unit. The processing unit 1 includes one or more processing components, for example any processor or microprocessor, general or specialized (for example a DSP, for Digital Signal Processor, or a GPU, for Graphics Processing Unit, or an NPU, for Neural Processing Unit), a microcontroller, or a programmable logic circuit such as an FPGA (for Field Programmable Gate Arrays) or an ASIC (for Application Specific Integrated Circuit).

[0049] Here, the processing unit 1 comprises two GPUs 2. The two GPUs 2 are for example both RTX™ 3090 components.

[0050] The processing unit 1 also comprises memories. At least one of these memories forms a computer-readable recording medium, on which is recorded at least one computer program comprising instructions which cause the processing unit 1 to execute at least some of the steps of the learning method.

[0051] The reference videos are transferred to the processing unit 1 directly by the 3 ultrasound machines in the hospital, or from another server that received them itself. The data is collected in DICOM format, and includes more than 10,000 reference videos from eight different ultrasound machines and more than 800 patients.

[0052] In these reference videos, the proportion of positive cases for cardiac amyloidosis is approximately 20%.

[0053] Only the grayscale reference videos were considered, in order to train the classification model exclusively on the “raw” ultrasound signal, and therefore without depending on any other information. This ensures that the classification carried out subsequently on a real patient, and using the inference model obtained, will be efficient and accurate by using videos of this type as input data.

[0054] The color echo-Doppler signal videos were therefore excluded from the set of reference videos, by analyzing the difference in pixel intensity between each of the three channels of the RGB image: step E2. Indeed, a grayscale image is characterized by almost identical RGB channels.

[0055] Furthermore, in order not to bias the classification model, it is preferable to have a more or less homogeneous distribution of the data according to the ultrasound scanners.

[0056] The reference videos form a set of videos, which is divided into three subsets: a set of training videos, a set of validation videos and a set of test videos: step E3.

[0057] Each ultrasound scanner must be equally represented in terms of the number of amyloidosis-positive and negative data. However, among the eight ultrasound scanners, only three produced films of sick patients and films of healthy patients.

[0058] Thus, in order to avoid any machine bias when training the classification model, the training video set and the validation video set only include videos from these three ultrasound scanners.

[0059] Furthermore, in order to guarantee a quasi-homogeneous distribution between positive and negative data, certain negative data from these three ultrasound scanners were not selected.

[0060] The test video set, on the other hand, includes the rest of the reference videos.

[0061] The distribution of data after data selection is illustrated in the his tograms of [Fig.3]. The reference videos were produced by the eight ultrasound scanners 3a, 3b, 3c, 3d, 3e, 3f, 3g, 3h. The training video set contains 915 videos corresponding to 62 patients, with 395 cases of cardiac amyloidosis and 520 control cases (or "control cases": patients not presenting with cardiac amyloidosis, and moreover not presenting any particular characteristic that could influence the onset of the disease). The validation video set contains 306 videos corresponding to 14 patients, with 131 cases of cardiac amyloidosis and 175 control cases. The video set The test contains 7268 ultrasound films corresponding to 722 patients, with 1351 cases of cardiac amyloidosis and 5917 control cases.

[0062] It is noted here that the collected videos can include images from any type of section: parasternal major axis section, minor axis section, apical two, three or four cavities, subcostal section, etc. The apical four-cavity section, called A4c, is the one that presents the most relevant information for the analysis of pathologies such as amyloidosis.

[0063] It is therefore possible to select, from among the images of the echocardiographic films, only those which correspond to the A4c section.

[0064] However, the type of cut is not information available in the DICOM standard.

[0065] We therefore used a classification model, pre-trained for the classification of echocardiographic slices, in order to automatically detect the slice of an echocardiographic film: step E4. The distribution of data containing only A4c slices is shown in [Fig.4]. The classification model used is a deep learning model. With the help of a cardiologist, the accuracy (which is translated here as "exactness") of the classification model was estimated at approximately 80%. This filter is therefore quite efficient.

[0066] This step of filtering the cuts is not obligatory for implementing the invention.

[0067] A processing phase Ph of the reference videos is then implemented in the processing unit 1.

[0068] An echocardiographic film generally displays the ultrasound signal by aggregating other textual or graphic information around it, such as a graduation scale or a signal describing electrical activity. The Ph processing phase makes it possible to get rid of this information, which could bias the learning, and therefore to keep only the ultrasound signal.

[0069] Each reference video comprises a plurality of reference images. Each reference image comprises an ultrasound signal and additional information.

[0070] The processing phase therefore makes it possible to process each reference image of the reference video to remove the additional information from said reference image, so as to produce a processed video from the reference video.

[0071] The processing phase Ph firstly comprises a trimming step E5, implemented in a first module 4a implemented in the processing unit 1.

[0072] With reference to [Fig.5], each reference image Iref of each reference video therefore comprises an ultrasonic signal 5, which is included in a useful portion 6 of the image. The useful portion 6 generally has the shape of an angular sector whose circular part is positioned at the bottom of the image.

[0073] The coordinates in the reference image of reference points which delimit the useful portion, that is to say in this case the three vertices SI, S2, S3 of the angular sector, are entered in the DICOM file.

[0074] The cropping step therefore consists, for each reference image of each reference video, in acquiring coordinates of these vertices SI, S2, S3, and in cropping the reference image Iref to retain only a part of the reference image passing through the reference points and comprising the useful portion. A cropped image Ir is obtained whose upper edge and lateral edges pass through the vertices SI, S2, S3 of the angular sector.

[0075] The additional information includes graduations 7 which surround the ultrasonic signal.

[0076] The processing phase then comprises the step of applying a binary mask to the useful portion 6 of the reference image to remove said graduations: step E6. This step is implemented in a second module 4b implemented in the processing unit 1.

[0077] First of all, it is necessary to detect the coordinates in the cropped image Ir of the three vertices of the angular sector.

[0078] The detection of the top vertex SI consists of selecting the point located in the middle of the upper edge of the cropped image. The detection of the other two vertices S2, S3 consists of detecting the intensity maxima of the two columns of pixels respectively to the left and right of the cropped image. Once these vertices have been detected, the binary mask is applied.

[0079] The additional information also includes an electrical signal representation 9.

[0080] This representation is formed of pixels having a color close to cyan blue.

[0081] The processing phase Ph comprises the steps of detecting in the reference image pixels corresponding to the electrical signal representation 9 (and therefore the pixels of color close to cyan blue), and of deleting said electrical signal representation 9 by applying a median filter to each of said pixels: step E7.

[0082] These steps are implemented in a third module 4c implemented in the processing unit 1.

[0083] A processed cropped image Irt is obtained comprising only the ultrasound signal and freed from additional information.

[0084] Advantageously, the processing phase Ph then comprises a step consisting of spatially resampling (this is a sub-sampling) each reference video to obtain videos having a standardized format, for example videos in which each image has a size of 112 x 112 pixels: step E8. This step is implemented in a fourth module 4d implemented in the processing unit 1.

[0085] The processing phase Ph then includes a step of standardizing the input data: step E9. This step is implemented in a fifth module 4e implemented in the processing unit 1. The standardization step consists of standardizing each reference video by associating with each pixel of each reference image of said reference video a standard intensity % standard- The standard intensity ^standard is obtained by subtracting from the intensity X of said pixel the average d of all the pixels of the training videos, and by dividing the result by the standard deviation -XstaM

[0086] The five modules 4a, 4b, 4c, 4d, 4e are electronic and / or software modules (here software), and are arranged to form a pipeline 10 of calculation modules. This pipelined architecture makes it possible to optimize the use of the calculation resources of the processing unit 1.

[0087] As we have seen, the detection algorithm solves a video classification problem. An echocardiographic loop, obtained during the examination, is a video, that is to say a succession of images also called frames. Thus, the neural network must analyze a succession of images to be able to predict a detection score.

[0088] The number of frames per video was set at 16, which corresponds, for a typical FPS (number of frames per second) of 50 frames / second, to 0.32 seconds. This duration corresponds approximately to the duration of half or a third of a heartbeat, which seems relevant for detecting the presence of cardiac amyloidosis.

[0089] The videos used for training, validation and testing therefore each comprise 16 images.

[0090] We are now interested in the classification model itself.

[0091] For example, to classify echocardiographic videos (videos of size W x H x T: width x height x time), it would have been possible to use convolutional neural networks of the “2D CNN” type (2D for two dimensions, and CNN for Convolutional Neural Network) by considering time as a color channel. It would also have been possible to use a 3D CNN type network, taking all 16 frames as input directly.

[0092] However, we chose to use a convolutional neural network implementing spatiotemporal convolutions of type (2+l)D.

[0093] More precisely, the classification model here is a convolutional neural network of type R(2+1)D based on a ResNet (for residual neural network).

[0094] ResNet networks are Deep Learning models, of the CNN type.

[0095] ResNet networks are characterized by the addition of residual links between certain layers. These residual links allow connection jumps to be made which notably solve the problem of the vanishing gradient, or Vanishing Gradient, encountered during learning.

[0096] By separating spatial and temporal convolutions, (2+1)D convolution allows for better isolation and extraction of domain-specific features (spatial and temporal), rather than mixing this information in a single 3D convolution. Also, this decomposition reduces computational complexity, making training and inference faster and more efficient.

[0097] The ResNet network used here is an 18-layer network (ResNet-18).

[0098] With reference to [Fig.6], the R(2+1)D models are therefore ResNet models in which, for each convolution layer, the 3D convolution is factored into a product of two convolutions: a 2D spatial convolution followed by a 1D temporal convolution.

[0099] The number N of two-dimensional filters (kernels) is chosen to obtain the same number of parameters as a 3D convolution block.

[0100] Such a model is described for example in the document Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2018). A dosage look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 6450-6459).

[0101] The training of the classification model is carried out in a training module 11 of the processing unit 1.

[0102] Some hyperparameters used during the training step (step E10) are provided.

[0103] The model was trained on 50 epochs, i.e. via 50 complete passes of the entire training videos in the model.

[0104] The batch size, that is to say the number of training videos passing together in the network, is here equal to 8.

[0105] The learning rate is equal to 0.0001 and is divided by 10 at epochs 15, 30 and 45.

[0106] The model weights were initialized using the weights of the pre-trained model from the Kinetics 400 database. The optimizer used is Adam, and the loss function is cross-entropy loss. Accuracy, precision, and recall are the three metrics that were calculated to monitor the model's performance.

[0107] [Fig.7] shows the performance of the model during the training phase and the validation phase.

[0108] Curve C1 represents the evolution of the loss function as a function of the epochs, during the training phase. Curve C2 represents the evolution of the loss function as a function of the epochs, during the validation phase.

[0109] Curve C3 represents the evolution of accuracy as a function of epochs, during the training phase. Curve C4 represents the evolution of accuracy as a function of epochs, during the validation phase.

[0110] Curve C5 represents the evolution of precision as a function of epochs, during the training phase. Curve C6 represents the evolution of precision as a function of epochs, during the validation phase.

[0111] Curve C7 represents the evolution of recall as a function of epochs, during the training phase. Curve C8 represents the evolution of recall as a function of epochs, during the validation phase.

[0112] We are now interested in the implementation of the detection assistance method on a “new” patient during an examination. We are therefore seeking to detect cardiac amyloidosis in this patient. The detection assistance method therefore uses the inference model which was obtained thanks to the different steps which have just been described. It should be noted that the detection assistance method is intended to help the doctor to establish a diagnosis, and not to establish said diagnosis.

[0113] With reference to [Fig.8], the detection assistance method is here carried out in real time and is implemented in a processing unit 12 which is here integrated into an ultrasound scanner 13. The ultrasound scanner 13 also comprises an ultrasound probe 14 which emits and receives ultrasound signals to produce ultrasound videos.

[0114] It is noted that the detection assistance method could also be carried out at a later stage, after the examination, using the images captured during said examination. The detection assistance method could also be implemented in a processing unit which is not integrated into an ultrasound scanner.

[0115] Again, the processing unit 12 is an electronic and software unit. The processing unit 12 comprises one or more processing components 15, and for example any processor or microprocessor, general or specialized (for example a DSP, for Digital Signal Processor, or a GPU, for Graphics Processing Unit, or even an NPU, for Neural Processing Unit), a microcontroller, or even a programmable logic circuit such as an FPGA (for Field Programmable Gate Arrays) or an ASIC (for Application Specific Integrated Circuit).

[0116] The processing unit 12 also comprises memories. At least one of these memories forms a computer-readable recording medium, on which is recorded at least one computer program comprising instructions which cause the processing unit 12 to execute at least some of the steps of the detection assistance method.

[0117] With reference to [Fig.9], the processing unit 12 therefore receives at least one current video Vc obtained from an echocardiography carried out on the patient: step E20.

[0118] Optionally, the current video is applied as input to the classification model for selecting the A4c cuts: step E21 (the classification model used here is the one for distinguishing an A4c cut from the other cuts, and not the video classification model).

[0119] The processing unit 12 then implements a processing phase Ph on the current video. The processing phase is the same as that previously described for the learning method.

[0120] The processing phase Ph therefore firstly consists of processing each current image of the current video to remove the additional information from said current image and retain only the ultrasonic signal.

[0121] The processing phase includes, for each current image of the current video, the trimming step: step E22.

[0122] The processing unit 12 acquires coordinates of reference points delimiting a useful portion of the current image comprising the ultrasound signal.

[0123] The current image is cropped by the first module 4a to retain only a part of the current image passing through the reference points and comprising the useful portion.

[0124] The processing phase then includes the step of applying a binary mask to the useful portion of the current image to remove the graduations: step E23. The binary mask is applied by the second module 4b.

[0125] Then, the processing phase comprises the steps of detecting in the current image the pixels corresponding to the electrical signal representation, and of removing said electrical signal representation by applying a median filter to each of said pixels: step E24. These steps are carried out in the third module 4c.

[0126] The current video is resampled to obtain a video in standardized format: step E25. This step is carried out in the fourth module 4d.

[0127] The current video is standardized: step E26. This step is implemented in the fifth module 4e.

[0128] The processing phase therefore consists of passing the current video through the same processing pipeline 10 as that used during the training phase.

[0129] The processing phase, applied to the current video, makes it possible to obtain a processed video from the current video.

[0130] Then, the processing unit 12 implements an inference phase.

[0131] The inference phase is implemented in an inference module 16 of the processing unit 12.

[0132] The processing unit 12 executes an inference of the previously trained classification model, by applying as input to the inference model at least one input video obtained from the processed video: step E27. The model is the convolutional neural network R(2+1)D which has just been described.

[0133] As seen, the model was trained with training videos each comprising 16 images. However, the current video has more than 16 images.

[0134] Several inference approaches have been evaluated.

[0135] A first approach consists of applying all the images of the processed video as input to the inference model. The input video is therefore the processed video. In this case, the output of the inference model provides a single prediction used as the final result. This first method of obtaining the final prediction is called "method 1".

[0136] A second approach consists in producing, from the processed video, a plurality of image packets, each comprising a first predetermined number of successive images, the image packets forming the input videos applied as input to the classification model. The inference phase thus comprises the step of producing a plurality of intermediate predictions each associated with a distinct image packet.

[0137] The first predetermined number is equal to 16. We therefore apply packets of 16 consecutive images as input to the inference model.

[0138] The image packets are defined successively, each image packet being shifted in time, relative to a previous image packet, by a second predetermined number of images. The second predetermined number is here equal to 1.

[0139] The center of each packet is therefore shifted by one image in time.

[0140] In this way, several predictions are obtained, each corresponding to one of the packets. If the processed video contains N images, N-16 predictions are obtained. The calculation of the final prediction can then be carried out using several methods.

[0141] The processing unit 12 can determine, among the N-16 predictions, the label which was predicted predominantly among the labels “amyloidosis” (positive) and “control” (negative). This majority label is assigned to the final prediction. This second method of obtaining the final prediction is called “method 2”.

[0142] If amyloidosis has been predicted at least once, the processing unit 12 may consider the final prediction positive for amyloidosis. This third method of obtaining the final prediction is called “method 3”.

[0143] If amyloidosis has been predicted at least once with a confidence score greater than a predetermined threshold, the processing unit 12 may consider the final prediction positive for amyloidosis. The predetermined threshold is for example equal to 60%. This fourth method of obtaining the final prediction is called “method 4”.

[0144] As we will see below, method 4 is chosen.

[0145] During the examination, to provide a visual interpretation of the prediction proposed by the detection assistance method, a visualization method is used: step E28. The visualization method consists of visualizing the activation functions of a a certain, well-chosen layer of the neural network, where the network analyzes the essential characteristics of the image.

[0146] The heat map is obtained from the last convolutional layer of the neural network used (see above).

[0147] The last convolutional layer has been extracted. This layer contains the information related to the classification task. The gradients of this layer are used to weight the activation maps, thus producing a heat map showing the regions that have aroused the model's interest.

[0148] Here we use the GradCam 3D method, described for example in the document Gotkowski, K., Gonzalez, C., Bûcher, A., & Mukhopadhyay, A. (2020). M3d-CAM: A PyTorch library to generate 3D data attention maps for medical deep learning. arXiv preprint arXiv:2007.00453.

[0149] We obtain a heat map superimposed on the original ultrasound signal which evolves dynamically with the video, thus showing image by image the location of the ultrasound signal which oriented the prediction. We see in [Fig. 10] the successive images II, 12, 13, 14, on each of which are superimposed hatchings which represent in a simplified manner the corresponding heat map. For each image, the smaller the spacing between the hatchings, the greater the intensity.

[0150] The visualization method allows the physician to visualize the parts of the ultrasound signal in which the classification model has focused its attention during its predictions.

[0151] In addition to providing the doctor with initial diagnostic assistance, the detection assistance method therefore uses the heat map to target the locations on the ultrasound likely to confirm or refute the diagnosis.

[0152] A confidence score is also predicted by the model and transmitted to the practitioner.

[0153] We are now interested in evaluating the performance of the classification model.

[0154] The test metrics that were used are: precision, recall, as well as Y AUC Precision - Recall (AUC for Area Under Curve, which is translated by Surface Under the Curve, Precision - Recall which is translated by Precision-Recall) and the AUC of the ROC curve (ROC for Receiver Operating Characteristic). The ROC curve is also called the sensitivity curve.

[0155] Since the training video set is unbalanced, as it contains much more control data than positive data, the accuracy will not be sufficiently representative of the model quality. Indeed, with a model consistently predicting the healthy label, relatively high accuracy would be achieved. The evaluation metrics used are more robust to class imbalance.

[0156] The results of training the classification model are presented, as a function of the four evaluation methods just described.

[0157] Table 1 provides the values ​​obtained from the different metrics (in %), for the different methods of obtaining the final prediction: method 1, method 2, method 3 and method 4. Accuracy Precision Recall AUC PR AUC ROC Method 1 92 94 62 0.86 0.93 Method 2 93 88 72 0.87 0.92 Method 3 94 81 87 0.93 0.97 Method 4 95 87 84 0.93 0.97

[0158] Table 1

[0159] It is noted that method 1 and method 2 have too low a recall, which is due to too high a number of false negatives.

[0160] Method 3 and Method 4 allow more false negatives to be transformed into true positives than true negatives into false positives, which increases the AUCs.

[0161] Method 3 and Method 4 present a good compromise between precision and recall. The best compromise is that obtained by Method 4.

[0162] These results give an indication of how the classification model detects amyloidosis during an ultrasound sequence. Amyloidosis detection is done punctually, at certain moments in the current video where certain characteristics of amyloidosis seem to appear, rather than globally and uniformly throughout the sequence. Indeed, if we observe the majority predictions, the classification model tends to predict the healthy label more often, and the number of false negatives is therefore greater for methods 1 and 2. If, on the other hand, we focus on local predictions, amyloidosis is better detected by reducing the number of false negatives with methods 3 and 4.

[0163] It is finally method 4 which was retained as the most optimal method and which is used in the simulations presented below.

[0164] Tables 2 and 3 present the results of simulations using different configurations: - configuration 1: model trained on the set of training videos corresponding to [Fig.l]; - configuration 2: model trained on the training video set filtered to select only A4c slices (therefore corresponding to [Fig.2]); - configuration 3: model trained with a second set of training videos, and validated with a second set of validation videos. The second set of training videos and the second set of validation videos contain a larger number of videos corresponding to cases of cardiac amyloidosis.

[0165] This second distribution of the reference videos is visible in [Fig. 11].

[0166] The amyloidosis cases come from a new ultrasound machine (the 3h ultrasound machine) with no control data. This new distribution is therefore somewhat biased because the distribution of the machines is not respected. It serves to validate the importance of the homogeneous distribution of data per ultrasound machine as explained above.

[0167] The last line of Tables 2 and 3 presents the results of configuration 3 by initializing this time the weights of the model with those of the model trained in configuration 1. We speak of a “pre-trained configuration”. This last simulation assumes that a model trained on an unbalanced data set (configuration 3) can still continue to learn reasonably by incorporating information from previous training. This technique is called “transfer learning”.

[0168] The last three columns of Tables 2 and 3 correspond to the test metrics evaluated on the entire data set with A4c filter ([Fig.2]). Configur-a tion Pre-trained Precision Recall Accuracy Precision (A4C) Recall (A4C) Accuracy (A4C) 1 ❖ 87.29 84.44 94.7 92.06 90.62 96.26 2 ❖ ❖ ❖ ❖ 71.43 95.70 90.74 3 ❖ 53.30 95.04 83.22 53.55 97.27 81.05 3 ❖ 68.36 91.70 90.35 67.75 97.66 89.38

[0169] Table 2

[0170] It can be seen that the processing phase carried out, and the classification model used, make it possible to obtain an accuracy of approximately 95% on the test data set, as well as a precision of approximately 87% and a recall of approximately 84%. These results are significantly superior to those obtained by the methods of the prior art. Configur-at ion Pre-trained AUC ROC AUC PR AUX ROC (A4C) AUC PR (A4C) 1 ❖ 0.97 0.93 0.99 0.97 2 ❖ ❖ ❖ 0.98 0.96 3 ❖ 0.96 0.90 0.97 0.94 3 ❖ 0.97 0.92 0.99 0.97

[0171] Table 3

[0172] Configuration 1 therefore gives the best result, corresponding to the best compromise between precision and recall. In addition, we observe that a non-balancing of the data by ultrasound (configuration 3) drastically degrades the performance of the model, because of the machine bias introduced, even when using transfer learning.

[0173] Regarding GradCam observations via heat maps, the model tends to often observe the mitral valve in systole, and the left ventricle as well as the septum in diastole. During a video sequence, the heat map is observed moving and pointing at different parts of the heart in a coordinated manner with the heartbeats (see [Fig. 10]). The fact that this movement of the heat map is observed dynamically with the beats seems to show that the characteristic signs of the disease differ depending on the moments of the cardiac cycle.

[0174] Of course, the invention is not limited to the embodiment described but encompasses any variant falling within the scope of the invention as defined by the claims.

[0175] The convolutional neural network used to detect cardiac amyloidosis is not necessarily an 18-layer ResNet. It can be a ResNet with a different number of layers, or a different network model. It is thus possible, for example, to integrate the (2+1)D convolution blocks into C3D (for Convolutional 3D) or I3D (for Inflated 3D ConvNet) type architectures.

[0176] The invention can of course be implemented with numerical values ​​different from those cited here (and in particular different values ​​of hyperparameters).

[0177] The architectures of the processing units can of course be different.

[0178] The steps of the learning method and the detection assistance method can be carried out in a different order (and in particular those of the processing phases).

Claims

Claims

1. Method for assisting in the detection of cardiac amyloidosis in a patient, implemented in a processing unit (12) and comprising: • a reception step, consisting of receiving at least one current video (Vc) obtained from an echocardiography performed on the patient, the current video comprising a plurality of current images each comprising an ultrasound signal (5) and additional information (7, 9); • a processing phase (Ph), comprising the step of processing each current image of the current video to remove the additional information from said current image, so as to produce a processed video from the current video;• an inference phase (E27), comprising the step of executing an inference of a previously trained classification model, by applying as input to the classification model at least one input video obtained from the processed video, the classification model being a convolutional neural network implementing spatiotemporal convolutions of type (2+l)D.;

2. A method for assisting detection according to claim 1, said convolutional neural network being based on a ResNet.

3. Detection assistance method according to one of the preceding claims, in which the inference phase comprises the step of producing, from the processed video, a plurality of image packets, each comprising a first predetermined number of successive images, the image packets forming input videos applied as input to the classification model, the inference phase thus comprising the step of producing a plurality of intermediate predictions each associated with a distinct image packet.

4. A method of assisting detection according to claim 3, wherein the image packets are defined successively, each image packet being shifted in time, relative to a previous image packet, by a second predetermined number of images.

5. Detection assistance method according to one of claims 3 or 4, in which if, for the current video, at least one intermediate prediction, obtained for a packet of images, is a positive prediction, which therefore detects a presence of amyloidosis, and which is associated with a confidence score greater than a predetermined threshold, the inference phase produces a final prediction which is also a positive prediction.

6. Detection assistance method according to one of the preceding claims, in which the processing phase comprises, for each current image of the current video, the step of acquiring coordinates of reference points (S1, S2, S3) delimiting a useful portion (6) of the current image comprising the ultrasonic signal.

7. Detection assistance method according to claim 6, in which the processing phase comprises the step (E22) of cropping the current image to retain only a part of the current image passing through the reference points and comprising the useful portion.

8. Detection assistance method according to one of claims 6 or 7, in which the additional information comprises graduations (7), the processing phase further comprising the step (E23) of applying a binary mask to the useful portion of the current image to remove said graduations.

9. Detection assistance method according to one of the preceding claims, in which the additional information comprises an electrical signal representation (9), the processing phase comprising the steps (E24) of detecting in the current image pixels corresponding to the electrical signal representation, and of removing said electrical signal representation by applying a median filter to each of said pixels.

10. Detection assistance method according to one of the preceding claims, further comprising the step (E28) of superimposing on a current image a heat map obtained from the last convolutional layer of the convolutional neural network.

11. Ultrasound scanner (13) comprising a processing unit (12) in which the detection assistance method according to one of the preceding claims is implemented.

12. Computer program comprising instructions which cause the processing unit (12) to execute the steps of the detection assistance method according to one of claims 1 to 10.

13. A computer-readable recording medium on which the computer program according to claim 12 is recorded.

14. Method for learning the classification model of the detection assistance method according to one of claims 1 to 10, implemented in a processing unit (1) and comprising: • a reception step, consisting of receiving a set of reference videos (Vref) obtained from echocardiographies carried out on reference patients; • a processing phase, implemented on each reference video, which is the same as that of the detection assistance process; • a training phase (E10), consisting of training the classification model using a set of training videos belonging to the set of reference videos.

15. A learning method according to claim 14, further comprising the preliminary step of applying, on reference images of the reference videos, another previously trained classification model, to retain only A4c sections among them.

Images