Attention learning from videos of an area of interest of a patient

Abstract

In some embodiments of the present disclosure, video frame data across one or more videos is processed by a pretrained encoder and the resulting frame-based embeddings are concatenated for analysis by a downstream attention-based deep learning network. In various embodiments, different types of self-supervised learning (SSL) pretraining techniques and different types of attention-based deep learning networks are utilized. In a particular example, embodiments of the present disclosure are applied to drawing inferences regarding pulmonary hypertension based on video data from a patient's transthoracic echocardiogram. In another example, embodiments of the present disclosure are applied to drawing inferences regarding anatomical bowel segments in an endoscopy video. The details of various examples are further described herein.

Description

Attention Learning from Videos of an Area of Interest of a PatientCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Applications 63 / 533,101, filed on August 16, 2023; 63 / 599,994, filed on November 16, 2023; and 63 / 555,883, filed on February 20, 2024. The contents of these applications are incorporated by reference herein.BACKGROUND

[0002] This disclosure relates generally to computerized technology in the medical space for making inferences from one or more videos of an area of interest of a patient.SUMMARY

[0003] In some medical image analysis contexts, “best” views are typically selected for making algorithmic-based inferences from patient data. For example, in the context of transthoracic echocardiography (TTE), many TTE videos are generated during a patient examination, but only a subset (or even just one) of those videos is selected (e.g., via computerized analysis) for more detailed analysis by the computerized system. However, video data from unselected views / videos might have useful information to inform computerized analysis. At the same time, video data, by its nature, is quite large and computerized analysis of video data can be resource intensive. Attention-based deep learning approaches require significant processing resources as well, and the resource consumption level is dependent on the level of granularity to which attention mechanisms are applied. Therefore, efficiently implementing attention-based learning in the context of analyzing medical video data benefits from thoughtful approaches to the levels of data aggregation that are used for attention-based analysis. Also, approaches are needed that can take advantage of information available across a plurality of videos, whether a particular video is determined to be a “best” view, or not.

[0004] In some embodiments of the present disclosure, video frame data across one or more videos is processed by a pretrained encoder and the resulting frame-based embeddings are concatenated for analysis by a downstream attention-based deep learning network. In various embodiments, different types of self-supervised learning (SSL) pretraining techniques and different types of attention-based deep learning networks are utilized. In one example, embodiments of the present disclosure are applied to drawing inferences regarding pulmonary hypertension (PH) based on video data from a patient’s transthoracic echocardiogram. In another example, embodiments of the present disclosure are applied to identifying the anatomic bowel segment corresponding to a frame of an endoscopy video. The details of various examples are further described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] FIG. 1 illustrates a medical video processing system in accordance with an embodiment of the present disclosure.

[0006] FIG. 2 illustrates a medical video processing system in accordance with another embodiment of the present disclosure.

[0007] FIG. 3 illustrates a self-supervised learning (SSL) pretraining method in accordance with one embodiment of the present disclosure.

[0008] FIG. 4 illustrates a method in accordance with the present disclosure for processing medical videos corresponding to a patient exam by a computerized deep learning network to make patient-related inferences based on the medical videos.

[0009] FIGs. 5a-5d illustrate frame-wise attention representations obtained from a medical exam video dataset at various levels of granularity.

[0010] FIG. 6 shows an example of a computer system, one or more of which may be used to implement one or more of the apparatuses, systems, and methods illustrated herein.

[0011] While embodiments of the present disclosure are described with reference to the above drawings, the drawings are intended to be illustrative. Other embodiments are consistent with the spirit and scope of the disclosure.DETAILED DESCRIPTION

[0012] The various embodiments now will be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific examples of practicing the embodiments. This specification may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this specification will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Among other things, this specification may be embodied as methods or devices. Accordingly, any of the various embodiments herein may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. The following specification is, therefore, not to be taken in a limiting sense.

[0013] FIG. 1 illustrates a medical video processing system 1000 in accordance with an embodiment of the present disclosure. This and other embodiments will be described with reference to videos from a patient’s transthoracic echocardiogram (TTE) examination. But the underlying principles of the disclosure are applicable to other medical image / video analysis applications.

[0014] This example shows processing of a plurality of videos including first video 11 having ml (number of) frames, second video 12 having m2 frames, and nthvideo 13 having mv frames. Each video corresponds to a different view from a medical procedure such as, for example, a TTE test. For example, video 11 corresponds to a view first view (#1), video 12 corresponds to a second view (#2), and nthvideo 13 corresponds to the nthview. In aparticular embodiment, several additional videos (not shown) are processed corresponding to views between second view #2 and nthview #n. In a typical TTE examination, for example, about 10-20 different TTE videos might be taken, each considered to correspond to a different potentially useful view.

[0015] Pre-processing block 110 pre-processes the video data corresponding to a patient’s TTE study. Specifically, block 110 processes the video data such that each frame has uniform dimensions, DI. In this example, each frame in each video has, after preprocessing, dimensions DI of 224 x 224 x 3 (if using RGB color channels). Other dimensions can be used in different implementations. Pre-processing block 110 prepares matrices, e.g., matrices 21, 22, and 23, including data for each video. The dimensions of each matrix are 224 x 224 x 3 (DI), x the number of frames in the video (e.g., ml, m2, mv).

[0016] The video data matrices are processed by encoder 120. In the illustrated embodiment, encoder 120 has been pre-trained using a self-supervised learning (SSL) technique. In a typical implementation, pre-training is performed by dividing frames into patches, and a self-supervised learning task is performed with the encoder, in some cases using augmented versions of the patches, to train the encoder to extract useful features from frame pixel data for downstream learning tasks. In some embodiments, the “SimCLR” approach is used, as described by Chen et al. in “A Simple Framework for Contrastive Learning of Visual Representations” (2020). In other embodiments, a “DINOvl” approach (described by Caron et al. in “Emerging Properties in Self-Supervised Vision Transformers” (2021)) or a “DIN0v2” approach (described by Oquab et al. in “DIN0v2: Learning Robust Visual Features without Supervision” (2023)) is used. All three papers are hereby incorporated by reference in their entirety. In other embodiments, any combination of the foregoing approaches (or any portion thereof) may be used.

[0017] In the SimCLR approach, the same image or image patch is differently augmented to generate two differently augmented images (or image patches) corresponding to the same image (or image patch) (referred to in the SimCLR paper as a “positive pair”). The learning network tries to minimize the result of a contrastive loss function value between encodings of each augmentation in the pair, the encoding being carried out by the same network (having the same initial parameters and the same training-adjusted parameters). In SimCLR, the augmented patch pairs are processed through the same encoding network (e.g., a ResNet) and the network’s parameters are then adjusted to minimize contrastive loss.

[0018] In the DINO approaches, different versions of the augmented images (and / or augmented patches) are passed through different networks (teacher and student networks) having the same architecture (e.g., a Vision Transformer (“ViT”)) but different starting parameters. During pre-training, parameters of the student network are adjusted to try to minimize the differences between its K-dimensional output embedding and the teacher’s output embedding. In a particular example, the teacher’s parameters are updated based on an exponential moving average of the student’s parameters as the student learns. After pretraining, the teacher network is used as the pre-trained encoder (for example encoder 120).

[0019] With reference to exemplary FIG. 1, encoder 120 (which may be, e.g., an SSL pretrained encoder) generates, from each frame’s data, embedding vectors having D2 dimensions that are collected into matrix representations, one embedding matrix for each video. In one implementation, using a ViT encoder and the DIN0v2 framework, D2 is 768. In the illustrated example, encoder 120 generates matrices for each video such as, for example, matrices 31, 32, and 33. The dimensions of each matrix correspond to the vector embedding dimension D2 and the number of frames for the given video. As shown, matrix 31 has dimensions D2 x ml, where ml is the number of frames processed for first video 11; matrix 32 has dimensions D2 x m2, where m2 is the number of frames processed for secondvideo 12; and matrix 33 has dimensions D2 x mv, where mv is the number of frames for nthvideo 13.

[0020] Block 130 unpacks the frame-level embedding vectors from embedding matrices output by encoder 120 and concatenates the frame embedding vectors, which each have D2 dimensions. Concatenate, in this context, simply means that the frame embeddings across all videos for a given patient’s TTE are linked for being processed together (whether are not the frames are from the same video) by the downstream parts of the network, and one having skill in the art will understand that another term may be used in place of concatenate to the same general meaning.

[0021] Positional encoding block 140 computes a position encoding value based on positional information corresponding to the frame. In the original transformer model (as described by Vaswani et al. in “Attention is All You Need”, 2017, hereby incorporated by reference herein), this encoding’s values include both sine and cosine functions of the frame’s position, the embedding’s output dimension, and an integer index value that increments from 0 to — 1, i.e., from 0 to one less than half the number of dimensions (d) of the output embedding vector. This results in a positional encoding vector that is the same length (i.e., has the same number of values) as a frame embedding vector, and the positional encoding vector can therefore be added to the frame embedding vector through simple vector addition. In the case of a frame embedding having D2 dimensions, the positional encoding (PE) is determined using the following equations:PE(pos,2i) =sin100002i / Dz)where pos is the frame’s position in a sequence of frames, i is the integer index, and D2 is the dimensions of the frame embedding. So, for example, a positional encoding vector for aframe embedding of 768 dimensions is a vector including the above sine and cosine values for each “i” index value from 0 to 383. This results in an embedding vector of length 768, i.e., a sine and cosine value corresponding to the respective equations above for each of the 384 values of “z”.

[0022] In one example, for each video, frame “pos” values start at zero and are incremented for each frame in the sequence of frame of the video. In an example where a plurality of videos corresponding to the same patient are processed, information that identifies which video a particular frame comes from is tracked, but that “which-video” information is not included in the pos value used for the positional encoding described above. Rather, the frame pos values restart again from zero for each video. In this manner, each video is encoded separately despite the frames being concatenated.

[0023] Of course, the foregoing is simply one example and any one or more of many other positional encoding methods can be used. For clarity, the positional encoding function referenced above is “fixed” in that it does not include any learnable parameters (i.e., parameters that are updated based on training data and backward propagation of error). However, other examples consistent with the present disclosure can additionally or alternately include a positional encoding function with learnable parameters (or a different fixed positional encoding function than the one referenced above).

[0024] Addition operation 150 performs vector addition to add the computed positional encoding vector to the frame embedding vector (both being the same length) and passes (or otherwise outputs) a resulting input vector 51 as input into multi-head attention encoder 160. A positional encoding value is added to every frame-level embedding for a patient’s medical videos and a set of resulting input vectors 51 are processed together by encoder 160.

[0025] In one embodiment, multi-head attention encoder 160 is implemented as a set transformer encoder as described by Lee et al. in “Set Transformer: A Framework forAtention-based Permutation-Invariant Neural Networks,” 2019 (incorporated by reference herein). The Multi-head Atention Block (MAB) for matrices X and Y, representing two d- dimensional vectors, integrates layer normalization and multi-head atention with parameters co as: MAB (X, Y) = LayerNorm (H + rFF(H)) where H = LayerNorm (X + Multihead (X, Y, Y; co)) and rFF is any row-wise feedforward layer. The Set Atention Block (SAB) in Lee is defined as: SAB(X) := MAB(X,X). (See Lee at 3.1). In a particular example applied to patient video analysis, “X” is a matrix that includes all the frame embeddings from all the videos, with each row being a frame embedding vector for a particular frame. This operation takes SSL-processed video frames, applies self-atention, and yields a set reflecting the interactions among input frames. Given its multi -head atention design, where Q = K = V = X, the SAB treats frames uniformly for permutation invariance.

[0026] Those skilled in the art will appreciate that the Set Transformer described in Lee is a variation of the original transformer described in Vaswani (referenced above). Another variation is the Vision Transformer (ViT) described in Dosovitskly et al. “An Image is Worth 16X16 Words”, 2021, hereby incorporated by reference herein. As shown in Dosovitskly, a transformer encoder such as that shown in Vaswani can be combined with processing that is suited for the image processing context (rather than using the transformer decoder of Vaswani, which is most applicable to language or other sequential data). In some embodiments, a ViT encoder similar to that taught be Dosovitskly is used. Those skilled in the art will appreciate that, in addition to the multi-head atention encoders taught in Lee and Dosovitskly (both of which are variations on the encoder in Vaswani), other variations on multi-head atention encoders can be additionally or alternately used to implement multihead atention encoder 160 in Lee. The referenced embodiments provide just some examples.

[0027] In the illustrated embodiment of FIG. 1, multi-head atention encoder 160 outputsatention-processed vectors 61 , each representing the output of processing an input vector 51 through encoder 160, whose processing atends to all other input vectors 51 to produce a particular output vector 61. In the illustrated embodiment, each output vector 61 corresponds to a particular video frame processed by system 1000.

[0028] Output vectors 61 are processed by an atention pooling layer comprising atentionweighting block 192 and aggregator 191. In the illustrated embodiment, atention-weighting block 192 assigns an atention value (which is a learnable parameter) to each frame, and then aggregator block 191 multiplies each vector 61 corresponding to that frame by the corresponding atention value received from atention block 250 and sums the resulting atention-weighted vectors for all frames in all videos of a patient’s TTE exam (or other exam) to obtain a single summarized vector 71 per patient, of size D2 x 1.

[0029] Multilayer perceptron (MLP) 170-2 processes vector 71 and outputs values that correspond to various desired output classifications (e.g., “PH”, “Not PH”, “indeterminate,” etc.). Softmax layer 170-1 converts the output of MLP 170-2 to a class probability value to provide PH inferences for each patient. An “inference” could, in some embodiments, be a prediction, estimate, score, suggestion, categorization, or other inference. During supervised training, learning block 180 computes a loss value (sometimes refer to as an “error”) and then back propagates that error to adjust learnable parameters in MLP 170-2, atentionweighting block 192, and multi-head atention encoder 160.

[0030] In some embodiments, an atention score for each frame is obtained from atentionweighting block 192 and that is provided to video review interface module 175 so that the frames most important to the inference can be reviewed through a computer user interface (interface not separately shown). An atention “score” in this context can simply be equal to or derived from the weight that is applied by block 192. This atention score reflects how heavily a particular frame was weighted in the atention-pooling operations carried out byblocks 192 and 191, and therefore how much a particular frame’s representation 61 contributed to the summarized representation vector 71. In one embodiment, the approach to attention pooling disclosed in Use et al., “Attention-based Deep Multiple-Instance Learning,” 2018 (hereby incorporated by reference herein) is used.

[0031] In alternate embodiments, aggregation of the input vectors for downstream classification may be performed with an additional input vector, referred to as a classification (“CLS”) token, which may be processed together with the input vectors representing images or image portions. In such an embodiment, CLS token has the same dimensions as the input image representation vectors, but is initialized to random or predetermined start values that are unrelated to the values of the image representation vectors. Because the transformer attention processing carried out by the encoder on a given input vector attends to the values of all the other input vectors, such a CLS token, after processing, can be used as an aggregated representation of all of other processed vectors. In such an example, the resulting processed CLS token output by the encoder can be used by the downstream classification portions of the network (e.g., in the context of FIG. 1, MPL 107- 2). Although the illustrated example does not use a CLS token to obtain aggregated (i.e., summarized) output for classification processing, alternatives to the illustrated example could be implemented using such a token for attention pooling without departing from the spirit and scope of the present disclosure.

[0032] One aspect of using the approach illustrated in FIG. 1, is that a specific score per frame is readily available. Thus, in addition to the system providing a computed classification (e.g., an inference or estimation), the system can also provide (e.g., output and / or display) at least some transparent, user-digestible information of how the computed classification was reached. As one example, in the context of a multi-frame medical video, the system can identify and / or output information comprising which frames were mostimportant (or otherwise significant, or a ranking of significance) to the analysis resulting in the computed classification, for example as an annotation, overlay, report, notes, or the like. Such identification can be useful for later professional medical review of a TTE video, or other medical videos. For example, the user of video review interface 175 can interactively access the frames which system 1000 determined to be most important (or otherwise significant) to the classification decision and the user can devote additional time to reviewing those frames.

[0033] FIG. 2 illustrates a medical video processing system 2000 that has some alternative aspects relative to the embodiment illustrated in FIG. 1. System components with the same reference number as those shown in FIG. 1 will not necessarily be separately described in the context of FIG. 2 and are of a similar nature and scope (including potential variations) as same-numbered elements in FIG. 1. The embodiment of FIG. 2 utilizes multi -instance attention with a convolutional neural network (CNN) encoder, in conjunction with weakly supervised learning. As illustrated for this example, frame embeddings 41 corresponding to frames across all videos of a patient’s TTE examination (or other medical examination) are processed by convolutional neural network (CNN) 240 (which may, in some examples, be a ResNet). Each frame embedding 41 is fed into CNN 240 and the resulting encoding vectors 53 each have, in one example, the same dimensions as a corresponding input frame embedding vector 41.

[0034] In the illustrated embodiment, attention block 250 assigns an attention value (which is a learnable parameter) to each frame, and then aggregator block 260 multiplies each vector 53 corresponding to that frame by the corresponding attention value received from attention block 250 and sums the resulting attention-weighted vectors for all frames in all videos of a patient’s TTE exam (or other exam) to obtain a single summarized vector 72 per patient, of size D2 x 1.

[0035] In similar fashion to that described in the context of FIG. 1, MLP 280-2 processes vector 72 and outputs a number of values that corresponds to the number of desired output classifications (e.g., “PH”, “Not PH”, “indeterminate,” etc.) and Softmax layer 280-1 converts the output of MLP 280-2 to a class probability value to provide PH inferences for each patient. During weakly supervised training, WSL block 290 computes a loss value (“error”) and then back propagates that error to adjust learnable parameters in MLP 280-2, attention block 250, and CNN 240.

[0036] In some embodiments, an attention score for each frame is obtained from block 250 and is provided to video review interface module 15 so that the frames most important to the inference can be reviewed through a computer user interface (interface details not separately shown).

[0037] FIG. 3 illustrates self-supervised learning (SSL) pretraining method 3000, in accordance with one embodiment of the present disclosure. Step 310 obtains a dataset representing video data corresponding to a plurality of views of an area of interest, the video data from each video comprising a plurality of frames. As one skilled in the art will appreciate, such a dataset need not be labelled. And, if it is labelled, the labels need not be used for SSL pretraining. Step 320 divides each frame into a plurality of tiles, sometimes called “patches.” Step 330 performs one or more augmentation operations on each patch. In the context of machine learning for image processing, “augmentations” refers to one or more modifications to an image such as, for example, randomly blurring, cropping, converting to greyscale, etc.

[0038] In some SSL processes, images (or image patches) are augmented, and the SSL encoder network tries to extract feature representations that will minimize the difference between representations of different augmentations of the same image, or the same image patch. In some techniques, such as SimCLR, the same network processes positive pairs ofaugmented images (or image patches), i.e., two different augmentations of the same image or image patch. In another set of techniques, e.g., DINO and DIN0v2 discussed above, different augmentations are processed by two different encoder networks (one a “teacher” and the other a “student”, initialized with different parameters) and training seeks to have the representations output by one encoder get closer to the representation output by another encoder.

[0039] Consistent with any one of various SSL techniques, step 340 encodes each augmented image (or image patch) with one or more encoders. Step 350 computes a loss (using a loss function), based on the output of the one or more encoders, in accordance with a defined SSL technique which may specify the SSL learning task and corresponding loss function. Step 350 adjusts learnable parameters of the one or more encoders in view of back propagation of the computed loss.

[0040] FIG. 4 illustrates a method 4000 in accordance with one example of the present disclosure for processing medical videos corresponding to a patient exam (a TTE exam, for example) by a computerized deep learning network to make patient-related inferences based on the medical videos.

[0041] Step 401 pre-processes a data set (or a next data set) obtained from a plurality of videos corresponding to a plurality of views of an area of interest of a patient. In this example, pre-processing includes parsing the video data for each video into a plurality of frames. In one example, the frames are pre-processed to make each frame a uniform pixel size. In one example, in the case of RGB color, each frame is pre-processed to be 224 x 224 x 3 in size. Other dimensions are consistent with the present disclosure, this is just one example.

[0042] Step 402 uses an SSL pre -trained encoder to embed each frame and obtain a plurality of embedding vectors, each representing features of a respective frame’s video data. Step403 concatenates the plurality of frame embedding vectors (in one example, across the plurality of videos corresponding to a patient medical exam such as a TTE exam) for processing together by an attention-based deep learning network.

[0043] Step 403 obtains an attention-based output vector from processing the plurality of input frame embeddings across the plurality of videos. In one embodiment, the output vector may be obtained by multi-instance attention learning processing / attention pooling in which, for example, learned weights are applied to a plurality of individual output vectors corresponding to individual frames and then summarized (e.g., multiplied by weights and then added) to obtain the attention-based output vector. In alternative embodiments, the attention-based output vector is the result of processing an initialized (e.g., to random values, to zeros, or to other initialization values) seed vector or classification token vector through an attention-based transformer deep learning network such as a vision transformer (ViT) or a set transformer such that the output vector corresponding to the input seed vector (or classification token vector) is obtained by attending to all other frames of the set of input embeddings.

[0044] Step 405 processes the attention-based output vector using one or more classification layers (e.g., an MLP) followed by a softmax function to obtain one or more classifications corresponding to the patient. If the relevant attention-based network is still undergoing supervised learning, then step 406 applies a loss function to the classifications and back propagates the computed loss to adjust learnable parameters of the classification layers and of the attention-based deep learning network. Processing then returns to step 401 to process the next data set.

[0045] FIGs. 5a-5d illustrate frame-wise attention for a dataset at various levels of granularity. FIG. 5a illustrates attention representations for a dataset of frames for 20 videos from a patient’s TTE exam. FIG. 5b illustrates attention representations for frames forselected videos, Video 3 and Video 14. FIG. 5c illustrates select frames from the selected videos. FIG. 5d illustrates DIN0v2 attention maps overlaid on the actual frames for framelevel interpretability.SELECTED RESULT EXAMPLES

[0046] Some embodiments implement methods and demonstrate results detailed in U.S. provisional application no. 63 / 599,994, fded on November 16, 2023 (previously incorporated by reference herein). In the examples described therein, two unique, private datasets were used. These datasets included subjects who were suspected of having PH and subsequently underwent both TTE imaging and the invasive RHC procedure to confirm a PH diagnosis. The first dataset, Sheffield (Hurdman et al. 2013), is a multi-center private collection consisting of 1024 subjects. On average, each individual in this dataset had 10 views, with some having as many as 20 views. The second dataset, CIPHER (Howard et al. 2020), is another multi-center private collection, which comprises 739 subjects. Here, each subject had an average of 10 views, with a maximum of 22 views for some. Further details regarding the data and the training splits can be found in below Table 1 :TABLE 1

[0047] Two distinctions between these datasets relative to others commonly used for PH risk detection are worth noting. First, right heart catheterization (RHC) -the invasive procedure that provides final PH diagnosis - is present, as opposed to proxies like medication usage or TTE-based diagnosis which are prone to both false positives and false negatives. Second, because all patients were suspect of having PH prior to the invasive procedure, even if a PH diagnosis is discarded, another heart condition is probably present. This last point — that the dataset consists of subjects with distinctive heart diseases rather than healthy controls - makes this real-world problem even more challenging to state-of-the- art implementations.

[0048] Data pre-processing: An echocardiography study typically consists of 10 videos containing multiple views of the heart. For both the CIPHER and Sheffield datasets, video pixel data found in the DICOM files were converted to RGB from either Y CRCB or gray colormaps and the region where the beam-formed cone was located was found in the “Sequence of Ultrasound Regions” DICOM tag. Finally, the frames were then saved into PNGs.

[0049] Implementation settings: All implementations were done in Pytorch and were trained on a server equipped with four NVIDIA A10 GPUs. Three distinct pre-training networks - a Vision Transformer model (ViT B / 16) [for DINO and DIN0v2] and ResNet-34 [for SimCLR] were used to extract features of varying lengths. For the examples in which a transformer is used for downstream supervised learning, the transformer-dropout was set to 0.25 and classification-dropout to 0.20. Batch size was set to 1 and the learning rate was set to le-6.

[0050] Evaluation Metrics: In diagnosing PH using TTE data, we compute AUC, Fl Score, and Accuracy in a 4-fold cross-validation setting. These respectively evaluate the model’s ability to distinguish between PH and non-PH cases, maintain precision recall balance, andcorrectly classify instances, with special consideration given to potential dataset imbalance. To binarize the continuous outputs of the model output, a threshold of 0.5 was used on the sigmoid outputs for each experiment.

[0051] View Classification: An off-the-shelf view classification algorithm (Zhang et al. 2018) was employed to identify, for each patient, the video with highest probability of being A4c- the de facto standard echocardiogram view, used by most traditional methods to acquire relevant anatomical and functional information for PH classification.Baseline Experiment: The goal of this experiment was to assess the efficacy of the systems and methods described herein at distinguishing patients with PH from non-PH patients. For benchmarking purposes, an off-the-shelf view-classification algorithm (Zhang et al. 2018) was used to single out one A4c view per patient, which was fed into a CNN for disease classification. When addressing the challenge posed by the varying lengths of videos, a WSL model that employs attention-based CNNs was used.

[0052] The performance metrics for the baseline using the CNN method revealed an AUC Score of 0.67 ± 0.05, an Fl Score of 0.56 ± 0.33, and an Accuracy of 0.52 ± 0.20. In contrast, the systems and methods described herein in the exemplary embodiment, integrating DIN0v2 and a custom transformer architecture, addresses the constraints of a variable number of videos and their lengths inherent in the baseline method. The performance enhancement is evident across all metrics with the systems and methods described herein in the exemplary embodiment recording an AUC Score of 0.80 ± 0.01, an Fl Score of 0.87 ± 0.01, and an Accuracy of 0.79 ± 0.02 and shown in Table 2:TABLE 2

[0053] The relatively low performance of traditional methods in this dataset can potentially be attributed to the difficulty in distinguishing different types of heart disease - a significantly more challenging problem than identifying PH disease from healthy controls. Analysis of Experiments: The preceding results indicated that the success of the exemplary embodiment could be attributed to one or more of the following architectural advancements: 1) the incorporation of multiple views, 2) the deployment of a potent pre-training encoder, and 3) the application of a spatio-temporal network. To evaluate these hypotheses, we methodically implemented the following architectural adjustments: 1) integration of multiple concatenated views within the WSL framework used as baseline, 2) utilization of diverse pre-training encoders, and 3) adoption of varied architectures for the downstream task. The outcomes of these experiments are detailed in Table 3 :TABLE 3

[0054] While SimCLR does not show significant improvement over traditional CNNs, DINO and DIN0v2 demonstrate large gains in performance across all three metrics. Comparing WSL with transformers suggests that extracting spatio-temporal features through the transformer model provides significant improvements for the PH classification task when SimCLR or DINO are used but less so when a stronger encoder is employed. These collective findings suggest that the coupling of DIN0v2 with a downstream application of the systems and methods described herein can considerably enhance the baseline performance of PH classification derived from TTE videos.

[0055] With SimCLR as the training mechanism, the systems and methods described herein in the exemplary embodiment achieves an AUC score of 0.77 ± 0.01, Fl score of 0.80 ± 0.01, and accuracy of 0.71 ± 0.02. Despite these metrics being lower than those achieved with the systems and methods described herein in the exemplary embodiment using DIN0v2, they still significantly surpass the baseline model. This comparison accentuates the efficacy of modem encoders for risk disease assessment from TTE.

[0056] Some examples of embodiments of the present disclosure have been described above in the context of TTE videos. However, embodiments of the disclosure are also applicable to other medical video contexts. For example, endoscopy videos can be processed using embodiments of the present disclosure including using SSL pre-trained encoders such as those described above to obtain frame level embeddings from the frames of endoscopy videos and then processing those embeddings using an attention-based deep learning network as described above, such as, for example, a transformer network, a multi-instance learning attention network, or other network. In one example, such networks can be trained and then used to classify each frame of an endoscopy video as belonging to a particular anatomic segment class, such as rectum (RM), left colon / sigmoid (LC, transvers colon (TC), right colon RC, and ileum (IL). In one example, such automated segmentation can be used to assist in automating determination of disease scores such as the Simple Endoscopic Score for Crohn’s Disease (SES-CD).

[0057] FIG. 6 shows an example of a computer system 6000, one or more of which may be used to implement one or more of the apparatuses, systems, and methods illustrated herein. Computer system 6000 executes instruction code contained in a computer program product 660. Computer program product 660 comprises executable code in an electronically readable medium that may instruct one or more computers such as computer system 6000 to perform processing that accomplishes the exemplary method steps performed.

[0058] The electronically readable medium may be any transitory or non-transitory medium that stores information electronically and may be accessed locally or remotely, for example via a network connection. The medium may include a plurality of geographically dispersed media each configured to store different parts of the executable code at different locations and / or at different times. The executable instruction code in an electronically readable medium directs the illustrated computer system 6000 to carry out various exemplary tasks described herein. The executable code for directing the carrying out of tasks described herein would be typically realized in software. However, it will be appreciated by those skilled in the art, that computers or other electronic devices might utilize code realized in hardware to perform many or all the identified tasks. Those skilled in the art will understand that many variations on executable code may be found that implement exemplary methods within the spirit and the scope of the disclosure.

[0059] The code or a copy of the code contained in computer program product 660 may reside in one or more storage persistent media (not separately shown) communicatively coupled to system 6000 for loading and storage in persistent storage device 670 and / or memory 610 for execution by processor 620. Computer system 600 also includes I / O subsystem 630 and peripheral devices 640. I / O subsystem 630, peripheral devices 640, processor 620, memory 610, and persistent storage device 670 are coupled via bus 650. Like persistent storage device 670 and any other persistent storage that might contain computer program product 660, memory 610 is a non-transitory media (even if implemented as a typical volatile computer memory device). Moreover, those skilled in the art will appreciate that in addition to storing computer program product 660 for carrying out processing described herein, memory 610 and / or persistent storage device 670 may be configured to store the various data elements referenced and illustrated herein.

[0060] Those skilled in the art will appreciate computer system 6000 illustrates just oneexample of a system in which a computer program product in accordance with the disclosure may be implemented. To cite but one example, execution of instructions contained in a computer program product may be distributed over multiple computers, such as, for example, over the computers of a distributed computing network.

[0061] Instructions for implementing an artificial neural network or other deep learning network may reside in computer program product 660. When processor 620 is executing the instructions of computer program product 660, the instructions, or a portion thereof, are typically loaded into working memory 610 from which the instructions are readily accessed by processor 620.

[0062] Processor 620 may comprise multiple processors which may comprise respective additional working memories (additional processors and memories not individually illustrated) including one or more graphics processing units (GPUs) comprising at least thousands of arithmetic logic units supporting parallel computations on a large scale. GPUs are often utilized in deep learning applications because they can perform the relevant processing tasks more efficiently than typical general-purpose processors (CPUs). Processor 620 may additionally or alternatively comprise one or more specialized processing units comprising systolic arrays and / or other hardware arrangements that support efficient parallel processing. Such specialized hardware may work in conjunction with a CPU and / or GPU to carry out the various processing described herein. Such specialized hardware may comprise application specific integrated circuits and the like (which may refer to a portion of an integrated circuit that is application-specific), field programmable gate arrays and the like, or combinations thereof. However, a processor such as processor 620 may be implemented as one or more general purpose processors (preferably having multiple cores) without necessarily departing from the spirit and scope of the present disclosure.

[0063] While the word inference or infer may be variously used herein, one having skill inthe art will understand that the systems and methods described herein are not so limited and that the term inference herein may indicate the performance of any of a variety of calculations and / or generation of a variety of outputs which may include, without limitation, any one or more of inferences, scores, estimates, predictions, projections, suggestions, recommendations, classifications, categorizations, annotations, conclusions, or the like or any combination of the foregoing.ADDITIONAL EXAMPLES

[0064] Example 1: A computer system comprising: one or more processors; a storage medium storing (a) instructions and (b) a dataset representing video data; an encoding module that uses the one or more processors to encode a set of frames from the dataset as highly-dimensional vector data, wherein (a) the set of frames represents data comprising a plurality of views of an area of interest, and (b) the highly-dimensional vector data contains information representing each of the plurality of views of the area of interest; one or more attention modules, comprising at least one of: (i) a first attention module that uses the one or more processors to generate, from the highly-dimensional vector data, a video-level attention map, (ii) a second attention module that uses the one or more processors to generate, from the highly-dimensional vector data, a frame-level attention map, (iii) a third attention module that uses the one or more processors to generate, from the highly-dimensional vector data, a pixel-level attention map; and an inference module that uses the one or more processors to output an inference based on one or more of: the video-level attention map, the frame-level attention map, and the pixel-level attention map.

[0065] Example 2: The computer system of example 1 wherein the video data comprises one or more of the following: transthoracic echocardiogram data, electrocardiogram data, and surgical video data.

[0066] Example 3: The computer system of example 1, wherein the video data comprisestransthoracic echocardiogram data.

[0067] Example 4. The computer system of example 1, wherein the highly-dimensional vector data comprises, for each frame of the set of frames, a respective vector representing the frame.

[0068] Example 5. The computer system of example 4, wherein the encoding module encodes, into each respective vector, a positional token.

[0069] Example 6. The computer system of example 5, wherein the one or more attention modules generate one or more attention maps based on the positional token of each respective vector.

[0070] Example 7. The computer system of example 1, wherein the set of frames comprises the entirety of the video data.

[0071] Example 8. The computer system of example 1, wherein the set of frames comprises video data from two or more distinct videos.

[0072] Example 9. The computer system of example 1, wherein the encoding module comprises a concatenation model, and wherein the concatenation model uses a positional token to locate each frame into the highly-dimensional vector data.

[0073] Example 10. The computer system of example 1, wherein the highly-dimensional vector data generated by the encoding module represents concatenated video data of the plurality of views of the area of interest.

[0074] Example 11. The computer system of example 1, wherein the first attention module, the second attention module, and the third attention module are implemented by one or more transformer architectures.

[0075] Example 12. The computer system of example 1, wherein the second attention module is implemented by a transformer architecture.

[0076] Example 13. The computer system of example 1, wherein the encoding module comprises one or more of the first attention module, the second attention module, or the third attention module.

[0077] Example 14. The computer system of example 1, further comprising a second encoding module, wherein the second encoding module comprises one or more of the first attention module, the second attention module, or the third attention module.

[0078] Example 15. The computer system of example 1, wherein one or more of the first attention module, the second attention module, or the third attention module is implemented by a self-supervised architecture.

[0079] Example 16. The computer system of example 1, wherein the inference is a disease severity estimation.

[0080] Example 17. The computer system of example 1, wherein the inference is an assessment of patient health.

[0081] Example 18. The computer system of example 1, wherein the inference is a measurement of disease progression.

[0082] Example 19. The computer system of example 1, wherein the inference is a medical diagnostic determination.

[0083] Example 20. The computer system of example 1, wherein the area of interest is the human heart and the plurality of views of the areas of interest comprise two or more views of the human heart.

[0084] Example 21. The computer system of example 1 , wherein the area of interest is one or more anatomical segments of the human body.

[0085] Example 22. The computer system of example 1, wherein each view of the plurality of views represents a distinct positional view of the area of interest.

[0086] Example 23. A computer system comprising: one or more processors; a storage medium storing (a) instructions and (b) a dataset; an encoding module that uses the one or more processors to encode the dataset as a highly-dimensional vector data, such that vector data representing one of the following types of data is concatenated with vector data representing another of the following types of data: transthoracic echocardiogram data, electrocardiogram data, surgical video data, video data, text data, and audio data; one or more attention modules, comprising at least one of: (i) a first attention module that uses the one or more processors to generate, from the highly-dimensional vector data, a video-level attention map, (ii) a second attention module that uses the one or more processors to generate, from the highly-dimensional vector data, a frame-level attention map, (iii) a third attention module that uses the one or more processors to generate, from the highly- dimensional vector data, a pixel-level attention map; and an inference module that uses the one or more processors to output an inference based on one or more of: the video-level attention map, the frame-level attention map, and the pixel-level attention map.

[0087] Example 24. The computer system of example 23, wherein the dataset comprises two or more of the following types of data: transthoracic echocardiogram data, electrocardiogram data, surgical video data, video data, medical image data, text data, and audio data.

[0088] Example 25. A computer system comprising: one or more processors; a storage medium storing (a) instructions and (b) a dataset representing video data; an encoding module that uses the one or more processors to encode each frame of the dataset as vector data; one or more attention modules, comprising at least one of: (i) a first attention module that uses the one or more processors to generate, from the vector data, a video-level attention map, (ii) a second attention module that uses the one or more processors to generate, from the vector data, a frame-level attention map, (iii) a third attention module that uses the one or more processors to generate, from the vector data, a pixel-level attention map; and aninference module that uses the one or more processors to output an inference based on one or more of: the video-level attention map, the frame-level attention map, and the pixel-level attention map.

[0089] Example 26. A computer system comprising: one or more processors; a storage medium storing (a) instructions and (b) a dataset; an encoding module that uses the one or more processors to encode the dataset as highly-dimensional vector data; a first attention module that uses the one or more processors to generate, from the highly-dimensional vector data, a token-level attention map; and an inference module that uses the one or more processors to output an inference based on the token-level attention map.

[0090] Example 27. The computer system of example 26, wherein the dataset comprises one or more of the following types of data: transthoracic echocardiogram data, electrocardiogram data, surgical video data, video data, medical image data, text data, and audio data.

[0091] Example 28. The computer system of example 26, wherein the dataset comprises echocardiogram data or electrocardiogram data, and wherein, in the token-level attention map, a token is equal to a period of time.

[0092] Example 29. The computer system of example 26, wherein, in the token-level attention map, a token is equal to a period of time.

[0093] Example 30. The computer system of any of examples 28 or 29, wherein the token is equal to one second.

[0094] Example 31. A method of, via a deep learning pipeline, generating a deep learning network configured to execute on one or more computers to make an inference using video data, the method comprising: applying a first machine learning model to the video data to encode a set of frames from the dataset as highly-dimensional vector data, wherein (a) theset of frames represents data comprising a plurality of views of an area of interest, and (b) the highly-dimensional vector data contains information representing each of the plurality of views of the area of interest; applying one or more second machine learning models to the highly-dimensional vector data to generate one or more of: a video-level attention map, a frame-level attention map, and a pixel-level attention map; and outputting an inference based on one or more of: the video-level attention map, the frame-level attention map, and the pixel-level attention map.

[0095] Example 32. A method for spatiotemporal inferences using video data, the method comprising: applying a first machine learning model to the video data to encode a set of frames from the dataset as highly-dimensional vector data; applying one or more second machine learning models to the highly-dimensional vector data to generate one or more of: a video-level attention map, a frame-level attention map, and a pixel-level attention map; and outputting a spatiotemporal inference based on one or more of: the video-level attention map, the frame-level attention map, and the pixel-level attention map.

[0096] Example 33. The method of example 32, wherein the one or more second machine learning models are implemented in a transformer architecture.

[0097] Example 34. The method of example 32, wherein the one or more second machine learning models are implemented in a temporal architecture.

[0098] Example 35. The method of example 32, wherein the video data comprises a plurality of large videos.

[0099] Example 36. The method of example 32, wherein the video data comprises one or more long form videos.

[0100] While the present disclosure has been particularly described with respect to theillustrated embodiments, it will be appreciated that various alterations, modifications, and adaptations may be made based on the disclosure and are intended to be within the scope of the disclosure. While the disclosure has been described in connection with what are presently considered to be the most practical and preferred embodiments, it is to be understood that the present disclosure is not limited to the disclosed embodiments but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the underlying principles of the invention as described by the various embodiments referenced above and below.

Claims

CLAIMS1. A method of using one or more computers to analyze one or more medical videos corresponding to one or more patient exams, the method comprising: pre-processing video data to obtain a plurality of video frames corresponding to the one or more medical videos corresponding to a patient exam of the one or more patient exams; encoding the plurality of video frames using a pretrained encoder to obtain a plurality of frame embeddings corresponding to respective frames of the plurality of video frames, wherein the pre-trained video encoder has been pre-trained using self-supervised learning; concatenating the plurality of frame embeddings; processing the plurality of frame embeddings in an attention-based deep learning network to obtain a summary vector resulting from attention processing that attends to each of the plurality of frame embeddings; and submitting the summary vector to a classification network to obtain one or more computed classifications corresponding to the one or more medical videos from the patient exam.

2. The method of claim 1 wherein the one or more medical videos comprise a plurality of videos such that the plurality of frame embeddings that are concatenated for further processing include frame embeddings from each of the plurality of videos.

3. The method of any of claims 1-2 wherein the patient exam is a transthoracic echocardiogram.

4. The method of any of claims 1-3 wherein the pretrained encoder comprises a vision transformer (ViT) encoder.

5. The method of any of claims 1-3 wherein the pretrained encoder comprises a convolutional neural network.

6. The method of any of claims 1-5 wherein the pretrained encoder is trained using SimCLR.

7. The method of any of claims 1-5 wherein the pretrained encoder is trained using DINO.

8. The method of any of claims 1-5 wherein the pretrained encoder is trained using DIN0v2.

9. The method of any of claims 1-8 wherein the attention-based deep learning network comprises a transformer multi-head attention encoder.

10. The method of claim 9 wherein the transformer multi -head encoder comprises a set transformer encoder.

11. The method of any of claims 1-8 wherein the attention-based deep learning network comprises a convolutional neural network encoder.

12. The method of any of claims 1-11 wherein the attention-based deep learning network comprises an attention pooling layer.

13. The method of claim 12 wherein the attention pooling layer comprises an attention block and an aggregator.

14. The method of claim 13 wherein the attention block associates respective learnable parameters with respective frame embedding outputs of an encoder of the attention-based deep learning network.

15. The method of claim 14 wherein the aggregator applies the respective learnable parameters to the respective frame embedding outputs and sums the resulting attention- scored frame embedding output to obtain the summary vector.

16. The method of any of claims 1-15 wherein the one or more computed classifications are regarding pulmonary hypertension.

17. A method of using one or more computers to analyze one or more medical videos corresponding to one or more patient exams, the method comprising: pre-processing video data to obtain a plurality of video frames corresponding to the one or more medical videos corresponding to a patient exam of the one or more patient exams; encoding the plurality of video frames using a pretrained encoder to obtain a first plurality of frame embeddings corresponding to respective frames of the plurality of video frames, wherein the pre-trained video encoder has been pre-trained using self-supervised learning; and processing the first plurality of embeddings in an attention-based deep learning network trained using supervised learning to obtain a second plurality of frame embeddings corresponding to respective ones of the first plurality of frame embeddings and to use a frame embedding of the second plurality of frame embeddings to make a computed classification regarding patient tissue corresponding to the frame embedding.

18. The method of claim 17 wherein the patient exam is an endoscopy exam.

19. The method of claim 18 wherein the computed classification is regarding which bowel segment of a plurality of bowel segments corresponds to the frame embedding.

20. A computer program product stored in a non-transitory tangible medium and comprising instructions executable on one or more processors of one or more computers to implement processing to analyze one or more medical videos corresponding to a patient exam, the processing comprising executing the method of any of claims 1-19.

21. A computer-executable deep learning network stored in a non-transitory computerreadable medium and configured to execute on one or more processor of one or more computers to process one or more medical videos corresponding to a patient exam, the computer-executable deep learning network comprising: a pretrained encoder configured to obtain a plurality of frame embeddings corresponding to respective frames of the one or more medical videos, wherein the pretrained video encoder has been pre-trained using self-supervised learning; an attention-based deep learning network configured to process the plurality of frame embeddings in an attention-based deep learning network to obtain a summary vector resulting from attention processing that attends to each of the plurality of frame embeddings; and a classifier network configured to process the summary vector to obtain one or more computed classifications corresponding to the one or more medical videos from the patient exam.

22. The computer-executable deep learning network of claim 21 wherein the one or more medical videos comprise a plurality of videos and the plurality of frame embeddings are concatenated prior to being processed by the attention-based deep learning network such that the plurality of frame embeddings are processed together by the attention-based deep learning network and include frame embeddings from each of the plurality of videos.

23. The computer-executable deep learning network of any of claims 21-22 wherein the patient exam is a transthoracic echocardiogram.

24. The computer-executable deep learning network of any of claims 21-23 wherein the pretrained encoder comprises a vision transformer (ViT) encoder.

25. The computer-executable deep learning network of any of claims 21-23 wherein the pretrained encoder comprises a convolutional neural network.

26. The computer-executable deep learning network of any of claims 21-25 wherein the pretrained encoder is trained using SimCLR.

27. The computer-executable deep learning network of any of claims 21-25 wherein the pretrained encoder is trained using DINO.

28. The computer-executable deep learning network of any of claims 21-25 wherein the pretrained encoder is trained using DIN0v2.

29. The computer-executable deep learning network of any of claims 21-28 wherein the attention-based deep learning network comprises a transformer multi-head attention encoder.

30. The computer-executable deep learning network of claim 29 wherein the transformer multi-head attention encoder is a set transformer encoder.

31. The computer-executable deep learning network of ay of claims 21-28 wherein the attention-based deep learning network comprises a convolutional neural network encoder.

32. The computer-executable deep learning network of any of claims 21-31 wherein the attention-based deep learning network comprises an attention pooling layer.

33. The computer-executable deep learning network of claim 32 wherein the attention pooling layer comprises an attention block and an aggregator.

34. The computer-executable deep learning network of claim 31 wherein the attention block associates respective learnable parameters with respective frame embedding outputs of an encoder of the attention-based deep learning network.

35. The computer-executable deep learning network of claim 34 wherein the aggregator applies the respective learnable parameters to the respective frame embedding outputs and sums the resulting attention-scored frame embedding output to obtain the summary vector.

36. The computer-executable deep learning network of any of claims 21-35 wherein the one or more computed classifications are regarding pulmonary hypertension.

Images