Attention learning from videos of patients' areas of interest

Abstract

In some embodiments of this disclosure, video frame data across one or more videos is processed by a pre-trained encoder, and the resulting frame-based embeddings are concatenated for analysis by a downstream attention-based deep learning network. Different types of self-supervised learning (SSL) pre-training techniques and different types of attention-based deep learning networks are utilized in various embodiments. In a particular embodiment, embodiments of this disclosure are applied to derive inferences about pulmonary hypertension based on video data from a patient's transthoracic echocardiography. In another embodiment, embodiments of this disclosure are applied to derive inferences about anatomical intestinal segments in endoscopic video. Details of various embodiments are further described herein.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Application No. 63 / 533,101, filed on August 16, 2023; U.S. Provisional Application No. 63 / 599,994, filed on November 16, 2023; and U.S. Provisional Application No. 63 / 555,883, filed on February 20, 2024. The contents of these applications are hereby incorporated by reference into this specification.

Background Art

[0002] The present disclosure generally relates to computerized techniques in a medical space for making inferences from one or more videos of a patient's region of interest.

Summary of the Invention

[0003] In some medical image analysis contexts, the "best" view is typically selected for algorithm-based inference from patient data. For example, in the context of transthoracic echocardiography (TTE), many TTE videos are generated during a patient's examination, but only a subset (or even just one) of those videos is selected (e.g., via computerized analysis) for more detailed analysis by a computerized system. However, video data from unselected views / videos may contain useful information to inform the computerized analysis. At the same time, video data is, by its nature, very large, and computerized analysis of video data can be resource-intensive. Attention-based deep learning approaches also require considerable processing resources, and the level of resource consumption depends on the level of granularity to which the attention mechanism is applied. Therefore, efficiently implementing attention-based learning in the context of analyzing medical video data benefits from a thoughtful approach to the level of data aggregation used for attention-based analysis. Furthermore, an approach is needed that can leverage information available across multiple videos, regardless of whether a particular video is determined to be the "best" view.

[0004] In some embodiments of this disclosure, video frame data across one or more videos is processed by a pre-trained encoder, and the resulting frame-based embeddings are concatenated for analysis by a downstream attention-based deep learning network. Different types of self-supervised learning (SSL) pre-training techniques and different types of attention-based deep learning networks are utilized in various embodiments. In one embodiment, embodiments of this disclosure are applied to derive inferences about pulmonary hypertension (PH) based on video data from a patient's transthoracic echocardiography. In another embodiment, embodiments of this disclosure are applied to identify anatomical intestinal segments corresponding to frames in an endoscopic video. Details of various embodiments are further described herein. [Brief explanation of the drawing]

[0005] [Figure 1] This disclosure shows a medical video processing system according to one embodiment. [Figure 2] Another embodiment of the medical video processing system according to this disclosure is shown. [Figure 3] This disclosure describes a self-supervised learning (SSL) pre-training method according to one embodiment of this disclosure. [Figure 4] This disclosure describes a method for processing medical videos corresponding to patient examinations using a computerized deep learning network and for performing patient-related inference based on the medical videos. [Figure 5] a-d represent frame-by-frame attentional representations obtained from medical examination video datasets at various levels of granularity. [Figure 6] Examples of computer systems are provided, one or more of which may be used to implement one or more of the devices, systems, and methods illustrated herein.

[0006] Embodiments of this disclosure are described with reference to the drawings above, which are intended to be illustrative. Other embodiments are consistent with the spirit and scope of this disclosure. [Modes for carrying out the invention]

[0007] Various embodiments are described more fully below with reference to the accompanying drawings, which form part of this specification and illustrate specific examples of implementations of the embodiments. However, this specification can be embodied in a wide variety of forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided so as to make this specification thorough and complete and to fully convey the scope of this disclosure to those skilled in the art. In particular, this specification can be embodied as a method or a device. Accordingly, any of the various embodiments described herein may take the form of all hardware embodiments, all software embodiments, or embodiments combining software and hardware aspects. Accordingly, the following description should not be construed as restrictive.

[0008] Figure 1 shows a medical video processing system 1000 according to one embodiment of the present disclosure. This embodiment and other embodiments are described with reference to video from a patient's transthoracic echocardiography (TTE) examination. However, the underlying principles of the present disclosure are applicable to other medical image / video analysis applications.

[0009] This embodiment illustrates the processing of multiple videos, including a first video 11 having m1 (several) frames, a second video 12 having m2 frames, and an nth video 13 having mv frames. Each video corresponds to a different view from a medical procedure, such as a TTE examination. For example, video 11 corresponds to the first view (#1), video 12 corresponds to the second view (#2), and the nth video 13 corresponds to the nth view. In certain embodiments, several additional videos (not shown) are processed to correspond to views between the second view #2 and the nth view #n. In a typical TTE examination, for example, about 10 to 20 different TTE videos are taken, each thought to correspond to a different potentially useful view.

[0010] The preprocessing block 110 preprocesses the video data corresponding to the patient's TTE examination. Specifically, block 110 processes the video data so that each frame has a uniform dimension D1. In this embodiment, each frame of each video has a dimension D1 of 224 × 224 × 3 after preprocessing (when using RGB color channels). In different embodiments, other dimensions can be used. The preprocessing block 110 prepares matrices containing data for each video, for example, matrices 21, 22, and 23. The dimension of each matrix is ​​224 × 224 × 3 (D1) × several frames in the video (e.g., m1, m2, mv).

[0011] The video data matrix is ​​processed by encoder 120. In the illustrated embodiment, encoder 120 is pre-trained using self-supervised learning (SSL) techniques. In a typical implementation, pre-training is performed by dividing the frames into patches, and possibly using extended versions of the patches, a self-supervised learning task is performed using the encoder to train the encoder and extract useful features from the frame pixel data for downstream learning tasks. In some embodiments, the "SimCLR" approach is used, as described by Chen et al. in "A Simple Framework for Contrastive Learning of Visual Representations" (2020). In other embodiments, the "DINOv1" approach (described by Caron et al. in "Emerging Properties in Self-Supervised Vision Transformers" (2021)) or the "DINOv2" approach (described by Oquab et al. in "DINOv2: Learning Robust Visual Features without Supervision" (2023)) is used. All three of these papers are incorporated herein by reference in their entirety. In other embodiments, any combination (or any part thereof) of the aforementioned approaches may be used.

[0012] In the SimCLR approach, the same image or image patch is augmented in a different way to generate two different augmented images (or image patches) corresponding to the same image (or image patch) (referred to as "positive pairs" in the SimCLR paper). The trained network attempts to minimize the result of the symmetric loss function value between the encodings of each augmentation in the pair, and the encoding is performed by the same network (with the same initial parameters and the same training tuning parameters). In SimCLR, the augmented patch pairs are processed through the same encoding network (e.g., ResNet), and the network's parameters are then tuned to minimize the symmetric loss.

[0013] In the DINO approach, different versions of an augmented image (and / or augmented patch) pass through different networks (teacher and student networks) that have the same architecture (e.g., Vision Transformer ("ViT")) but different starting parameters. During pre-training, the student network's parameters are tuned to minimize the difference between its K-dimensional output embedding and the teacher's output embedding. In certain embodiments, the teacher's parameters are updated based on an exponential moving average of the student's parameters as the student learns. After pre-training, the teacher network is used as a pre-trained encoder (e.g., encoder 120).

[0014] Referring to the illustrative Figure 1, the encoder 120 (which may be, for example, an SSL pre-trained encoder) generates an embedding vector having a D2 dimension collected from the data of each frame, with one embedding matrix for each video. In one implementation, the ViT encoder and the DINOv2 framework are used, where D2 is 768. In the illustrated embodiment, the encoder 120 generates matrices for each video, such as matrices 31, 32, and 33. The dimensions of each matrix correspond to the vector embedding dimension D2 and the number of frames for a given video. As shown in the figure, matrix 31 has a dimension D2 × m1, where m1 is the number of frames processed for the first video 11; matrix 32 has a dimension D2 × m2, where m2 is the number of frames processed for the second video 12; and matrix 33 has a dimension D2 × mv, where mv is the number of frames for the nth video 13.

[0015] Block 130 unpacks the frame-level embedding vectors from the embedding matrix output by encoder 120 and concatenates the frame embedding vectors, each having a D2 dimension. In this context, concatenation simply means that the frame embeddings across all videos of a given patient's TTE are linked together so that they can be processed together by the downstream part of the network (regardless of whether the frames are from the same video), and those skilled in the art will understand that other terms may be used instead of concatenation in the same general sense.

[0016] The position coding block 140 calculates a position coding value based on the position information corresponding to the frame. In the original transformer model (described by Vaswani et al. in "Attention is All You Need," 2017, and incorporated herein by reference), the value of this coding is based on the position of the frame, the output dimension of the embedding, and from 0 to

[0017]

number

[0018]

number

[0019] In one embodiment, for each video, the frame "pos" value starts at zero and is incremented for each frame within the sequence of frames of the video. In embodiments where multiple videos corresponding to the same patient are processed, information identifying which video a particular frame came from is tracked, but that "which video" information is not included in the pos value used for the position encoding described above. Rather, the frame pos value restarts from zero for each video. In this way, each video is encoded separately, even though the frames are concatenated.

[0020] Of course, the above is merely an example, and any one or more of many other position encoding methods can be used. For clarity, the position encoding function referred to above is "fixed" in that it does not include any learnable parameters (i.e., parameters updated based on training data and backpropagation of error). However, other embodiments consistent with the present disclosure can additionally or alternatively include a position encoding function with learnable parameters (or a fixed position encoding function different from that referred to above).

[0021] Addition operation 150 performs a vector addition to add the calculated position-encoded vector to the frame embedding vector (both of the same length), and passes (or outputs in other ways) the resulting input vector 51 as an input to the multi-head attention encoder 160. The position-encoded values are added to all frame-level embeddings for the patient's medical video, and the resulting set of input vectors 51 is processed together by the encoder 160.

[0022] In one embodiment, the multi-head attention encoder 160 is implemented as a set transformer encoder as described by Lee et al. in "Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks", 2019 (incorporated herein by reference). The multi-head attention block (MAB) for matrices X and Y representing two d-dimensional vectors integrates layer normalization with parameters ω and multi-head attention as follows. MAB(X,Y)=LayerNorm(H+rFF(H)), where H=LayerNorm(X+Multihead(X,Y,Y);ω)), and rFF is an arbitrary row-wise feed-forward layer. The Set Attention Block (SAB) in Lee is defined as SAB(X):=MAB(X,X). (See 3.1 of Lee). In a particular example applied to patient video analysis, "X" is a matrix containing all frame embeddings from all videos, and each row is a frame embedding vector for a particular frame. Considering its multi-head attention design where Q=K=V=X, the SAB treats the frames uniformly for permutation invariance.

[0023] Those skilled in the art will understand that the set transformer described in Lee is a variation of the original transformer described in Vaswani (see above). Another variation is the vision transformer (ViT) described in Dosovitskly et al., “An Image is Worth 16X16 Words” (2021), which is incorporated herein by reference. Transformer encoders, such as those shown in Vaswani, can be combined with processing appropriate to the context of image processing (rather than using Vaswani’s transformer decoder, which is most applicable to language or other sequential data), as shown in Dosovitskly. In some embodiments, a ViT encoder similar to that taught in Dosovitskly is used. Those skilled in the art will understand that, in addition to the multi-head attention encoders taught in Lee and Dosovitskly (both of which are variations of the encoder in Vaswani), other variations of the multi-head attention encoder may be used additionally or alternatively to implement the multi-head attention encoder 160 in Lee. The referenced embodiments provide only a few examples.

[0024] In the exemplary embodiment shown in Figure 1, the multi-head attention encoder 160 outputs attention-processed vectors 61, each attention-processed vector 61 representing an output processed through the encoder 160, where the encoder 160 directs attention to all other input vectors 51 in order to generate a specific output vector 61. In the illustrated embodiment, each output vector 61 corresponds to a specific video frame processed by the system 1000.

[0025] The output vector 61 is processed by an attention pooling layer including an attention weighting block 192 and an aggregator 191. In the illustrated embodiment, the attention weighting block 192 assigns attention values ​​(which are learnable parameters) to each frame, and the aggregator block 191 then multiplies each vector 61 corresponding to that frame by the corresponding attention value received from the attention block 250, and sums the resulting attention weighting vectors for all frames in all videos of the patient's TTE examination (or other examination) to obtain a single, summarized vector 71 per patient of size D2 × 1.

[0026] The multilayer perceptron (MLP) 170-2 processes vector 71 and outputs values ​​corresponding to various desired output classifications (e.g., "PH", "Not PH", "Uncertain"). The softmax layer 170-1 converts the output of MLP 170-2 into class probability values ​​to provide PH inferences for each patient. "Inference" can be a prediction, estimation, score, suggestion, categorization, or other inference in some embodiments. During supervised training, the learning block 180 calculates a loss value (sometimes called "error") and then backpropagates that error to adjust the learnable parameters in MLP 170-2, attention weighting block 192, and multi-head attention encoder 160.

[0027] In some embodiments, the attention score for each frame is obtained from the attention weighting block 192 and provided to the video review interface module 175, so that the frames most important for inference can be reviewed via a computer user interface (the interface is not shown separately). The attention “score” in this context is simply equal to or can be derived from the weights applied by block 192. This attention score reflects how heavily a particular frame was weighted in the attention pooling operation performed by blocks 192 and 191, and therefore how much the representation 61 of that particular frame contributed to the summarized representation vector 71. In one embodiment, the approach to attention pooling disclosed in Ilse et al., “Attention-based Deep Multiple-Instance Learning,” 2018 (incorporated herein by reference) is used.

[0028] In alternative embodiments, the aggregation of input vectors for downstream classification may be performed using additional input vectors called classification ("CLS") tokens, which may be processed together with input vectors representing images or portions of images. In such embodiments, the CLS tokens have the same dimensions as the input image representation vectors but are initialized to random or predetermined starting values ​​independent of the values ​​of the image representation vectors. Since the transformer attention processing performed by the encoder on a given input vector follows the values ​​of all other input vectors, such CLS tokens may, after processing, be used as an aggregated representation of all other processed vectors. In such embodiments, the resulting processed CLS tokens output by the encoder may be used by the downstream classification portion of the network (e.g., MPL107-2 in the context of Figure 1). The illustrated embodiment does not use CLS tokens to obtain an aggregated (i.e., summarized) output for classification processing, but alternatives to the illustrated embodiment may be implemented using such tokens for attention pooling without departing from the spirit and scope of the disclosure.

[0029] One aspect of using the approach shown in Figure 1 is that a specific score for each frame is readily available. Thus, in addition to the system providing a calculated classification (e.g., inference or estimation), the system can also provide at least some transparent, user-understandable information (e.g., output and / or display) about how the calculated classification was arrived at. In one embodiment, in the context of multi-frame medical video, the system can identify and / or output information, such as annotations, overlays, reports, notes, etc., that includes which frames were most important (or otherwise important, or ranked in importance) for the analysis that resulted in the calculated classification. Such identification can be useful for professional medical reviews after TTE video or other medical video. For example, a user of the video review interface 175 can interactively access the frames that the system 1000 determined to be most important (or otherwise important) for the classification decision, and the user can spend additional time reviewing those frames.

[0030] Figure 2 shows a medical video processing system 2000 having several alternative embodiments to the embodiment shown in Figure 1. System components having the same reference numbers as those shown in Figure 1 are not necessarily described separately in the context of Figure 2 and are of similar nature and scope (including potential variations) to the elements of the same number in Figure 1. The embodiment in Figure 2 utilizes multi-instance attention with a convolutional neural network (CNN) encoder, along with weakly supervised learning. As shown in this embodiment, frame embeddings 41 corresponding to frames across all video of a patient's TTE examination (or other medical examination) are processed by a convolutional neural network (CNN) 240 (which may be ResNet in some embodiments). Each frame embedding 41 is fed into the CNN 240, and the resulting encoded vectors 53 each have the same dimensions as the corresponding input frame embedding vector 41 in one embodiment.

[0031] In the illustrated embodiment, attention block 250 assigns attention values ​​(which are learnable parameters) to each frame, and then aggregater block 260 multiplies each vector 53 corresponding to that frame by the corresponding attention value received from attention block 250, and sums the resulting attention-weighted vectors for all frames in all videos of the patient's TTE examination (or other examination) to obtain a single summarized vector 72 per patient of size D2 × 1.

[0032] In a similar manner to that described in the context of Figure 1, MLP280-2 processes vector 72 and outputs several values ​​corresponding to several desired output classifications (e.g., "PH", "Not PH", "Uncertain"), and the softmax layer 280-1 converts the output of MLP280-2 into class probability values ​​to provide PH inference for each patient. During weakly supervised training, WSL block 290 calculates a loss value ("error") and then backpropagates that error to adjust the learnable parameters in MLP280-2, attention block 250, and CNN240.

[0033] In some embodiments, the attention score for each frame is obtained from block 250 and provided to the video review interface module 275, so that the frames most important for inference can be reviewed via a computer user interface (details of the interface are not shown separately).

[0034] Figure 3 shows a self-supervised learning (SSL) pre-training method 3000 according to one embodiment of the present disclosure. Step 310 obtains a dataset representing video data corresponding to multiple views of a region of interest, with the video data from each video containing multiple frames. As will be understood by those skilled in the art, such datasets do not need to be labeled. Also, if they are labeled, the labels do not need to be used for SSL pre-training. Step 320 divides each frame into multiple tiles, sometimes called “patches.” Step 330 performs one or more augmentation operations on each patch. In the context of machine learning for image processing, “augmentation” refers to one or more modifications to an image, such as randomly blurring, cropping, or converting to grayscale.

[0035] In some SSL processes, an image (or image patch) is augmented, and the SSL encoder network attempts to extract feature representations that minimize the difference between representations of different augments of the same image or image patch. In some techniques, such as SimCLR, the same network handles positive pairs of augmented images (or image patches), i.e., two different augments of the same image or image patch. In another set of techniques, such as DINO and DINOv2 discussed above, different augments are handled by two different encoder networks (one being the "teacher" and the other the "student," initialized with different parameters), and training attempts to make the representation output by one encoder approximate the representation output by the other encoder.

[0036] Step 340 encodes each augmented image (or image patch) using one or more encoders, matching one of various SSL techniques. Step 350 calculates the loss (using the loss function) based on the output of one or more encoders, according to a defined SSL technique that may specify an SSL learning task and a corresponding loss function. Step 350 adjusts the learnable parameters of one or more encoders, taking into account the backpropagation of the calculated loss.

[0037] Figure 4 shows a method 4000 according to one embodiment of the present disclosure, which processes medical video corresponding to a patient examination (e.g., TTE examination) using a computerized deep learning network and performs patient-related inference based on the medical video.

[0038] Step 401 preprocesses a dataset (or subsequent dataset) obtained from multiple videos corresponding to multiple views of the patient's region of interest. In this embodiment, preprocessing includes parsing the video data of each video into multiple frames. In one embodiment, the frames are preprocessed to make each frame a uniform pixel size. In one embodiment, for RGB color, each frame is preprocessed to have a size of 224 × 224 × 3. Other dimensions are consistent with the present disclosure, and this is merely one embodiment.

[0039] Step 402 uses an SSL pre-trained encoder to embed each frame, obtaining multiple embedding vectors, each representing a feature of the video data for that frame. Step 403 concatenates the multiple frame embedding vectors (in one embodiment, across multiple videos corresponding to patient medical examinations such as TTE examinations) for processing together by an attention-based deep learning network.

[0040] Step 403 obtains an attention-based output vector from processing multiple input frame embeddings across multiple videos. In one embodiment, the output vector may be obtained by multi-instance attention learning / attention pooling, for example, where learned weights are applied to multiple individual output vectors corresponding to individual frames, which are then aggregated (e.g., multiplied by weights and then added) to obtain an attention-based output vector. In an alternative embodiment, the attention-based output vector is the result of processing a seed vector or classification token vector initialized (e.g., to random values, zero, or other initial values) via an attention-based transformer deep learning network such as a vision transformer (ViT) or set transformer, such that an output vector corresponding to the input seed vector (or classification token vector) is obtained by paying attention to all other frames in the set of input embeddings.

[0041] Step 405 processes the attention-based output vector using one or more classification layers (e.g., MLPs), and then uses a softmax function to obtain one or more classifications corresponding to patients. If the associated attention-based network is still undergoing supervised learning, Step 406 applies a loss function to the classification and backpropagates the calculated loss to adjust the learnable parameters of the classification layer and the attention-based deep learning network. The process then returns to Step 401 to process the next dataset.

[0042] Figures 5a–5d show frame-level attention to the dataset at various granularity levels. Figure 5a shows the attentional representation of the dataset of frames from 20 videos from a patient's TTE examination. Figure 5b shows the attentional representation of frames from selected video, video 3, and video 14. Figure 5c shows selected frames from the selected video. Figure 5d shows the DINOv2 attention map overlaid on the actual frames for frame-level interpretability.

[0043] Examples of selected results Several embodiments demonstrate the methods and results described in U.S. Provisional Application No. 63 / 599,994, filed November 16, 2023 (incorporated herein by prior reference). The embodiments described therein used two unique private datasets. These datasets included subjects suspected of having PH and subsequently undergoing both TTE imaging and invasive RHC procedures to confirm a PH diagnosis. The first dataset, Sheffield (Hurdman et al., 2013), is a multicenter private collection of 1024 subjects. On average, each individual in this dataset had 10 views, with some having as many as 20 views. The second dataset, CIPHER (Howard et al., 2020), is another multicenter private collection containing 739 subjects. Here, each subject had an average of 10 views, with some having up to 22 views. Further details regarding the data and training splits can be found in Table 1 below.

[0044] [Table 1]

[0045] Two distinctions between these datasets and other datasets commonly used for PH risk detection are noteworthy. First, there is right heart catheterization (RHC), an invasive procedure that provides a definitive PH diagnosis, in contrast to proxies such as drug use or TTE-based diagnoses, which are prone to both false positives and false negatives. Second, since all patients were suspected of having PH before the invasive procedure, even if the PH diagnosis was discarded, another cardiac condition likely exists. This last point, namely that the dataset consists of subjects with specific cardiac conditions rather than healthy controls, further complicates this real-world problem for state-of-the-art implementations.

[0046] Data preprocessing: Echocardiography typically consists of 10 videos containing multiple views of the heart. For both the CIPHER and Sheffield datasets, video pixel data found in the DICOM files was converted from either YCRCB or grayscale colormap to RGB, and the regions where beamforming cones were located were identified within the "Sequence of Ultrasound Regions" DICOM tag. Finally, the frames were saved as PNGs.

[0047] Implementation Settings: All implementations were done in PyTorch and trained on a server with four NVIDIA A10 GPUs. Three different pre-trained networks, namely the vision transformer model (ViT B / 16) [for DINO and DINOv2] and ResNet-34 [for SimCLR], were used to extract features of varying lengths. In the examples where the transformer was used for downstream supervised learning, the transformer dropout was set to 0.25 and the classification dropout to 0.20. The batch size was set to 1 and the learning rate to 1e-6.

[0048] Evaluation metrics: When diagnosing PH using TTE data, the inventors calculate AUC, F1 score, and accuracy in a 4-fold cross-validation setup. These evaluate the model's ability to distinguish between PH and non-PH cases, maintain accuracy recall balance, and correctly classify instances, with special consideration given to potential dataset imbalances. A threshold of 0.5 was used for the sigmoid output for each experiment to binarize the continuous output of the model output.

[0049] View Classification: A pre-existing view classification algorithm (Zhang et al., 2018) was used to identify, for each patient, the video with the highest probability of being A4c (the de facto standard echocardiographic view, used by the most traditional methods to obtain relevant anatomical and functional information for PH classification).

[0050] Baseline Experiment: The objective of this experiment was to evaluate the effectiveness of the system and method described herein in distinguishing PH patients from non-PH patients. For benchmarking purposes, a ready-made view classification algorithm (Zhang et al., 2018) was used to select one A4c view per patient, which was then fed into a CNN for disease classification. A WSL model employing an attention-based CNN was used to address the challenges posed by variations in video length.

[0051] Baseline performance metrics using the CNN method revealed an AUC score of 0.67±0.05, an F1 score of 0.56±0.33, and an accuracy of 0.52±0.20. In contrast, the system and method described herein in exemplary embodiments, integrating DINOv2 with a custom transformer architecture, addresses the variable video count and length constraints inherent in the baseline method. Performance improvements are evident across all metrics using the system and method described herein in exemplary embodiments, recording an AUC score of 0.80±0.01, an F1 score of 0.87±0.01, and an accuracy of 0.79±0.02, as shown in Table 2.

[0052] [Table 2]

[0053] The relatively poor performance of conventional methods on this dataset may potentially stem from the difficulty in distinguishing between different types of heart disease, which is significantly more challenging than identifying PH disease from healthy controls.

[0054] Experimental Analysis: The results described above indicated that the success of the exemplary embodiment may be attributable to one or more of the following architectural advancements: 1) the incorporation of multiple views, 2) the deployment of a robust pre-trained encoder, and 3) the application of a spatiotemporal network. To evaluate these hypotheses, the inventors systematically performed the following architectural adjustments: 1) the integration of multiple connected views within the WSL framework used as a baseline, 2) the utilization of a variety of pre-trained encoders, and 3) the adoption of various architectures for downstream tasks. The results of these experiments are detailed in Table 3.

[0055] [Table 3]

[0056] While SimCLR does not show a significant improvement over conventional CNNs, DINO and DINOv2 demonstrate substantial performance gains across all three metrics. Comparing WSL to transformers suggests that extracting spatiotemporal features via transformer models provides a significant improvement to PH classification tasks when SimCLR or DINO are used, but not as much when stronger encoders are employed. These collective findings suggest that combining DINOv2 with downstream applications of the systems and methods described herein can significantly improve the baseline performance of PH classification derived from TTE video.

[0057] When SimCLR is used as the training mechanism, the systems and methods described herein in exemplary embodiments achieve an AUC score of 0.77 ± 0.01, an F1 score of 0.80 ± 0.01, and an accuracy of 0.71 ± 0.02. These metrics still significantly outperform the baseline model, even though they are lower than the metrics achieved by the systems and methods described herein in exemplary embodiments using DINOv2. This comparison highlights the effectiveness of modern encoders for risk disease assessment from TTE.

[0058] Some embodiments of the embodiments of the present disclosure are described above in the context of TTE video. However, embodiments of the present disclosure are also applicable in the context of other medical videos. For example, endoscopic video can be processed using embodiments of the present disclosure, which include using an SSL pre-trained encoder, such as the one described above, to obtain frame-level embeddings from the frames of the endoscopic video, and then processing those embeddings using an attention-based deep learning network, such as the one described above, such as a transformer network, a multi-instance learning attention network, or other networks. In one embodiment, such a network may be trained to classify each frame of the endoscopic video as belonging to a specific anatomical segment class, such as the rectum (RM), left colon / sigmoid colon (LC), transverse colon (TC), right colon (RC), and ileum (IL). In one embodiment, such automated segmentation may be used to help automate the determination of disease scores, such as the Simple Endoscopic Score for Crohn's Disease (SES-CD).

[0059] Figure 6 shows embodiments of computer system 6000, one or more of which may be used to implement one or more of the apparatus, systems, and methods illustrated herein. Computer system 6000 executes instruction code contained in computer program product 660. Computer program product 660 includes executable code in an electronically readable medium that can instruct one or more computers, such as computer system 6000, to perform processes to achieve exemplary method steps to be performed.

[0060] An electronically readable medium can be any temporary or non-temporary medium for electronically storing information, and may be accessed locally or remotely, for example, via a network connection. The medium may include multiple geographically distributed mediums, each configured to store different parts of the executable code in different locations and / or at different times. The executable instruction code in the electronically readable medium instructs the illustrated computer system 6000 to perform various exemplary tasks described herein. The executable code for instructing the performance of the tasks described herein is typically implemented in software. However, a person skilled in the art will understand that a computer or other electronic device may utilize code implemented in hardware to perform many or all of the identified tasks. A person skilled in the art will understand that many variations of the executable code can be found that implement the exemplary methods within the spirit and scope of this disclosure.

[0061] The code or a copy of the code contained in the computer program product 660 may reside in one or more persistent storage media (not shown separately) communicatively coupled to the system 6000 for loading and storing in the persistent storage device 670 and / or memory 610 for execution by the processor 620. The computer system 600 also includes an I / O subsystem 630 and peripheral devices 640. The I / O subsystem 630, peripheral devices 640, processor 620, memory 610, and persistent storage device 670 are coupled via a bus 650. Like any other persistent storage that may contain the persistent storage device 670 and the computer program product 660, memory 610 is a non-temporary medium (even when implemented as a typical volatile computer memory device). Furthermore, those skilled in the art will understand that, in addition to storing the computer program product 660 for performing the operations described herein, memory 610 and / or persistent storage device 670 may be configured to store various data elements referenced and illustrated herein.

[0062] Those skilled in the art will understand that computer system 6000 merely illustrates one embodiment of a system in which the computer program product of this disclosure may be implemented. For example, the execution of instructions contained in the computer program product may be distributed among multiple computers, such as computers in a distributed computing network.

[0063] Instructions for implementing an artificial neural network or other deep learning network may reside within the computer program product 660. When the processor 620 is executing instructions in the computer program product 660, the instructions or parts thereof are typically loaded into working memory 610, from which the processor 620 can easily access them.

[0064] The processor 620 may comprise multiple processors, each having additional working memory (additional processors and memory not individually illustrated), including one or more graphics processing units (GPUs) having at least several thousand arithmetic logic units to support large-scale parallel computing. GPUs are often used in deep learning applications because they can perform the associated processing tasks more efficiently than typical general-purpose processors (CPUs). The processor 620 may also comprise one or more dedicated processing units having systolic arrays and / or other hardware arrangements to support efficient parallel processing, either additionally or alternatively. Such dedicated hardware may operate with the CPU and / or GPUs to perform the various processing described herein. Such dedicated hardware may comprise application-specific integrated circuits (which may refer to parts of application-specific integrated circuits), field-programmable gate arrays, or a combination thereof. However, processors such as the processor 620 may be implemented as one or more general-purpose processors (preferably having multiple cores) without necessarily departing from the spirit and scope of this disclosure.

[0065] The terms inferring or estimating may be used in various ways herein, but those skilled in the art will understand that the systems and methods described herein are not so limited, and that the term inferring herein may indicate any performance of any of the calculations and / or the generation of any of the outputs, including but not limited to inference, score, estimate, prediction, forecast, suggestion, recommendation, classification, categorization, annotation, conclusion, or any combination thereof.

[0066] Additional examples Example 1: A computer system comprising one or more processors, a storage medium for storing a dataset representing (a) instructions and (b) video data, an encoding module that encodes a set of frames from a dataset as high-dimensional vector data using one or more processors, wherein (a) the set of frames represents data including multiple views of a region of interest, and (b) the high-dimensional vector data includes information representing each of the multiple views of the region of interest, and at least one of the following: (i) a first attention module that generates a video-level attention map from the high-dimensional vector data using one or more processors, (ii) a second attention module that generates a frame-level attention map from the high-dimensional vector data using one or more processors, and (iii) a third attention module that generates a pixel-level attention map from the high-dimensional vector data using one or more processors, and an inference module that outputs an inference based on one or more of the video-level attention map, frame-level attention map, and pixel-level attention map using one or more processors.

[0067] Example 2: The computer system according to Example 1, wherein the video data includes one or more of the following: transthoracic echocardiogram data, electrocardiogram data, and surgical video data.

[0068] Example 3: The computer system according to Example 1, wherein the video data includes transthoracic echocardiographic data.

[0069] Example 4. The computer system according to Example 1, wherein the high-dimensional vector data includes, for each frame in a set of frames, a vector representing the frame.

[0070] Example 5. The computer system according to Example 4, wherein the encoding module encodes position tokens into each respective vector.

[0071] Example 6. The computer system according to Example 5, wherein one or more attention modules generate one or more attention maps based on the position tokens of each respective vector.

[0072] Example 7. The computer system described in Example 1, wherein the set of frames includes the entire video data.

[0073] Example 8. The computer system according to Example 1, wherein the set of frames includes video data from two or more separate videos.

[0074] Example 9. The computer system according to Example 1, wherein the encoding module includes a concatenated model, and the concatenated model uses position tokens to place each frame within high-dimensional vector data.

[0075] Example 10. The computer system according to Example 1, wherein high-dimensional vector data generated by the encoding module represents concatenated video data of multiple views of the region of interest.

[0076] Example 11. The computer system according to Example 1, wherein the first attention module, the second attention module, and the third attention module are implemented by one or more transformer architectures.

[0077] Example 12. The computer system described in Example 1, wherein the second attention module is implemented by a transformer architecture.

[0078] Example 13. The computer system according to Example 1, wherein the encoding module includes one or more of the first attention module, the second attention module, or the third attention module.

[0079] Example 14. The computer system according to Example 1, further comprising a second encoding module, wherein the second encoding module includes one or more of the first attention module, the second attention module, or the third attention module.

[0080] Example 15. The computer system according to Example 1, wherein one or more of the first attention module, the second attention module, or the third attention module are implemented by a self-supervised architecture.

[0081] Example 16. The computer system according to Example 1, wherein the inference is disease severity estimation.

[0082] Example 17. The computer system described in Example 1, wherein the inference is an assessment of the patient's health.

[0083] Example 18. The computer system described in Example 1, wherein the inference is the measurement of disease progression.

[0084] Example 19. The computer system described in Example 1, wherein the inference is a medical diagnosis decision.

[0085] Example 20. The computer system according to Example 1, wherein the region of interest is a human heart, and the multiple views of the region of interest include two or more views of the human heart.

[0086] Example 21. The computer system according to Example 1, wherein the region of interest is one or more anatomical segments of the human body.

[0087] Example 22. The computer system according to Example 1, wherein each of the multiple views represents a separate location view of the region of interest.

[0088] Example 23. A storage medium that stores (a) instructions and (b) a dataset, and an encoding module that uses one or more processors to encode the dataset as high-dimensional vector data, thereby concatenating vector data representing one of the types of data, transthoracic echocardiogram data, electrocardiogram data, surgical video data, video data, text data, and audio data, with vector data representing another type of data, transthoracic echocardiogram data, electrocardiogram data, surgical video data, video data, text data, and audio data, and (i) uses one or more processors to encode the high-dimensional vector data A computer system comprising one or more attention modules, each including: (ii) a first attention module that generates a video-level attention map from data; (ii) a second attention module that generates a frame-level attention map from high-dimensional vector data using one or more processors; and (iii) a third attention module that generates a pixel-level attention map from high-dimensional vector data using one or more processors; and an inference module that outputs an inference based on one or more of the video-level attention map, frame-level attention map, and pixel-level attention map using one or more processors.

[0089] Example 24. The computer system according to Example 23, wherein the dataset includes two or more of the following data types: transthoracic echocardiogram data, electrocardiogram data, surgical video data, video data, medical image data, text data, and audio data.

[0090] Example 25. A computer system comprising one or more processors, a storage medium for storing a dataset representing (a) instructions and (b) video data, an encoding module that encodes each frame of the dataset as vector data using one or more processors, at least one of the following: (i) a first attention module that generates a video-level attention map from the vector data using one or more processors, (ii) a second attention module that generates a frame-level attention map from the vector data using one or more processors, and (iii) a third attention module that generates a pixel-level attention map from the vector data using one or more processors, and an inference module that outputs an inference based on one or more of the video-level attention map, frame-level attention map, and pixel-level attention map using one or more processors.

[0091] Example 26. A storage medium for storing (a) instructions and (b) a dataset, an encoding module that uses one or more processors to encode the dataset as high-dimensional vector data, and a first attention module that uses one or more processors to generate a token-level attention map from the high-dimensional vector data. A computer system comprising an inference module that uses one or more processors to output inferences based on a token-level attention map.

[0092] Example 27. The computer system according to Example 26, wherein the dataset includes one or more of the following data types: transthoracic echocardiogram data, electrocardiogram data, surgical video data, video data, medical image data, text data, and audio data.

[0093] Example 28. The computer system according to Example 26, wherein the dataset includes echocardiogram data or electrocardiogram data, and in a token-level attention map, tokens are equal to a certain period of time.

[0094] Example 29. The computer system according to Example 26, in which a token in a token-level attention map is equal to a certain period of time.

[0095] Example 30. The computer system according to Example 28 or 29, wherein a token is equal to one second.

[0096] Example 31. A method for generating a deep learning network configured to run on one or more computers to perform inference using video data via a deep learning pipeline, the method comprising: applying a first machine learning model to video data to encode a set of frames from a dataset as high-dimensional vector data, wherein (a) the set of frames represents data containing multiple views of a region of interest, and (b) the high-dimensional vector data contains information representing each of the multiple views of the region of interest; applying one or more second machine learning models to the high-dimensional vector data to generate one or more of the video-level attention map, frame-level attention map, and pixel-level attention map; and outputting an inference based on one or more of the video-level attention map, frame-level attention map, and pixel-level attention map.

[0097] Example 32. A method for spatiotemporal inference using video data, wherein the method applies a first machine learning model to video data to encode a set of frames from a dataset as high-dimensional vector data. A method comprising: applying one or more second machine learning models to high-dimensional vector data to generate one or more of the video-level attention maps, frame-level attention maps, and pixel-level attention maps; and outputting spatiotemporal inference based on one or more of the video-level attention maps, frame-level attention maps, and pixel-level attention maps.

[0098] Example 33. The method according to Example 32, wherein one or more second machine learning models are implemented in the transformer architecture.

[0099] Example 34. The method according to Embodiment 32, wherein one or more second machine learning models are implemented in a temporal architecture.

[0100] Example 35. The method according to Example 32, wherein the video data includes multiple large videos.

[0101] Example 36. The method according to Example 32, wherein the video data includes one or more long-form videos.

[0102] While this disclosure is described in particular with respect to the illustrated embodiments, it will be understood that various changes, modifications, and adaptations may be made based on this disclosure and are intended to be within the scope of this disclosure. Although this disclosure has been described in relation to the embodiments currently considered most practical and preferred, it should be understood that this disclosure is not limited to the disclosed embodiments, but rather is intended to cover various changes and equivalent arrangements that fall within the scope of the basic principles of the invention as described by the various embodiments referenced above and below.

Claims

1. A method for using one or more computers to analyze one or more medical videos corresponding to one or more patient examinations, wherein the method is Preprocessing video data to obtain multiple video frames corresponding to one or more medical videos corresponding to one of the patient examinations mentioned above, Encoding the plurality of video frames using a pre-trained encoder in order to obtain a plurality of frame embeddings corresponding to each of the plurality of video frames, wherein the pre-trained video encoder is pre-trained using self-supervised learning. Connecting the aforementioned multiple frame embeddings, To obtain a summary vector resulting from attention processing that directs attention to each of the aforementioned multiple frame embeddings, the process of the multiple frame embeddings is performed in an attention-based deep learning network, A method comprising presenting the summary vectors to a classification network in order to obtain one or more calculated classifications corresponding to one or more medical videos from the patient examination.

2. The method according to claim 1, wherein the one or more medical videos include a plurality of videos, and thereby the plurality of frame embeddings, which are concatenated for further processing, include frame embeddings from each of the plurality of videos.

3. The method according to claim 1 or 2, wherein the patient examination is transthoracic echocardiography.

4. The method according to any one of claims 1 to 3, wherein the pre-trained encoder includes a vision transformer (ViT) encoder.

5. The method according to any one of claims 1 to 3, wherein the pre-trained encoder includes a convolutional neural network.

6. The method according to any one of claims 1 to 5, wherein the pre-trained encoder is trained using SimCLR.

7. The method according to any one of claims 1 to 5, wherein the pre-trained encoder is trained using DINO.

8. The method according to any one of claims 1 to 5, wherein the pre-trained encoder is trained using DINOv2.

9. The method according to any one of claims 1 to 8, wherein the attention-based deep learning network includes a transformer multi-head attention encoder.

10. The method according to claim 9, wherein the transformer multi-head encoder includes a set transformer encoder.

11. The method according to any one of claims 1 to 8, wherein the attention-based deep learning network includes a convolutional neural network encoder.

12. The method according to any one of claims 1 to 11, wherein the attention-based deep learning network includes an attention pooling layer.

13. The method according to claim 12, wherein the attention pooling layer includes a attention block and an aggregater.

14. The method according to claim 13, wherein the attention block associates each learnable parameter with each frame embedding output of the encoder of the attention-based deep learning network.

15. The method according to claim 14, wherein the aggregater applies the respective learnable parameters to the respective frame embedding outputs in order to obtain the summary vector, and sums the resulting attention-scored frame embedding outputs.

16. The method according to any one of claims 1 to 15, wherein the one or more calculated classifications relate to pulmonary hypertension.

17. A method for using one or more computers to analyze one or more medical videos corresponding to one or more patient examinations, wherein the method is Preprocessing video data to obtain multiple video frames corresponding to one or more medical videos corresponding to one of the patient examinations mentioned above, Encoding a plurality of video frames using a pre-trained encoder in order to obtain a first plurality of frame embeddings corresponding to each of the plurality of video frames, wherein the pre-trained video encoder is pre-trained using self-supervised learning. A method comprising processing the first plurality of embeddings in an attention-based deep learning network trained using supervised learning, to obtain a second plurality of frame embeddings corresponding to each of the first plurality of frame embeddings, and to use the frame embeddings from the second plurality of frame embeddings to perform a computed classification of patient tissue corresponding to the frame embeddings.

18. The method according to claim 17, wherein the patient examination is an endoscopic examination.

19. The method according to claim 18, wherein the calculated classification relates to which of the multiple intestinal segments corresponds to the frame embedding.

20. A computer program product stored on a non-temporary tangible medium, comprising instructions executable on one or more processors of one or more computers for performing processing to analyze one or more medical videos corresponding to a patient examination, wherein the processing includes performing the method according to any one of claims 1 to 19.

21. A computer-executable deep learning network, which is stored in a non-temporary computer-readable medium and configured to run on one or more processors of one or more computers for processing one or more medical videos corresponding to patient examinations, wherein the computer-executable deep learning network is A pre-trained encoder configured to acquire multiple frame embeddings corresponding to each frame of one or more medical videos, wherein the pre-trained video encoder is pre-trained using self-supervised learning, An attention-based deep learning network, wherein the attention-based deep learning network is configured to process the plurality of frame embeddings in order to obtain a summary vector resulting from attention processing that directs attention to each of the plurality of frame embeddings, A computer-executable deep learning network comprising: a classifier network configured to process the summary vectors in order to obtain one or more calculated classifications corresponding to one or more medical videos from the patient examination;

22. The computer-executable deep learning network according to claim 21, wherein the one or more medical videos include a plurality of videos, and the plurality of frame embeddings are concatenated before being processed by the attention-based deep learning network, so that the plurality of frame embeddings are processed together by the attention-based deep learning network, including frame embeddings from each of the plurality of videos.

23. The computer-executable deep learning network according to claim 21 or 22, wherein the patient examination is transthoracic echocardiography.

24. The computer-executable deep learning network according to any one of claims 21 to 23, wherein the pre-trained encoder includes a vision transformer (ViT) encoder.

25. The computer-executable deep learning network according to any one of claims 21 to 23, wherein the pre-trained encoder includes a convolutional neural network.

26. The computer-executable deep learning network according to any one of claims 21 to 25, wherein the pre-trained encoder is trained using SimCLR.

27. The computer-executable deep learning network according to any one of claims 21 to 25, wherein the pre-trained encoder is trained using DINO.

28. The computer-executable deep learning network according to any one of claims 21 to 25, wherein the pre-trained encoder is trained using DINOv2.

29. The computer-executable deep learning network according to any one of claims 21 to 28, wherein the attention-based deep learning network includes a transformer multi-head attention encoder.

30. The computer-executable deep learning network according to claim 29, wherein the transformer multi-head attention encoder is a set transformer encoder.

31. A computer-executable deep learning network according to any one of claims 21 to 28, wherein the attention-based deep learning network includes a convolutional neural network encoder.

32. The computer-executable deep learning network according to any one of claims 21 to 31, wherein the attention-based deep learning network includes an attention pooling layer.

33. The computer-executable deep learning network according to claim 32, wherein the attention pooling layer includes attention blocks and an aggregater.

34. The computer-executable deep learning network according to claim 31, wherein the attention block associates each learnable parameter with the respective frame embedding output of the encoder of the attention-based deep learning network.

35. The computer-executable deep learning network according to claim 34, wherein the aggregater applies the respective learnable parameters to the respective frame embedding outputs in order to obtain the summary vector, and sums the resulting attention-scored frame embedding outputs.

36. A computer-executable deep learning network according to any one of claims 21 to 35, wherein one or more of the calculated classifications relate to pulmonary hypertension.

Images