Ultrasound video feature detection using learning from untagged data

Abstract

A processor trains a neural network using self-supervised learning with a first enhanced ultrasound video and a second enhanced ultrasound video. The processor performs spatial and / or temporal enhancement on frame (s) of untagged ultrasound video to generate the first enhanced ultrasound video and a second enhanced ultrasound video. The training is based on comparing outputs associated with the first enhanced ultrasound video and the second enhanced ultrasound video with each other. A neural network is implemented during the inference to identify a pathology associated with the anatomical structure. A processor generates a heat map identifying a location of a pathology in the frame (s) of the ultrasound video using the neural network, and provides a screen display with the heat map to a display.

Description

Technical Field

[0001] The subject matter described herein relates to devices, systems, and methods for locating and visualizing features (e.g., anatomical features, such as pathologies) in frames of ultrasound videos. Features are detected using a machine learning (ML) algorithm (such as a neural network) that is trained using self-supervised learning (SSL) using unlabeled ultrasound videos that have been spatially and / or temporally enhanced. Background Art

[0002] Ultrasound imaging is often used for diagnostic purposes in an office or hospital setting. For example, lung ultrasound (LUS) is an imaging technology deployed at the point of care to help evaluate lung diseases and infectious diseases, including COVID-19 pneumonia. Important clinical features (such as B-lines, merged B-lines, pleural line changes, consolidation, and pleural effusion) can be visualized under LUS, but accurately identifying these clinical features can be a challenging skill. Therefore, it is beneficial to automatically identify and visualize spectral features through machine learning models.

[0003] However, the data and annotation challenges for artificial intelligence (AI) and machine learning (ML) applications are daunting. For example, AI algorithms developed on medical images require large amounts of training data with expert human annotations of high clinical quality. In particular, for ultrasound, frame-by-frame annotation of a single dynamic loop video takes hours to complete for a trained clinician, making annotation at scale both challenging and expensive. Therefore, despite the recent success of lung ultrasound algorithms (such as automatic B-line detection), further development of these AI tools requires a large amount of manual annotation work.

[0004] The information included in this background section of the specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be construed as subject matter by which the scope of the present disclosure is to be limited. Summary of the invention

[0005] An ultrasound video feature detection system having a machine learning algorithm (e.g., a neural network) is disclosed, and the machine learning algorithm is trained using self-supervised learning (SSL) using unlabeled data with spatial and / or temporal enhancement. The SSL ultrasound video feature detection system disclosed herein has a specific but non-exclusive utility for finding pixels that may have features of interest (e.g., pathology) within a frame of an ultrasound video (such as a lung ultrasound video). The SSL ultrasound video feature detection system includes a training mode in which the machine learning algorithm is trained using unlabeled ultrasound video data. The unlabeled ultrasound video does not have live real annotations made by an expert user. Instead, SSL with spatial and / or temporal enhancement is performed on the ultrasound video. The SSL ultrasound video feature detection system also includes an inference mode in which the machine learning algorithm generates a heat map that identifies pixels of interest (e.g., suspected pathology) within (one or more) frames of the ultrasound video. For example, the heat map can be superimposed on the video.

[0006] In one example aspect, a system is provided. The system includes a display; and a processor configured to communicate with the display, wherein the processor is configured to: receive an ultrasound video of an anatomical structure acquired by an ultrasound probe, wherein the ultrasound video includes a plurality of frames; provide the ultrasound video to a neural network during inference, wherein the neural network is trained using self-supervised learning using an unlabeled ultrasound video, wherein the unlabeled ultrasound video includes at least one of: a plurality of spatial enhancements or a plurality of temporal enhancements; use the neural network to generate a heat map, wherein the heat map identifies the location of a pathology in one or more of the plurality of frames; and provide a screen display including the heat map to a complaint display.

[0007] In some aspects, the neural network is trained using only unlabeled ultrasound videos. In some aspects, the screen display also includes the ultrasound video. In some aspects, the heat map is superimposed on the one or more frames of the multiple frames of the ultrasound video. In some aspects, the heat map includes a single heat map, and the single heat map is superimposed on each of the multiple frames of the ultrasound video. In some aspects, the processor is configured to generate a plurality of heat maps, and the screen display includes the plurality of heat maps superimposed on the multiple frames of the ultrasound video, respectively. In some aspects, the direct output of the neural network includes a visual representation associated with the one or more frames of the ultrasound video, and the dimension of the visual representation is different from the dimension of the multiple frames of the ultrasound video. In some aspects, in order to generate the heat map, the processor is configured to convert the dimension of the visual representation to the dimension of the ultrasound video. In some aspects, the processor is configured to: provide the visual representation to another neural network, the other neural network is configured for at least one of classification, regression, object detection or segmentation; and, use the other neural network to generate an output. The output is associated with at least one of classification, regression, object detection, or segmentation. The output is different from a visual representation and a heat map. The screen display includes a visualization based on the output. In some aspects, the system also includes an ultrasound probe, and the processor is configured to control the ultrasound probe to obtain ultrasound video.

[0008] In one example aspect, a system is provided. The system includes: a memory including an ultrasound video of an anatomical structure acquired by an ultrasound probe, wherein the ultrasound video is unlabeled and includes a plurality of frames; and a processor configured to communicate with the memory, wherein the processor is configured to: retrieve the ultrasound video from the memory; perform a first enhancement on the ultrasound video to generate a first enhanced ultrasound video; perform a second enhancement on the ultrasound video to generate a second enhanced ultrasound video, wherein the first enhancement and the second enhancement include at least one of: spatial enhancement of the plurality of frames or temporal enhancement of the plurality of frames; train a neural network having a first plurality of weights using the first enhanced ultrasound video and the second enhanced ultrasound video, wherein the training of the neural network includes self-supervised learning; and, after the training, provide the neural network having a second plurality of weights, wherein the neural network having the second plurality of weights is configured to be implemented during inference to identify a pathology associated with the anatomical structure, wherein to train the neural network, the processor is configured to: generate a first output associated with the neural network based on the first enhanced ultrasound video; generate a second output associated with the neural network based on the second enhanced ultrasound video; and determine the second plurality of weights based on a comparison between the first output and the second output.

[0009] In some aspects, to train the neural network, the processor is configured to implement a first processing path associated with a first enhanced ultrasound video and a second processing path associated with a second enhanced ultrasound video. In some aspects, the memory further includes a plurality of ultrasound videos, and the processor is configured to: obtain a plurality of ultrasound videos from the memory; and, train the neural network based on the plurality of ultrasound videos, and most of the plurality of ultrasound videos are unlabeled. In some aspects, all of the plurality of ultrasound videos are unlabeled. In some aspects, spatial enhancement includes a change in how image content is depicted in a plurality of frames, and temporal enhancement includes a change in the order in which a plurality of frames are arranged. In some aspects, the first enhancement is different from the second enhancement, and the ultrasound video, the first enhanced ultrasound video, and the second enhanced ultrasound video are different from each other. In some aspects, the first output associated with the neural network includes a first one-dimensional (1D) projection feature set, the second output associated with the neural network includes a second 1D projection feature set, and the processor is configured to determine a second plurality of weights to maximize the consistency between the first 1D projection feature set and the second 1D projection feature set. In some aspects, to train the neural network, the processor is configured to: generate a first initial output of the neural network based on the first enhanced ultrasound video before generating the first output; and generate a second initial output of the neural network based on the second enhanced ultrasound video before generating the second output, the first initial output comprising a first plurality of two-dimensional (2D) visual representations, and the second initial output comprising a second plurality of 2D visual representations. In some aspects, to train the neural network, the processor is configured to: compress the first plurality of 2D visual representations using another neural network to generate a first 1D projection feature set, and compress the second plurality of 2D visual representations using another neural network to generate a second 1D projection feature set. In some aspects, the neural network comprises a convolutional neural network, and the other neural network comprises a multilayer perceptron.

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A more extensive presentation of features, details, utilities, and advantages of the SSL ultrasound video feature detection system as defined in the claims is provided in the following written description of various aspects of the disclosure and illustrated in the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Illustrative aspects of the present disclosure will be described with reference to the accompanying drawings, in which:

[0012] Figure 1 is a schematic diagrammatic representation of an ultrasound imaging system in accordance with at least one aspect of the present disclosure.

[0013] Figure 2 is a schematic diagram of a processor circuit according to aspects of the present disclosure.

[0014] Figure 3 is a schematic diagrammatic representation of a radiology video clip in accordance with at least one aspect of the present disclosure.

[0015] Figure 4 is a schematic diagrammatic representation of a labeled ultrasound data set in accordance with at least one aspect of the present disclosure.

[0016] Figure 5 is a schematic diagrammatic representation of an unlabeled ultrasound dataset in accordance with at least one aspect of the present disclosure.

[0017] Figure 6 Schematic diagrammatic representation of extracting sub-segments from an ultrasound video segment according to at least one aspect of the present disclosure.

[0018] Figure 7 is a schematic illustration of a 3D spatio-temporal enhancement method for ultrasound video data according to at least one aspect of the present disclosure.

[0019] Figure 8 is a schematic illustrative overview in block diagram form of a self-supervised learning model for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure.

[0020] Fig. 9 is a schematic illustrative overview in block diagram form of a self-supervised learning model for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure.

[0021] Fig.10 is a flow chart of an example SSL ultrasound video feature detection system training method according to at least one aspect of the present disclosure.

[0022] Fig.11 is a schematic illustrative overview in block diagram form of an inference mode for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure.

[0023] Fig.12 is a schematic illustrative overview in block diagram form of an inference mode for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure.

[0024] Fig.13 is a schematic illustrative overview in block diagram form of an inference mode for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure.

[0025] Fig.14is a schematic diagrammatic illustration of a formatted heat map superimposed on an incoming video frame to produce a highlighted output frame in accordance with at least one aspect of the present disclosure.

[0026] Fig.15 is a flowchart of an example inference mode SSL video annotation method according to at least one aspect of the present disclosure.

[0027] Fig.16 is an exemplary representation of highlighting features in a video frame through a supervised learning algorithm according to at least one aspect of the present disclosure.

[0028] Fig.17 is an exemplary representation of highlighting features in a video frame through a self-supervised learning (SSL) algorithm according to at least one aspect of the present disclosure.

[0029] Fig.18 is a graph illustrating the test accuracy of a partially trained machine learning model according to at least one aspect of the present disclosure.

[0030] Fig.19 is a graph showing mean values ​​of test sensitivity and test specificity for a partially trained machine learning model according to at least one aspect of the present disclosure.

[0031] Fig. 20 is a graph illustrating the test area under the curve (AUC) of a partially trained machine learning model according to at least one aspect of the present disclosure.

[0032] Fig.21 is a schematic illustrative overview in block diagram form of an inference mode for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure.

[0033] Fig. 22 is a schematic illustrative overview in block diagram form of an inference mode for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure.

[0034] Fig.23 is a schematic illustrative overview in block diagram form of an inference mode for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure.

[0035] Fig.24 An exemplary screen display including both a highlighted image frame and a classification output according to at least one aspect of the present disclosure is shown.

[0036] Fig.25 is an exemplary screen display including both a highlighted image frame and a regression output (eg, a numeric score indicating the probability or severity of a detected pathology) in accordance with at least one aspect of the present disclosure.

[0037] Fig.26 2 is an exemplary screen display including both a highlighted image frame and object detection output (eg, a bounding box indicating a region of interest) according to at least one aspect of the present disclosure.

[0038] Fig. 27 is a schematic illustrative overview in block diagram form of at least a portion of a training mode for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure. DETAILED DESCRIPTION

[0039] According to at least one aspect of the present disclosure, a self-supervised learning (SSL) video annotation system is provided that can identify clinically interesting features in radiology videos (such as lung ultrasound) and can be trained using little or no human-annotated data. Other types of radiology videos can include X-ray videos, camera videos, computer-assisted tomography (CT) scans, magnetic resonance imaging (MRI) scans, etc.

[0040] Accurately identifying clinical features in ultrasound videos can be a challenging skill. Therefore, it is beneficial to automatically identify and visualize spectral features through machine learning models. However, supervised learning for AI or ML algorithms requires large amounts of frame-by-frame, manually annotated data, which is time-consuming and expensive to develop, especially for video data. Therefore, it is substantially easier to obtain large amounts of unlabeled medical image data than to create properly labeled datasets. As a result, unlabeled training data often exceeds labeled training examples by a factor of 100:1 or more.

[0041] Self-supervised learning (SSL) methods are a promising alternative to standard (supervised) AI training because these methods require no or very limited labeled data. Instead, SSL methods exploit unlabeled data to learn meaningful visual representations, relying on extensive data augmentation during training without ground truth. Variants of SSL for processing 2D natural and medical images have been described in the art. SSL has also been combined with traditional supervised training for 2D images, thereby improving prediction performance using both labeled and unlabeled training datasets.

[0042] Applying SSL to ultrasound videos is challenging. Currently, successful applications of SSL have been reported on 2D natural images, using 2D data augmentation techniques suitable for natural images. Unlike natural images, ultrasound data include video sequences (3D data format, including X-axis, Y-axis and time axis) with many factors affecting image quality, including scan settings, scan angles, probe contact, probe motion, body constitution and many other factors.

[0043] For example, applying SSL on lung ultrasound videos for medical applications (e.g., detecting consolidation associated with COVID-19 infection) is not a trivial task. In the present disclosure, a SSL model to be trained on ultrasound data is developed using a dedicated data augmentation process that mimics the full variability seen in ultrasound imagery. Furthermore, the data augmentation and model architecture are compatible with videos, as opposed to 2D images. Such methods are novel in the field of ultrasound video sequences.

[0044] To apply SSL on challenging lung ultrasound applications, the SSL ultrasound video feature detection system adopts a 3D enhancement method based on joint spatial and temporal image transformation, which simulates the changing image appearance and scanning scenes in real-world environments. The SSL ultrasound video feature detection system combines the 3D enhancement method with the SSL training process to generate meaningful visual representations (such as heat maps indicating the clinical significance of different regions) without requiring labeled data during training. The visual representation is then upsampled to the resolution of the original image and displayed to the user. The AI ​​model trained in this way can highlight potential clinically relevant areas within each frame of the ultrasound video. The AI ​​model can be developed without annotation work or with limited annotation work.

[0045] For example, automatic identification and highlighting of relevant features in ultrasound videos based on AI can assist in clinical decision making. Therefore, the present disclosure provides a system for locating and visualizing relevant features in ultrasound videos by learning from unlabeled data. The SSL ultrasound video feature detection system utilizes a self-supervised learning (SSL) method in combination with spatial-temporal 3D video enhancement to generate meaningful visual graphs from radiology videos such as ultrasound video clips or dynamic loops. In the software terminal product, the highlighted areas detected with high probability by the SSL-trained model are displayed to the user as heat maps. These heat maps are learned directly from the video data and do not require manual annotation during training. For example, the system can be trained using dynamic loops of lung ultrasound videos, and can generate heat maps based on live or stored videos, which are strongly associated with areas of pathology (e.g., consolidation, pleural effusion, etc.) in the ultrasound video, indicating its usefulness for computer-aided detection.

[0046] Therefore, the SSL ultrasound video feature detection system locates and visualizes relevant features in ultrasound videos by learning from unlabeled data, which is implemented by using 3D video augmentation within a self-supervised learning (SSL) training process that requires no or very limited ground truth annotations. The input is an ultrasound dynamic cycle, and the output is a heat map for each frame of the dynamic cycle, highlighting the areas detected by SSL. The heat map can be superimposed on B-mode ultrasound for display.

[0047] A proper data augmentation process is a key module for SSL training. Unlike natural 2D images, lung ultrasound is fundamentally a 3D temporal data type (e.g., XY-time). The SSL ultrasound video feature detection system introduces a 3D augmentation method that applies joint spatial and temporal image transformations to simulate the variability expected during real-world ultrasound scanning. The 3D augmentation method enables SSL training on videos. The SSL algorithm is used to train a deep neural network model using unlabeled data.

[0048] Unlike standard supervised training that maximizes the consistency between model predictions and labels, the SSL algorithm disclosed herein maximizes the consistency between visual representations from input video data that undergo different 3D augmentations, as described below. Thus, during the training process, the SSL algorithm learns to focus on regions within the video that have potential clinical relevance and highlight these regions, ignoring the background. SSL can be trained entirely using unlabeled data or using a combination of labeled and unlabeled data.

[0049] The specific mechanism of learning from unlabeled data is described below. During training, sub-segments of ultrasound cycles are extracted and applied to two different 3D enhancements. The two enhanced sub-segments are then sent through a backbone network (e.g., a convolutional neural network or CNN) to collect a 2D visual representation, which is then converted into a 1-dimensional projection feature. SSL training aims to maximize the consistency between these two projection features.

[0050] During deployment of the model (e.g., inference mode or clinical use mode), ultrasound videos are sent directly to the backbone network, and heat maps are generated to locate and visualize clinical features. Importantly, the entire process can be trained using unlabeled data (with or without a small portion of labeled data to provide supervision). Therefore, the method greatly reduces the need for expensive manual annotation.

[0051] The AI ​​software developed using the SSL method produces heat maps that highlight high-probability (high-activity) areas of the image. These heat maps are the output displayed to the user.

[0052] The present disclosure substantially assists in the automatic identification of pathologies by improving the ability to train machine learning systems using unlabeled data. Implemented on a processor in communication with an ultrasound imaging system or a database of stored radiology videos, the SSL ultrasound video feature detection system disclosed herein provides practical assistance to clinicians, especially those with limited training in identifying features of interest. This improved feature recognition converts an expensive, time-consuming, labor-intensive training process into one that can be performed with little or no human supervision and without the usual conventional need for humans to label training videos frame by frame. This unconventional approach improves the functionality of ultrasound imaging systems by providing detailed information about the identified pathologies in real time or near real time.

[0053] The SSL ultrasound video feature detection system can be implemented as a process that is at least partially viewable on a display and operated by a control process executing on a processor that accepts user input from a keyboard, mouse, or touch screen interface and communicates with one or more ultrasound sensors. In this regard, the control process performs certain specific operations in response to different inputs or selections made at different times. Certain structure, functions, and operations of the processor, display, sensor, and user input system are known in the art, while other structure, functions, and operations are described herein to implement novel features or aspects of the present disclosure with particularity.

[0054] Apply machine learning to radiology images and videos (including but not limited to ultrasound images and videos), including feature detection, classification, regression, segmentation, and / or object detection.

[0055] These descriptions are provided for exemplary purposes only and should not be considered to limit the scope of the SSL ultrasound video feature detection system. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

[0056] In order to promote the understanding of the principles of the present disclosure, reference will now be made to the aspects shown in the accompanying drawings, and specific language will be used to describe these aspects. However, it should be understood that it is not intended to limit the scope of the present disclosure. Any changes and further modifications to the described devices, systems and methods and any further applications of the principles of the present disclosure are fully contemplated and included in the present disclosure, as would be normally expected by a person skilled in the art to which the present disclosure relates. In particular, it is fully conceivable that the features, components and / or steps described with respect to one embodiment can be combined with the features, components and / or steps described with respect to other embodiments of the present disclosure. However, for the sake of brevity, multiple iterations of these combinations will not be described separately.

[0057] Figure 1is a schematic diagrammatic representation of an ultrasound imaging system 100 according to at least one aspect of the present disclosure. For example, the ultrasound imaging system 100 may be used to acquire ultrasound video clips that may be used to train an SSL ultrasound video feature detection system, or may be analyzed and highlighted by an SSL ultrasound video feature detection system in a clinical setting (whether in real-time, near real-time, or as post-processing of stored video clips).

[0058] The ultrasound imaging system 100 is used to scan an area or volume of a subject's body. The subject may include a patient of an ultrasound imaging procedure, or any other person, or any suitable living or non-living tissue or structure. The ultrasound imaging system 100 includes an ultrasound imaging probe 110 that communicates with a host 130 via a communication interface or link 120. The probe 110 may include a transducer array 112, a beamformer 114, a processor circuit 116, and a communication interface 118. The host 130 may include a display 132, a processor circuit 134, a communication interface 136, and a memory 138 for storing subject information.

[0059] In some aspects, the probe 110 is an external ultrasound imaging device that includes a housing 111 configured for handheld operation by a user. The transducer array 112 can be configured to obtain ultrasound data when the user grasps the housing 111 of the probe 110 so that the transducer array 112 is positioned adjacent to or in contact with the subject's skin. The probe 110 is configured to obtain ultrasound data of anatomical structures within the subject's body when the probe 110 is positioned outside the subject's body for general imaging (such as for abdominal imaging, liver imaging, etc.). In some aspects, the probe 110 can be an external ultrasound probe, a transthoracic probe, and / or a curved array probe.

[0060] In other aspects, the probe 110 can be an internal ultrasound imaging device and can include a housing 111 configured to be positioned within a lumen of a subject's body for general imaging, such as for abdominal imaging, liver imaging, etc. In some aspects, the probe 110 can be a curved array probe. The probe 110 can be in any suitable form for any suitable ultrasound imaging application including external and internal ultrasound imaging.

[0061] In some aspects, various aspects of the present disclosure may be implemented using medical images of an object obtained using any suitable medical imaging device and / or modality. Examples of medical images and medical imaging devices include X-ray images (angiographic images, fluoroscopic images, images with or without contrast) obtained by an X-ray imaging device, computed tomography (CT) images obtained by a CT imaging device, positron emission tomography-computed tomography (PET-CT) images obtained by a PET-CT imaging device, magnetic resonance images (MRI) obtained by an MRI device, single photon emission computed tomography (SPECT) images obtained by a SPECT imaging device, optical coherence tomography (OCT) images obtained by an OCT imaging device, and intravascular photoacoustic (IVPA) images obtained by an IVPA imaging device. The medical imaging device may obtain the medical image when positioned outside the subject's body, spaced apart from the subject's body, adjacent to the subject's body, in contact with the subject's body, and / or inside the subject's body.

[0062] For ultrasound imaging equipment, the transducer array 112 transmits ultrasound signals toward the anatomical object 105 of the object, and receives echo signals reflected from the object 105 back to the transducer array 112. The ultrasound transducer array 112 may include any suitable number of acoustic elements, including one or more acoustic elements and / or multiple acoustic elements. In some cases, the transducer array 112 includes a single acoustic element. In some cases, the transducer array 112 may include an acoustic element array of any number of acoustic elements with any suitable configuration. For example, the transducer array 112 may include between 1 acoustic element and 10,000 acoustic elements, including values ​​such as 2 acoustic elements, 4 acoustic elements, 36 acoustic elements, 64 acoustic elements, 128 acoustic elements, 500 acoustic elements, 812 acoustic elements, 1000 acoustic elements, 3000 acoustic elements, 8000 acoustic elements, and / or other values ​​greater and lesser. In some cases, the transducer array 112 may include an acoustic element array of any number of acoustic elements with any suitable configuration, such as a linear array, a planar array, a curved array, a curved array, a circumferential array, an annular array, a phased array, a matrix array, a one-dimensional (1D) array, a 1.x-dimensional array (e.g., a 1.5D array) or a two-dimensional (2D) array. The array of acoustic elements (e.g., one or more rows, one or more columns and / or one or more orientations) may be uniformly or independently controlled and activated. The transducer array 112 may be configured to obtain a one-dimensional, two-dimensional and / or three-dimensional image of the anatomical structure of the object. In some aspects, the transducer array 112 may include a piezoelectric micromachined ultrasonic transducer (PMUT), a capacitive micromachined ultrasonic transducer (CMUT), a single crystal, lead zirconate titanate (PZT), a PZT composite material, other suitable transducer types and / or combinations thereof.

[0063] The object 105 may include any anatomical structure or anatomical feature, such as a kidney, a liver, and / or any other anatomical structure of the object. The present disclosure may be implemented in the context of any number of anatomical locations and tissue types, including, but not limited to: organs, including liver, kidney, gallbladder, pancreas, lungs; ducts; intestines; nervous system structures, including brain, dural sac, spinal cord, and peripheral nerves; urinary tract; and valves within blood vessels, blood, abdominal organs, and / or other systems of the body. In some aspects, the object 105 may include a malignancy, such as a tumor, cyst, lesion, hemorrhage, or blood pool within any part of the human anatomy. The anatomical structure may be a blood vessel, such as an artery or vein of the vascular system of the object, including the cardiac vasculature, peripheral vasculature, neural vasculature, renal vasculature, and / or any other suitable lumen within the body. In addition to natural structures, the present disclosure may be implemented in the context of artificial structures, such as, but not limited to, heart valves, stents, shunts, filters, implants, and other devices.

[0064] The beamformer 114 is coupled to the transducer array 112. The beamformer 114 controls the transducer array 112, for example, for the transmission of ultrasound signals and the reception of ultrasound echo signals. In some aspects, the beamformer 114 can apply a time delay to the signal sent to the individual acoustic transducers within the array in the transducer 112 so that the acoustic signal is directed in any suitable direction of propagation away from the probe 110. The beamformer 114 can also provide an image signal to the processor circuit 116 based on the response of the received ultrasound echo signal. The beamformer 114 can include multi-level beamforming. Beamforming can reduce the number of signal lines used to couple to the processor circuit 116. In some aspects, the transducer array 112 combined with the beamformer 114 can be referred to as an ultrasound imaging component.

[0065] The processor 116 is coupled to the beamformer 114. The processor 116 may also be described as a processor circuit, which may include other components in communication with the processor 116, such as a memory, the beamformer 114, a communication interface 118, and / or other suitable components. The processor 116 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 116 may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. The processor 116 is configured to process beamformed image signals. For example, the processor 116 may perform filtering and / or orthogonal demodulation to adjust the image signal. The processor 116 and / or 134 may be configured to control the array 112 to obtain ultrasound data associated with the object 105.

[0066] The communication interface 118 is coupled to the processor 116. The communication interface 118 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuits for transmitting and / or receiving communication signals. The communication interface 118 may include hardware components and / or software components that implement a specific communication protocol suitable for delivering signals to the host 130 via the communication link 120. The communication interface 118 may be referred to as a communication device or a communication interface module.

[0067] The communication link 120 may be any suitable communication link. For example, the communication link 120 may be a wired link, such as a Universal Serial Bus (USB) link or an Ethernet link. Alternatively, the communication link 120 may be a wireless link, such as an Ultra Wideband (UWB) link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 WiFi link, or a Bluetooth link.

[0068] At host 130, communication interface 136 may receive the image signal. Communication interface 136 may be substantially similar to communication interface 118. Host 130 may be any suitable computing and display device, such as a workstation, a personal computer (PC), a laptop, a tablet computer, or a mobile phone.

[0069] The processor 134 is coupled to the communication interface 136. The processor 134 may also be described as a processor circuit, which may include other components that communicate with the processor 134, such as a memory 138, a communication interface 136, and / or other suitable components. The processor 134 may be implemented as a combination of software components and hardware components. The processor 134 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a controller, an FPGA device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 134 may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. The processor 134 may be configured to generate image data based on image signals received from the probe 110. The processor 134 may apply advanced signal processing and / or image processing techniques to the image signal. In some aspects, the processor 134 may form a three-dimensional (3D) volume image based on the image data. In some aspects, processor 134 can perform real-time processing on the image data to provide a streaming video of an ultrasound image of subject 105. In some aspects, host 130 includes a beamformer. For example, processor 134 can be part of and / or otherwise communicate with such a beamformer. The beamformer in host 130 can be a system beamformer or a main beamformer (providing one or more subsequent stages of beamforming), while beamformer 114 is a probing beamformer or a microwave beamformer (providing one or more initial stages of beamforming).

[0070] Memory 138 is coupled to processor 134. Memory 138 may be any suitable storage device, such as cache memory (e.g., cache memory of processor 134), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, a solid-state memory device, a hard disk drive, a solid-state drive, other forms of volatile and nonvolatile memory, or a combination of different types of memory.

[0071] The memory 138 may be configured to store object information, measurements, data, or files related to the medical history of the subject, the history of the procedures performed, anatomical or biological features, characteristics, or medical conditions associated with the subject, computer readable instructions (such as code, software, or other applications), and any other suitable information or data. The memory 138 may be located within the host 130. The object information may include measurements, data, files, other forms of medical history, such as, but not limited to: ultrasound images, ultrasound videos, and / or any imaging information related to the anatomical structure of the subject. The object information may include parameters related to the imaging process, such as anatomical scan windows, probe orientation, and / or the position of the object during the imaging process. The memory 138 may also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to the implementation of image recognition algorithms, image quantification algorithms, and / or image acquisition guidance algorithms for detecting / segmenting anatomical structures, including those described herein.

[0072] The display 132 is coupled to the processor circuit 134. The display 132 may be a monitor or any suitable display. The display 132 is configured to display an ultrasound image of the subject 105, an image video, and / or any imaging information.

[0073] The ultrasound imaging system 100 can be used to assist ultrasound physicians in performing ultrasound scanning. Scanning can be performed in a point-of-care environment. In some cases, the host 130 is a console or a movable cart. In some cases, the host 130 can be a mobile device, such as a tablet, a mobile phone, or a portable computer. During the imaging process, the ultrasound system can acquire ultrasound images of a specific region of interest within the anatomical structure of the object. The ultrasound imaging system 100 can then analyze the ultrasound image to identify various parameters associated with the acquisition of the image, such as a scan window, a probe orientation, an object position, and / or other parameters. The ultrasound imaging system 100 can then store the image and these associated parameters in a memory 138. During subsequent imaging, the ultrasound imaging system 100 can retrieve previously acquired ultrasound images and associated parameters to display to the user, which can be used to guide the user of the ultrasound imaging system 100 to use the same or similar parameters during subsequent imaging, as will be described in more detail below.

[0074] In some aspects, the processor 134 can utilize a deep learning-based prediction network to identify parameters of the ultrasound image, including anatomical scan windows, probe orientation, object position, and / or other parameters. In some aspects, the processor 134 can receive metrics related to the physiological state of the imaged region of interest or object or perform various calculations related to the physiological state of the imaged region of interest or object during the imaging process. These metrics and / or calculations can also be displayed to a sonographer or other user via the display 132.

[0075] Before continuing, it should be noted that the above examples are provided for purposes of illustration and are not intended to be limiting. Other devices and / or device configurations may be utilized to perform the operations described herein.

[0076] Figure 2 2 is a schematic diagram of a processor circuit 250 according to various aspects of the present disclosure. The processor circuit 250 may be implemented in the ultrasound imaging system 100 or other device or workstation (e.g., a third-party workstation, a network router, etc.), or on a cloud processor or other remote processing unit, as required for implementing the method. As shown, the processor circuit 250 may include a processor 260, a memory 264, and a communication module 268. For example, these elements may communicate directly or indirectly with each other via one or more buses.

[0077] The processor 260 may include any combination of a central processing unit (CPU), a digital signal processor (DSP), an ASIC, a controller or a general purpose computing device, a reduced instruction set computing (RISC) device, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other related logic devices (including mechanical and quantum computers). The processor 260 may also include another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 260 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0078] Memory 264 may include cache memory (e.g., cache memory of processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory devices, hard disk drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. In one aspect, memory 264 includes a non-transitory computer-readable medium. Memory 264 may store instructions 266. Instructions 266 may include instructions that, when executed by processor 260, cause processor 260 to perform the operations described herein. Instructions 266 may also be referred to as code. The terms "instructions" and "code" should be broadly interpreted to include any type of (one or more) computer-readable statements. For example, the terms "instructions" and "code" may refer to one or more programs, routines, subroutines, functions, procedures, etc. The terms "instructions" and "code" may include a single computer-readable statement or multiple computer-readable statements.

[0079] The communication module 268 may include any electronic circuitry and / or logic circuitry to facilitate direct or indirect communication of data between the processor circuit 250 and other processors or devices. In this regard, the communication module 268 may be an input / output (I / O) device. In some cases, the communication module 268 facilitates direct or indirect communication between the processor circuit 250 and / or various elements of the ultrasound imaging system 100. The communication module 268 may communicate within the processor circuit 250 via a variety of methods or protocols. Serial communication protocols may include, but are not limited to, the United States Serial Protocol Interface (US SPI), Inter-Integrated Circuit (IIC), and the like. 2 C), Recommended Standard 232 (RS-232), RS-485, Controller Area Network (CAN), Ethernet, Aeronautical Radio Corporation 429 (ARINC429), MODBUS, Military Standard 1553 (MIL-ST D-1553), or any other suitable method or protocol. Parallel protocols include, but are not limited to: Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), Institute of Electrical and Electronics Engineers 488 (IEEE-488), IEEE-1284, and other suitable protocols. Where appropriate, serial and parallel communications may be bridged by a Universal Asynchronous Receiver / Transmitter (UART), a Universal Synchronous Receiver / Transmitter (USART), or other appropriate subsystems.

[0080] External communications (including but not limited to software updates, firmware updates, model sharing between the processor and the central server, or readings from the ultrasound imaging system 100) can be implemented using any suitable wireless or wired communication technology, such as a cable interface (such as a universal serial bus (USB), micro USB, lightning or firewire interface), Bluetooth, Wi-Fi, ZigBee, Li-Fi, or a cellular data connection (such as 2G / GSM (Global System for Mobile), 3G / UMTS (Universal Mobile Telecommunications System), 4G, Long Term Evolution (LTE), WiMax, or 5G). For example, a Bluetooth Low Energy (BLE) radio can be used to establish a connection with a cloud service for transmitting data and for receiving software patches. The controller can be configured to communicate with a remote server or a local device (such as a laptop, tablet, or handheld device), or can include a display capable of showing state variables and other information. Information can also be transmitted on a physical medium such as a USB flash drive or memory stick.

[0081] Figure 33 is a schematic diagrammatic representation of a radiology video, dynamic loop or video segment 310 (e.g., an ultrasound video segment) according to at least one aspect of the present disclosure. The ultrasound video segment 310 includes a plurality of frames 320. In an example, the ultrasound video segment 310 is between 1 second and 60 seconds in length, with a frame rate of 30 frames per second, and thus may include between 30 frames and 1800 frames 320. Each frame as a Y-axis or height 330 and an X-axis or width 340 are spatial dimensions representing a 2D cross-section of an object imaged by an ultrasound imaging system. In addition, the ultrasound video segment 310 includes a depth or time axis 350, which represents the time at which each frame 320 of the video segment 310 was captured. Therefore, the ultrasound video segment 310 can be viewed as a 3D data structure. The video segment 310 can be any suitable modality with 2D image frames over time, such as X-ray, MRI, CT, etc.

[0082] In some aspects, the video segment 310 may include 4D data (X, Y, Z, time). For example, the 4D data may be 3D ultrasound (X, Y, Z are spatial dimensions) + time or other imaging modalities of 3D (X, Y, Z are spatial dimensions) + time, such as MRI, CT, etc. In other cases, the video segment 310 may include a 4D multi-modality / multi-imaging type image (X, Y are spatial dimensions in one imaging type of the modality, + Z is an imaging type dimension in the modality, with an imaging type different from the X, Y dimension + time). For example, the 4D multi-modality / multi-imaging type image may be 2D ultrasound (X, Y are spatial dimensions in B-mode ultrasound) + color Doppler ultrasound (Z) + time. In general, the "Z" dimension may be any suitable imaging type (e.g., Doppler, elastic imaging, etc.) different from the X, Y dimensions (e.g., B-mode).

[0083] Figure 4 4 is a schematic diagrammatic representation of a labeled ultrasound dataset 400 according to at least one aspect of the present disclosure. The labeled ultrasound dataset 400 includes a plurality of video clips 405. Each video clip 405 includes a title 410 and a plurality of frames 420. Each frame 420 includes a frame number 430 and an annotation 440. For example, the annotation 440 may indicate whether there is visible pathology in the frame 420. If pathology is present, the annotation 440 may also include one or more pathology locations 450, and the frame 420 may include one or more bounding boxes 460 indicating those locations on the image.

[0084] Such frame-by-frame labeling is typically performed manually by highly skilled clinicians in order to generate training data for conventional machine learning (ML) models. However, given the large number of frames in even a short video clip, and given the high value of a trained clinician's time, such labeled ultrasound datasets 400 are expensive and labor intensive to produce, and thus a limited amount of labeled data is available for any given pathology, organ, or anatomical system.

[0085] Figure 5 is a schematic diagrammatic representation of an unlabeled ultrasound dataset 500 according to at least one aspect of the present disclosure. The unlabeled ultrasound dataset 500 includes a plurality of video clips 505. As with the labeled dataset 400, each video 500 includes a title 410 and a plurality of frames 420, and each frame includes a frame number 430. However, the video clips 505 of the unlabeled dataset 500 do not include annotations 440 (or, alternatively, the annotations 440 remain blank). The unlabeled data 500 is easy to collect and store, and therefore can be cheaper and more readily available than the labeled data 400. A given video clip 505 may include pathology, or may include only healthy tissue. The video clips 505 should all belong to the same anatomical system (e.g., all lung tissue), but may be captured at different locations, depths, angles, image settings, etc.

[0086] Figure 6 is a schematic diagrammatic representation of extracting sub-segments 610 from an ultrasound video segment 400 in accordance with at least one aspect of the present disclosure. In a first example 620, a video segment 405 includes a plurality of frames 420 that are divided into sub-segments 610 of equal size, with possible remaining segments 615 having a smaller size. In the example 620, the video segment 405 includes 30 frames that are divided into three normal sub-segments 610, each of which includes eight frames 420. For example, eight frames may be a sub-segment size that is large enough for training an ML algorithm, but small enough to keep the computational burden within reasonable parameters. However, because 30 frames 420 are not divisible by 8, the division leaves a remaining segment 615 of six frames.

[0087] In example 630, video segment 405 includes 30 frames, which are divided into four sub-segments 610, each of which includes eight frames 420. Taking into account the fact that 30 is not divisible by 8, each sub-segment 610 includes between 1 and 3 overlapping frames 635 shared with another sub-segment 610.

[0088] In example 640 , in order to save data and speed up the training process, the 30-frame video clip 405 is divided into two five-frame sub-segments 610 , each of which includes only one-third of the frames 420 from the video clip 405 , so that two-thirds of the frames 420 are excluded from frames 650 .

[0089] In example 660 , the 15-frame ultrasound video segment 405 is directly used as the 15-frame sub-segment 610 .

[0090] Figure 7 is a schematic illustration of a 3D spatio-temporal enhancement method 700 for ultrasound video data in accordance with at least one aspect of the present disclosure. Enhancement is a key component of the SSL training process described below. For lung ultrasound and other radiology video data, the input is 3D data for which a 3D data enhancement method is required.

[0091] exist Figure 7 In the example shown, sub-segment 710 is enhanced using two different data enhancements 720a and 720b. The same first data enhancement 720a is applied to one, multiple or all frames in the sub-segment 710, thereby generating a first enhanced sub-segment 710', and the second data enhancement 720b is applied to one, multiple or all frames in the sub-segment 710, thereby generating a second enhanced sub-segment 710". In the example, the data enhancement is randomly selected from a list, which may include, for example: random affine transformation (e.g., translation and / or rotation of the image), random horizontal flip, random color jitter (which may, for example, include random small color modifications to each pixel in the image, or changes to the entire image, such as hue, gain, brightness, contrast, etc.), random noise (e.g., Gaussian or specular noise scattered throughout the image), random erasing of frames in the sub-segment, random time reversal of frames in the sub-segment, random shuffling of frames in the sub-segment, or random dropping of some frames in the sub-segment and replacing them with the first frame. These enhancements are called "3D" enhancements because the frames of the sub-segment are 2D images and time is used as the third dimension.

[0092] exist Figure 7 In the example shown, a second different sub-segment 730 is enhanced using two different randomly selected data enhancements 720c and 720d, resulting in two enhanced video segments 730' and 730". Therefore, it should be understood that video segments 710, 710', 710", 730, 730' and 730" are all different from each other.

[0093] Applying these 3D enhancements to the sub-segments helps provide a variety of unrelated data to the learning algorithm so that the training process teaches the learning algorithm to distinguish clinically relevant details that might otherwise be lost in noise, anatomical motion, motion of the ultrasound probe, and normal anatomical structures. Other amounts of data augmentation or other types of data augmentation may be used instead of or in addition to those described herein.

[0094] Affine transformation, flipping, color jitter, and noise can be considered spatial enhancements, while frame deletion, time reversal, and substitution can be considered temporal enhancements. One or more options are selected for each enhancement, so the combination of options results in a diverse set of enhanced video data for training AI models using SSL. In other words, the randomly selected enhancements can include any combination of spatial and / or temporal enhancements.

[0095] Figure 8 is a schematic diagrammatic overview in block diagram form of a self-supervised learning model 800 for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure. Figure 8 In the example shown, the training data set 810 includes unlabeled ultrasound video data 820 and optionally includes a small amount of labeled ultrasound video data 830 (see below). Fig. 27 ). The amount of labeled data 830 can be relatively less than the amount of unlabeled data 820 (e.g., between 1% and 49%, between 1% and 25%, between 1% and 10%, and / or other values ​​of greater and lesser magnitudes of the total amount of data). In some aspects, unlabeled ultrasound data 830 is fully employed, such that all training data 810 is unlabeled data 820. The training data 810 is enhanced using 3D data augmentation 720, and the resulting enhanced data is fed to a machine learning model 850, such as a convolutional neural network (CNN) or an encoder network.

[0096] The output 860 of the machine learning model 850 (e.g., a 2D matrix for each image frame indicating the level of clinical interest at different locations on the frame) is then fed to a second machine learning model 870 (or a different part of the same learning model), which can be, for example, a multilayer perceptron. The second machine learning model 870 then converts the output 860 into outputs 880 (e.g., a single vector for each video segment, representing (one or more) heat maps of the video segment). These outputs 880 are then passed to a consistency step 890, which adjusts the weights of the machine learning models 850, 870 until a specified level of consistency is reached (e.g., determined using cosine similarity, mean squared error, mutual information, absolute difference test, and / or other suitable metrics).

[0097] As will be appreciated by those of ordinary skill in the art, the machine learning model / network 850 may implement or include any suitable type of learning network. For example, in some aspects, the machine learning model 850 may include a neural network, such as a convolutional neural network (CNN). In addition, additionally or alternatively, the convolutional neural network may be an encoder-decoder type network, or may utilize a backbone architecture based on other types of neural networks, such as an object detection network, a classification network, etc. An example backbone network is the Darknet YOLO backbone, which may be used for object detection. For example, a CNN may include a set of N convolutional layers, where N may be any positive integer. When a CNN is the backbone, the fully connected layer may be omitted. A CNN may also include a maximum pooling layer and / or an activation layer. Each convolutional layer 520 may include a set of filters configured to extract features from an input (e.g., a frame from an ultrasound video). The value N and the size of the filter may vary according to various aspects. In some cases, the convolutional layer may utilize any nonlinear activation function, such as a leaky rectified nonlinear (ReLU) activation function and / or batch normalization. The max pooling layer gradually reduces the high-dimensional output to the dimension of the desired result (e.g., the heat map of model 850).

[0098] Fully connected layers may be referred to as perceptual or sensory layers. In some aspects, perceptual / sensory and / or fully connected layers may be found in additional machine learning models / networks or projection heads 870 (e.g., multilayer perceptrons) to allow downstream processing (e.g., classification, regression, detection, segmentation, etc.). Network 870 may also include max pooling layers and / or activation layers. Max pooling layers gradually reduce high-dimensional outputs to the dimensions of the desired result (e.g., heat maps, classification outputs, regression outputs, bounding boxes of regions of interest, etc. for network 870).

[0099] These descriptions are included for exemplary purposes; one of ordinary skill in the art will understand that other types of learning models having features similar to or different from those described above may be used alternatively or additionally without departing from the spirit of the present disclosure. It should also be noted that the block diagrams provided herein are for exemplary purposes; one of ordinary skill in the art will recognize countless variations that still fall within the scope of the present disclosure. For example, a block diagram may illustrate a specific arrangement of components, modules, services, steps, processes, or layers that result in a particular data flow. It should be understood that some aspects of the systems disclosed herein may include additional components, some aspects may not have some of the components shown, and the arrangement of components may be different from that shown, resulting in different data flows while still performing the methods described herein.

[0100] Fig. 9900 is a schematic diagram of a self-supervised learning model 900 for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure. The SSL ultrasound video feature detection system includes a training model (herein Fig. 9 ) and the inference model or clinical use model (shown below in Fig.12 and 13 ).

[0101] In an example, the training mode 900 can train a learning algorithm 850, which can be, for example, or include an artificial intelligence (AI), a neural network, a backbone network, or a deep learning network, such as a convolutional neural network (e.g., a micro Yolov3) algorithm. For example, the training mode 900 can be executed on a computer accessing a database of stored video clips (e.g., ultrasound or other radiology clips). The training mode 900 can use tens, hundreds, or thousands of video clips to train the learning algorithm 850. However, the amount of data represented by these video clips can be large. For example, in the case where the video clips are ultrasound videos of the internal anatomical structures of a patient, each clip can include 0.2 seconds to 60 seconds of video captured at 30 frames per second, for example, 30-1,800 individual images (image size is approximately hundreds of pixels by hundreds of pixels), each image having a resolution of approximately 0.01 megapixels, 0.1 megapixels, 1 megapixel, 1.5 megapixels, and / or other values ​​of greater and lesser. In an example, in order to limit the amount of time and computing power required to train the learning algorithm, a single sub-segment 610 can be provided to the learning algorithm at a time. In an example, the sub-segment 610 may include one-third of the frames 420 of the 24-frame segment at a time (8 total frames 420 for a single video).

[0102] The sub-segments may be selected randomly (which may be considered a form of data augmentation) or based on fixed parameters. During training, sub-segments are automatically generated from the full video as part of the data processing pipeline. In an example, the system may select frame index i±n*X (n=0, 1, ...) based on a random first frame index i and a fixed gap of frame X, uniformly sample 8 frames from the video, so that a total of 8 frames are randomly selected from the video by the computer. The video sub-segment has a width W (e.g., in pixels or millimeters), a height H (e.g., in pixels or millimeters), and a depth or duration D (e.g., in frames or milliseconds).

[0103] exist Fig. 9In the example shown, the video sub-segment 610 undergoes two different randomly selected 3D data augmentations 720e and 720f, resulting in two different enhanced sub-segments 910e (including frame 920e) and 910f (including frame 920f). Each enhanced sub-segment 910e, 910f is then fed into the machine learning model 850, where each frame of the sub-segment is processed according to the weights within the learning model 850 to produce corresponding visual representations 930e and 930f, each visual representation including a 2D matrix 940 for each frame of the corresponding video sub-segment. For example, the 2D matrix 940 can be a w×h heat map that identifies locations within the image that contain suspicious features of interest. In some cases, the 2D matrix can have the same resolution as the incoming frame or image (e.g., w=W and h=H). In other cases, the 2D matrix can have a smaller resolution in one or both of the 2D dimensions (e.g., w<W and / or h<H). In an example, there may be a downsampled output of the model 850 so that the model 870 does not have too many parameters to train well. In some aspects, the depth d (e.g., the number of matrices) of the visual representations 931e, 930f is equal to the depth D (e.g., d=D) of the ultrasound video sub-segment 610. In other aspects, the depth d of the visual representation is less than the depth D of the ultrasound video sub-segment 610 (e.g., d<D), or is uniform (e.g., d=1).

[0104] although Fig. 9 Two examples of model 850 and model 870 are shown, but it should be understood that the self-supervised learning model 900 includes one model 850 and one model 870 (e.g., Figure 8 ). Two instances of model 850 and model 870 are shown in Fig. 9 850 and 870. As described below, training includes adjusting the values ​​of the weights of the neural network based on the consistency between the two branches (compared to the consistency between the labeled data and the output of model 850 and / or model 870).

[0105] After all frames 920e, 920f of the sub-segments 910e, 910f have been fed through the machine learning model 850, there will be one matrix or visual representation 940 for each frame. These visual representations 930e, 930f are then processed by the projection head 870. For example, the projection head 870 can be a fixed algorithm, or it can be part of the learning algorithm 850, such as a multi-layer perceptron. The projection head 870 reduces the visual representation (e.g., eight 2D matrices for each sub-segment) to a single "projected feature" vector 950e, 950f representing the features identified in the enhanced sub-segment. The size of the vector 950e can be kx 1. In one example, k=18. However, in other cases, k can be a positive integer parameter assigned by humans and is therefore a value other than 18 that is fixed by the perceptron architecture. Depending on the implementation, too small a k value does not represent image features, while too large a k value will increase model complexity / parameters.

[0106] The two projected feature vectors 950e, 950f (e.g., a vector representing a first enhancer segment and a vector representing a second enhancer segment) are then compared in a consistency step 960, and the weights of the learning algorithm 850 are iteratively adjusted until the two projected feature vectors 950e, 950f are within a specified consistency level (e.g., as measured by cosine similarity, mean squared error, mutual information, absolute difference test, and / or other suitable metrics). The training is based on stochastic gradient descent, which means that in each step the sub-segments 610 are processed one by one, and the weights are updated stepwise according to each sub-segment 610 to gradually improve the consistency. The training of the learning algorithm can be considered complete when the consistency level no longer improves by more than a threshold as new sub-segments 610 are processed.

[0107] In some cases, the output of the model 850 (e.g., visual representations 930e, 930f) may not be suitable for direct comparison with each other in some cases. The machine learning model 870 generates outputs (two projected feature vectors 950e, 950f) that can be compared with each other to determine consistency. This is advantageous when training is done using unlabeled data because the two projected feature vectors 950e, 950f can be compared with each other (rather than comparing the output of the model 850 to labeled ultrasound data).

[0108] In the example, the training of the machine learning model can be or include SimCLR (Simple Framework for Contrastive Learning of Visual Representations) as the SSL training method, although other methods can be used alternatively or additionally. In the case where the video data is 4D instead of 3D, the spatial enhancement option needs to be upgraded to 3D instead of 2D spatial enhancement, but the overall process remains the same.

[0109] In some cases, the machine learning model 850 and the projection head 870 are combined here, and there is no intermediate output or visual representation 930e, 930f. Instead, the machine learning model 850 directly outputs feature vectors 950e, 950f. For example, this can be done when it is not desired that the system generate a heat map.

[0110] Fig.10 1 is a flow chart of an example SSL ultrasound video feature detection system training method 1000 according to at least one aspect of the present disclosure. It should be understood that the steps of method 1000 may be performed in the same manner as described above. Fig.10 The method 1000 may be performed in a different order than that shown, additional steps may be provided before, during, and after the steps, and / or some of the steps described may be replaced or eliminated in other aspects. One or more steps of the method 1000 may be performed by one or more devices and / or systems described herein (such as components of the ultrasound imaging system 100 and / or the processor circuit 250).

[0111] In step 1010, method 1000 includes retrieving (e.g., from a memory or database) a plurality of ultrasound videos of a selected anatomical structure type for use in training a machine learning model. In some aspects, all ultrasound videos are unlabeled. In some aspects, a few ultrasound videos are labeled. Each ultrasound video includes a plurality of 2D frames, each frame representing a different time within a time period of the video. In some aspects, all frames in the plurality of frames are unlabeled. In some aspects, a few frames in the plurality of frames are labeled. Figure 3-6 Examples of videos, video clips, or video sub-clips can be seen in .

[0112] Steps 1020 to 1070 may be repeated for a large number of video clips or video sub-segments in order to fully train the machine learning algorithm or learning model.

[0113] In step 1020, the method 1000 includes performing a first randomly selected enhancement on the currently selected ultrasound video to generate a first enhanced ultrasound video (see above). Figure 7 ).

[0114] In step 1030, the method 1000 includes performing a second randomly selected enhancement on the currently selected ultrasound video to generate a second enhanced ultrasound video (see above). Figure 7 ).

[0115] In step 1035, method 1000 includes training a neural network (e.g., Figure 8 and Fig. 9850 in the model. The neural network initially includes a first plurality of weights (e.g., a first plurality of values ​​for the weights). Step 1035 may include sub-steps 1040, 1050, 1060, 1070, and 1080.

[0116] In step 1040, method 1000 includes performing a first enhanced ultrasound video (see above) based on the first enhanced ultrasound video. Fig. 9 ) generates a first plurality of 2D visual representations (e.g., a first initial output).

[0117] In step 1050, method 1000 includes performing a step based on the second enhanced ultrasound video (see above Fig. 9 ) generates a second plurality of 2D visual representations (e.g., a second initial output).

[0118] In step 1060, method 1000 includes using another neural network (see, e.g., above Figure 8 and Fig. 9 870 in the model 870) to compress the first plurality of 2D visual representations to generate a first 1D projection feature set (e.g., a first feature vector or a first output).

[0119] In step 1070, method 1000 includes using the additional neural network (see, e.g., above Figure 8 and Fig. 9 870 in the model 870) to compress the second plurality of 2D visual representations to generate a second 1D projection feature set (e.g., a second feature vector or a second output).

[0120] In step 1080, method 1000 includes determining a second plurality of weights (e.g., a second plurality of values ​​of the weights) for the neural network and the additional neural network based on a comparison between the first 1D projection feature set and the second 1D projection feature set (see above Fig. 9 The comparison may include a measure of consistency between the first 1D projection feature set and the second 1D projection feature set (see e.g. Figure 8 Consistency in 890 and Fig. 9 consistency in 960).

[0121] Steps 1040 to 1080 may be iteratively performed on the first enhanced ultrasound video and the second enhanced ultrasound video, continuously updating and refining the values ​​of the weights of the neural network until a desired level of consistency is reached between the first 1D projection feature set and the second 1D projection feature set (e.g., measured using cosine similarity, mean square error, mutual information, absolute difference test, and / or other suitable metrics).

[0122] In step 1090, method 1000 includes providing or outputting a neural network having a second plurality of weights. The second plurality of weights is an updated version (e.g., updated values) of the first plurality of weights, reflecting iterative training of the network as additional video data is fed to the network. During iteration, the second value of the plurality of weights replaces the first value of the plurality of weights, and may then be replaced by the new "second" value of the plurality of weights in the next iteration of the loop. Providing or outputting a neural network having a second plurality of weights may include writing data representing the neural network having a second plurality of weights accessible in memory to a memory (e.g., as in step 1095, for implementation during inference). Other examples of providing or outputting may include a processor circuit transmitting data representing the neural network having a second plurality of weights to a different computer or processor circuit. For example, the training of the neural network may be performed by a manufacturer of an ultrasound system using a processor circuit at a research and development or manufacturing location. The trained neural network (having a second plurality of weights) may be transmitted (online data transfer, data transfer via a physical medium, wired or wireless data transfer) to a different processor circuit to be used to identify pathology in ultrasound video data of a patient. The different processor circuit may be an ultrasound console or another computer in a hospital, physician's office, or other clinical setting where patient data with potential pathology is collected and / or evaluated.

[0123] In step 1095, method 1000 includes implementing a neural network having a second plurality of weights during inference (e.g., real-time annotation of ultrasound video captured in a clinical setting) to identify pathologies associated with the anatomical structure under investigation (see below). Figure 11-15 ). In some cases, only the neural network (and no additional neural networks) is used during inference. The neural network can be implemented during inference by an ultrasound console or another computer in a hospital, doctor's office, or other clinical setting.

[0124] Note that the flowcharts provided herein are for exemplary purposes; those of ordinary skill in the art will recognize countless variations that still fall within the scope of the present disclosure. For example, the logic of the flowchart may be shown as sequential. However, similar logic may be parallel, massively parallel, object-oriented, real-time, event-driven, cellular automata, etc., while implementing the same or similar functions. In order to perform the methods described herein, the processor may divide each step described herein into multiple machine instructions, and these instructions may be run at a rate of hundreds, thousands, millions, or billions per second in a single processor or across multiple processors. This fast operation may be necessary in order to run the methods described herein in real time or near real time. For example, real-time annotation of ultrasound video may require analysis of megapixel or more data thirty times per second.

[0125] Fig.11 is a schematic illustrative overview in the form of a block diagram of an inference mode 1100 for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure. In the inference mode 1100, a trained machine learning algorithm or backbone network 850 is placed into clinical use, such as embedded in one or more processors of an ultrasound imaging system 100. In this configuration, the learning algorithm or backbone network 850 is used to analyze a clinical video stream (such as ultrasound video) in real-time, near real-time, or to post-process recorded video. For example, a video (e.g., ultrasound video, whether live or recorded) is sent directly to a trained learning algorithm or backbone network, which generates a visual representation or heat map for each frame of the video to identify the locations of features suspected by the learning algorithm or backbone network to be of clinical interest.

[0126] exist Fig.11 In the illustrated example, ultrasound video data 1110 (e.g., live video captured from a patient in real time, or recorded video captured from a patient at an earlier time) is fed into a trained machine learning algorithm 850, which produces an output 930 (e.g., a heat map for each frame of the video 1110, typically at a lower resolution than the original video frame). Fig. 9 The visual display of the heat map is implemented using an encoder neural network and projection head 870) to upsample the heat map to the original resolution of the video frame. The heat map indicates high activation areas within the frame of the ultrasound video. For a well-performing SSL model, the highlighted strong activation areas are associated with clinically relevant areas (e.g., pathological areas within each frame of the video). The processing step 1120 also performs any other associated processing that prepares the output 930 and / or the ultrasound data 1110 for display, such as transparency (e.g., the heat map can be transparent), gain, changes in brightness or sharpness, smoothing over time (e.g., blending between frames), applying a color map or color key for display, etc. The processing step 1120 can also overlay the upsampled heat map directly onto the original video frame, thereby producing a highlighted video, which is then displayed on the display 1130. In some aspects, the heat map and ultrasound video can be displayed side by side instead of being overlaid, or in other configurations useful to clinicians.

[0127] Fig.12 is a schematic diagrammatic overview in block diagram form of an inference model 1100 for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure. Fig.12In the example shown, an input video stream 1210 (whether real-time, near real-time, or recorded) comprising a large number of frames 420 is fed into a machine learning algorithm 850, which produces a visual representation (e.g., a heat map) 1230. In some cases, the heat map can have the same resolution as the incoming video frame. In other cases, the heat map can be upsampled to match the resolution of the incoming video frame. In either case, the result is a stream of formatted heat maps 1240 that can be matched to the frames 420 of the incoming video stream 1210. The formatted heat maps 1240 can then be superimposed on the image frames 420 that generated them, thereby producing an output stream of highlighted video 1250. In an example, the learning algorithm can operate at a 30 Hz cycle so that an incoming video stream captured at 30 Hz can be processed and highlighted in real time (e.g., with a delay of less than 33 milliseconds between the time the image frame is captured and the time the image frame is highlighted using the heat map).

[0128] Importantly, this entire process can be trained using unlabeled data (with or without a small portion of labeled data to provide supervision). Thus, the method greatly reduces the need for expensive manual annotation.

[0129] Fig.13 is a schematic illustrative overview in block diagram form of an inference model 1100 for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure. Fig.13 The data flow is similar to Fig.12 data stream, except that only a single visual representation 1330 is generated, thereby producing a single formatted heat map 1340 which can be repeatedly superimposed on frames 420 of the incoming video stream 1210, thereby producing an output stream of highlighted video 1350.

[0130] Fig.14 is a schematic diagrammatic view of a formatted heat map 1240 superimposed on an incoming video frame 420 to produce a highlighted output frame 1450 in accordance with at least one aspect of the present disclosure.

[0131] Fig.15 1 is a flow chart of an exemplary inference mode or clinical use mode SSL video annotation method 1500 according to at least one aspect of the present disclosure. It should be understood that the steps of method 1500 can be similar to Fig.15 The method 1500 may be performed in a different order than shown, additional steps may be provided before, during, and after the steps, and / or some of the steps described may be replaced or eliminated in other aspects. One or more steps of the method 1500 may be performed by one or more devices and / or systems described herein (such as components of the ultrasound imaging system 100 and / or the processor circuit 250).

[0132] In step 1510, method 1500 includes receiving an ultrasound video of an anatomical structure acquired by an ultrasound probe (e.g., in a clinical setting). The video may be real-time, near real-time, or may be stored in a volatile or non-volatile memory. In an example, the ultrasound video includes a plurality of 2D frames over a time period of the ultrasound video. Figure 3-6 Examples of ultrasound videos, video clips, or video sub-clips are found in.

[0133] In step 1520, method 1500 includes providing the ultrasound video to a neural network for inference, for example, for highlighting features in the ultrasound video to assist clinical analysis (see above). Fig.12 ).

[0134] In step 1530, method 1500 includes directly outputting from the neural network one or more visual representations having dimensions that are the same as or different from the dimensions of the plurality of frames of the ultrasound video (see above). Fig.12 ).

[0135] In step 1540, method 1500 includes converting the dimensions of one or more visual representations to the dimensions of the ultrasound video to generate a heat map (e.g., one heat map per frame, one heat map per several frames, one heat map for the entire video, etc.) if the dimensions of the visual representation are different from the dimensions of the plurality of frames. The heat map identifies one or more locations of a pathology in one or more frames of the ultrasound video (see above Fig.12 ). In some cases, the heat map identifies the location of features or anatomical features suspected of having pathology. In other cases, the heat map identifies the absence of pathology (e.g., if no pathology is present in the image). In the example, the neural network trained using SSL does not explicitly know what, if anything, is present at the identified pixel (e.g., the dark spot of the heat map). Instead, the neural network trained using SSL knows that the image content of the pixel (the dark spot of the heat map) has different content than the image content in other pixels. In some aspects, other neural networks (e.g., classification, trained using supervised learning using labeled data) can probabilistically identify whether the dark spot in the heat map has pathology (or which pathology). In some aspects, additional neural networks are configured for at least one of classification, regression, object detection, or segmentation. These neural networks are trained using supervised learning (i.e., labeled data) using weights from SSL training as a starting point.

[0136] In step 1550, method 1100 includes providing a screen display including the heat map and / or the ultrasound video to a display. In some aspects, the heat map is superimposed on a frame of the ultrasound video (see above). Fig.14 ).

[0137] Fig.16is an exemplary representation of highlighting features in a video frame through a supervised learning algorithm according to at least one aspect of the present disclosure. Fig.16 In the example shown, input frame 420 is used to generate formatted heat map 1640 , which is then superimposed on input frame 420 to produce output frame 1650 .

[0138] Fig.17 is an exemplary representation of highlighting features in a video frame through a self-supervised learning (SSL) algorithm according to at least one aspect of the present disclosure. Fig.16 In the example shown, input frame 420 is used to generate formatted heat map 1640 , which is then superimposed on input frame 420 to produce output frame 1650 .

[0139] exist Fig.16 and Fig.17 In the specific example of , the incoming video stream is an ultrasound image of lung tissue containing consolidation (a pathological feature associated with pneumonia and other lung conditions). Fig.16 In , a standard deep learning AI model trained without SSL and using only labeled data generates a heat map 1640 that is noisy and does not clearly identify pathology, whereas in Fig.17 , the SSL ultrasound video feature detection system has clearly identified two regions of possible materialization 1710. Therefore, the SSL ultrasound video feature detection system shows better heat maps than the supervised learning process using manually labeled data.

[0140] Fig.18 18 is a graph 1800 illustrating test accuracy of partially trained machine learning models according to at least one aspect of the present disclosure. Three different machine learning models are represented: a fully supervised learning model 1810, an SSL feature extraction model 1820, and a fine-tuned SSL model 1830. Only labeled data is used to train the fully supervised learning model 1810. Only unlabeled data is used to train the SSL feature extraction model 1830. Both labeled data and unlabeled data are used to train the fine-tuned SSL model 1830.

[0141] Graph 1800 shows that when fully trained with 100% of the necessary training data, the fine-tuned SSL model 1830 produces comparable results to the fully supervised learning model 1810, while the SSL feature extraction model 1820 is approximately 5% less accurate. However, when only 20% of the training data has been input into the machine learning algorithm, the fine-tuned SSL model 1830 is already almost 80% accurate in identifying lung consolidation, while the fully supervised learning model 1810 has only 50% accuracy. Thus, the SSL ultrasound video feature detection system shows clear advantages over existing machine learning video annotation systems when large training datasets are not available.

[0142] Fig.19 1900 is a graph showing the average values ​​of test sensitivity and test specificity for a partially trained machine learning model according to at least one aspect of the present disclosure. Fig.18 , three different machine learning models are shown: a fully supervised learning model 1810, an SSL feature extraction model 1820, and a fine-tuned SSL model 1830. Graph 1800 shows that when fully trained with 100% of the necessary training data, the fine-tuned SSL model 1830 produces comparable results to the fully supervised learning model 1810, while the SSL feature extraction model 1820 is approximately 5% less accurate. However, when only 20% of the training data has been input into the machine learning algorithm, the fine-tuned SSL model 1830 is already almost 85% accurate in identifying lung consolidation, while the fully supervised learning model 1810 has only 50% accuracy. Therefore, the SSL ultrasound video feature detection system shows clear advantages over existing machine learning video annotation systems when large training datasets are not available.

[0143] Fig. 20 2000 is a graph showing the test area under the curve (AUC) of a partially trained machine learning model according to at least one aspect of the present disclosure. Three different machine learning models are represented: a fully supervised learning model 1810, an SSL feature extraction model 1820, and a fine-tuned SSL model 1830. Graph 1800 shows that when fully trained with 100% of the necessary training data, the fine-tuned SSL model 1830 produces comparable results to the fully supervised learning model 1810, while the SSL feature extraction model 1820 is about 5% less accurate. However, when only 20% of the training data has been input into the machine learning algorithm, the fine-tuned SSL model 1830 is already almost 95% accurate in identifying lung consolidation, while the fully supervised learning model 1810 has only about 60% accuracy. Therefore, when a large training data set is not available, the SSL ultrasound video feature detection system shows a clear advantage over existing machine learning video annotation systems.

[0144] When the relative amount of labeled training data is high, the SSL method has similar performance compared to the baseline (fully supervised) model. However, when the amount of labeled training data is reduced (e.g., less than 30% of the total dataset), the benefit of using the SSL method is more obvious. The fully supervised method cannot be trained well with very limited data, so the test performance is poor. However, both SSL methods still work with minor changes in performance (i.e., the accuracy of the SSL fine-tuned method remains unchanged regardless of the amount of labeled data used).

[0145] Fig.21is a schematic overview in block diagram form of an inference model 2100 for an SSL ultrasound video feature detection system according to at least one aspect of the present disclosure. The inference model 2100 is similar to Fig.11 The inference pattern 1100 shown in FIG. 1 is similar to that shown in FIG. 1 , except that the output 930 from the machine learning algorithm 850 is fed into a second machine learning algorithm 2110 (or a second portion of the machine learning algorithm 850 ) that is used as a classifier.

[0146] For example, a convolutional neural network (CNN) may additionally or alternatively be or include a multi-class classification network. The fully connected layers of a CNN may be non-linear and may gradually reduce the high-dimensional output to the dimension of the desired result (e.g., the classification output 2120). Therefore, the fully connected layer may also be referred to as a classifier. The classification output may indicate a confidence score for each class based on the input image. Classes indicating high confidence scores indicate that the input image or a portion or pixel of the image may include anatomical objects / features of the class. Conversely, classes indicating low confidence scores indicate that the input image or a portion or pixel of the image is unlikely to include anatomical objects / features of the class.

[0147] The classified output 2120 is then included in a visual processing step 2130 that generates a highlighted video for display 2140. Fig.24 An example output frame video including the classifier output is shown in . Examples of classification networks can be found, for example, in U.S. Provisional Patent Application No. 63 / 293,232, entitled “Methods and Systems for clinical Scoring alung ultrasound,” filed on December 23, 2021, and U.S. Provisional Patent Application No. 63 / 294,501, entitled “Machine-learning image processing independent of reconstruction filter,” filed on December 29, 2021, each of which is incorporated by reference as if fully set forth herein.

[0148] In an example, a multi-class classification network may include an encoder path that utilizes a convolutional layer to process an incoming image frame so that the size is reduced. The resulting low-dimensional representation of the image may be used to generate a fully connected layer to regress and output one or more categories 542. In some aspects, a fully connected layer may process the output of an encoder or convolutional layer. Additionally, the fully connected layer 530 may be referred to as a task layer or a regression layer, etc.

[0149] In some cases, the two machine learning algorithms 850 and 2110 may be combined so that there is no intermediate output 930. Instead, the combined machine learning algorithm 850 may directly display the processed image.

[0150] Reference again Fig.18 , Fig.19 and Fig. 20 , the fully supervised learning model 1810, the SSL feature extraction model 1820, and the fine-tuned SSL model 1830 may be examples for providing the output of the SSL-trained neural network to the classifier neural network, such as Fig.21 For example, during training, once an SSL is trained (e.g., Figure 8 and Fig. 9 850 and neural network 870 in the classification neural network), then the fully connected layer after the output (for example, of the neural network 870) is classified. In this task, the fully supervised learning model 1810, the SSL feature extraction model 1820, and the fine-tuned SSL model 1830 can be trained for classification according to whether the weights trained by SSL are used as initial weights and whether the weights of the neural network 850 and the neural network 870 are frozen. The fully supervised learning model 1810, the SSL feature extraction model 1820, and the fine-tuned SSL model 1830 are trained using labeled classification labels. The fully supervised learning model 1810 does not use the weights trained by SSL, and the weights of the neural network 850 and the neural network 870 are not frozen when training the classification neural network. The SSL feature extraction model 1820 uses the weights trained by SSL, and the weights of the neural network 850 and the neural network 870 are frozen when training the classification neural network. The fine-tuned SSL model 1830 uses the weights trained by SSL, and the weights of the neural network 850 and the neural network 870 are not frozen when training the classification neural network.

[0151] Fig. 22 is a schematic diagrammatic overview in block diagram form of an inference model 2200 for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure. The inference model 2200 is similar to Fig.11 , except that the output 930 from the machine learning algorithm 850 is fed into a second machine learning algorithm 2210 (or a second portion of the machine learning algorithm 850) used as a regressor.

[0152] In an example, the fully connected layers of the CNN may be non-linear and may gradually reduce the high-dimensional output to the dimension of the desired result (e.g., regression output 2220). The regression output 2220 is then included in a visual processing step 2230 that generates a highlighted video for display 2240. Fig.25An example output frame video including regression output is shown in . An example of a regression network can be found in U.S. Provisional Patent Application 63 / 293,215, entitled “Methods and systems for clinical scoring of a lung ultrasound,” filed on December 23, 2021, which is incorporated by reference as if fully set forth herein.

[0153] In some cases, the two machine learning algorithms 850 and 2210 may be combined so that there is no intermediate output 930. Instead, the combined machine learning algorithm 850 may directly display the processed image.

[0154] Fig.23 is a schematic diagrammatic overview in block diagram form of an inference model 2300 for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure. The inference model 2300 is similar to Fig.11 , except that the output 930 from the machine learning algorithm 850 is fed into a second machine learning algorithm 2310 (or a second portion of the machine learning algorithm 850) that functions as an object detector, bounding box generator, or segmentor / segmenter.

[0155] For example, a heat map can show the relative importance of different regions of an image (typically at a pixel-by-pixel level) as a function of the model input. Heat maps are not necessarily specific to one or more target classes, although they may be if the mechanism used to generate the heat maps is conditioned on the class. In contrast, bounding boxes indicate “positive” regions where objects of one or more target classes have been detected. Bounding boxes do not provide pixel-by-pixel information across the entire image. Examples of bounding box object detection can be found in Indian Patent Application No. 202141034243 (International Application No. PCT / EP2022 / 070410), filed on July 29, 2021, entitled “Generating location data,” which is incorporated by reference as if fully set forth herein.

[0156] A related output type is segmentation, which is similar to a binarized version of a heatmap or a pixel-by-pixel implementation of a bounding box. Segmentation is also pixel-by-pixel, where each pixel is classified into one or more target categories. The output is therefore a pixel mask that "color codes" different regions based on which category they belong to. Examples of image or video segmentation can be found, for example, in U.S. Provisional Patent Application No. 63 / 325,660, entitled "Methods and systems for ultrasound-based structure localization using image and user inputs," filed on March 31, 2022, and U.S. Patent Publication No. 2022 / 0198669, entitled "Segmentation and view guidance in ultrasound imaging and associated devices, systems, and methods," each of which is incorporated by reference as if fully set forth herein.

[0157] Furthermore, a convolutional neural network may be additionally or alternatively trained to recognize features within an image. In an example, the weights of the encoder neural network trained with SSL may also be a good initialization for downstream feature video detection tasks (e.g., bounding box detection), another important task for ultrasound applications. The fully connected layers of the CNN may be non-linear and may gradually reduce the high-dimensional output to the dimensions of the desired result (e.g., bounding box output 2320).

[0158] Bounding box 2320 is then included in a visual processing step 2330 that generates a highlighted video for display 2340. Fig.26 An example output frame video including bounding box output is shown in FIG.

[0159] Therefore, the encoder neural network can be followed by an object detection layer (e.g., based on YOLO), which can be trained to detect solid regions given bounding box labels. The SSL-trained encoder contains meaningful visual representations, which helps the network to locate important regions based on ultrasound videos, thus outperforming the baseline detection model if trained from scratch in detection performance.

[0160] In some cases, the two machine learning algorithms 850 and 2310 may be combined so that there is no intermediate output 930. Instead, the combined machine learning algorithm 850 may directly display the processed image.

[0161] Fig.2424 is an exemplary screen display 2440 including both a highlighted image frame 1450 and a classification output 2120 in accordance with at least one aspect of the present disclosure. For example, the classification output may report whether a given anatomical structure or pathology is believed to be present in the image frame.

[0162] Fig.25 25 is an exemplary screen display 2540 including both a highlighted image frame 1450 and a regression output 2220 (e.g., a numerical score indicating the probability or severity of a detected pathology) in accordance with at least one aspect of the present disclosure. For example, the regression output may report a severity value (within a range of values) for a given pathology believed to be present in the image frame.

[0163] Fig.26 2640 is an exemplary screen display including both a highlighted image frame 1450 and object detection output according to at least one aspect of the present disclosure. For example, the object detection output may be or include a bounding box 2320 indicating a region of interest and / or a label 2321 identifying the region of interest.

[0164] Fig. 27 is a schematic illustrative overview in block diagram form of at least a portion of a training mode 900 for an SSL ultrasound video feature detection system in accordance with at least one aspect of the present disclosure. Fig. 27 The data flow is similar to Fig. 9 , except that for clarity only one branch (eg, 3D enhancement 720e) is shown.

[0165] However, Fig. 27 The training mode 900 includes training when some labeled data is included. The labeled data may include annotations with labeled boxes in one or more frames of the input video 420 (e.g., see Figure 4 ). In such cases, the machine learning algorithm 850 passes the labeled data to a detector head 2710 (e.g., a YoloV3 learning network, a neural network, or other machine learning algorithm), which outputs a series of bounding boxes 2730, each with an x ​​position, y position, width, and height (e.g., all measured in pixels or millimeters). Each bounding box 2730 may also include a confidence level or confidence score. These bounding boxes are then used for joint consistency and detection losses in training, although not in inference mode.

[0166] For the detector / labeled data branch, the output is a bounding box prediction (x, y, width, height). These predicted boxes are compared to the ground truth boxes using an IOU loss (a metric that measures the overlap of boxes). During this training, the weights of the CNN (850) need to be adjusted to maximize the overlap while maximizing the consistency between the outputs from the two unlabeled branches.

[0167] When both labeled and unlabeled data are available, the training mode 900 has two loss parts to optimize (the loss of the detector head 2710 of the labeled box from the labeled data and the SSL loss for consistency from the two branches of the unlabeled data). The two losses are added together to train the model. The training minimizes the total loss by adjusting the weight values ​​of the neural network(s). In some cases, weights can be added (e.g., multiplied by a given loss that is different from the weight values ​​of the neural network) to control the contribution from each part of the loss so that the sum of the losses becomes a weighted sum.

[0168] Despite Fig. 27 are not shown, but these features (when present) are replicated for Fig. 9 Another branch of (e.g., 3D enhancement 720f).

[0169] As one of ordinary skill in the art will readily appreciate after becoming familiar with the teachings herein, the SSL ultrasound video feature detection system advantageously allows highly accurate video annotation machine learning algorithms to be trained in less time and with less labor and expense using primarily or solely unlabeled training data.

[0170] Many variations are possible in the above examples and aspects. For example, the systems, methods, and devices described herein are not limited to lung ultrasound applications. Instead, the same techniques can be applied to images of other organs or anatomical systems (such as the heart, brain, digestive system, vascular system, etc.). In addition, the techniques disclosed herein are also applicable to other medical imaging modalities where 3D data is available, such as other ultrasound applications, camera-based videos, X-ray videos, and 3D volumetric images, such as computer-assisted tomography (CT) scans, magnetic resonance imaging (MRI) scans, optical coherence tomography (OCT) scans, or intravenous ultrasound (IVUS) retraction sequences. The output of the SSL-trained AI model is a heat map that highlights areas of strong network activation associated with features of clinical relevance (e.g., pathology) in dynamic cycles. The heat map output is easily detectable. 3D-enhanced dynamic cycles can be provided to the SSL-trained AI to test whether the model predictions are sensitive to these enhancements. If the AI ​​predictions are robust to these 3D enhancements, there is a high likelihood of using 3D spatial and temporal enhancements. The technology described in this article can be used in a variety of settings, including emergency departments, critical care, inpatient and out-of-hospital settings.

[0171] Therefore, the logical operations constituting the various aspects of the technology described herein are variously referred to as operations, steps, objects, layers, elements, components, algorithms, or modules. Furthermore, it should be understood that these may occur or be executed or arranged in any order unless otherwise explicitly stated or a specific order is inherently required by the claim language.

[0172] All directional references (e.g., up, down, inside, outside, upward, downward, left, right, lateral, front, back, top, bottom, above, below, vertical, horizontal, clockwise, counterclockwise, proximal and distal) are used for identification purposes only to help the reader understand the claimed subject matter and do not create limitations, particularly with respect to the location, orientation or use of the SSL ultrasound video feature detection system. Connection references (e.g., attached, coupled, connected, joined or "communicating") should be interpreted broadly and, unless otherwise specified, may include intermediate members between sets of elements and relative movement between elements. Thus, connection references do not necessarily cover two elements being directly connected and in a fixed relationship to each other. The term "or" should be interpreted to mean "and / or" rather than "exclusive or". The word "comprising" does not exclude other elements or steps, and the indefinite article "one" or "an" does not exclude a plurality. Unless otherwise indicated in the claims, the values ​​should be interpreted as merely illustrative and should not be considered limiting.

[0173] The above specification, examples, and data provide a complete description of the structure and use of an exemplary embodiment of an SSL ultrasound video feature detection system as defined in the claims. Although various embodiments of the claimed subject matter have been described above with a certain degree of particularity or with reference to one or more individual embodiments, those skilled in the art may make many changes to the disclosed embodiments without departing from the spirit or scope of the claimed subject matter.

[0174] Other embodiments are also contemplated. It is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative only of particular embodiments and not limiting. Changes may be made in detail or structure without departing from the essential elements of the subject matter defined in the appended claims.

Claims

1. A system comprising: monitor; as well as a processor configured to communicate with the display, wherein the processor is configured to: receiving an ultrasound video of an anatomical structure acquired by an ultrasound probe, wherein the ultrasound video includes a plurality of frames; providing the ultrasound video to a neural network during inference, wherein the neural network is trained using self-supervised learning using an unlabeled ultrasound video, the unlabeled ultrasound video comprising at least one of: a plurality of spatial enhancements or a plurality of temporal enhancements; generating a heat map using the neural network, the heat map identifying the location of pathology in one or more of the plurality of frames; and A screen display including the heat map is provided to the display.

2. The system according to claim 1, wherein: Only unlabeled ultrasound videos are used to train the neural network.

3. The system according to claim 1, wherein: The screen display also includes the ultrasound video.

4. The system according to claim 3, wherein: The heat map is superimposed on the one or more frames of the plurality of frames of the ultrasound video.

5. The system according to claim 4, in, The heat map comprises a single heat map, Wherein, the single heat map is superimposed on each of the multiple frames of the ultrasound video.

6. The system according to claim 4, in, The processor is configured to generate a plurality of heat maps, and The screen display includes the multiple heat maps respectively superimposed on multiple frames of the ultrasound video.

7. The system according to claim 1, in, The direct output of the neural network comprises a visual representation associated with the one or more frames of the ultrasound video, and Wherein a dimension of the visual representation is different from a dimension of the plurality of frames of the ultrasound video.

8. The system according to claim 7, wherein: To generate the heat map, the processor is configured to convert the dimensions of the visual representation to the dimensions of the ultrasound video.

9. The system according to claim 7, wherein: The processor is configured to: providing the visual representation to a further neural network configured for at least one of classification, regression, object detection, or segmentation; and generating an output using the additional neural network, wherein the output is associated with at least one of the classification, the regression, the object detection, or the segmentation, and wherein the output is different from the visual representation and the heat map; and Wherein, the screen display includes a visualization based on the output.

10. The system according to claim 1, Also includes the ultrasound probe, in, The processor is configured to control the ultrasound probe to obtain the ultrasound video.

11. A system comprising: a memory comprising an ultrasound video of an anatomical structure acquired by an ultrasound probe, wherein the ultrasound video is unlabeled and comprises a plurality of frames; a processor configured to communicate with the memory, wherein the processor is configured to: retrieving the ultrasound video from the memory; performing a first enhancement on the ultrasound video to generate a first enhanced ultrasound video; performing a second enhancement on the ultrasound video to generate a second enhanced ultrasound video, wherein the first enhancement and the second enhancement include at least one of: spatial enhancement on the plurality of frames or temporal enhancement on the plurality of frames; training a neural network having a first plurality of weights using the first enhanced ultrasound video and the second enhanced ultrasound video, wherein the training of the neural network comprises self-supervised learning; and providing a second plurality of weights to the neural network after the training, wherein the neural network having the second plurality of weights is configured to be implemented during inference to identify a pathology associated with the anatomical structure, Wherein, in order to train the neural network, the processor is configured to: generating a first output associated with the neural network based on the first enhanced ultrasound video; generating a second output associated with the neural network based on the second enhanced ultrasound video; and The second plurality of weights is determined based on a comparison between the first output and the second output.

12. The system according to claim 11, wherein: To train the neural network, the processor is configured to implement a first processing path associated with a first enhanced ultrasound video and a second processing path associated with the second enhanced ultrasound video.

13. The system according to claim 11, in, The memory also includes a plurality of ultrasound videos, Wherein, the processor is configured to: Obtaining the plurality of ultrasound videos from the memory; and training the neural network based on the plurality of ultrasound videos, and Among them, most of the multiple ultrasound videos are unlabeled.

14. The system according to claim 13, wherein: All of the plurality of ultrasound videos are unlabeled.

15. The system according to claim 11, in, The spatial enhancement comprises changes to how image content is depicted in the plurality of frames, and The time enhancement includes changing the arrangement order of the multiple frames.

16. The system according to claim 11, in, The first enhancement is different from the second enhancement, and The ultrasound video, the first enhanced ultrasound video and the second enhanced ultrasound video are different from each other.

17. The system according to claim 11, in, The first output associated with the neural network comprises a first one-dimensional (1D) projected feature set, wherein the second output associated with the neural network comprises a second set of 1D projection features, and The processor is configured to determine the second plurality of weights to maximize consistency between the first 1D projection feature set and the second 1D projection feature set.

18. The system according to claim 17, in, To train the neural network, the processor is configured to: Before generating the first output, generating a first initial output of the neural network based on the first enhanced ultrasound video; and before generating the second output, generating a second initial output of the neural network based on the second enhanced ultrasound video, wherein the first initial output comprises a first plurality of two-dimensional (2D) visual representations, and Wherein the second initial output comprises a second plurality of 2D visual representations.

19. The system of claim 18, wherein: To train the neural network, the processor is configured to: compressing the first plurality of 2D visual representations using a further neural network to generate the first set of 1D projection features, and The second plurality of 2D visual representations are compressed using the further neural network to generate the second set of 1D projection features.

20. The system according to claim 19, in, The neural network comprises a convolutional neural network, and Wherein, the additional neural network comprises a multilayer perceptron.

Images