System and method for processing electronic medical images for diagnostic or interventional applications - Patents.com

Abstract

Disclosed herein are systems and methods for processing ultrasound images to identify objects for diagnostic and / or interventional applications. For example, an ultrasound image of an anatomical structure may be received from a computing device of an ultrasound imaging system. The ultrasound image may be input to a machine learning model trained to identify a plurality of objects in the ultrasound image of the anatomical structure. The plurality of objects may include anatomical features, destructive features, and / or instruments. A prediction of one or more objects from the plurality of objects identified in the ultrasound image may be received as an output of the machine learning model. An indication of the prediction may be provided to the computing device for display on a display of the ultrasound imaging system.

Description

[Technical field]

[0001] Various techniques presented herein relate generally to the processing of electronic medical images to provide clinical diagnoses, measurements, and / or observations using artificial intelligence (AI). More specifically, certain techniques of the present disclosure relate to systems and methods for training and using machine learning models to predict clinical diagnoses, measurements, and / or observations related to musculoskeletal disorders from diagnostic or interventional ultrasound images. [Background technology]

[0002] Musculoskeletal disorders have a significant impact on quality of life both in the United States and worldwide. Imaging of anatomical structures affected by musculoskeletal disorders may be used to facilitate clinical diagnosis of the disorder and / or as part of (e.g., guide) an intervention to treat the disorder. However, patients are becoming concerned about the overuse and / or misuse of expensive advanced medical imaging such as computed tomography (CT), magnetic resonance imaging (MRI) and positron emission tomography (PET), subjecting patients to scheduling delays, additional costs, and unnecessary radiation exposure. Thus, alternative imaging techniques such as ultrasound may be increasingly used for diagnostic and interventional imaging related to musculoskeletal disorders.

[0003] The foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit the present disclosure. The background description provided herein is intended to generally present the context of the present disclosure. Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application and is not admitted to be prior art or a proposal of prior art by inclusion in this section. Summary of the Invention

[0004] According to certain aspects of the present disclosure, systems and methods are disclosed for processing electronic images, such as ultrasound images, to identify objects for diagnostic and / or interventional applications.

[0005] In one exemplary aspect, a system for processing ultrasound images to identify objects is described. The exemplary system may include a processor and a memory coupled to the processor. The memory may store instructions that, when executed by the processor, cause the system to perform operations. The operations may include receiving an ultrasound image of an anatomical structure from a computing device of an ultrasound imaging system, providing the ultrasound image as an input to a machine learning model trained to identify a plurality of objects in the ultrasound image of the anatomical structure. The plurality of objects may include anatomical features, destructive features, and / or instruments. The operations may further include receiving a prediction of one or more objects from the plurality of objects identified in the ultrasound image as an output of the machine learning model, and providing an indication of the prediction to the computing device for display on a display of the ultrasound imaging system.

[0006] In another exemplary aspect, a method for processing an ultrasound image to identify an object is described. The exemplary method may include receiving an ultrasound image of an anatomical structure from a computing device of an ultrasound imaging system, and providing the ultrasound image as an input to a machine learning model trained to identify a plurality of objects in the ultrasound image of the anatomical structure. The plurality of objects may include anatomical features, destructive features, and / or instruments. The method may further include receiving a prediction of one or more objects from the plurality of objects identified in the ultrasound image as an output of the machine learning model, and providing an indication of the prediction to the computing device for display on a display of the ultrasound imaging system.

[0007] In a further exemplary aspect, a non-transitory computer-readable medium for processing an ultrasound image to identify an object is described. The exemplary non-transitory computer-readable medium may store instructions that, when executed by a processor, cause the processor to perform operations for processing an ultrasound image to identify an object. The operations may include receiving an ultrasound image of an anatomical structure from a computing device of an ultrasound imaging system, providing the ultrasound image as an input to a machine learning model trained to identify a plurality of objects in the ultrasound image of the anatomical structure. The plurality of objects may include anatomical features, destructive features, and / or instruments. The operations may further include receiving a prediction of one or more objects from the plurality of objects identified in the ultrasound image as an output of the machine learning model, and providing an indication of the prediction to the computing device for display on a display of the ultrasound imaging system.

[0008] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed. [Brief description of the drawings]

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments. [Figure 1] FIG. 1 is an example block diagram of a network environment for processing electronic medical images according to techniques described herein. [Diagram 2] 1 is a block diagram of an example process performed by a medical image processing system in accordance with techniques described herein. [Diagram 3] 1 is a flowchart illustrating an example method for training a machine learning model to predict a diagnosis of a musculoskeletal disorder, according to the technology described herein. [Figure 4]1 is a flow chart illustrating an exemplary method for predicting a diagnosis of a musculoskeletal disorder, according to the technology described herein. [Figure 5A] FIG. 1 is a conceptual diagram depicting the process of training a machine learning model to predict a diagnosis of a shoulder injury according to techniques described herein. [Figure 5B] FIG. 1 illustrates an example application user interface displaying a predicted diagnosis of a shoulder injury according to the techniques described herein. [Figure 6] 1 is a flowchart illustrating an example method for training a machine learning model to identify objects in medical images of anatomical structures, according to techniques described herein. [Figure 7] 4 is a flowchart illustrating an example method for identifying objects in a medical image of an anatomical structure, according to techniques described herein. [Figure 8] 1 is a flowchart illustrating an example method for training a machine learning model to identify optimal images of objects identified in medical images of anatomical structures, according to techniques described herein. [Figure 9] 5 is a flowchart illustrating an example method for determining whether a medical image is a best image of an object identified therein, according to techniques described herein. [Figure 10] 1 illustrates an example application user interface displaying objects identified in a medical image that may be displayed in accordance with the techniques presented herein. [Figure 11] 1 is a flowchart illustrating an example method for training a machine learning model to predict an instrument position according to techniques described herein. [Figure 12] 1 is a flowchart illustrating an example method for predicting a position of an instrument, according to techniques described herein. [Figure 13] 1 is a flowchart illustrating an example method for training a machine learning model to predict an instrument trajectory in accordance with techniques described herein. [Figure 14]1 is a flowchart illustrating an example method for predicting an instrument trajectory in accordance with techniques described herein. [Figure 15] 11A-11C illustrate an example application user interface displaying predicted instrument position and trajectory in accordance with the techniques described herein. [Figure 16] 1 is a flowchart illustrating an example method for training a machine learning model to identify anatomical features, according to techniques described herein. [Figure 17] 1 is a flowchart illustrating an example method for visualizing anatomical features of an identified anatomical structure in a medical image, according to techniques described herein. [Figure 18] FIG. 1 illustrates an example application user interface displaying visualization information of anatomical features of an anatomical structure identified in a medical image of the anatomical structure in accordance with the techniques described herein. [Figure 19] 1 is a flowchart illustrating an example method for training a machine learning model to detect probe alignment with anatomical features identified in a medical image, according to techniques described herein. [Figure 20] 1 is a flowchart illustrating an example method for detecting probe alignment with one or more anatomical features of an anatomical structure identified in a medical image of the anatomical structure, according to techniques described herein. [Figure 21] 1 is a flowchart illustrating an example method for training a machine learning model to identify optimal image frames, according to techniques described herein. [Figure 22] 4 is a flowchart illustrating an example method for capturing optimal image frames in accordance with techniques described herein. [Figure 23] FIG. 1 illustrates an example system that may perform the techniques presented herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or like parts.

[0011] The systems, devices, and methods disclosed herein are described in detail by way of example and with reference to the drawings. The examples described herein are merely illustrative and are provided to aid in the explanation of the apparatus, devices, systems, and methods described herein. None of the features or components shown in the drawings or described below should be considered essential for any particular embodiment of any of these devices, systems, or methods, unless specifically designated as essential.

[0012] Also, with respect to any method described, whether or not the method is described in conjunction with a flow diagram, unless otherwise specified or required by context, it should be understood that any explicit or implicit ordering of steps performed in performing the method does not imply that the steps must be performed in the order presented, but instead may be performed in a different order or in parallel.

[0013] As used herein, the term "exemplary" is used in the sense of "example" rather than "ideal." Additionally, the terms "a" and "an" as used herein denote the presence of one or more of the referenced items, but rather than denote a limitation of quantity.

[0014] In diagnostic imaging, a physician may evaluate electronic images during a patient's imaging examination to facilitate a diagnosis of a disorder, injury, and / or condition, including any classification thereof (e.g., category, stage, phase, grade, etc.). In interventional imaging, a physician may utilize electronic images during a procedure, for example, to visualize an instrument inserted into the patient's body to assist the physician in safely guiding the instrument to the intended target area. Traditionally, advanced medical imaging such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) have been used for diagnostic and / or interventional imaging. However, alternative imaging techniques such as ultrasound may also be used for diagnostic and interventional imaging, particularly for musculoskeletal disorders. Ultrasound imaging systems are more readily available due to their lower cost (e.g., a physician may utilize multiple ultrasound systems, whereas only one CT imaging system), which reduces scheduling delays and provides lower costs to patients. In addition, ultrasound imaging avoids exposing patients to radiation. Ultrasound devices may also be portable, allowing the devices to be delivered to remote locations or otherwise allow for more rapid transportation or access in remote or difficult geographic areas.

[0015] The techniques disclosed herein provide AI tools for diagnostic and / or interventional ultrasound imaging. For example, multiple machine learning models may be trained and deployed to predict diagnoses, observations, and / or measurements related to musculoskeletal disorders from ultrasound images. Exemplary observations may include identification of objects in ultrasound images, such as anatomical features, features not normally present in the anatomy that may disrupt bodily functions, and / or foreign objects (e.g., instruments) inserted into the body. Observations may also include predictions of object locations and / or trajectories, and / or predictions of whether an optimal image of an object has been captured. Visualization information based on the predictions may be generated and provided to a physician in real time as the physician is performing a diagnostic test on a patient and / or as the physician is performing an ultrasound-guided procedure that may be for treating a diagnosed disorder.

[0016] Although specific examples included throughout this disclosure include ultrasound imaging, it should be understood that the techniques according to the present disclosure may be adapted to other types of imaging modalities, for example, the techniques may be adapted to any medical imaging modality, such as MRI, CT, PET, X-ray, etc.

[0017] environment 1 illustrates an example block diagram of an environment 100 for processing electronic medical images in accordance with an example technique of this disclosure. The environment may include a server system 110 in communication with one or more imaging systems 130, one or more user computing devices 140, one or more picture archiving and communication (PAC) systems 150, and / or a medical imaging database 160 via a network 120.

[0018] The server system 110 may include a processing device 112 and a storage device 114. The processing device 112 may be configured to implement a medical image processing system 116 (hereinafter system 116). The system 116 may apply AI, machine learning, and / or image processing techniques to medical images received over the network 120, for example, from the imaging system 130, the user computing device 140, or the PAC system 150. Alternatively, the system 116, the trained machine learning model 118, or other features described with the server system 110 may be located with the imaging system 130 itself. Additionally, techniques described herein as being performed by the system 116 may be performed, for example, by the imaging computing device 136 or the user computing device 140.

[0019] For example, the system 116 may include a training image platform configured to generate and train a plurality of trained machine learning models 118 based on a dataset of training medical images received, for example, from one or more medical imaging databases 160 via the network 120. The training medical images may be images of human and / or animal (e.g., in a veterinary context) anatomical structures. The training medical images may be real images or synthetically generated images to compensate for data sparsity as needed. The received training medical images may be annotated by physicians and / or other medical professionals. For a given training medical image of an anatomical structure, an annotation may be made of anatomical features of the anatomical structure, features not normally present in the anatomical structure that may disrupt the function of the body, foreign objects, measurements related to features and / or the body, diagnoses identifiable from the image, and / or image view type (e.g., probe orientation), as described in detail elsewhere herein. The training medical images may be annotated using one or more of known or future data annotation techniques, such as polygons, brushes / erasers, bounding boxes, keypoints, keypoint skeletons, lines, ellipses, cuboids, classification tags, attributes, instance / object tracking identifiers, free text, and / or direction vectors, to train any one or more of known or future model types, such as image classifiers, video classifiers, image segmentation, object detection, object orientation, instance segmentation, semantic segmentation, volumetric segmentation, composite objects, keypoint detection, keypoint mapping, 2D / 3D and 6 DOF object pose, pose estimation, regression networks, ellipsoid regression, 3D cube estimation, optical character recognition, text detection, and / or artifact detection.

[0020] The trained machine learning models 118 may include convolutional neural networks (CNNs), support vector machines (SVMs), generative adversarial networks (GANs), and / or other similar types of models trained using supervised, unsupervised, and / or reinforcement learning techniques. For example, as used herein, a "machine learning model" generally encompasses instructions, data, and / or models configured to receive an input and apply one or more of weights, biases, classifications, or analyses to the input to generate an output. The output may include, for example, a classification of the input, an analysis based on the input, a design, process, prediction, or recommendation related to the input, or any other suitable type of output. A machine learning system or model may be trained using training data (e.g., empirical data and / or samples of input data) that is supplied to the system to establish, adjust, or modify one or more aspects of the system (e.g., weights, biases, criteria for forming classifications or clusters, etc.). The training data may be generated, received, and / or otherwise obtained from internal or external resources. Aspects of the machine learning system may operate on inputs linearly, in parallel, via a network (e.g., a neural network), or via any suitable configuration.

[0021] The implementation of the machine learning system may include the introduction of one or more machine learning techniques, such as linear regression, logistic regression, random forest, gradient boosted machine (GBM), deep learning, and / or deep neural networks (e.g., multi-layer perceptron (MLP), CNN, recurrent neural networks). Supervised and / or unsupervised training may be used. For example, supervised learning may include providing training data and labels corresponding to the training data, e.g., as ground truth. The training data may include medical images annotated by human technicians and / or other medical professionals. Unsupervised techniques may include clustering, classification, etc. K-means clustering or K-nearest neighbors may also be used, which may be supervised or unsupervised. A combination of K-nearest neighbors and unsupervised cluster techniques may also be used. Any suitable type of training may be used, such as, for example, stochastic, gradient boosted, random seed, recursive, epoch-based, or batch-based. Alternatively, training may be performed using reinforcement learning. For example, reinforcement learning may involve training an agent to interact with an environment to make decisions based on the current state of the environment, receive feedback (e.g., positive or negative rewards based on the accuracy of the decision), adjust its decisions to maximize the rewards, and repeat again until a loss function is optimized.

[0022] The trained machine learning models 118 may be stored by the storage device 114 to enable subsequent retrieval and use by the system 116, for example, when a medical image is received for processing. In other techniques, a third party system may generate and train multiple trained machine learning models 118. The server system 110 may receive the trained machine learning models 118 from the third party system and store them in the storage device 114.

[0023] The imaging system 130 may include systems implementing multiple different imaging modalities. For example, one of the imaging systems 130 may be an ultrasound imaging system 132. The ultrasound imaging system 132 may include one or more probes 134 (e.g., transducers), an imaging computing device 136 communicatively connected to the probe 134, and a display 138. When the probe is placed in contact with the skin of a patient (human or animal) near the anatomical structure to be imaged, it emits sound waves into the patient's body and receives reflected sound waves from which an image may be created by the imaging computing device 136. For example, the probe 134 may generate an electrical signal based on the reflected sound waves, which may be transmitted to the imaging computing device 136 to generate an image. The image may then be displayed on the display 138. The frequency and depth at which the sound waves are transmitted by the probe 134 may be adjustable settings of the ultrasound imaging system 132. The image may be a live image. Control of the ultrasound imaging system 132 may allow live images to be frozen and captured as still images. Other exemplary imaging systems implement an X-ray imaging system, a CT system, an MRI system, and / or a PET system.

[0024] In some examples, images generated by the imaging system 130 may be transmitted over the network 120 to a user computing device 140 for review by a physician. For example, after imaging of a patient is performed using the ultrasound imaging system 132 (e.g., by a technician qualified to operate the ultrasound imaging system 132), the generated images may be transmitted to one or more user computing devices 140 (e.g., a physician's computing device) for initial analysis. The user computing devices 140 may include desktop computers, laptop computers, tablets, smart cellular phones (e.g., mobile phones), smart watches, or other electronic wearables, etc. Additionally or alternatively, images generated by the imaging system 130 may be transmitted over the network 120 to one of the PAC systems 150 for storage.

[0025] At least a portion of the one or more instructions stored in memory of the imaging computing device 136 of the ultrasound imaging system 132 and / or the memory of the user computing device 140 may include instructions for executing an application (e.g., a client application) associated with the system 116 configured to communicate with the server system 110 over the network 120. As one illustrative example, when a patient is being imaged using the ultrasound imaging system 132, the application may execute on the imaging computing device 136 to enable real-time processing by the system 116 of generated images. As another illustrative example, the application may execute on the user computing device 140, and a user (e.g., a physician) may select previously captured and stored images (e.g., from the PAC system 116) for processing by the system 150. In some examples, the application may be capable of capturing and processing voice commands from the physician (e.g., via a voice interface).

[0026] Additionally, one or more components of imaging computing device 136 and / or computing device 140 may generate or cause to be generated one or more graphic user interfaces (GUIs) based on instructions / information stored in memory, instructions / information received from other systems in environment 100, and / or the like, and cause the GUIs to be displayed via the respective device's display. The GUI may be, for example, a mobile application interface or a browser user interface, and may include text, input text boxes, selection controls, and / or the like. The display may include a display with a touch screen or other input system (e.g., mouse, keyboard, etc.) for the respective device's management contacts and / or guest contacts to control their functions.

[0027] The network 120 over which one or more components of the environment 100 communicate may include one or more wired and / or wireless networks such as a wide area network ("WAN"), a local area network ("LAN"), a personal area network ("PAN"), a cellular network (e.g., 3G network, 4G network, 5G network, etc.), or the like. In one technique, the network 120 includes the Internet, and the information and data provided between the various systems occurs online. "Online" may mean connecting to or accessing source data or information from a location remote from other devices or networks coupled to the Internet. Alternatively, "online" may refer to connecting to or accessing an electronic network (wired or wireless) via a mobile communication network or device. The server system 110, the imaging system, the computing device 140, the PAC system 150, and / or the medical image database 160 may be connected over the network 120 using one or more standard communication protocols.

[0028] 1 as separate components, it should be understood that components or portions of components within the systems of the exemplary environment 100 may, in some embodiments, be integrated with or incorporated into one or more other components. For example, the display 138 may be integrated with the imaging computing device 136, such as an ultrasound imaging system. In some embodiments, the operations or aspects of one or more of the components described above may be distributed among one or more other components. Any suitable arrangement and / or integration of the various systems and devices of the exemplary environment 100 may be used.

[0029] In the disclosure that follows, various operations may be described as being performed or carried out by components of FIG. 1, such as server system 110, imaging system, computing device 140, or components thereof. However, it should be understood that in various embodiments, various components of the exemplary environment 100 described above may execute instructions or perform operations including those described below. Operations performed by a device may be considered to be performed by a processor, actuator, etc. associated with that device. Furthermore, it should be understood that in various embodiments, various steps may be added, omitted, and / or rearranged in any suitable manner.

[0030] A high-level overview of medical image processing FIG. 2 illustrates a block diagram of an exemplary process 200 performed by the system 116. The process 200 may begin when an input 202 is received at the system 116. The input 202 may include one or more medical images of an anatomical structure. The medical images may be received over the network 120 by an application associated with the system 116 running on a computing device, such as the imaging computing device 136 and / or the user computing device 140. Next, in step 204, the input 202 may be preprocessed. As part of the preprocessing, any relevant health data may be extracted from the medical image (e.g., to de-identify) and the medical image may be converted to a lossless image format, such as portable graphics format (PNG). Additionally, in some examples, the medical image may be fed through a generative adversarial network (GAN) to increase image clarity (e.g., to reconstruct and denoise the image).

[0031] The pre-processed medical image of the anatomical structure may then be provided as input to a trained machine learning model 118A from a plurality of machine learning models 118 stored in the storage device 114. The trained machine learning model 118 may be trained to predict at least one of a diagnosis of the musculoskeletal system affecting the anatomical structure, an anatomical feature of the anatomical structure, a feature not normally present in the body that may disrupt the function of the body, and / or an observation related to one or more objects, such as a foreign object inserted into the body, and / or a measurement related to the observation, such as an area or volume of the feature and / or body. The observation related to the object may include identification and / or contouring / visual representation of the object or region of interest, the object's location, the object's trajectory, and / or image quality (e.g., is it an optimal image of a given object, such as appropriate image depth, image focal zone, and image gain, and recognition of sonography artifacts including anisotropy, shadowing, refractive shadowing, back echo enhancement or increased transmission, back reflection, and ring-down artifacts). Also, one or more of the trained machine learning models 118 may be trained to predict optimal image frame capture for a given diagnostic or interventional procedure.

[0032] The predicted output by the trained machine learning model 118A may then be post-processed in step 206 to generate the output 208. Additionally or alternatively (as indicated by the dotted line), the predicted output by the trained machine learning model 118A may be provided as an input to another trained machine learning model 118B from multiple machine learning models 118 stored in the storage device 114, whose output may also be post-processed in step 206 to generate the output 208. Although only one or two of the trained machine learning models 118 are shown in the process 200, in other examples, any number of trained machine learning models may be implemented.

[0033] Post-processing in step 206 may, for example, generate results based on the trained machine learning model 118A and / or the prediction(s) from the trained machine learning model to generate output 208. In other words, post-processing step 206 converts the predictions into an information format and / or display consumable by a physician or other medical professional. Exemplary information formats and / or displays may include heat maps, text overlays superimposed on images, numeric table formats, league table formats, text tables, highlight tables, and / or bar graphs. In some examples, an external (e.g., third-party) system may be utilized to generate the results.

[0034] One exemplary result may include visualization information showing predictions within the medical image to assist a physician in making a diagnosis and / or intervention. The system 116 may provide these visualization information via an application for display on a computing device from which the medical image is received, such as the imaging computing device 136 or the user computing device 140. As previously mentioned, the medical image may be fed through a GAN in step 204 to increase the clarity of the image input to the trained machine learning model. Additionally or alternatively, the medical image may be fed through a GAN in step 206 to improve the quality of the output of the visualization information to a physician or other medical professional.

[0035] Another exemplary result generated may include a prioritized list of cases for physician attention, for example, with diagnoses suspected to be of elevated severity or classification being prioritized. A further exemplary result generated may include a pre-populated written report of the medical image analysis that has been reviewed and authenticated by a physician. Additionally, results may include the generation and transmission of a communication including the medical image analysis to other parties in the clinical setting, such as the original physician, the patient themselves, the patient's own caregiver or family, surgeons, physical therapists, etc.

[0036] Diagnostic prediction for musculoskeletal disorders Diagnostic images captured during a patient examination may facilitate a physician's diagnosis of the injury, injury, and / or condition, including any classification (e.g., category, stage, phase, grade, etc.) of the injury, injury, and / or condition. The techniques described in Figures 3-4 involve training and using machine learning models to diagnose musculoskeletal disorders. Figures 5A and 5B show an exemplary machine learning model trained to diagnose a shoulder injury, such as a rotator cuff tear.

[0037] 3 shows a flowchart illustrating an example method 300 for training a machine learning model (e.g., one of the trained machine learning models 118) to predict a diagnosis of a musculoskeletal disorder according to example techniques presented herein. The example method 300 (e.g., steps 302-306) may be performed by the system 116. The example method 300 may include one or more of the following steps:

[0038] In step 302, a plurality of labeled training medical images may be received (e.g., from the medical image database 160 via the network 120). The training medical images may include ultrasound images of specific anatomical structures affected by one or more musculoskeletal disorders. The training medical images may be labeled with annotations from a physician that at least indicate one or more musculoskeletal disorders (if any) present in the image. The annotations may also indicate anatomical features (e.g., bones, tendons, ligaments, muscles, nerves, etc.) and / or destructive features that are not normally present in the body and may be associated with one or more musculoskeletal disorders present in the image. Exemplary destructive features associated with muscle, ligament, and / or tendon injuries may include bleeding, muscle edema, hematoma, fluid collection, lesion, scar, inflammation, defect, tendinosis, ligamentopathy, tendonitis, and / or rupture. Exemplary destructive features associated with bone damage may include stress fractures, avulsions at tendon and ligament attachments, callus formation, fracture pseudarthrosis, growth plate injury, and / or tendon screw impingement. Other exemplary destructive features may include cellulitis and / or abscesses associated with infection, arthritis (e.g., rheumatoid arthritis, psoriatic arthritis, gout, or osteoarthritis), myositis and diabetic myocardial infarction, soft tissue foreign bodies (e.g., wood, plastic, metal, glass, organic matter, and / or plants), peripheral nerve entrapment, soft tissue masses (e.g., lipomas, peripheral nerve sheath tumors, vascular anomalies, ganglion cysts, lymph nodes, and / or malignant soft tissue tumors), and bone masses. Additionally, annotations may indicate the area and / or volume of any of these features. The training medical images may be preprocessed (similar to the preprocessing described in step 204 of FIG. 2). In some examples, as part of the pre-processing, annotations may be extracted or otherwise identified from the training medical images to form labels separate from the training medical images, hi other examples, the annotations may be received as labels separate from the training medical images.

[0039] For a particular type of musculoskeletal disorder, there may be multiple classifications for a given type of disorder. The classifications may include categories, stages, phases, grades, and / or the like. For example, as described in more detail with reference to FIG. 5A, to diagnose a shoulder injury such as a rotator cuff pathology, the rotator cuff may be classified as normal (e.g., absent pathology), tendinitis, tendinosis, calcific tendinitis, calcific tendinosis, delamination, articular tear, intraarticular tear, bursal tear, or full thickness tear. Similar injuries may be classified for any part of the anatomy, and the training images may include multiple examples of each classification. For example, in addition to shoulder injuries, injuries may be classified for the elbow, wrist, hand, finger, hip, thigh, knee, ankle, foot, and / or lower leg. If the musculoskeletal disorder of interest includes multiple classifications, the received labeled training medical images may be composed of subsets of labeled training medical images, each subset may correspond to a distinct classification from the multiple classifications, and the corresponding labels for the training medical images in each subset may include annotations further indicating known classifications from the multiple classifications for the musculoskeletal disorder present in the image.

[0040] In step 304, a machine learning model for predicting a diagnosis associated with a musculoskeletal disorder affecting an anatomical structure may be generated and trained using the plurality of labeled training medical images. For example, the training medical images may be input to the machine learning model. The machine learning model may be any of the exemplary types previously listed herein. The machine learning model may predict a diagnosis associated with a musculoskeletal disorder. In some examples, the predicted diagnosis may be based on a predicted identification of an object for a given anatomical structure associated with a musculoskeletal disorder. For example, a predicted diagnosis of calcific tendonitis may be based on predicted calcium deposits on the rotator cuff tendons. The machine learning model may output, for each training image, at least a prediction of whether a musculoskeletal disorder is present. Furthermore, if the predicted diagnosis may be based on a predicted identification of an object for a given anatomical structure associated with a musculoskeletal disorder, a predicted object of interest may be output in addition to or instead of a predicted diagnosis by the machine learning model (e.g., calcium deposits on the tendons). In other examples, if the disorder includes multiple classifications, the machine learning model may output, for each training image, a score (e.g., probability) for each classification that represents the likelihood that the training medical image exhibits a distinct classification of musculoskeletal disorder.

[0041] To train the machine learning model, a predicted diagnosis associated with a musculoskeletal disorder output by the machine learning model for a training medical image may be compared with a label corresponding to the training medical image to determine a loss or error. For example, a predicted diagnosis for a first training image may be compared with a known diagnosis in the first training image identified by a corresponding label. The machine learning model may be modified or altered (e.g., weights and / or biases may be adjusted) based on the error to improve the accuracy of the machine learning model. This process may be repeated for each training image, or at least until the determined loss or error is below a predetermined threshold. In some examples, at least a portion of the training images and corresponding labels may be retained and used to further validate or test the trained machine learning model.

[0042] To provide an illustrative example, FIG. 5A is a conceptual diagram 500 illustrating the training process of a machine learning model for predicting a diagnosis of a shoulder injury. To diagnose a shoulder injury, such as a rotator cuff tear, the rotator cuff tear may be classified as normal (e.g., no tear), articular side tear, intra-articular tear, bursal side tear, or full thickness tear. Thus, the labeled training medical images used to generate and train the machine learning model for predicting the diagnosis of a rotator cuff tear may be composed of subsets of labeled training medical images corresponding to each individual category. For example, a first subset 502 may include images of shoulders without rotator cuff tears, a second subset 504 may include images of shoulders with articular side tears, a third subset 506 may include images of shoulders with bursal side tears 508, and a fourth subset 510 may consist of images of shoulders with full thickness tears. Exemplary numbers of images included in each of the above subsets, and the representation (e.g., percentage) of each subset among all images, are shown in table 512. Table 512 also includes a breakdown of the number of images in each subset that were used in the training set and the number of images that were held back for the validation set.

[0043] Returning to FIG. 3, once the machine learning model is sufficiently trained, in step 306, the trained machine learning model may be stored (e.g., as one of the trained machine learning models 118 stored in the storage device 114) for subsequent use. In some examples, the trained machine learning model may be a single machine learning model that has been generated and trained to predict the diagnosis of multiple different musculoskeletal disorders affecting a given anatomical structure. In other examples, the exemplary method 300 may be executed to generate and train a collection of machine learning models, each model predicting the diagnosis of a particular musculoskeletal disorder affecting a given anatomical structure (e.g., one model for identifying rotator cuff tears, one model for identifying calcific tendinitis in the rotator cuff, etc.). When deployed to evaluate medical images of anatomical structures, the collection of machine learning models may be executed in parallel.

[0044] 4 shows a flow chart illustrating an example method 400 for predicting a diagnosis of a musculoskeletal disorder according to example techniques presented herein. The example method 400 (e.g., steps 402-408) may be performed by the system 116. The example method 400 may include one or more of the following steps:

[0045] In step 402, a medical image of an anatomical structure may be received from a computing device. The medical image may be an ultrasound image or any imaging modality described herein, and the computing device may include the user computing device 140 or the imaging computing device 136 of the ultrasound imaging system 132. For example, the computing device may execute an application (e.g., a client application) associated with the medical image processing system 110. In some aspects, the medical image may be a previously captured and stored image, which is selected from the computing device's local storage or a remote data storage system (e.g., the PAC system 150) or transmitted to the system 116 via the application. In other aspects, the medical image may be a live image captured in real time (e.g., by the ultrasound imaging system 132 as the patient is being imaged) and transmitted to the system 116 via the application. In some examples, the user may also select, via the user interface of the application, a type of musculoskeletal disorder that may affect the anatomical structure captured in the medical image. The type of musculoskeletal disorder may be selected based on symptoms reported by the patient and / or signs detected by a physician during a physical examination and / or initial review of medical images.

[0046] In step 404, the medical image may be provided as an input to a trained machine learning model, such as a machine learning model trained using the method 300 described with reference to FIG. 3, to predict a diagnosis associated with a musculoskeletal disorder affecting an anatomical structure. In examples where the exemplary method 300 is used to generate and train a collection of machine learning models, the medical image may be provided as an input to each machine learning model of the collection of machine learning models operating in parallel. In some examples, if the type of musculoskeletal disorder is selected by a physician, only the machine learning model trained to identify the type of musculoskeletal disorder may be run (e.g., to conserve computing resources).

[0047] In step 406, a predicted diagnosis may be received from the trained machine learning model. In some examples, the predicted diagnosis may include at least an indication of the presence or absence of a musculoskeletal disorder in the medical image. In the case of a musculoskeletal disorder with a classification, the predicted diagnosis may indicate a predicted classification. Additionally or alternatively, in examples where the predicted diagnosis may be based on a predicted identification of an object for a given anatomical structure associated with the musculoskeletal disorder, the predicted diagnosis may include an identified predicted object (object identification is described in more detail below). Additionally or alternatively, the predicted diagnosis may have an associated score representing a confidence associated with the prediction. Similarly, in the case where the musculoskeletal disorder includes a classification, the predicted diagnosis may include a score for each classification representing the likelihood of the medical image indicating a respective classification for the musculoskeletal disorder, and the classification with the highest score may be the predicted diagnosis.

[0048] In step 408, the medical image and the predicted diagnosis may be provided to a computing device for display. For example, the medical image and the predicted diagnosis may be received via an application executing on a computing device (e.g., user computing device 140 and / or imaging computing device 136) and displayed within a user interface of the application, such as the exemplary user interface shown in FIG.

[0049] FIG. 5B illustrates an example application user interface 550 that displays a predicted diagnosis of a shoulder injury. The application user interface 550 may be a user interface of an application associated with the system 116 executing on a computing device, such as the user computing device 140 and / or the imaging computing device 136. One of the machine learning models 118 may be generated and trained to predict a diagnosis of a rotator cuff tear using the example training medical images described with reference to FIG. 5A. A medical image, such as a medical image 552 of a rotator cuff of a shoulder, may be received for processing by the system 116 (e.g., using the example method 400). The medical image 552 may include tissues, muscles, and tendons of the rotator cuff structure, specifically the supraspinatus / tendon complex. Once processed, the system 116 may provide at least the medical image 552 and the predicted diagnosis 554 to an application for display in the application user interface 550. For example, the predicted diagnosis 554 may indicate a tear and include a predicted classification of a full thickness tear.

[0050] In addition, a score 556 associated with the predicted diagnosis 554 may be displayed. The score 556 may indicate a likelihood or probability that the predicted diagnosis 554 for the musculoskeletal disorder is present in the medical image 552. The score 556 may assist the physician in the diagnostic process. For example, the physician may feel more confident using the predicted diagnosis 554 as a factor in the diagnostic process when there is a high value of the score 556 of the predicted diagnosis 554 that corresponds to (or at least does not contradict) what the physician can visualize with his or her own eyes. In addition, for musculoskeletal disorders having multiple classifications (e.g., categories), such as a rotator cuff tear, a table 558 or other similar graphical element may be displayed on the user interface 550 listing each of the classifications 560 and associated scores 562. For example, as shown in the table 558, individual scores may be included for display indicating that the medical image 552 is a normal rotator cuff (no tear), an articular tear, an intra-articular tear, a bursal tear, or a full thickness tear.

[0051] The application user interface 550 described above is provided by way of example only and may include additional, less, different, or differently organized information than that shown in FIG. 5B.

[0052] The schematic techniques for diagnosing musculoskeletal disorders described above with reference to Figures 3-5B may have many possible specific applications or use cases, as described in more detail below. For example, similar techniques may be used to identify objects in medical images to facilitate (guide) orthopedic surgery in joints, muscles, tendons and ligaments. Additionally or alternatively, the techniques described above may be used in non-surgical orthopedics or osteopathic manipulative treatment, which may include clinician-directed treatment techniques (such as dilation, light compression and light resistance) as administered to a patient by a treating physician. In the techniques described above, ultrasound images are the image modality input to the trained machine learning model, while in other techniques, digital images from another medical imaging modality, such as images from a CT scanner, X-ray images, etc., may be used to diagnose injuries in other areas of the patient's body, such as brain injuries, bone injuries, etc.

[0053] Identifying, measuring, and visualizing objects for diagnostic or interventional observation The techniques described in Figures 6-7 involve training and using machine learning models to identify objects in medical images of anatomical structures. Objects may include, for example, anatomical features of the anatomical structures, features not normally present in the body that may disrupt bodily function, and / or foreign objects inserted during a procedure. Identification of objects may be useful for diagnosis as well as in guiding post-diagnosis interventions. Additionally, for billing purposes, physicians may be required to submit images in which certain objects are identified to provide evidence of a given procedure. For example, when performing a needle biopsy procedure, an image including an identified needle may be submitted as part of the billing documentation.

[0054] Once an object is identified, measurements related to the object, such as the area and / or volume of the object, may also be determined. The measurements may be useful to a physician in determining whether a medical condition is normal or abnormal and may be used in the diagnostic process of a musculoskeletal disorder. In addition, by obtaining measurements of the object of interest over time (e.g., during subsequent patient visits), the progression and / or regression of a patient associated with a musculoskeletal disorder may be tracked by a physician. For example, with respect to carpal tunnel syndrome, the area of ​​the median nerve may be a useful indicator for diagnosis and may be monitored over time to determine whether the carpal tunnel syndrome is regressing (e.g., after treatment). As another example, with respect to a rotator cuff tear, the amount of tendon separated from the bone material may be monitored over time to determine whether the injury is regressing (e.g., after treatment). As a further example, after identifying a destructive feature that is not normally present in the body and may disrupt the body's function, such as a ganglion cyst, pleural effusion, calcium deposits, masses, lesions, etc., at least a portion of the structure may be surgically removed and post-operative images may be captured. Measurements of this feature in the pre-operative and post-operative images can be automatically compared by the system to provide objective evidence of how much of the object of interest has been removed. For example, images from a previous imaging session can be automatically co-registered with images from a current imaging session. Multiple co-registration points can be automatically identified, and corresponding objects that can be indicated in at least one of the image sets by a user can be identified in both the past and current image sets. Calcium deposits in a previous image set can be automatically associated with the same calcium deposits in the current imaging session, and measurements can be automatically taken to determine disease progression or regression. In some cases, an object such as a cyst or mass may be completely missing from the images of the current session due to surgery or other procedures. In this case, the system can automatically determine that an object that was an object of interest in the previous imaging session is no longer present.

[0055] In some examples, the identified object may be input to another process. For example, as described in Figures 8-9, a machine learning model may be trained and used to determine whether a medical image is an optimal image for the identified object (e.g., an optimal viewing angle for the object) and to predict a probe orientation to capture the image that results in the viewing angle. If the medical image is not optimal, a proposed probe movement to capture a more optimal image for the identified object of interest may be determined based on the predicted probe orientation (e.g., current probe orientation) and the learned probe orientation associated with capturing the optimal image.

[0056] Further, visualization information may be provided for display to indicate the identified objects and any measurements associated with the objects, as shown and described with reference to Figure 10. Additionally, if the medical image is determined to be sub-optimal for one or more of the identified objects, the visualization information may further include a directional indicator indicating suggested probe movement to capture a more optimal image.

[0057] 6, a flowchart illustrates an example method 600 for training a machine learning model (e.g., one of the trained machine learning models 118) to identify objects in a medical image of an anatomical structure, according to example techniques presented herein. The example method 600 (e.g., steps 602-606) may be performed by the system 116. The example method 600 may include one or more of the following steps:

[0058] At step 602, a plurality of labeled training medical images may be received (e.g., from the medical image database 160 via the network 120). The training medical images may include ultrasound images of particular anatomical structures. The training medical images may be labeled with annotations from a physician indicating a plurality of objects within the individual images. The annotated objects may include anatomical features of anatomical structures such as bones, tendons, ligaments, cartilage, muscles, nerves, veins, arteries, etc. The annotated objects may also include features not normally present in anatomical structures that may disrupt bodily function (e.g., destructive features), such as ganglions, calcifications, pleural effusions, ruptures, masses, lesions, restrictions, impingements, compressions, etc. The annotated objects may further include foreign objects that may be inserted into the body as part of a procedure, such as inflatable balloons, needles, knives, scalpels, fingers, stents, intravascular devices, catheters, surgical instruments, etc. Additionally, the annotations may indicate the area and / or volume of any of the above-mentioned objects. The training medical images may represent a variety of different types of objects. For example, a first subset of training images may include ganglions, a second subset may include calcifications, a third set may include an inflatable balloon, a fourth set may include needles, etc. In some examples, the training medical images may be further labeled with annotations indicating measurements associated with one or more of the objects in the individual images. For example, the images may be annotated with the area and / or volume of the object. If the object volume is annotated for a given object, then for a given training image, there may be a subset of image sequences, where each image in the subset of image sequences may be captured at a different depth and frequency setting of the ultrasound imaging system 132 without moving the position of the probe 134.

[0059] The training medical images may be pre-processed (similar to the pre-processing described in step 204 of FIG. 2). In some examples, as part of the pre-processing, annotations may be extracted or otherwise identified from the received training medical images to form labels separate from the training medical images. In other examples, the annotations may be received as labels separate from the training medical images. The corresponding labels for the training medical images may include the known types of each of the annotated objects.

[0060] At step 604, a machine learning model for identifying objects may be generated and trained using the plurality of labeled training medical images. For example, the training medical images may be input to the machine learning model. The machine learning model may identify one or more objects in the training medical images. In some examples, the machine learning model may predict measurements (e.g., area and / or volume) associated with the objects.

[0061] To train the machine learning model, the output by the machine learning model for the training medical images may be compared to the labels corresponding to the training medical images to determine a loss or error. For example, the object(s) identified in the first training image may be compared to the type of known object identified by the corresponding label. Furthermore, if the prediction includes measurements related to the identified object(s), the predicted measurements for the object(s) identified in the first training image may be compared to the known measurements of the type of known object included in the first training image identified by the corresponding label. The machine learning model may be modified or altered (e.g., weights and / or biases may be adjusted) based on the error to improve the accuracy of the machine learning model. This process may be repeated for each training image, or at least until the determined loss or error is below a predetermined threshold. In some examples, at least a portion of the training images and corresponding labels may be retained and used to further validate or test the trained machine learning model.

[0062] Once the machine learning model is sufficiently trained, in step 606, the trained machine learning model may be stored (e.g., in storage device 114) for subsequent use. In some examples, the trained machine learning model may be a single machine learning model that is generated and trained to identify each of the different types of objects that may be present in a medical image of an anatomical structure. In other examples, the exemplary method 600 may be performed to generate and train a collection of machine learning models, each model identifying a particular type of object (e.g., one system for identifying ganglions, another system for identifying calcifications, etc.). When deployed to evaluate medical images of an anatomical structure, the collection of machine learning models may be run in parallel.

[0063] 7 shows a flow chart illustrating an example method 700 for identifying objects in a medical image of an anatomical structure, according to example techniques presented herein. The example method 700 (e.g., steps 702-708) may be performed by the system 116. The example method 700 may include one or more of the following steps:

[0064] In step 702, a medical image of an anatomical structure may be received from a computing device. The medical image may be an ultrasound image, and the computing device may include the user computing device 140 or the imaging computing device 136 of the ultrasound imaging system 132. For example, the computing device may execute an application (e.g., a client application) associated with the medical image processing system 110. In some aspects, the medical image may be a previously captured and stored image, which is selected from the computing device's local storage or a remote data storage system (e.g., the PAC system 150) or transmitted to the system 116 via the application. In other aspects, the medical image may be a live image captured in real time (e.g., by the ultrasound imaging system 132 as the patient is being imaged) and transmitted to the system 116 via the application. In some examples, more than one medical image may be received (e.g., a subset of an image sequence of an anatomical structure captured at various depth and frequency combinations may be received).

[0065] In some examples, the physician may also select, via the application's user interface, one or more particular types of objects that the physician desires to confirm are present in the image and / or distinguish in the image. As one illustrative example, in certain surgeries, an inflatable balloon filled with a fluid, such as saline, may be inserted into the body to inflate the balloon during the procedure. It may be difficult for the physician's eye to distinguish between the balloon and another feature (such as a ganglion) in an ultrasound image. Thus, the physician may select the balloon as an object of particular interest to facilitate the guidance of a needle to fill the balloon (not a ganglion-like feature) with fluid during the procedure. For example, upon receiving the physician's selection, the balloon (not a ganglion) may be visually emphasized (e.g., highlighted) along with the needle, and the needle may be visualized in a similar but distinguishable manner (e.g., highlighted in a different color) as the balloon, thus indicating to the observer the relative positions and / or orientations of the balloon and the needle.

[0066] In step 704, the medical image may be provided as an input to a trained machine learning model (e.g., one of the trained machine learning models 118), such as a trained machine learning model trained using the method 600 described with reference to Figure 6. In examples where the exemplary method 600 is used to generate and train a collection of machine learning models, the medical image may be provided as an input to each machine learning model of the collection of machine learning models operating in parallel. In some examples, once one or more particular types of objects of interest are selected by a physician, only the machine learning models trained to identify those particular types of objects of interest may be executed (e.g., to conserve computational resources).

[0067] At step 706, a prediction of one or more of the identified objects may be received from the trained machine learning model. In addition, the prediction may have an associated score that represents a confidence associated with the prediction (e.g., the likelihood that each type of identified object is actually an object in the medical image). In some examples, the machine learning model may also be trained to output measurements such as area and volume associated with one or more of the identified objects. The area may represent the cross-sectional area of ​​the object, and the system may prompt the user to move the probe 134 to obtain the optimal angle to obtain the cross-sectional area. In cases where the volume of one or more of the identified objects is predicted, multiple medical images (e.g., a subset of an image sequence of an anatomical structure captured at various depth and frequency combinations) may be received at step 702 and input to the model at step 704 to enable volumetric prediction. In such examples, the prediction may also include predicted measurements associated with the identified objects.

[0068] At step 708, an indication of the prediction may be provided to a computing device for display (e.g., via an application executing on the computing device 140 and / or the imaging computing device 136). For example, visualization information may be generated based on the prediction in the post-processing step 206 described with reference to FIG. 2. The visualization may label, highlight, or otherwise enhance the objects identified in the medical image, and the visualization information may be provided for display within an application user interface. In addition, the visualization information may include a score associated with the prediction for each object. Furthermore, the visualization information may include any measurements associated with one or more of the identified objects. Exemplary visualization information is shown in FIG. 10 as described below.

[0069] As described above, in some examples, the machine learning model may be trained to output measurements such as area and volume associated with one or more of the identified objects. In other examples, measurements associated with a given identified object may be obtained independent of the prediction received in step 706. As one non-limiting example, in some aspects, the visualization information provided to the computing device (e.g., in step 708) may also include a tracer tool control element in the application user interface. The tracer tool control element may be selectable by a physician viewing the medical image to trace along the boundary of the object in the displayed medical image. The system 116 may then be able to determine the area of ​​the object using the traced boundary and provide the area of ​​the object to the computing device for display.

[0070] As another non-limiting example, in some embodiments, volumetric sweep imaging may be performed to determine the volume of an object. For example, multiple ultrasound images of an anatomical structure captured at various depths and frequencies may be received as two-dimensional (2D) slices having positions and orientations. The system 116 may use the 2D slices to reconstruct three-dimensional (3D) visualization information of the object, and the volume of the object may be determined from the 3D visualization information. The volume may then be provided to a computing device for display.

[0071] Additionally, in some examples, after displaying the visualization information of the medical image with the identified object(s), the operator may select to visually lock onto one of the identified objects (e.g., an object of interest to the user) so that when the operator moves the probe, the identified object remains labeled, highlighted, or otherwise emphasized in the new medical image after the probe movement. For example, the operator may provide a touch input to the display of the computing device in an area of ​​the medical image corresponding to the object to be locked. The position of the object relative to the position of the probe capturing the medical image (e.g., the current probe position) may be identified for use as a reference. The object may be stationary, so that a second position of the object relative to the probe movement may be determined by recognizing a first position (e.g., a reference) of the object relative to the probe prior to the probe movement and determining the probe movement. The probe movement may be detected using a sensor attached to the probe, such as an accelerometer, a gyroscope, and / or an inertial sensor. The identified object may be labeled, highlighted, or otherwise emphasized in a second position in the new medical image after the probe movement. For example, the visually locked identified object may be a lesion or mass to be removed as part of an ultrasound-guided procedure, such that as the probe is moved during the procedure, the lesion or mass continues to be labeled, highlighted, or otherwise enhanced to enable the surgeon to maintain a view of the lesion or mass.

[0072] 8 shows a flowchart illustrating an example method 800 for training a machine learning model (e.g., one of the trained machine learning models 118) to identify optimal images for identified objects in a medical image of an anatomical structure, according to example techniques presented herein. The example method 800 (e.g., steps 802-806) may be performed by the system 116. The example method 800 may include one or more of the following steps:

[0073] In step 802, a plurality of labeled training medical images of an anatomical structure may be received (e.g., from the medical image database 160 via the network 120). The training medical images may include a plurality of objects that are annotated within the images. The annotated objects may include anatomical features of the anatomical structure, such as bones, tendons, ligaments, cartilage, muscles, nerves, veins, arteries, etc. The annotated objects may also include features that are not normally present within the anatomical structure that may disrupt bodily function, such as ganglions, calcifications, pleural effusions, tears, masses, lesions, restrictions, impingements, compressions, etc. The annotated objects may further include foreign objects that may be inserted into the body as part of a procedure, such as inflatable balloons, needles, knives, scalpels, fingers, stents, intravascular devices, catheters, surgical instruments, etc.

[0074] The trained medical images may represent a variety of different types of objects that have been annotated within the images. For a given object type, a subset of training medical images that includes that given object type may be composed of images with multiple different viewing angles of the object within the image (e.g., different viewing angles based on the orientation of the probe of the imaging system used to capture the images). Certain viewing angles may be more optimal than others for an object type based on the presence or absence of musculoskeletal ultrasound artifacts such as anisotropy, posterior enhancement or shadowing, edge artifacts, etc. Using anisotropy as an example, a tissue is anisotropic if the properties of the tissue change when measured from different viewing directions. For example, when a tendon or ligament is imaged perpendicular to the ultrasound beam, a characteristic hyperechoic (visually displayed as a light gray) fibrous appearance can be seen. However, when the ultrasound beam is at an angle of 2-3 degrees to the long axis of the tendon, the normal hyperechoic (light gray) is lost and the tendon becomes more hypoechoic (dark gray) as the angle increases. Once the tendon is identified, the anisotropy can be corrected to eliminate pathology.

[0075] The training medical images may have corresponding ground truth labels. For example, each image may be labeled as a known optimal image or a known non-optimal image of a type of object. In some examples, the ground truth labels may also include a known orientation of the probe when the training medical image was captured. The training medical images may also be pre-processed (similar to the pre-processing described in step 204 of FIG. 2).

[0076] In step 804, a machine learning model may be generated and trained to predict whether a medical image is an optimal image of an object identified in the medical image using the plurality of labeled training medical images. In some examples, the machine learning model may also be generated and trained to predict the orientation of an associated probe used to capture the medical image. To train the machine learning model, predictions output by the machine learning model for the training medical images may be compared to the ground truth label(s) corresponding to the training medical images to determine a loss or error. For example, a prediction of an optimal or non-optimal image for a first training medical image may be compared to a corresponding ground truth label for a training medical image that is a known optimal or known non-optimal image of the object therein. In addition, if the prediction output also includes a predicted orientation of the probe, the predicted orientation may be compared to a known orientation of the probe included in the corresponding ground truth label. The machine learning model may be modified or altered (e.g., weights and / or biases may be adjusted) based on the error to improve the accuracy of the machine learning model. This process may be repeated for each training image, or at least until the determined loss or error is below a predetermined threshold. In some examples, at least a portion of the training images and corresponding labels may be retained and used to further validate or test the trained machine learning model.

[0077] Once the machine learning model is sufficiently trained, in step 806, the trained machine learning model may be stored (e.g., as one of the trained machine learning models 118 stored in the storage device 114) for subsequent use. In some examples, the trained machine learning model may be a single machine learning model generated and trained to predict whether a medical image of an anatomical structure is an optimal image of a plurality of different object types that may be contained therein. In other examples, the exemplary method 800 may be executed to generate and train a collection of machine learning models, each model predicting whether a medical image of an anatomical structure is an optimal image of a particular object type identified therein. When deployed to evaluate a medical image of an anatomical structure, the collection of machine learning models may be executed in parallel. In some examples, when a particular object type of an object is identified (e.g., received as an output of another process such as method 700), only the machine learning model trained to predict whether a medical image of an anatomical structure is an optimal image of a particular object type may be executed to conserve computing resources.

[0078] 9 shows a flow chart illustrating an example method 900 for determining whether a medical image is an optimal image of an object identified therein, according to example techniques presented herein. The example method 900 (e.g., steps 902-912) may be performed by the system 116. The example method 1900 may include one or more of the following steps:

[0079] At step 902, a medical image of an anatomical structure may be received from a computing device. The medical image may be an ultrasound image, and the computing device may include the user computing device 140 or the imaging computing device 136 of the ultrasound imaging system 132. For example, the computing device may execute an application (e.g., a client application) associated with the medical image processing system 110. In some examples, the medical image may be a previously captured and stored image, which is selected from the computing device's local storage or a remote data storage system (e.g., the PAC system 150) or transmitted to the system 116 via the application; in other examples, the medical image may be a live image captured in real time (e.g., by the ultrasound imaging system 132 as the patient is being imaged) and transmitted to the system 116 via the application. In some examples, the user may also select, via the user interface of the application, one or more particular types of objects that the user desires to verify are present in the medical image and / or to distinguish within the medical image.

[0080] In step 904, the medical image may be provided as an input to a first trained machine learning model to obtain a prediction of one or more objects identified (e.g., as an output of the first trained machine learning model). The first trained machine learning model may be a machine learning model trained and used to identify objects in medical images of anatomical structures as described with reference to Figures 6 and 7. For example, the first trained machine learning model may predict that an object identified in an ultrasound image includes a calcium deposit located on a tendon of the shoulder.

[0081] In step 906, the medical image may be provided to a second trained machine learning model for identifying whether the medical image is an optimal image of the identified object(s) (e.g., a particular type of object identified and received as a prediction from the first trained machine learning model). The second trained machine learning model may be a trained machine learning model trained using the method 800 described with reference to FIG. 8. Continuing with the above example, if the identified object includes a calcification located on a tendon in the shoulder, the second trained machine learning model may be trained to at least identify an optimal image of the calcification.

[0082] At step 908, a prediction of whether the medical image is an optimal image of the identified object(s) may be received as an output of the second trained machine learning model. In some examples, the prediction output may also include a predicted orientation of the probe capturing the medical image (e.g., a current probe orientation that results in a viewing angle of the identified object(s) that is either optimal or non-optimal).

[0083] In step 910, an indication of the prediction may be provided to a computing device for display. For example, visualization information may be generated based on the prediction in the post-processing step 206 described with reference to FIG. 2. In some examples, if the medical image is predicted to be an optimal image of the identified object(s), instructions may be provided to the computing device to cause a positive visual indicator to be displayed along with the medical image. In other examples, if the medical image is predicted to be a non-optimal image of the identified object(s), instructions may be provided to the computing device to cause a prompt to be displayed for the operator to make a probe adjustment (e.g., reorient the probe) to enable capture of an optimal image of the identified object(s).

[0084] The prompts may include directional instructions to guide the operator to adjust the current orientation of the probe to correspond to the predicted orientation to achieve capture of an optimal image. For example, as part of training, the trained machine learning model may learn the probe orientation associated with the optimal viewing angle (e.g., resulting in an optimal image) of a particular type of identified object(s) from the labeled training medical images. Thus, the learned probe orientation associated with capture of an optimal image of a particular type of identified object(s) may be compared to the current probe orientation to generate directional instructions to adjust the current probe orientation to the learned probe orientation. In some examples, the directional instructions may be in the form of a probe orientation indicator, as shown with reference to FIG. 10.

[0085] 10 is an example application user interface 1000 that displays objects identified in a medical image and associated object measurements. The application user interface 1000 may be a user interface of an application displayed on the user computing device 140 and / or the imaging computing device 136. A machine learning model, such as the machine learning model trained using the example method 600 described with reference to FIG. 6, may be generated and trained to identify objects in ultrasound images of the shoulder joint.

[0086] An ultrasound image of the patient's shoulder may be received from the application for processing by the system 116 (e.g., using the example method 600) to identify tendons 1004 (e.g., anatomical features) and calcifications 1006 (e.g., features not normally present in the body that may be disruptive to bodily function), among other objects in the ultrasound image. Once processed, the system 116 may provide visualization information including at least the medical image 1002 with the tendons 1004 and calcifications 1006 labeled, highlighted, or otherwise enhanced to the application for display via the application user interface 1000.

[0087] Additionally, measurements 1008 associated with one or more of the identified objects, such as the calcification 1006, obtained as an additional output of the trained machine learning model or obtained independently (e.g., using the above-mentioned tracing to obtain area and / or volumetric sweep imaging techniques to obtain volume), may be displayed. In some examples, the measurements 1008 may be displayed as annotations on the medical image 1002 along with the calcification 1006, as shown.

[0088] Additionally, the application user interface 1000 may include a probe direction indicator 1010 that suggests a direction for moving the probe 134 of the ultrasound imaging system 132, for example, to capture a more optimal image of the calcific deposit 1006 than the current image displayed. The determination of whether the medical image 1002 is an optimal image of the calcific deposit 1006, and if not, the proposed direction for probe movement, may be obtained using a process similar to the exemplary method 900 described with reference to FIG. 9. The probe direction indicator 1010 may include an arrow indicating that the operator of the ultrasound imaging system 132 should slide the probe 134 (e.g., forward, backward, right, or left) and / or rotate the probe 134. In some examples, the probe direction indicator 1010 may be animated such that the arrow pointing in the direction of the proposed movement is highlighted, flashes, or is otherwise visually directed to the user's attention.

[0089] The application user interface 1000 described above is provided by way of example only, and may include additional, less, different, or differently organized information than that shown in FIG.

[0090] Predicting and visualizing the position and trajectory of instruments in interventional observation For example, physicians may utilize electronic medical imaging during a procedure to visualize an instrument inserted into a patient's body and assist the physician in safely guiding the instrument to the intended target area. An instrument may include any device, tool, or object that is inserted into the body as part of a procedure, such as a medical treatment or surgery. For example, a physician may insert the needle of a drug syringe containing a steroid into a muscle of the shoulder as part of a treatment to reduce pain. In such an example, a physician may use ultrasound imaging to guide the needle to a specific area of ​​the muscle. In addition, ultrasound-guided surgery is becoming increasingly common. For example, an ultrasound-guided needle biopsy of the shoulder may be performed to remove at least a portion of a mass identified within the shoulder to determine whether the mass is cancerous, precancerous, or benign.

[0091] Often, the critical portion of the instrument to locate is the distal end of the instrument (e.g., the tip, or end of the instrument furthest from the operator) that is used to perform a given action on a target area of ​​the body. For example, in a needle biopsy, the distal end of the instrument may be the tip of a biopsy needle that removes tissue from a target area, such as a mass or lesion. Thus, precise location of the distal end may be necessary to prevent the distal end from acting on a different non-target area of ​​the body (e.g., to prevent puncturing an adjacent organ). Currently, for an operator of an ultrasound imaging device to locate the distal end of the instrument, the operator must manually manipulate the probe of the imaging device (e.g., by rotating, sliding, or heel-toing the probe) until the position of the probe captures an image that allows the operator to confirm where the distal end of the instrument is within the body. The operator may need to perform these maneuvers repeatedly while advancing the instrument toward the target in order to continue visualization of the distal end of the instrument.

[0092] To improve upon current manual techniques, the techniques presented in Figures 11-12 describe the training and use of a machine learning model to predict (e.g., in real-time) the position of an instrument during a procedure based on input medical images. In addition, as described in Figures 13-14, another machine learning model may be trained and used to predict the trajectory of the instrument to the intended target. As shown in Figure 15, visualization information including the predicted position and trajectory of the instrument overlaid on the medical image, among other information, may be displayed to the physician.

[0093] 11 shows a flowchart illustrating an example method 1100 for training a machine learning model (e.g., one of the trained machine learning models 118) to predict a position of an instrument according to example techniques presented herein. The example method 1100 (e.g., steps 1102-1106) may be performed by the system 116. The example method 1100 may include one or more of the following steps:

[0094] In step 1102, a plurality of training medical images may be received (e.g., from the medical image database 160 via the network 120). The training medical images may include ultrasound images of an anatomical structure in which an instrument is present. For example, the instrument may be a biopsy needle. The training medical images may be comprised of subsets of image sequences. Each image in the subset of image sequences may be captured at a different setting of the ultrasound imaging system 132 without moving the position of the probe 134. The different settings may include varying depths and corresponding frequencies (e.g., because the frequency is set lower as the depth increases). For example, the subset of image sequences may include a first image of the patient's anatomy captured at a first depth-frequency combination, a second image of the patient's anatomy captured at a second depth-frequency combination, a third image of the patient's anatomy captured at a third depth-frequency combination, and so on. The image sequences in each subset may include at least two images of the same patient's anatomy captured at two different depth-frequency combinations.

[0095] In some examples, the training medical images may be labeled such that supervised learning techniques may be performed to generate and train the machine learning model in step 1104. For example, the images may include annotations that identify at least a portion of the instrument. In some examples, the identified portion may include at least a distal end of the instrument. The annotations may indicate anatomical structures present in the image. In other examples, the training medical images may be unlabeled and unsupervised learning techniques may be performed to train the machine learning model in step 1104. The training medical images may also be pre-processed (similar to the pre-processing described in step 204 of FIG. 2).

[0096] At step 1104, a machine learning model for predicting a position of the instrument may be generated and trained using the plurality of training medical images. For example, a subset of a first image sequence of training medical images may be input to the machine learning model. The machine learning model may predict a position of at least a portion of an instrument included within the subset. In some examples, the prediction may include a predicted position of a distal end of the instrument. For example, the machine learning model may identify at least a portion of the instrument in the image and then determine an orientation and / or length of the instrument.

[0097] In some examples, if the training medical images are labeled and supervised learning techniques are implemented to train the machine learning model, the predicted position of the instrument output by the machine learning model for the training medical images may be compared to the label corresponding to the training medical images to determine a loss or error. For example, the predicted position of the instrument in the subset of the first image sequence may be compared to the known position of at least a portion of the instrument in the subset of the first image sequence identified by the corresponding label. The machine learning model may be modified or altered (e.g., weights and / or biases may be adjusted) based on the error to improve the accuracy of the machine learning model. This process may be repeated for each subset of image sequences in the training medical images, or at least until the determined loss or error is below a predetermined threshold. In some examples, the subset of image sequences in the training images and at least a portion of the corresponding labels may be retained and used to further validate or test the trained machine learning model. In other examples, where unsupervised learning techniques are implemented to train the machine learning model, for a given subset of image sequences, the machine learning model may extract at least a portion of the instrument as a feature from each image in the respective subset of image sequences and identify a pattern within the subset based on the identification of the instrument as a feature (e.g., using clustering). In either supervised or unsupervised examples, once at least a portion of the instrument has been identified, the location of the distal end may be determined. In some examples, the determination of the distal end may be further facilitated by other information, such as the known length of the instrument or other images from the subset of image sequences.

[0098] Once the machine learning model is sufficiently trained, in step 1106, the trained machine learning model may be stored (e.g., as one of the trained machine learning models 118 stored on the storage device 114) for subsequent use. In some examples, the trained machine learning model may be a single machine learning model generated and trained to predict instrument positions associated with different types of instruments (e.g., instruments having different shapes, sizes, etc.). In other examples, the exemplary method 1100 may be executed to generate and train a collection of machine learning models, each model predicting instrument positions associated with a particular type of instrument (e.g., needles). When deployed to evaluate medical images, the collection of machine learning models may be executed in parallel.

[0099] 12 shows a flow chart illustrating an example method 1200 for predicting a position of an instrument according to example techniques presented herein. The example method 1200 (e.g., steps 1202-1208) may be performed by the system 116. The example method 1200 may include one or more of the following steps:

[0100] In step 1202, a medical image sequence after the insertion of the instrument may be received from a computing device. The medical image sequence may be an ultrasound image sequence, and the computing device may be the imaging computing device 136 of the ultrasound imaging system 132. For example, the ultrasound imaging system 132 may guide a procedure involving the insertion of an instrument into the patient's body to reach a target. One exemplary procedure may be a needle biopsy procedure in which a needle is inserted to reach a target mass or lesion for biopsy. To provide the physician with visualization information of the instrument inserted during the procedure, the probe 134 of the ultrasound imaging system 132 may be used to transmit and receive sound waves at adjustable frequencies and depths at which images are created by the imaging computing device 136. In some examples, a long-axis approach may be performed to generate an image having a long-axis (LAX) view of the instrument. The LAX view may be a longitudinal view of the instrument inserted into the body, for example, allowing the length of the instrument to be viewed. The sequence of medical images received in step 1202 may include at least two medical images created by the imaging computing device 136 from acoustic waves transmitted and received by the probe 134 at different frequencies and depths while the position of the probe 134 remains fixed.

[0101] In some examples, the system 116 may provide instructions to the imaging computing device 136 to cause an operator of the ultrasound imaging system 132 to display audio or visual prompts to capture a sequence of medical images following insertion of the instrument, and the sequence of medical images may be received in response to the prompts. For example, an application running on the imaging computing device 136 may include an instrument location function for locating an instrument inserted within the body, and instructions for displaying the prompts may be provided and displayed via an application user interface on the display 138 of the ultrasound imaging system 132 in response to the operator selecting the instrument location function. In some examples, as part of the selection of the instrument location function, the operator may also provide additional inputs, such as a type of instrument (e.g., a biopsy needle) and / or instrument size parameters (e.g., a length and / or diameter of the biopsy needle) that may be used in subsequent steps of the method 1200. The displayed prompts may instruct the operator on various depth and frequency combinations at which acoustic waves are transmitted and received by the probe 134 to create the medical image sequence. For example, the instructions may include a first depth and frequency at which acoustic waves are initially transmitted and received by the probe 134 to create a first sequence of images, a second depth and frequency at which acoustic waves are later transmitted and received by the probe 134 to create a second sequence of images, etc. The prompts may further instruct the operator to keep the probe 134 positioned in the same position as the operator adjusts the depth and frequency settings between image captures.

[0102] In other examples, the application (e.g., based on instructions from system 116) may provide a signal to the imaging computing device 136 that causes automatic adjustment of the depth and frequency at which sound waves are transmitted by the probe 134 to enable automatic capture of a sequence of medical images.

[0103] In step 1204, the sequence of medical images may be provided as input to a trained machine learning model for predicting the location of an instrument, such as a trained machine learning model trained using the method 1100 described with reference to FIG. 11. In examples where the exemplary method 1100 is used to generate and train a collection of machine learning models, the medical images may be provided as input to each machine learning model of the collection of machine learning models operating in parallel. In some examples, when the type of instrument is indicated by the physician as part of the selection of the instrument location function, only the machine learning model trained to identify the particular type of instrument may be operated. Additionally, any size parameters of the instrument entered by the operator as part of the selection of the instrument location function may be provided as further input to the trained machine learning system.

[0104] In step 1206, a predicted location may be received as an output from the trained machine learning model. The predicted location may include at least the distal end of the instrument. The predicted location may also include the orientation and / or length of the instrument. For example, the machine learning model may identify at least a portion of the instrument in the image and then determine the orientation and / or length of the instrument. In some examples, the predicted location output from the trained machine learning model may then be provided as an input to another process, such as a process for predicting the trajectory of the instrument, which is described in detail with reference to FIGS. 13 and 14 below.

[0105] At step 1208, a visualization of the predicted position of the instrument may be provided to the computing device for display. For example, the visualization may be generated based on the prediction in post-processing step 206 described with reference to FIG. 2 and provided for display via an application executing on the imaging computing device 136. The visualization shows the predicted position of at least the distal end of the instrument relative to the anatomical structures, allowing the physician to verify that no unintended contact has occurred between the instrument and the anatomical structures.

[0106] In some examples, the visualization information may include at least two medical images simultaneously showing orthogonal views of the instrument. For example, the first image may be an LAX view of the instrument where the length of the instrument may be visible, and the first image may be one image from a sequence of received images (e.g., an image at a given depth and frequency that is more easily visible by the human eye). The second image may be a short-axis (SAX) view of the instrument orthogonal to the LAX view such that only the distal end of the instrument is visible. The SAX view may be captured simultaneously by the same probe as the LAX view if the probe is capable of biplane imaging, and / or may be captured simultaneously with the LAX view by a different probe of the ultrasound imaging system 132. In each image, a predicted position of the instrument, and in particular the predicted position of the distal end of the instrument, may be visually indicated. An exemplary application user interface displaying visualization information with two medical images showing orthogonal views of the instrument is shown in FIG. 15, as described below.

[0107] In other examples, the visualization information may include a three-dimensional (3D) representation of the anatomical structure. The 3D representation may be generated from a sequence of medical images. In addition, a predicted position of the instrument output by a trained machine learning model may be utilized as part of the generation process such that the 3D representation visually indicates the predicted position of the instrument, and in particular the distal end of the instrument.

[0108] In some examples, the exemplary method 1200 may be repeated as the instrument is inserted further into the body in order to continue to track at least the distal end of the instrument as it is moved. As described above with reference to Figures 11-12, the trained machine learning model may be trained to receive medical image sequences captured at different depths and frequencies when the position of the instrument is predicted. For example, if the probe 134 is not aligned with the long axis of the instrument for a given captured image, the model may only learn / predict the position of the cross-section of the instrument at the given depth / frequency at which the image was captured. Thus, multiple medical images at different depths and frequencies may be required to allow the full length of the instrument to be learned and predicted by the trained machine learning model.

[0109] In other aspects, a machine learning model may be trained to predict the position of the instrument using one medical image (e.g., when the medical image is captured when the probe 134 is substantially aligned with the long axis of the instrument). Additionally, if the probe 134 is not substantially aligned with the long axis of the instrument in the currently captured medical image, a prompt may be provided to the operator to adjust the position and / or orientation of the probe to capture an image with a better viewing angle of the instrument (e.g., a viewing angle showing the long axis of the instrument). For example, a determination of whether the image of the instrument is optimal and a subsequent determination of a proposed direction for moving the probe if the image is not optimal may be determined using a similar process described above with reference to FIGS. 8-9. Once a medical image with a better viewing angle of the instrument is captured, the medical image may be provided as an input to a machine learning model trained to predict the position of the instrument using one medical image.

[0110] 13 shows a flowchart illustrating an example method 1300 for training a machine learning model (e.g., one of the trained machine learning models 118) to predict an instrument trajectory according to example techniques presented herein. The example method 1300 (e.g., steps 1002-1006) may be performed by the system 116. The example method 1300 may include one or more of the following steps:

[0111] At step 1302, a plurality of labeled training medical images may be received (e.g., from medical image database 160 via network 120), each including an instrument and a target for the instrument. For example, the training medical images may be ultrasound images of a patient's anatomy, including the instrument at a given position after insertion of the instrument into the patient, and the target for the instrument relative to the anatomical structure. The training medical images may be labeled with annotations identifying the anatomical structure, the instrument, the target, the angle of entry (e.g., the insertion angle) of the inserted instrument, the position of at least the distal end of the instrument approaching the target, and / or the position of the target. Additionally, each of the training medical images may have a corresponding ground truth label indicating a known trajectory of the instrument from the distal end of the instrument to the target.

[0112] The training medical images may be pre-processed (similar to the pre-processing described in step 204 of FIG. 2). In some examples, as part of the pre-processing, annotations may be extracted or otherwise identified from the training medical images to form labels separate from the training medical images. In other examples, the annotations may be received as labels separate from the training medical images.

[0113] In step 1304, a machine learning model for predicting the trajectory of the instrument may be generated and trained using the plurality of labeled training medical images. For example, a training medical image including an instrument and a target for the instrument may be input to the machine learning model. The machine learning model may predict the trajectory of the instrument from the distal end of the instrument to the target. To train the machine learning model, a trajectory output by the machine learning model for the training medical image may be compared to the ground truth labels corresponding to the training medical image to determine a loss or error. For example, a predicted trajectory of a first medical training image may be compared to a known trajectory of the instrument from the distal end of the instrument to the target identified by the corresponding ground truth labels of the first training medical image. The machine learning model may be modified or altered (e.g., weights and / or biases may be adjusted) based on the error to improve the accuracy of the machine learning model. This process may be repeated for each training medical image, or at least until the determined loss or error is below a predetermined threshold. In some examples, at least a portion of the training medical images and corresponding labels may be retained and used to further validate or test the trained machine learning model.

[0114] Once the machine learning model is sufficiently trained, in step 1306, the trained machine learning model may be stored (e.g., as one of the trained machine learning models 118 stored on the storage device 114) for subsequent use. In some examples, the trained machine learning model may be a single machine learning model generated and trained to predict instrument trajectories associated with different types of instruments (e.g., instruments having different shapes, sizes, etc.). In other examples, the exemplary method 1300 may be executed to generate and train a collection of machine learning models, each model predicting instrument trajectories associated with a particular type of instrument (e.g., needles). When deployed to evaluate medical images, the collection of machine learning models may be executed in parallel.

[0115] 14 shows a flow chart illustrating an example method for predicting an instrument trajectory according to example techniques presented herein. The example method 1400 (e.g., steps 1402-1412) may be performed by the system 116. The example method 1400 may include one or more of the following steps:

[0116] In step 1402, a sequence of medical images including the inserted instrument and the target of the instrument may be received from a computing device. The computing device may be the imaging computing device 136 of the ultrasound imaging system 132. The ultrasound imaging system 132 may guide a procedure involving the insertion of an instrument into the patient's body to reach a target, such as a needle biopsy procedure in which a needle is inserted to reach a target mass or target lesion for biopsy. To provide the physician with visualization information of the inserted instrument during the procedure, the probe 134 may be used to transmit and receive acoustic waves at adjustable frequencies and depths at which images are created by the imaging computing device 136. In some examples, a long axis approach may be performed to generate an image having a LAX view of the instrument, as described in more detail above with reference to FIG. 12. The sequence of medical images received in step 1402 may include at least two medical images created by the imaging computing device 136 from acoustic waves transmitted and received by the probe 134 at different depths and frequencies while the location or position of the probe 134 remains fixed.

[0117] As described in detail with reference to step 1202 of method 1200, in some examples, the sequence of medical images may be received in response to a prompt provided to an operator via an application user interface to capture each sequence of images at varying depth-frequency combinations. The prompt may be displayed upon selection of a selection of an instrument location function, and may cause the operator to provide additional input, such as an instrument type (e.g., biopsy needle) and / or instrument size parameters (e.g., biopsy needle length and / or diameter) that may be used in subsequent steps of method 1400. In other examples, the application (e.g., based on instructions from system 116) may provide signals to imaging computing device 136 that cause automatic adjustment of the depth and frequency at which acoustic waves are transmitted by probe 134 to enable automatic capture of a sequence of medical images.

[0118] In step 1404, the sequence of medical images may be provided as input to a first trained machine learning model to obtain a predicted position of the instrument. The predicted position of the instrument may include at least a distal end of the instrument. The first trained machine learning model may be the machine learning model trained and used to predict the position of the instrument described with reference to FIGS. 12 and 13.

[0119] In step 1406, an insertion angle of the instrument may be determined based on the insertion position of the instrument and a predicted position of the instrument (particularly the distal end of the instrument). The insertion position of the instrument may be extracted from a sequence of medical images. In some examples, an operator may input the insertion position of the instrument as part of the selection of the tracking function.

[0120] In step 1408, the medical image sequence, the predicted position of the instrument, and the insertion angle of the instrument may be provided as input to a second trained machine learning model for predicting the trajectory of the instrument. Optionally, any size parameters of the instrument provided as part of the selection of the instrument location function may also be provided as input. The second trained machine learning model may be a trained machine learning model trained using the method 1300 described with reference to FIG. 13. In an example where the exemplary method 1300 is used to generate and train a collection of machine learning models, the medical images may be provided as input to each machine learning model of the collection of machine learning models operating in parallel. In some examples, when the type of instrument is indicated as part of the selection of the instrument location function, only the machine learning models trained to identify a particular type of instrument may be operated.

[0121] At step 1410, a predicted trajectory of the instrument to reach the target may be received as an output of the second trained machine learning model. For example, the predicted trajectory may be a path from the distal end of the instrument to the target. At step 1412, a visualization of the predicted trajectory may be provided to the computing device for display. For example, the visualization may be generated based on the prediction in the post-processing step 206 described with reference to FIG. 2 and provided for display (e.g., on the display 138) via an application running on the imaging computing device 136.

[0122] In some examples, the visualization information may include at least two medical images simultaneously showing orthogonal views of the instrument. For example, the first image may be an LAX view of the instrument in which the length of the instrument may be visible, and the first image may be one image from a sequence of received medical images. In some examples, the image selected from the sequence may be an image at a given depth and frequency that is most easily visible by the human eye. The second image may be a short-axis (SAX) view of the instrument orthogonal to the LAX view, such that only the distal end of the instrument is visible. The SAX view may be captured simultaneously by the same probe as the LAX view if the probe is capable of simultaneous bidirectional imaging, and / or may be captured simultaneously with the LAX view by a different probe. In at least the first image, including the LAX view, the predicted trajectory of the instrument is visual. In addition, in each of the first and second images, the predicted position of the instrument, particularly the predicted position of the distal end of the instrument, may be visually indicated. An exemplary application user interface displaying visualization information having two medical images showing orthogonal views of an instrument is shown in FIG. 15, as described below.

[0123] In other examples, the visualization information may include a three-dimensional (3D) representation of the anatomical structure. The 3D representation may be generated from a sequence of medical images. In addition, a predicted trajectory of the instrument and / or a predicted position of the instrument output by a trained machine learning model may be utilized as part of the generation process such that the 3D representation visually shows a predicted trajectory of the instrument and / or a predicted position of the instrument (particularly the distal end of the instrument) to reach the target.

[0124] The exemplary method 1400 may be repeated as the operator moves the probe 134 to allow continuous visualization of the instrument's position and its trajectory to reach the target throughout the procedure.

[0125] The examples described above in Figures 13 and 14 include training and implementing a second machine learning model for predicting the trajectory of the instrument that is separate from the first machine learning model for predicting the position of the instrument, such as the models described in Figures 11 and 12. In other examples, rather than training and implementing a second machine learning model for predicting the trajectory of the instrument, the system 116 may computationally determine and visually project the trajectory of the instrument based on the position of the instrument predicted by the first machine learning model for predicting the position of the instrument (e.g., based on the determined orientation and / or length of the instrument, and the insertion angle).

[0126] FIG. 15 is an exemplary application user interface 1500 displaying a predicted instrument position and trajectory. For example, during a procedure such as a needle biopsy procedure, the application user interface 1500 may be presented on the display 138 via an application associated with the system 116 running on the imaging computing device 136 of the ultrasound imaging system 132. The user interface 1500 may simultaneously display images 1502, 1504 having orthogonal views of an instrument 1506, such as a needle. For example, the first image 1502 may be a LAX view of the instrument 1506, and the second image 1504 may be a SAX view of the instrument 1506 orthogonal to the LAX view, such that only the distal end 1508 of the instrument 1506 is visible. In each of the first and second images 1502, 1504, the position of the distal end 1508 of the instrument predicted using the exemplary method 1100 described with reference to FIG. 11 may be visually indicated. For example, the distal end 1508 may be highlighted, shaded, or colored to distinguish the distal end 1508 from the remainder of the instrument 1506.

[0127] Additionally, a target 1510 of the instrument (e.g., a mass for the needle to biopsy) may also be displayed as shown in the first image 1502, and a predicted trajectory 1512 for the instrument (e.g., from the distal end 1508) to the target 1510. The predicted trajectory 1512 may be determined and provided based on the example method 1300 described with reference to FIG.

[0128] Additionally, the user interface 1500 may include a probe direction indicator 1514 that suggests a direction for moving the probe 134 of the ultrasound imaging system 132, for example, to capture a more optimal image of the instrument 1506 (particularly the distal end 1508 of the instrument 1506). The probe direction indicator 1514 may include an arrow indicating that the operator should slide the probe (e.g., forward, backward, right, or left) and / or rotate the probe. The probe direction indicator 1514 may be highlighted, flashing, or otherwise animated such that the arrow pointing in the direction of the suggested movement can be visually directed to the operator's attention. The determination of whether the image of the instrument is optimal, and the subsequent determination of a suggested direction for moving the probe if the image is not optimal, may be determined using similar processes as described above with reference to FIGS. 8-9.

[0129] Anatomical feature identification, visualization, and probe-anatomical feature alignment for diagnostic and interventional observations The techniques presented in Figures 16-17 describe the training and use of a machine learning model that enables real-time identification and visualization of labeled anatomical features within ultrasound images displayed by the ultrasound imaging system 132 (e.g., via the display 138). Additionally, upon identification and visualization of anatomical features of interest, the anatomical features may be marked as the probe 134 of the ultrasound imaging system 132 is moved to provide context-aware imaging to the operator. The techniques presented in Figures 18-19 describe another machine learning model that is trained and used to detect probe alignment with anatomical features identified as output of, for example, the machine learning model described with reference to Figures 16-17.

[0130] To provide an illustrative example, as an operator of the ultrasound imaging system 132 moves a probe along the median nerve into the carpal tunnel, the distal tunnel bifurcates into smaller nerves following a change in depth within the carpal tunnel, making it difficult for the human eye to see at the distal tunnel. For example, certain image-guided procedures for treating carpal tunnel syndrome may involve inserting a sharp instrument, such as a knife, into this area of ​​the patient's body. Real-time identification and visual labeling of anatomical features (e.g., veins, arteries, nerves, tendons, ligaments, cartilage, muscles, bones, etc.) present within the displayed images to guide the procedure can increase the physician's confidence that the knife is being inserted against the desired anatomical feature (and not an unintended anatomical feature).

[0131] 16 shows a flowchart illustrating an example method 1600 for training a machine learning model (e.g., one of the trained machine learning models 118) to identify anatomical features according to example techniques presented herein. The example method 1300 (e.g., steps 1602-1606) may be performed by the system 116. The example method 1600 may include one or more of the following steps:

[0132] In step 1602, a plurality of training medical images may be received (e.g., from the medical image database 160 via the network 120). The training medical images may include ultrasound images of an anatomical structure having a plurality of anatomical features. For example, the ultrasound images may be of a wrist, and the anatomical features may include bones, nerves, ligaments, and / or tendons (e.g., depending on the view of the image). In some examples, the annotations for the anatomical features may be broader categories of anatomical features, such as veins, arteries, blood vessels, bones, nerves, ligaments, tendons, cartilage, etc., rather than specific anatomical features for that structure. In other words, continuing with the example where the image is of a wrist, the image may be annotated with nerves rather than the median nerve, and arteries rather than the ulnar artery. Training medical images may be received for a plurality of different types of anatomical structures (e.g., wrist, knee joint, shoulder joint, etc.).

[0133] The training medical images may be pre-processed (similar to the pre-processing described in step 204 of FIG. 2). In some examples, as part of the pre-processing, annotations may be extracted as features from the training medical images or otherwise removed from the training medical images and stored separately as ground truth labels for use in training the machine learning model in step 804. Alternatively, the annotations may be received separate from, but corresponding to, the training medical images.

[0134] At step 1604, a machine learning model may be generated and trained to identify a plurality of anatomical features in a target image of an anatomical structure using a plurality of training medical images. For example, the training medical images of the anatomical structure may be input to the machine learning model. The machine learning model may output the anatomical features of the anatomical structure as a prediction.

[0135] To train the machine learning model, anatomical features output by the machine learning model for the training medical images may be compared to ground truth labels (e.g., annotations of anatomical features) corresponding to the training medical images to determine loss or error. For example, anatomical features identified in a first training image of an anatomical structure may be compared to known anatomical features in the first training image of the anatomical structure indicated by annotations in the corresponding ground truth labels. The machine learning model may be modified or altered (e.g., weights and / or biases may be adjusted) based on the error to improve the accuracy of the machine learning model. This process may be repeated for each training image, or at least until the determined loss or error is below a predetermined threshold. In some examples, at least a portion of the training images and corresponding labels may be retained and used to further validate or test the trained machine learning model.

[0136] Once the machine learning model is sufficiently trained, in step 1606, the trained machine learning model may be stored (e.g., as one of the trained machine learning models 118 stored in the storage device 114) for subsequent use. In some examples, the trained machine learning model may be a single machine learning model that has been generated and trained to identify anatomical structures within a plurality of different anatomical structures. In other examples, the exemplary method 1600 may be executed to generate and train a collection of machine learning models, each model identifying anatomical features of a particular anatomical structure (e.g., one model may identify anatomical features of a shoulder joint, another model may identify anatomical features of a wrist, a further model may identify anatomical features of a knee joint, etc.). When deployed to evaluate medical images, the collection of machine learning models may be executed in parallel.

[0137] 17 shows a flowchart illustrating an example method 1700 for visualizing anatomical features of an identified anatomical structure in a medical image according to example techniques presented herein. The example method 1700 (e.g., steps 1702-1708) may be performed by the system 116. The example method 1700 may include one or more of the following steps:

[0138] In step 1702, a medical image (e.g., a target image) of an anatomical structure may be received from a computing device. The computing device may be the imaging computing device 136 of the ultrasound imaging system 132. The ultrasound imaging system 132 may facilitate diagnosis and / or guide intervention of a patient under examination. The medical image may be generated by the imaging computing device 136 from sound waves transmitted and received from the probe 134, and may be received in real time, for example, via an application associated with the system 116 running on the imaging computing device 136. In some examples, the operator may also input the type of anatomical structure to be imaged (e.g., wrist, shoulder joint, knee joint, etc.) via the application user interface.

[0139] In step 1704, the medical image may be provided as input to a trained machine learning model for identifying anatomical features of an anatomical structure, such as a trained anatomical feature identification machine learning model trained using the method 1600 described with reference to FIG. 16. In examples where the exemplary method 1600 is used to generate and train a collection of machine learning models, the medical image may be provided as input to each machine learning model of the collection of machine learning models operating in parallel. In some examples, if a type of anatomy is selected by a physician, only the machine learning model trained to identify anatomical features of the selected type of anatomy is run (e.g., to conserve computational resources).

[0140] In step 1706, anatomical features identified in the medical image may be received as output (e.g., as predictions) from the trained machine learning model. Based on the level of specificity of the annotations used to train the machine learning model, the anatomical features identified may be broad categories of anatomical features (e.g., nerves, arteries, bones) or may be specific anatomical features of a given anatomical structure (e.g., median nerve, ulnar artery, pisiform carpal bone).

[0141] In step 1708, visualization information of anatomical features identified in the medical image may be provided to a computing device for display. For example, the visualization information may be generated based on predictions in the post-processing step 206 described with reference to FIG. 2. In some examples, within the visualization, the identified anatomical features may be labeled or differentiated from one another using a visual scheme. For example, different colors, shading, patterning, and / or highlighting may be used for each category of anatomical features. For example, a first color may be used to identify bones, a second color may be used to identify nerves, a third color may be used to identify tendons, etc. An exemplary application user interface displaying the visualization information is shown in FIG. 18 as described below. Anatomical features may be labeled in all axes.

[0142] In some examples, the anatomical features of the anatomy identified and output by the trained machine learning model may be provided as input to another process. One exemplary process may include a process for detecting probe alignment with one or more of the identified anatomical features, as described with reference to FIG. 19 and FIG. 20 below. Another exemplary process may enable context-aware imaging. For example, one or more anatomical features identified in a medical image may be marked for continued observation as an operator moves the probe to capture subsequent images of the anatomy. In some examples, an operator may provide an input, for example, via an application, indicating which anatomical features should be marked for continued observation. For example, an operator may provide a touch input to a display of a computing device in an area of ​​the medical image corresponding to the anatomical feature to be marked. The location of the individual anatomical features marked for continued observation relative to the location of the probe relative to the medical image may be identified for use as a reference.

[0143] The position of the anatomical structure itself is fixed; that is, only the probe is moving, and no anatomical structure in the patient's body is moving. Thus, by knowing a first position (e.g., a fiducial) of the anatomical feature relative to the probe prior to the probe movement and determining the probe movement, a second position of the anatomical feature relative to the probe movement can be determined. The probe movement can be detected using a sensor attached to the probe, such as an accelerometer, gyroscope, and / or inertial sensor. Once determined, the second position of the anatomical feature can be visually indicated in a corresponding medical image generated during the probe movement.

[0144] 18 is an example application user interface 1800 that displays visualization information 1804 of anatomical features of an anatomical structure identified within a medical image 1802 of the anatomical structure. The application user interface 1800 may be a user interface of an application associated with the system 116 running on a computing device, such as the imaging computing device 136 of the ultrasound imaging system 132. When the ultrasound imaging system 132 is operating to image a patient's anatomical structure, the application user interface 1800 may be displayed on the display 138 such that an operator (e.g., a physician) can observe the visualization information 1804 while continuing to image the patient. In this illustrative example, the anatomical structure being imaged may be the patient's wrist.

[0145] Anatomical features of the wrist identified in the medical image 1802 (e.g., via the method 1700 described above with reference to FIG. 17) in the visualization information 1804 may be labeled or differentiated from one another using a visual scheme. The application user interface 1800 may include a legend 1806 or other similar key to inform a physician viewing the image of the visual scheme used to label each of the identified anatomical features. For example, the anatomical features of the wrist identified in the medical image 1802 may include, among other examples, nerves 1808, bones 1810, and tendons 1812, and each category of anatomical features may be labeled with a different outline scheme in the medical image 1802. For example, as shown in the legend, a first outline pattern may be used to identify the nerves 1808, a second outline pattern may be used to identify the bones 1810, and a third outline pattern may be used to identify the tendons 1812 in the medical image 1802. In other examples, rather than patterning, the visual scheme may be based on color, shading, highlighting, and / or other similar visual schemes.

[0146] The application user interface 1800 described above is provided by way of example only and may include additional, less, different, or differently configured information and / or visual schemes than that shown in FIG. 18.

[0147] 19 shows a flowchart illustrating an example method 1900 for training a machine learning model (e.g., one of the trained machine learning models 118) to detect probe alignment with an anatomical feature identified in a medical image according to example techniques presented herein. The example method 1900 (e.g., steps 1902-1906) may be performed by the system 116. The example method 1600 may include one or more of the following steps:

[0148] At step 1902, a plurality of labeled training medical images may be received (e.g., from medical image database 160 via network 120). The training medical images may include ultrasound images of an anatomical structure having a plurality of anatomical features, each of which has been generated by an ultrasound imaging system similar to ultrasound imaging system 132. In some examples, the training medical images may represent a plurality of different anatomical structures. The training medical images may include annotations for at least one or more anatomical features of interest, such as an anatomical feature to which a probe of an ultrasound imaging system should be aligned to achieve an optimal image. As one illustrative example, to obtain an optimal image, the probe may be in parallel alignment with tendon or ligament fibers (generally referred to herein as fibers) of a joint. The sagittal cross-sectional area of ​​the fiber (e.g., a cross section diagonally down along the fiber) decreases as the probe approaches parallel (e.g., in-line) alignment with the fiber. In contrast, as the probe moves away from in-line alignment, the sagittal cross-sectional area of ​​the fiber increases again. In addition, the training medical images may have corresponding ground truth labels that indicate whether one or more anatomical features are sufficiently aligned with the probe, and in some examples, the corresponding ground truth labels may also include details of the orientation associated with the probe that gave rise to a given alignment.

[0149] The training medical images may be pre-processed (similar to the pre-processing described in step 204 of FIG. 2). In some examples, as part of the pre-processing, annotations may be extracted or otherwise identified from the training medical images to form labels separate from the training medical images. In other examples, the annotations may be received as labels separate from the training medical images.

[0150] In step 1904, a machine learning model may be generated and trained to detect whether there is a probe alignment with an anatomical feature using the plurality of labeled training medical images. For example, training medical images of an anatomical structure may be input to the machine learning model. The machine learning model may output as a prediction whether the probe is in sufficient alignment with one or more anatomical features of interest. For example, the output may be either a prediction of sufficient alignment or a prediction of insufficient alignment. In some examples, the output may also include a predicted orientation of the probe that will result in an alignment.

[0151] To train the machine learning model, the alignment predictions output by the machine learning model for the training medical images may be compared to the ground truth labels corresponding to the training medical images to determine a loss or error. For example, the alignment prediction for a first training medical image may be compared to a known sufficiency or insufficiency of the alignment of the probe with one or more anatomical features of interest identified by the corresponding ground truth label. In addition, if the alignment prediction output also includes a predicted orientation of the probe resulting in an alignment state, the predicted orientation may be compared to the known orientation of the probe included in the corresponding ground truth label. The machine learning model may be modified or altered (e.g., weights and / or biases may be adjusted) based on the error to improve the accuracy of the machine learning model. This process may be repeated for each training image, or at least until the determined loss or error is below a predetermined threshold. In some examples, at least a portion of the training images and corresponding labels may be retained and used to further validate or test the trained machine learning model.

[0152] Once the machine learning model is sufficiently trained, in step 1906, the trained machine learning model may be stored (e.g., as one of the trained machine learning models 118 stored in the storage device 114) for subsequent use. In some examples, the trained machine learning model may be a single machine learning model that has been generated and trained to predict probe-anatomical feature alignment for a number of different anatomical structures. In other examples, the exemplary method 1900 may be executed to generate and train a collection of machine learning models, each of which predicts whether there is a probe alignment with a particular anatomical feature of interest for a given anatomical structure (e.g., one model for detecting probe-anatomical feature alignment for the shoulder joint, another model for identifying probe-anatomical feature alignment for the wrist). When deployed to evaluate medical images of anatomical structures, the collection of machine learning models may be executed in parallel.

[0153] 20 is a flow chart illustrating an example method 2000 for detecting probe alignment with one or more anatomical features of an anatomical structure identified in a medical image of the anatomical structure, according to example techniques presented herein. The example method 2000 (e.g., steps 2002-2010) may be performed by the system 116. The example method 2000 may include one or more of the following steps:

[0154] In step 2002, a medical image (e.g., a target image) of an anatomical structure may be received from a computing device. The computing device may be the imaging computing device 136 of the ultrasound imaging system 132. The ultrasound imaging system 132 may facilitate diagnosis and / or guide intervention of a patient under examination. The medical image may be generated by the imaging computing device 136 from sound waves transmitted and received from the probe 134, and may be received in real time, for example, via an application associated with the system 116 running on the imaging computing device 136. In some examples, the operator may also input the type of anatomical structure to be imaged (e.g., wrist, shoulder joint, knee joint, etc.) via the application user interface.

[0155] In step 2004, the medical image may be provided as an input to a first trained machine learning model to obtain anatomical features identified in the medical image as an output of the first trained machine learning model. The first trained machine learning model may be a trained machine learning model for identifying anatomical features of an anatomical structure as described with reference to Figures 16 and 17.

[0156] In step 2006, the medical image annotated with the identified anatomical features output by the first trained machine learning model may be provided as input to a second trained machine learning model for predicting whether there is a probe-anatomical feature alignment. That is, the second trained machine learning model may predict whether the probe 134 capturing the medical image is aligned with one or more anatomical features of interest to the anatomical structure. The second trained machine learning model may be a trained machine learning model trained using the method 1900 described with reference to FIG. 19. In an example where the exemplary method 1900 is used to generate and train a collection of machine learning models, the medical image may be provided as input to each machine learning model of the collection of machine learning models operating in parallel. In some examples, if a type of anatomical structure is selected by a physician, only the machine learning models trained to predict probe-anatomical feature alignment associated with the type of anatomical structure (e.g., trained based on a particular anatomical feature(s) of interest for that type of anatomical structure) may be executed (e.g., to conserve computing resources).

[0157] In some examples, the medical image may be annotated with all identified anatomical features output by the first trained machine learning model before being input to the second trained machine learning model. In other examples, the received medical image may be annotated with only the particular anatomical feature(s) of interest (e.g., from the identified anatomical features) for a given anatomical structure.

[0158] In step 2008, an alignment prediction may be received as an output of the second trained machine learning model. The alignment prediction may include whether the probe is in sufficient or insufficient alignment with the anatomical feature(s) of interest relative to the anatomical structure. Additionally, the alignment prediction may include a predicted orientation of the probe that will result in an aligned or misaligned state.

[0159] In step 2010, an indication of the alignment prediction may be provided to the computing device for display (e.g., via an application executing on the imaging computing device 136). For example, visualization information may be generated based on the alignment prediction in the post-processing step 206 described with reference to FIG. 2. As an example, if the prediction is that the probe is in sufficient alignment with the anatomical feature(s) of interest to the anatomy, a positive alignment indicator may be overlaid on the medical image to inform the operator that the probe is in alignment and therefore an optimal image has been captured. As another example, if the prediction is that the probe is in poor alignment with the anatomical feature(s) of interest to the anatomy, a negative alignment indicator may be overlaid on the medical image to inform the operator that the probe is not in alignment and therefore another, more optimal image should be captured. The alignment indicator may include text, graphics, animation, and / or combinations thereof.

[0160] In some examples, if the probe is poorly aligned and the alignment prediction further includes a predicted orientation of the probe resulting in misalignment, directional instructions may be generated to guide the operator to adjust the current orientation of the probe corresponding to the predicted orientation to achieve alignment and capture of optimal images. For example, as part of training, the second trained machine learning model may learn the probe orientation associated with the alignment of the probe with the anatomical feature from the labeled training medical images. Thus, the learned probe orientation associated with the alignment of the probe with the anatomical feature may be compared to the current probe orientation to generate directional instructions to adjust the current probe orientation to the learned probe orientation. In some examples, the directional instructions may be in the form of a probe orientation indicator similar to the probe orientation indicators described above with reference to Figures 10 and 15. The exemplary method 2000 may then be repeated iteratively as the probe is reoriented (e.g., adjusted in response to the directional instructions).

[0161] To provide an illustrative example of an operator's experience while method 2000 is performed iteratively, the operator may place the probe 134 in contact with the patient's skin in a first orientation, a second orientation, a third orientation, etc. (e.g., as part of a scanning operation). Medical images may be captured at each orientation and processed (e.g., via the steps of method 2000) to predict alignment of the probe with the anatomical feature, and alignment indicators and / or directional indications may be displayed on the display 138. Returning to the illustrative example in which the probe may be in parallel alignment with the fibers of a joint to obtain an optimal image, the sagittal cross-sectional area of ​​the fibers decreases as the probe approaches parallel (e.g., in-line) alignment with the fibers and increases again as the probe moves away from in-line alignment. Continuing with this example, when the probe is in a first orientation, the operator may be instructed to reorient the probe to a second orientation based on the predicted misalignment. Given the misalignment, the sagittal cross-sectional area of ​​the fibers has a first area in the medical image that corresponds to the first orientation. When in the second orientation, the operator may be instructed to redirect the probe to a third orientation (e.g., an orientation between the first and second orientations) based on the predicted opposite misalignment. For example, given a misalignment, the sagittal cross-sectional area of ​​the fiber may have a second area in the medical image corresponding to the second orientation, which may be greater than the first area in the medical image corresponding to the first orientation, indicating that the operator's reorientation overshot the alignment during the scan.

[0162] Determining optimal image capture for diagnostic and interventional observations 21 shows a flowchart illustrating an example method 2100 for training a machine learning model (e.g., one of the trained machine learning models 118) to identify optimal image frames according to example techniques presented herein. The example method 2100 (e.g., steps 2102-2106) may be performed by the system 116. The example method 2100 may include one or more of the following steps:

[0163] At step 2102, a plurality of labeled training medical images of an anatomical structure affected by a musculoskeletal disorder may be received. The training medical images may be comprised of a plurality of image subsets corresponding to a minimum number of image types to be acquired as part of a procedure related to a musculoskeletal disorder. For example, each image in an image subset corresponds to an image type to be acquired as part of the procedure. In some examples, the procedure may be a diagnostic procedure or a therapeutic procedure. The image types to be acquired may include a particular view angle of an anatomical feature in the anatomical structure. As one illustrative example, the procedure may be a diagnostic procedure for carpal tunnel syndrome in which at least three images of a wrist should be acquired from a first view angle, a second view angle, and a third view angle. In some examples, the training medical images may represent a plurality of different procedures related to a plurality of different types of musculoskeletal disorder.

[0164] The training medical images may be annotated with anatomical features of the anatomy, including at least any anatomical features to be included in one or more of the image types. The training medical images may have corresponding ground truth labels. For example, each image in the image subset may be labeled as an optimal image or a non-optimal image for a respective image type. In some techniques, the ground truth labels may also include the probe orientation (e.g., affecting the captured viewing angle) that results in an optimal or non-optimal image. The training medical images may be pre-processed (similar to the pre-processing described in step 204 of FIG. 2). In some examples, as part of the pre-processing, annotations may be extracted or otherwise identified from the training medical images to form labels separate from the training medical images. In other examples, the annotations may be received as labels separate from the training medical images.

[0165] In step 2104, a machine learning model for predicting an optimal image for each image type may be generated using the plurality of labeled training medical images. To train the machine learning model, a prediction output by the machine learning model for the training medical images may be compared to a ground truth label corresponding to the training medical image to determine a loss or error. For example, an optimal or non-optimal prediction for a first training medical image of a first image subset corresponding to a first image type may be compared to a corresponding optimal or non-optimal ground truth label for the training medical image. Furthermore, if the prediction output also includes a predicted orientation of the probe resulting in an optimal or non-optimal image, the predicted orientation may be compared to a known orientation of the probe included in the corresponding ground truth label. The machine learning model may be modified or altered (e.g., weights and / or biases may be adjusted) based on the error to improve the accuracy of the machine learning model. This process may be repeated for each training image, or at least until the determined loss or error is below a predetermined threshold. In some examples, at least a portion of the training images and corresponding labels may be retained and used to further validate or test the trained machine learning model.

[0166] Once the machine learning model is sufficiently trained, in step 2106, the trained machine learning model may be stored (e.g., as one of the trained machine learning models 118 stored on the storage device 114) for subsequent use. In some examples, the trained machine learning model may be a single machine learning model that has been generated and trained to predict optimal images for each image type to be acquired for a plurality of different procedures associated with a plurality of different musculoskeletal disorders. In other examples, the exemplary method 2100 may be executed to generate and train a collection of machine learning models, each model predicting optimal images for each image type to be acquired for a particular procedure associated with a particular musculoskeletal disorder (e.g., one model may predict optimal images for each image type to be acquired to diagnose carpal tunnel, another model may detect optimal images for each image type to be acquired during needle biopsy of a region of interest on the shoulder joint, etc.). When deployed to evaluate medical images of anatomical structures, the collection of machine learning models may be executed in parallel.

[0167] 22 shows a flowchart illustrating an example method 2200 for capturing an optimal image frame according to an example technique presented herein. The example method 2200 (e.g., steps 2202-2210) may be performed by a system. The example method 2200 may include one or more of the following steps:

[0168] In step 2202, a type of procedure being performed may be received from a computing device. The computing device may include an imaging computing device 136 of the ultrasound imaging system 132 on which an application associated with the system 116 is running. A user of the computing device may select a type of procedure via a user interface of the application. In some examples, upon receiving a selection of the type of procedure, the application may provide instructions to the operator indicating a minimum set of image types to be acquired for the selected type of procedure. In addition, the application may provide instructions prompting the operator to place the probe 134 in contact with the patient's skin and orient the probe 134 to capture a medical image of the individual image type (e.g., the individual image type is one image type in the minimum set of image types). In step 2204, a medical image of the individual image type (e.g., a live image received in real time) may be received from the computing device.

[0169] In step 2206, the medical images of the individual image types may be provided as input to a trained machine learning model, such as the trained machine learning model trained using method 2100 described with reference to FIG. 21. In some examples, a collection of machine learning models for predicting optimal images for each image type to be acquired for a plurality of different procedures associated with a plurality of different musculoskeletal disorders may be generated, trained, and stored (e.g., as disclosed in method 2100), for example, in storage device 114. In such examples, system 116 may query storage device 114 for the type of procedure received in step 2202 to obtain a corresponding trained machine learning model for predicting optimal images for each image type (e.g., including individual image types) acquired for the type of procedure.

[0170] At step 2208, a prediction of whether the medical image is an optimal image for a particular image type may be received as an output of the trained machine learning model. In some examples, the prediction output may include a predicted orientation of the probe that will result in an optimal or non-optimal image. Based on the prediction, at step 2210, instructions may be provided to a computing device (e.g., the imaging computing device 136) to cause the computing device to perform an action related to the medical image or to prompt the operator to make a probe adjustment.

[0171] For example, if the medical image is predicted to be an optimal image for the individual image type, the instructions provided to the computing device may be commands to cause the computing device to automatically freeze the live medical image and capture (e.g., record and save) a still image. In other examples, rather than the computing device automatically performing these actions, the instructions may instead cause an operator to be prompted to perform these actions (e.g., freeze and capture a still image using control elements on the application user interface). Once a still image for the individual image type has been captured, steps 2204-2208 may be repeated for the next image type in the minimum set of image types to be acquired for the type of procedure received in step 2202. This may continue until still images have been acquired for all image types in the minimum set of image types.

[0172] As another example, if the medical image is predicted to be a non-optimal image for the individual image type, the instructions provided to the computing device may cause a prompt to be displayed for the operator to make a probe adjustment (e.g., reorient the probe) to enable capture of an optimal image for the individual image type. For example, the prompt may include directional instructions to guide the operator to adjust the current orientation of the probe to correspond to the predicted orientation to achieve capture of an optimal image. For example, as part of training, the trained machine learning model may learn a probe orientation associated with an optimal image for the individual image type from the labeled training medical images. Thus, the learned probe orientation associated with capture of an optimal image for the individual image type may be compared to the current probe orientation to generate directional instructions to adjust the current probe orientation to the learned probe orientation. In some examples, the directional instructions may be in the form of a probe orientation indicator similar to the probe orientation indicators described above with reference to Figures 10 and 15. Steps 2204-2208 of the exemplary method 2200 may then be repeated iteratively as the probe is reoriented (e.g., adjusted in response to the directional instructions) until an optimal image for the individual image type is predicted and captured.

[0173] FIG. 23 illustrates an exemplary system or device 2300 that may perform the techniques presented herein. The device 2300 may include a central processing unit (CPU) 2320. The CPU 2320 may be any type of processor device, including, for example, any type of dedicated or general-purpose microprocessor device. As will be appreciated by those skilled in the art, the CPU 2320 may also be a single processor in a multi-core / multi-processor system, such as a system operating alone, or in a cluster of computing devices operating in a cluster or server farm. The CPU 2320 may be connected to a data communications infrastructure 2310, for example, a bus, a message queue, a network, or a multi-core message passing scheme.

[0174] The device 2300 may also include a main memory 2340, e.g., random access memory (RAM), and may also include a secondary memory 2330. The secondary memory 2330, e.g., read only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such removable storage drives may include, for example, floppy disk drives, magnetic tape drives, optical disk drives, flash memory, and the like. The removable storage drives in this example read from and / or write to the removable storage units in a known manner. The removable storage may include floppy disks, magnetic tapes, optical disks, and the like that are read from and written to by the removable storage drives. As will be appreciated by those skilled in the art, such removable storage units generally include computer usable storage media having computer software and / or data stored thereon.

[0175] In alternative implementations, the secondary memory 2330 may include similar means for allowing computer programs or other instructions to be loaded into the device 2300. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that allow software and data to be transferred from removable storage units to the device 2300.

[0176] The device 2300 may also include a communications interface ("COM") 2360. The communications interface 2360 allows software and data to be transferred between the device 2300 and external devices. The communications interface 2360 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. The software and data transferred via the communications interface 2360 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by the communications interface 2360. These signals may be provided to the communications interface 2360 over a communications path of the device 2300, which may be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channel.

[0177] The hardware elements, operating systems, and programming languages ​​of such equipment are conventional in nature, and are presumed to be fully familiar to those skilled in the art. The device 2300 may also include input and output ports 2350 for connecting with input and output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Of course, the various server functions may be implemented in a distributed manner on several similar platforms to distribute the processing load. Alternatively, the server may be implemented by suitable programming of one computer hardware platform.

[0178] Throughout this disclosure, references to components or modules generally refer to items that may be logically grouped together to perform a function or group of related functions. Like reference numbers are generally intended to refer to the same or similar components. The components and / or modules may be implemented in software, hardware, or a combination of software and / or hardware.

[0179] The tools, modules, and / or functions described above may be executed by one or more processors. A "storage" type medium may include tangible memory of a computer, a processor, or any or all of their associated modules, such as various semiconductor memories, tape drives, disk drives, etc., that may provide non-transitory storage at any time for software programming.

[0180] The software may be communicated through the Internet, a cloud service provider, or other telecommunications network. For example, the communication may enable the software to be loaded from one computer or processor to another. As used herein, unless limited to non-transitory tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0181] The foregoing general description is exemplary and explanatory only and is not intended to limit the present disclosure. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and practice of the invention be considered as exemplary only.

Claims

1. 1. A system for processing ultrasound images to identify objects, comprising: A processor; a memory coupled to the processor for storing instructions that, when executed by the processor, cause the system to: receiving an ultrasound image of an anatomical structure from a computing device of the ultrasound imaging system; providing the ultrasound image as an input to a machine learning model trained to identify a plurality of objects within the ultrasound image of the anatomical structure, where the plurality of objects includes an anatomical feature, a destructive feature, and / or an instrument; receiving as an output of the machine learning model a prediction of one or more objects from the plurality of objects identified in the ultrasound image; providing an indication of the prediction to the computing device for display on a display of the ultrasound imaging system.

2. The operation includes: generating a visualization based on the prediction, where the visualization includes a labeling of the one or more objects identified in the ultrasound image; The system of claim 1 , further comprising providing the visualization as an instruction.

3. The operation includes: determining a measurement associated with at least one of the identified one or more objects; The system of claim 1 , further comprising providing the measurements for display on the display of the ultrasound imaging system along with the identified at least one object.

4. The machine learning model is a first machine learning model, and the operation comprises:

2. The system of claim 1, further comprising providing the ultrasound image as an input to a second machine learning model trained to identify whether the ultrasound image is an optimal image of the one or more objects identified by the first machine learning model.

5. If the second machine learning model outputs a prediction that the ultrasound image is a non-optimal image, the operation includes: generating a prompt for an operator to adjust an orientation of a probe of the ultrasound imaging system from a current orientation to a new orientation learned by the second machine learning model in order to capture an optimal image of the identified one or more objects; The system of claim 4 , further comprising providing the prompt to the computing device for display on the display of the ultrasound imaging system.

6. The operation includes: receiving an operator input from the computing device of the ultrasound imaging system indicating one or more types of objects to be identified; 2. The system of claim 1, further comprising providing the one or more object types as input to the machine learning model such that the prediction of the one or more objects identified in the ultrasound image includes at least the one or more object types.

7. 2. The system of claim 1, wherein the ultrasound images are received during a diagnostic procedure, and the prediction of the identified one or more objects includes anatomical features and disruptive features to the anatomical structures indicative of a musculoskeletal disorder, and the indication of the prediction provided to the computing device for display on the display of the ultrasound imaging system facilitates diagnosis of the musculoskeletal disorder.

8. 2. The system of claim 1, wherein the ultrasound images are received during an interventional procedure, and the interventional procedure is guided by the indication of the prediction provided to the computing device for display on the display of the ultrasound imaging system.

9. The system of claim 1 , wherein the anatomical features include bones, tendons, ligaments, cartilage, muscles, nerves, veins, and / or arteries.

10. The system of claim 1 , wherein the disruptive features include ganglions, pleural effusions, calcifications, masses, and / or lesions.

11. The system of claim 1 , wherein the instrument comprises a needle, scalpel, knife, tool, and / or balloon.

12. 1. A method for processing ultrasound images to identify objects, comprising: receiving an ultrasound image of an anatomical structure from a computing device of the ultrasound imaging system; providing the ultrasound image as an input to a machine learning model trained to identify a plurality of objects within the ultrasound image of the anatomical structure, where the plurality of objects includes an anatomical feature, a destructive feature, and / or an instrument; receiving as an output of the machine learning model a prediction of one or more objects from the plurality of objects identified in the ultrasound image; providing an indication of the prediction to the computing device for display on a display of the ultrasound imaging system.

13. 1. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform operations for processing an ultrasound image to identify an object, the operations including: receiving an ultrasound image of an anatomical structure from a computing device of the ultrasound imaging system; providing the ultrasound image as an input to a machine learning model trained to identify a plurality of objects within the ultrasound image of the anatomical structure, where the plurality of objects includes an anatomical feature, a destructive feature, and / or an instrument; receiving as an output of the machine learning model a prediction of one or more objects from the plurality of objects identified in the ultrasound image; and providing an indication of the prediction to the computing device for display on a display of the ultrasound imaging system.

Images