Mixed reality guidance for ultrasound probes

Abstract

Techniques and systems are described for mixed reality (MR) guidance of an ultrasound probe that acquire reference data depicting a patient's bones, determine a physical position of an ultrasound probe, generate alignment data based on first ultrasound data generated by the ultrasound probe to align virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones, generate virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, where the virtual guidance provides guidance to a clinician regarding how the ultrasound probe is positioned relative to a target position at which the ultrasound probe can generate second ultrasound data that provides information about the patient's soft tissue structure, and cause an MR visualization device to output the virtual guidance to the clinician.

Description

[Technical Field]

[0001]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 273,008, filed October 28, 2021, the entire contents of which are incorporated by reference. [Background technology]

[0002]

[0002] Planning and performing surgery, diagnosing a condition, or performing other types of medical tasks may involve obtaining information about a patient's anatomy. Information about the patient's anatomy may include information about the patient's bones, such as the size, shape, and position of the patient's bones. Additionally, information about the patient's anatomy may also include information about the patient's various soft tissue structures, such as the location and quality of muscles, tendons, ligaments, cartilage, retinaculum, blood vessels, etc. Obtaining high-quality information about both the patient's bones and the patient's soft tissue structures may involve different skill sets. Summary of the Invention

[0003]

[0003] This disclosure describes a technique in which mixed reality (MR) guidance is used to assist a clinician in positioning an ultrasound probe to obtain information about soft tissue structures involved in a procedure, such as an orthopedic surgery. As described herein, a computing system may acquire reference data depicting at least one bone of a patient. Exemplary types of reference data may include one or more computed tomography (CT) images, magnetic resonance imaging (MRI) images, nuclear magnetic resonance (NMR) images, etc. Furthermore, the computing system may use the reference data to generate virtual guidance. The virtual guidance provides guidance to the clinician regarding how to position the ultrasound probe relative to a target position where the ultrasound probe can generate second ultrasound data that provides information about the patient's soft tissue structure. For example, the virtual guidance may instruct the clinician on how to move the ultrasound probe so that it is in a target position for generating ultrasound data that provides information about the patient's soft tissue structure. The computing system may cause a head-mounted MR visualization device to output the virtual guidance to the clinician.

[0004]

[0004] In one example, the present disclosure describes a method comprising obtaining reference data depicting a patient's bones; determining a physical position of an ultrasound probe; generating alignment data based on first ultrasound data generated by the ultrasound probe, the alignment data aligning virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones; generating virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the virtual guidance provides guidance to a clinician regarding how the ultrasound probe is positioned relative to a target position at which the ultrasound probe can generate second ultrasound data that provides information about the patient's soft tissue structure; and causing a head-mounted mixed reality (MR) visualization device to output the virtual guidance to the clinician.

[0005]

[0005] In another example, the present disclosure describes a system comprising a memory configured to store reference data depicting bones of a patient, and a processing circuit, the processing circuit configured to: determine a physical position of an ultrasound probe; generate alignment data based on first ultrasound data generated by the ultrasound probe, that aligns virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones; generate virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the virtual guidance provides guidance to a clinician regarding how the ultrasound probe is positioned relative to a target position at which the ultrasound probe can generate second ultrasound data that provides information about the patient's soft tissue structure; and cause a head-mounted mixed reality (MR) visualization device to output the virtual guidance to the clinician.

[0006]

[0006] The details of various examples of the disclosure are set forth in the accompanying drawings and the description below. Various features, objects, and advantages will be apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0007] [Figure 1]

[0007] FIG. 1 is a conceptual diagram illustrating an example system in which one or more techniques of the present disclosure may be implemented. [Figure 2]

[0008] FIG. 1 is a conceptual diagram illustrating an example computing system in accordance with one or more techniques of the present disclosure. [Figure 3]

[0009] 1 is a flowchart illustrating an example operation of a system in accordance with one or more techniques of this disclosure. [Figure 4]

[0010] 1 is a flowchart illustrating an example operation of a system for generating alignment data, in accordance with one or more techniques of this disclosure. [Figure 5]

[0011] FIG. 1 is a conceptual diagram illustrating curve matching in accordance with one or more techniques of the present disclosure. [Figure 6]

[0012] FIG. 10 is a conceptual diagram illustrating an example virtual guidance during an ultrasound examination of a patient's shoulder, in accordance with one or more techniques of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008]

[0013] Clinicians, such as surgeons, may need to obtain information about a patient's bones and soft tissues before, during, or after performing a medical task, such as surgery. For example, when planning a shoulder replacement, a surgeon may need to obtain information about the scapula, humerus, and rotator cuff muscles. Computed tomography (CT) images and three-dimensional (3D) models generated based on CT images provide highly accurate depictions of a patient's bones. However, because CT images are generated using X-rays, which easily pass through most soft tissue structures, CT images often cannot provide high-quality information about a patient's soft tissue structures. On the other hand, ultrasound images, while capable of providing high-quality information about soft tissue structures, provide less accurate information about bones than CT images.

[0009]

[0014] Clinicians may require specialized training to acquire the ability to position an ultrasound probe to obtain high-quality ultrasound images. For example, it may be difficult for an untrained clinician to position an ultrasound probe to gather useful information about a particular muscle or tendon. Therefore, the need for a trained ultrasound technician may increase costs and delays associated with performing a procedure. Robotic probe positioning systems have been developed to position ultrasound probes. However, access to such robotic probe positioning systems may be limited and expensive. Furthermore, robotic probe positioning systems may be unsightly and distract the surgeon during surgery.

[0010]

[0015] This disclosure describes techniques that can improve the process of using an ultrasound probe to gather information for a medical task. As described herein, a computing system may acquire reference data depicting a patient's bones. The reference data may include one or more CT images (e.g., multiple CT images) of the bones, a three-dimensional (3D) model of the bones, or another type of medical image depicting the patient's bones. Furthermore, the computing system may determine a physical position of the ultrasound probe. Based on first ultrasound data generated by the ultrasound probe, the computing system may generate registration data that aligns virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones. Furthermore, the computing system may generate virtual guidance based on the reference data, the registration data, and the physical position of the ultrasound probe. The virtual guidance may provide guidance to a clinician regarding how to position the ultrasound probe relative to a target position at which the ultrasound probe can generate second ultrasound data that provides information about the patient's soft tissue structure. For example, the virtual guidance may instruct the clinician on how to move the ultrasound probe so that the ultrasound probe is in a target position to generate second ultrasound data that provides information about the patient's soft tissue structure. The computing system may also cause a head-mounted mixed reality (MR) visualization device to output the virtual guidance to the clinician. In this manner, the clinician may be able to both see the patient and the virtual guidance. Using the virtual guidance in this manner may help the clinician obtain information about the soft tissue structure.

[0011]

[0016] Further, in some examples, the computing system may generate a virtual model (e.g., a two-dimensional (2D) or 3D model) of the soft tissue structure based on the data generated by the ultrasound probe. The MR visualization device may output the virtual model of the soft tissue structure such that the virtual model of the soft tissue structure appears to the clinician as superimposed on the patient in the actual location of the soft tissue structure. The MR visualization device may also output a virtual model of one or more bones of the patient such that the virtual bones of the patient appear to the clinician as superimposed on the patient in the actual location of the bones. In this manner, the clinician may easily grasp the location of hidden soft tissue structures and bones of the patient. In some examples, the computing system may output the virtual model of the soft tissue structure and the virtual model of the bone and display the virtual model of the soft tissue structure and the virtual model of the bone on a monitor. In such examples, the clinician can use these virtual models for various purposes, such as preoperative planning.

[0012]

[0017] FIG. 1 is a conceptual diagram illustrating an example system 100 in which one or more techniques of the present disclosure may be implemented. In the example of FIG. 1, the system 100 includes one or more computing devices 102, an MR visualization device 104, an ultrasound probe 106, and a medical imaging system 108. A clinician 110 is using the ultrasound probe 106 to perform an examination on a patient 112 positioned on a table 114. The clinician 110 may be a surgeon, nurse, technician, medical student, doctor, or other type of medical professional or participant. The clinician 110 and the patient 112 do not form part of the system 100. The MR visualization device 104 may use markers 116A, 116B (collectively referred to as “markers 116”) to determine the position of the patient 112. Although the example of FIG. 1 shows the clinician 110 performing an ultrasound examination on the shoulder of the patient 112, the techniques of the present disclosure may be applicable to other parts of the patient's 112 body, such as the foot, ankle, knee, hip, elbow, spine, wrist, hand, chest, etc.

[0013]

[0018] Generally, a clinician 110 performs an ultrasound examination by positioning an ultrasound probe 106 on the skin of a patient 112. The ultrasound probe 106 generates ultrasound waves and detects returning ultrasound waves. The returning ultrasound waves may include reflections of the ultrasound waves generated by the ultrasound probe 106. The ultrasound probe 106 may generate data based on the detected returning ultrasound waves. The data generated by the ultrasound probe 106 may be processed, for example, by the ultrasound probe 106, the computing device 102, or another device or system, to generate an ultrasound image. In some examples, the ultrasound probe 106 is a linear array ultrasound probe that detects returning ultrasound waves along a single plane oriented orthogonal to the direction of propagation of the ultrasound waves. A linear array ultrasound probe may generate a 2D ultrasound image. In some examples, the ultrasound probe 106 may be configured to perform 3D ultrasound, for example, by rotating a linear array of ultrasound transducers.

[0014]

[0019] The MR visualization device 104 may use various visualization techniques to display the MR visualization to the clinician 110. The MR visualization may comprise one or more virtual objects that the user can view simultaneously with real-world objects. Thus, what the clinician 110 sees is a mix of real and virtual objects.

[0015]

[0020] The MR visualization device 104 may comprise various types of devices for presenting MR visualizations. For example, in some examples, the MR visualization device 104 may be a Microsoft HOLOLENS™ headset, such as the HOLOLENS2 headset available from Microsoft Corporation (Redmond, Washington, USA), or a similar device, such as a similar MR visualization device including a waveguide. A HOLOLENS™ device may be used to present 3D virtual objects through holographic lenses or waveguides, while allowing a user to view actual objects in a real-world scene, i.e., the real-world environment, through holographic lenses. In some examples, the MR visualization device 104 may be a holographic projector, a head-mounted smartphone, a dedicated MR visualization device, or another type of device for presenting MR visualizations. In some examples, the MR visualization device 104 includes a head-mounted unit and a backpack unit that perform at least some of the processing functions of the MR visualization device 104. In other examples, all of the functions of the MR visualization device 104 are performed by hardware present in the head-mounted unit. The descriptions in this disclosure of actions performed by system 100 may be performed by one or more computing devices 102, MR visualization device 104, or a combination of one or more computing devices and MR visualization device 104 of system 100.

[0016]

[0021] Processing circuitry that performs computing tasks of system 100 may be distributed among one or more of computing device 102, MR visualization device 104, ultrasound probe 106, and / or other computing devices. Further, in some examples, system 100 may include multiple MR visualization devices. Computing device 102 may include server computers, personal computers, smartphones, tablet computers, laptop computers, and other types of computing devices. Computing device 102 may communicate with MR visualization device 104 via one or more wired or wireless communication links. In the example of FIG. 1, lightning bolt 118 represents a wireless communication link between computing device 102 and MR visualization device 104.

[0017]

[0022] According to one or more techniques of the present disclosure, the system 100 may obtain reference data depicting one or more bones of the patient 112. The medical imaging system 108 may generate the reference data. The medical imaging system 108 may generate the reference data prior to an ultrasound examination. In some examples, the medical imaging system 108 generates computed tomography (CT) data. In other examples, the medical imaging system 108 may generate magnetic resonance imaging (MRI) data or other types of medical images.

[0018]

[0023] Additionally, the system 100 may determine the spatial relationship between the ultrasound probe 106 and the bone. In other words, the system 100 may determine where the ultrasound probe 106 is located relative to the actual bones of the patient 112. The system 100 may determine this spatial relationship based on reference data and ultrasound data generated by the ultrasound probe 106. The ultrasound probe 106 generates ultrasound data while using the ultrasound probe 106 on the patient 112. The ultrasound data may include an ultrasound image, or the system 100 may generate an ultrasound image based on the ultrasound data generated by the ultrasound probe 106.

[0019]

[0024] As part of determining the spatial relationship between the ultrasound probe 106 and the bone, the system 100 may determine the current physical position of the ultrasound probe 106. The current physical position of the ultrasound probe 106 may be expressed in terms of coordinates in a real-world coordinate system. The real-world coordinate system may represent the position of the patient 112 within the physical environment. In some examples, the system 100 uses data from one or more sensors (e.g., depth sensors, visible light sensors, etc.) included in the MR visualization device 104 to determine the current physical position of the ultrasound probe 106. In some examples, the system 100 may use data from one or more other sensors in the examination room to determine the current physical position of the ultrasound probe 106. In some examples, one or more markers attached to the ultrasound probe 106 assist the system 100 in determining the current physical position of the ultrasound probe 106.

[0020]

[0025] In some examples, to determine the spatial relationship between the ultrasound probe 106 and bone, the system 100 may acquire one or more ultrasound images based on ultrasound data generated by the ultrasound probe 106. The ultrasound images may represent structures within the patient 112 in slices aligned with the detection plane (or axis) of the ultrasound probe 106. Generally, the transducer of the ultrasound probe 106 emits pulses of ultrasound onto the skin of the patient 112. In some examples, gel may be applied to the skin of the patient 112 to increase the penetration of the ultrasound waves generated by the ultrasound probe 106 into the interior of the patient 112. When the pulse of ultrasound strikes a first structure (e.g., muscle, tendon, ligament, blood vessel, cartilage, bone, etc.) within the patient 112, the first structure may reflect a portion of the ultrasound wave of the pulse back toward the transducer of the ultrasound probe 106, which may then detect the reflected portion of the ultrasound wave. However, the first structure may also allow a portion of the ultrasound wave of the pulse to transmit through the first structure. A second structure may reflect a portion of the ultrasound pulse that was transmitted through the first structure, transmit another portion of the ultrasound pulse, etc. Based on one or more estimated speeds of travel of the ultrasound waves through the patient 112 and based on the time required for the ultrasound waves reflected by the structure to travel back to the transducer of the ultrasound probe 106, the distance of the structure from the transducer of the ultrasound probe 106 may be estimated.

[0021]

[0026] The system 100 may acquire ultrasound images based on estimated distances to structures within the patient 112. For example, the ultrasound image may include pixels corresponding to distances from the transducer of the ultrasound probe 106. In a typical ultrasound image, pixels corresponding to distances of structures that reflect ultrasound are shown in white, while other pixels remain dark.

[0022]

[0027] In examples where the ultrasound probe 106 is a linear array ultrasound probe, the ultrasound probe 106 includes an array of transducers arranged in a single line along a detection plane of the ultrasound probe 106. The transducers may be arranged in a fan-shaped configuration. Thus, an ultrasound image generated by the linear array ultrasound probe may represent structures within a fan-shaped slice of the patient 112 aligned with the detection plane. In some examples, a 3D ultrasound image of a cone-shaped portion of the patient 112 may be generated by rotating the linear array of transducers of the ultrasound probe 106.

[0023]

[0028] The structures represented in the ultrasound image may include soft tissue structures and bones. System 100 may analyze the ultrasound image to identify structures represented in the ultrasound image that have the same contours as bones represented in the reference data. For example, system 100 may analyze the ultrasound image to identify a curve of the structure represented in the ultrasound image. System 100 may then attempt to match that curve to the curve of the bone represented in the reference data. If system 100 finds a match, the structure represented in the ultrasound image is likely to be the bone represented in the reference data.

[0024]

[0029] Furthermore, if the system 100 finds a match, the system 100 may determine the real-world coordinates of the bone. The system 100 may determine the real-world coordinates of the bone based on the distance of the bone from the ultrasound probe 106 (determined using the ultrasound image) and the real-world coordinates of the ultrasound probe 106. The bone points depicted in the reference data may be defined by a virtual coordinate system. Because the system 100 can match the curves of the bone represented in the reference data with the curves of the bone represented in the ultrasound image, the system 100 can therefore determine the relationship between the virtual coordinate system of the reference data and the real-world coordinate system. In other words, the system 100 may generate registration data that aligns the reference data with the real-world coordinate system.

[0025]

[0030] After registering the reference data with the real-world coordinate system, the system 100 may generate virtual guidance based on the reference data, the registration data, and the physical position of the ultrasound probe 106. The virtual guidance may provide guidance to the clinician 110 regarding how to position the ultrasound probe 106 relative to a target position where the ultrasound probe 106 can generate ultrasound data that provides information about the patient's soft tissue structure. For example, the virtual guidance may instruct the clinician 110 on how to move the ultrasound probe 106 so that the ultrasound probe 106 is at a target position for generating ultrasound data that provides information about the patient's soft tissue structure. In some examples, the virtual guidance may provide the clinician 110 with information that the ultrasound probe 106 is currently positioned at the target position. The system 100 may then cause the MR visualization device 104 to output the virtual guidance to the clinician 110.

[0026]

[0031] The system 100 may generate various types of virtual guidance. For example, a clinician 110 may be preparing for a shoulder replacement. In this example, the clinician 110 may need to take into account the characteristics of various soft tissue structures when determining how to select and implant a glenoid prosthesis and / or a humeral prosthesis. For example, laxity in the rotator cuff muscles (e.g., supraspinatus, infraspinatus, teres minor, and subscapularis) may suggest the use of a reverse total shoulder arthroplasty rather than an anatomical total shoulder arthroplasty. Thus, in this example, it may be beneficial for the clinician 110 to understand the location and size of the rotator cuff muscles. A single ultrasound image representing a 2D slice of the patient 112 may show the edges of the rotator cuff muscles but may not show enough of the entire rotator cuff muscles to enable the clinician 110 to understand the location and size of the rotator cuff muscles of the patient 112. Thus, in this example, the virtual guidance generated by the system 100 may instruct the clinician 110 on how to move the ultrasound probe 106 to one or more positions that will enable the ultrasound probe 106 to generate ultrasound data that provides more information about the rotator cuff muscles of the patient 112.

[0027]

[0032] As described above, the system 100 can generate virtual guidance based on reference data. Generally, the reference data provides a more complete and accurate representation of bones than can be generated by the ultrasound probe 106. Generally, the system 100 can predict the positions of various soft tissue structures based on the shapes and positions of bones represented in the reference data. Thus, because both the reference data and the ultrasound probe 106 are aligned with a real-world coordinate system, the system 100 can generate virtual guidance that provides guidance to the clinician 110 regarding how to position the ultrasound probe 106 relative to a target position where the ultrasound probe 106 can generate ultrasound data that provides information about the soft tissue structures of the patient 112. For example, the virtual guidance can instruct the clinician 110 to move the ultrasound probe 106 to the predicted positions of the soft tissue structures. The system 100 can update the virtual guidance as the clinician 110 moves the ultrasound probe 106 from position to position. Thus, the clinician 110 can get real-time feedback on how to move the ultrasound probe 106 so that the ultrasound probe 106 can generate ultrasound data. In this way, the ultrasound probe 106 can generate ultrasound data about the portion of the soft tissue structure that is of interest to the clinician 110.

[0028]

[0033] In some examples, the system 100 may generate a virtual model of the soft tissue structure of the patient 112 based on the ultrasound data related to the soft tissue structure. For example, in one example, the virtual guidance may instruct the clinician 110 to slide the ultrasound probe 106 along the skin of the patient 112 over a predicted location of the soft tissue structure. The system 100 may acquire a series of ultrasound images based on the ultrasound data generated by the ultrasound probe 106 as the clinician 110 slides the ultrasound probe 106 over the predicted location of the soft tissue structure. The system 100 may segment the ultrasound images to isolate portions of the ultrasound images that correspond to the soft tissue structure. In some examples, the system 100 may use machine learning (ML)-based computer vision techniques (e.g., convolutional neural networks) to segment the ultrasound images to isolate portions of the ultrasound images that correspond to the soft tissue structure. The system 100 may then process the portions of the ultrasound images that correspond to the soft tissue structure to form a virtual model of the soft tissue structure.

[0029]

[0034] The MR visualization device 104 may output a virtual model of the soft tissue structure such that the virtual model of the soft tissue structure appears to the clinician 110 as being superimposed on the patient 112 in the actual location of the soft tissue structure. The MR visualization device 104 may also output a virtual model of one or more bones of the patient 112 such that the virtual bones appear to the clinician 110 as being superimposed on the patient 112 in the actual location of the bones of the patient 112. In this way, the clinician 110 may easily grasp the location of hidden soft tissue structures and bones of the patient 112. The ability to view the virtual models of the soft tissue structures and bones on the MR visualization device 104 may be particularly beneficial during surgery.

[0030]

[0035] 2 is a conceptual diagram illustrating an example computing system 200 in accordance with one or more techniques of the present disclosure. The components of the computing system 200 of FIG. 2 may be included in one of the computing device 102 (FIG. 1), the MR visualization device 104, or the ultrasound probe 106. In the example of FIG. 2, the computing system 200 includes a processing circuit 202, a memory 204, a communication interface 206, and a display 208.

[0031]

[0036] Examples of processing circuitry 202 include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), hardware, or any combination thereof. In general, processing circuitry 202 may be implemented as fixed-function circuitry, programmable circuitry, or a combination thereof. A fixed-function circuit refers to a circuit that provides a specific function and is preconfigured with respect to the operations it can perform. A programmable circuit refers to a circuit that can be programmed to perform various tasks and provide flexibility in the operations it can perform. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. A fixed-function circuit may perform and execute software instructions (e.g., to receive or output parameters), but the types of operations the fixed-function circuit performs are generally unchanged. In some examples, one or more of the units may be separate circuit blocks (fixed function or programmable), and in some examples, one or more units may be integrated circuits.

[0032]

[0037] Processing circuitry 202 may include an arithmetic logic unit (ALU), an elementary function unit (EFU), digital circuits, analog circuits, and / or a programmable core formed from programmable circuitry. In examples where operations of processing circuitry 202 are performed using software executed by programmable circuitry, memory 204 may store object code for the software received and executed by processing circuitry 202, or a separate memory (not shown) within processing circuitry 202 may store such instructions. Examples of software include software designed for surgical planning. Processing circuitry 202 may perform actions attributed to computing system 200 in this disclosure.

[0033]

[0038] The memory 204 may store various types of data used by the processing circuit 202. The memory 204 may include any of a variety of memory devices, such as synchronous dynamic random access memory (DRAM), including DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Examples of the display 208 include a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0034]

[0039] The communications interface 206 enables the computing system 200 to output data and instructions to and receive data and instructions from the MR visualization device 104, the medical imaging system 108, or other devices over one or more communications links or networks. The communications interface 206 may be hardware circuitry that enables the computing system 200 to communicate (e.g., wirelessly or using wires) with other computing systems and devices, such as the MR visualization device 104. Exemplary networks may include various types of communications networks, including one or more wide area networks, such as the Internet, and local area networks. In some examples, the network may include wired and / or wireless communications links.

[0035]

[0040] In the example of Figure 2, memory 204 stores reference data 210, positioning data 212, ultrasound data 214, registration data 215, planning data 216, and virtual guidance data 218. Additionally, in the example of Figure 2, memory 204 stores registration unit 220, virtual guidance unit 222, and virtual modeling unit 224. In other examples, memory 204 may store more, fewer, or different types of data or units. Additionally, the data and units illustrated in the example of Figure 2 are provided for illustrative purposes and may not represent how the data is actually stored or how the software is actually implemented. Registration unit 220, virtual guidance unit 222, and virtual modeling unit 224 may comprise instructions executable by processing circuitry 202. For ease of explanation, the present disclosure may be described as the alignment unit 220, virtual guidance unit 222, and virtual modeling unit 224 performing various actions when the processing circuitry 202 executes the instructions of the alignment unit 220, virtual guidance unit 222, and virtual modeling unit 224.

[0036]

[0041] Generally, the reference data 210 includes previously acquired data depicting one or more bones of the patient 112. For example, the reference data 210 may include one or more CT images of the bones. In some examples, the reference data 210 may include a three-dimensional model of the bones. The three-dimensional model of the bones may be generated based on multiple CT images. The computing system 200 may obtain the reference data 210 from the medical imaging system 108 or another source. For example, the computing system 200 may generate the reference data 210 based on data received from the medical imaging system 108 or another source, or the computing system 200 may receive the reference data 210 from the medical imaging system 108 or another source.

[0037]

[0042] The positioning data 212 may include data indicating the positions of the ultrasound probe 106, the patient 112, and / or other real-world objects. The computing system 200 may acquire the positioning data 212 based on one or more sensors, such as depth sensors or cameras, located on the MR visualization device 104 and / or other devices. The ultrasound data 214 may include ultrasound images or other types of data generated by the ultrasound probe 106. In some examples, the computing system 200 may use the data generated by the ultrasound probe 106 to generate ultrasound images. The planning data 216 may include data related to planning a medical task. For example, the planning data 216 may indicate which soft tissue structures are relevant to the medical task.

[0038]

[0043] As described in more detail elsewhere herein, the registration unit 220 may determine the physical position of the ultrasonic probe 106. Furthermore, the registration unit 220 may generate registration data based on the first ultrasound data generated by the ultrasonic probe 106 that aligns virtual positions of the bones of the patient 112 depicted in the reference data 210 with corresponding physical positions of the bones of the patient 112. The virtual guidance unit 222 may generate virtual guidance data 218 based on the reference data 210, the registration data 215, and the physical position of the ultrasonic probe 106 (e.g., the positioning data 212). The virtual guidance data 218 may provide guidance to the clinician 110 regarding how the ultrasonic probe 106 is positioned relative to a target position where the ultrasonic probe 106 can generate ultrasound data that provides information about the patient's soft tissue structure. For example, the virtual guidance data 218 may instruct the clinician 110 on how to move the ultrasound probe 106 to a target position for generating ultrasound data that provides information about the soft tissue structure of the patient 112. The virtual guidance unit 222 may cause the MR visualization device 104 to output virtual guidance to the clinician 110. The virtual modeling unit 224 may generate a virtual model and, in some examples, may cause the MR visualization device 104 to output the virtual model.

[0039]

[0044] 3 is a flowchart illustrating an example operation of system 100 in accordance with one or more techniques of this disclosure. The flowcharts of this disclosure illustrate example operations. In other examples, the operations may include more, fewer, or different actions.

[0040]

[0045] 3, a computing system 200 may acquire 300 reference data 210 depicting at least one bone of a patient 112. As described elsewhere in this disclosure, the computing system 200 may acquire the reference data 210 from a medical imaging system 108 or another source.

[0041]

[0046] Additionally, the registration unit 220 may determine 302 the physical location of the ultrasound probe 106. In some examples, the registration unit 220 may determine the physical location of the ultrasound probe 106 based on data from one or more sensors of the MR visualization device 104. For example, the MR visualization device 104 may include one or more visible light cameras and a depth sensor. The depth sensor may be configured to detect the distance from the depth sensor to an object, such as the ultrasound probe 106. The depth sensor may be implemented in one of a variety of ways. For example, the depth sensor may include an infrared light emitter and a detector. The infrared light emitter may emit a pulse of infrared light. The reflection of the infrared light is detected by a detector of the depth sensor. The depth sensor may determine the distance from the depth sensor to the object based on the time of flight of the pulse of infrared light to the object and back from the object to the detector. In some examples, the registration unit 220 may be configured to use a signal from the visible light sensor to identify the ultrasound probe 106. In some examples, optical markers may be attached to the ultrasound probe 106 to enhance the ability of the registration unit 220 to identify the ultrasound probe 106 based on signals from the visible light sensors of the MR visualization device 104. Determining the position of the ultrasound probe 106 based on data from the sensors of the MR visualization device 104, as opposed to other types of devices, may be advantageous because using the data from the sensors of the MR visualization device 104 may eliminate the need for other objects in the operating room that may need to be sterilized or otherwise shielded. Furthermore, using the data from the sensors of the MR visualization device 104 may be advantageous because the sensors of the MR visualization device 104 can detect the ultrasound probe 106 from the perspective of the clinician 110 using the ultrasound probe 106. Thus, the clinician 110 does not block the view of the ultrasound probe 106 from other sensors.

[0042]

[0047] The registration unit 220 may indicate the physical position of the ultrasound probe 106 with respect to coordinates in a real-world coordinate system. The real-world coordinate system may be a coordinate system that describes the positions of objects in the physical environment of the MR visualization device 104 and the patient 112. The MR visualization device 104 may establish the real-world coordinate system by implementing a Simultaneous Localization and Mapping (SLAM) algorithm. The SLAM algorithm also determines the current position of the MR visualization device 104 with respect to the real-world coordinate system.

[0043]

[0048] The registration unit 220 may generate 304 registration data based on the ultrasound data generated by the ultrasound probe 106 that aligns virtual positions of the bones of the patient 112 depicted in the reference data 210 with corresponding physical positions of the bones of the patient 112. The registration unit 220 may generate the registration data in one of a variety of ways. For example, FIG. 4, detailed elsewhere in this disclosure, is a flowchart illustrating an example operation of a system for generating registration data.

[0044]

[0049] Further, the virtual guidance unit 222 may generate 306 virtual guidance based on the reference data 210, the alignment data 215, and the physical position of the ultrasound probe 106. The virtual guidance may provide guidance to the clinician 110 regarding how to position the ultrasound probe 106 relative to a target position where the ultrasound probe 106 can generate ultrasound data that provides information about the patient's soft tissue structure. For example, the virtual guidance may instruct the clinician 110 on how to move the ultrasound probe 106 so that the ultrasound probe 106 is in a target position for generating ultrasound data that provides information about the patient's soft tissue structure.

[0045]

[0050] The planning data 216 ( FIG. 2 ) may include data information describing a plan the clinician 110 should follow for the patient 112. In some examples, the planning data 216 may include surgical planning data describing a process for preparing and performing surgery on the patient 112. In some examples, the planning data 216 may be limited to only an ultrasound examination of the patient 112. In either case, the planning data 216 may indicate which soft tissue structures should be scanned during the ultrasound examination. For example, the planning data 216 may indicate that the supraspinatus muscle should be scanned during the ultrasound examination. In light of the planning data 216 indicating which soft tissue structures should be scanned during the ultrasound examination, the virtual modeling unit 224 may obtain (e.g., generate or receive) estimated models of the soft tissue structures based on the reference data. For example, the virtual modeling unit 224 may use statistical shape models of bones depicted in the reference data 210 as the basis for the estimated model of the soft tissue structures. In other words, the virtual guidance unit 222 may generate an estimated model of the soft tissue structure as a statistical shape model (SSM) of the soft tissue structure based on the reference data 210. Broadly speaking, when the virtual modeling unit 224 uses a statistical shape model to generate an estimated model of the soft tissue structure, the virtual modeling unit 224 may use statistics about the bones to determine the expected size and shape of the soft tissue structure.

[0046]

[0051] In some examples, the statistical shape model is implemented using a machine learning (ML) model. For example, in examples where the ML model is a neural network, the virtual modeling unit 224 may train the neural network to generate as output an estimated model of the soft tissue structure (or other data sufficient to characterize the soft tissue structure). Input to the neural network may include information about one or more bones (e.g., a model of the bone, data characterizing the one or more bones), patient demographic data, and / or other types of data. The neural network may be trained based on data from many people. Thus, the estimated model of the soft tissue structure generated by the neural network may be considered a prediction of the soft tissue structure given the corresponding soft tissue structure and bones of many other people.

[0047]

[0052] Furthermore, the virtual guidance unit 222 may generate additional alignment data (e.g., second alignment data) that aligns virtual positions on the estimated model of the soft tissue structure with corresponding physical positions of the bones based on the alignment data that aligns virtual positions of the bones of the patient 112 depicted in the reference data 210 with corresponding physical positions of the bones of the patient 112. For example, the virtual guidance unit 222 may determine positions on the bones depicted in the reference data 210 of expected attachment points of the soft tissue structures to the bones. Furthermore, in this example, the virtual guidance unit 222 may determine corresponding attachment points of the soft tissue structures to the bones in the estimated model of the soft tissue structure. Because the virtual guidance unit 222 has first alignment data that aligns the virtual positions of the bones of the patient 112 depicted in the reference data 210 with the corresponding physical positions of the bones of the patient 112, the virtual guidance unit 222 can therefore determine, using a virtual coordinate system, how the positions on the estimated model of the soft tissue structure relate to the virtual positions of the bones of the patient 112 depicted in the reference data 210, and therefore, how the positions on the estimated model of the soft tissue structure relate to a real-world coordinate system (i.e., the physical positions of the patient's bones).

[0048]

[0053] Furthermore, the virtual guidance unit 222 may determine a direction to move the ultrasonic probe 106 so that the ultrasonic probe 106 is at the target position based on the additional alignment data and the physical position of the ultrasonic probe 106. In some examples, the direction may be a lateral movement of the ultrasonic probe 106 across the skin of the patient 112. In some examples, the direction may be a rotation of the ultrasonic probe 106. In some examples, the direction may be a change in angle of the ultrasonic probe 106 relative to the surface of the skin of the patient 112.

[0049]

[0054] In some examples, to generate the virtual guidance, the virtual guidance unit 222 may track which portions of the soft tissue structure have been scanned during the ultrasound examination. For example, the virtual guidance unit 222 may determine which surfaces of an estimated model of the soft tissue structure are not yet within the detection plane of the ultrasound probe 106. The virtual guidance unit 222 may then generate virtual guidance that instructs the clinician 110 to position the ultrasound probe 106 so that the unscanned portions of the soft tissue structure are within the detection plane of the ultrasound probe 106. Thus, when a sufficient portion of the soft tissue structure has been scanned, the ultrasound examination of the soft tissue structure may be completed. Note that the virtual guidance unit 222 may generate virtual guidance that instructs the clinician 110 to rotate or tilt the ultrasound probe 106 to scan a certain portion of the soft tissue structure. Thus, in some examples, the virtual guidance instructing the clinician 110 on how to move the ultrasound probe 106 indicates how to adjust the angle of the ultrasound probe 106 relative to the patient 112 so that the ultrasound probe 106 is in a target position to generate additional ultrasound data. In some examples, the virtual guidance may indicate to the clinician 110 that the ultrasound probe 106 is at the correct angle to generate additional ultrasound data.

[0050]

[0055] Further, the virtual guidance unit 222 may cause the MR visualization device 104 to output the virtual guidance to the clinician 110 (308). For example, the virtual guidance unit 222 may send a signal to the MR visualization device 104 instructing the MR visualization device 104 to display the virtual guidance. In some examples, the virtual guidance unit 222 may cause the MR visualization device 104 to output the virtual guidance such that the virtual guidance appears to the clinician 110 as being superimposed on the patient 112.

[0051]

[0056] In some examples, the virtual guidance unit 222 may generate updated virtual guidance (310). The updated virtual guidance may instruct the clinician 110 to move the ultrasound probe 106 to a next target position so that the ultrasound probe 106 can generate additional ultrasound data related to the soft tissue structure or a different soft tissue structure. In some examples, the updated virtual guidance may indicate to the clinician 110 that the ultrasound probe 106 has not yet reached the next target position. The virtual guidance unit 222 may then cause the MR visualization device 104 to display the updated virtual guidance (308). This process may continue until the ultrasound probe 106 has generated sufficient ultrasound data. In this manner, if the virtual guidance is considered the first virtual guidance, the virtual guidance unit 222 may acquire second ultrasound data and determine the second virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe 106. In some examples, the second virtual guidance may instruct the clinician 110 on how to move the ultrasound probe 106 to a second target position for generating third ultrasound data that provides additional information about the soft tissue structure of the patient 112. In some examples, the second virtual guidance may indicate whether the ultrasound probe 106 is in the second target position. The virtual guidance unit 222 may then cause the MR visualization device to output the second virtual guidance to the clinician 110.

[0052]

[0057] The virtual guidance unit 222 may generate updated virtual guidance based on second ultrasound data generated by the ultrasound probe 106 when the ultrasound probe 106 is in the target position. For example, the virtual guidance unit 222 may refine an estimated model of the soft tissue structure based on the ultrasound data generated by the ultrasound probe 106 when the ultrasound probe 106 is in the target position. In this example, the virtual guidance unit 222 may generate updated virtual guidance based on the refined estimated model.

[0053]

[0058] The virtual guidance unit 222 may refine the estimation model in various ways. For example, the virtual guidance unit 222 may implement a machine learning (ML) model such as an artificial neural network. Input to the ML model may include data representing a 3D model of the soft tissue structure and data derived from the ultrasound data. The initial 3D model of the soft tissue structure may be generated using a statistical shape model based on the reference data and, in some examples, other factors such as the age, sex, weight, and other characteristics of the patient 112. The output of the ML model may include data representing an updated 3D model of the soft tissue structure. The data derived from the ultrasound data may include data indicative of the measured position of the soft tissue structure, the thickness of the soft tissue structure, the density of the soft tissue structure, and other types of information that the system 100 can derive from the second ultrasound data. Subsequently, as the virtual guidance unit 222 acquires more new ultrasound data, the virtual guidance unit 222 may use the updated 3D model of the soft tissue structure and data based on the new ultrasound data as input to the artificial neural network. The artificial neural network may be various types of artificial neural networks, such as a convolutional neural network or a fully connected deep neural network. In some examples, the virtual guidance unit 222 may use an image stitching technique to detect boundaries between the acquired ultrasound images. In some examples, the virtual guidance unit 222 may use a feature-based detector to detect shared features between the ultrasound images. Exemplary feature-based detectors include SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and PHOW (Pyramidal Histogram of Visual Words).

[0054]

[0059] 4 is a flowchart illustrating an example operation of the computing system 200 for generating registration data in accordance with one or more techniques of this disclosure. In the example of FIG. 4, the registration unit 220 may acquire an ultrasound image based on ultrasound data (400). The ultrasound probe 106 may generate the ultrasound data while the ultrasound probe 106 is in an initial physical position.

[0055]

[0060] Further, the registration unit 220 may determine 402 a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data 210. In some examples, the registration unit 220 may perform a curve matching process to determine a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data 210. An example of a curve matching process is described below in connection with the example of FIG. 5. In other examples, the registration unit 220 may use deep learning or a convolutional neural network to perform the curve matching process. In some examples, the registration unit 220 may use a wavelet transform to determine a feature vector that characterizes texture in the ultrasound image. The registration unit 220 may form elements of the feature vector by a wavelet transform at one or more decomposition levels. The registration unit 220 may implement a classifier that may use the feature vector to recognize structures in different ultrasound images.

[0056]

[0061] 4, the registration unit 220 may generate 404 displacement data describing the spatial displacement between the ultrasound probe 106 and a portion of a bone depicted in the ultrasound image. For example, the registration unit 220 may generate a displacement vector including components indicative of the displacement in the detection plane of the ultrasound probe 106 between the transducer of the ultrasound probe 106 and the location of the portion of the bone that reflected the ultrasound wave back to the transducer. In some examples, the components may include distance and angle values ​​indicative of the angle of the transducer with respect to the midline of the array of transducers of the ultrasound probe 106. In some examples, the components may include a first value indicative of the displacement of the location of the bone along a line perpendicular to the midline of the array of transducers of the ultrasound probe 106 and a second value indicative of the displacement of the location of the bone along the midline of the array of transducers of the ultrasound probe 106.

[0057]

[0062] The registration unit 220 may generate registration data based on the initial physical position of the ultrasound probe 106 and the displacement data (406). For example, the initial physical position of the ultrasound probe 106 may be expressed in terms of real-world coordinates. In this example, the displacement data may also be expressed in real-world coordinates or converted to real-world coordinates. Thus, the positions of the bones may be expressed in terms of real-world coordinates by adding the real-world coordinates of the ultrasound probe 106 and the displacement data. Furthermore, the registration unit 220 may determine virtual coordinates of the corresponding positions of the bones in the reference data (i.e., coordinates that define their positions in the registration data). Thus, the registration unit 220 may generate the registration data by determining the relationship between the real-world coordinates of the positions of the bones and the virtual coordinates of the corresponding positions of the bones in the reference data.

[0058]

[0063] FIG. 5 is a conceptual diagram illustrating curve matching according to one or more techniques of the present disclosure. The example of FIG. 5 shows an ultrasound image 500 and reference data 502. The reference data 502 includes a reference model 504 of the scapula of the patient 112. The reference model 504 may be a three-dimensional model of the scapula. Although described in FIG. 5 with respect to the scapula, the process described with respect to FIG. 5 may be applicable to other bones, such as the pelvis, humerus, tibia, fibula, femur, patella, radius, ulna, talus, metatarsals, phalanges, cuneiform bones, cuboid bones, calcaneus, carpal bones, etc.

[0059]

[0064] To determine a portion of a bone depicted in the ultrasound image 500 that corresponds to a portion of a bone (e.g., a scapula) depicted in the reference data 502, the registration unit 220 may generate curve data characterizing a curve 506 of the bone depicted in the ultrasound image 500. The curve 506 may correspond to the outer surface of the bone as viewed along the detection plane of the ultrasound probe 106. The registration unit 220 may then search the bone depicted in the reference data 502 for a curve that matches the curve of the bone depicted in the ultrasound image 500. In other words, the registration unit 220 may analyze the reference data 502 to identify a curve that matches the curve of the bone depicted in the ultrasound image 500.

[0060]

[0065] To generate the curve data, the registration unit 220 may apply an edge detection algorithm to the ultrasound image 500. The edge detection algorithm detects edges in the ultrasound image 500. The registration unit 220 may apply one or more of various known edge detection algorithms, such as a Canny edge detector, a quadratic edge detector, or another edge detection algorithm. The registration unit 220 may then perform curve fitting on the detected edges; for example, the registration unit 220 may perform polynomial regression or other types of regression to perform the curve fitting. Furthermore, the registration unit 220 may perform curve fitting on a surface of the reference model 504 taken along multiple slices through the reference model at multiple angles. As part of searching the bone depicted in the registered reference data 502 for a curve that matches the curve of the bone depicted in the ultrasound image 500, the registration unit 220 may compare the curve 506 with the curve of the surface of the reference model 504. For example, the registration unit 220 may compare the coefficients of a polynomial function generated by performing polynomial regression on the curve 506 and the curve of the surface of the reference model 504. In the example of FIG. 5, the registration unit 220 may determine that the curve 508 on the reference model 504 corresponds to the curve 506 in the ultrasound image 500.

[0061]

[0066] Because the curves 506 and 508 are lines and not single points, the registration unit 220 may therefore determine the rotation of the reference model 504 relative to the bones depicted in the ultrasound image 500. The registration unit 220 may use the rotation of the reference model 504 relative to the bones depicted in the ultrasound image 500 as part of generating registration data that aligns virtual positions of the bones of the patient 112 depicted in the ultrasound image 500 with corresponding physical positions of the bones of the patient 112. In some examples, generating the registration data includes generating a transformation matrix.

[0062]

[0067] 6 is a conceptual diagram illustrating an example virtual guidance during an ultrasound examination of the shoulder of a patient 112, in accordance with one or more techniques of the present disclosure. In the example of FIG. 6, a clinician 110 applies an ultrasound probe 106 to the skin of the patient 112. In the example of FIG. 6, only the hands of the clinician 110 are shown.

[0063]

[0068] The MR visualization device 104 (not shown in the example of FIG. 6) displays a scapula model 600 representing the scapula of the patient 112. The scapula model 600 is a virtual model and is positioned at a position corresponding to the actual scapula of the patient 112. The virtual modeling unit 224 (FIG. 2) may generate the scapula model 600 based on reference data depicting the scapula of the patient 112.

[0064]

[0069] Additionally, the MR visualization device 104 displays a supraspinatus model 602 representing the supraspinatus muscle of the patient 112. The supraspinatus model 602 is a virtual model and is positioned at a position corresponding to the actual supraspinatus muscle of the patient 112. The virtual modeling unit 224 may generate the supraspinatus model 602 based on the reference data 210. For example, the virtual modeling unit 224 may use bone parameters depicted in the reference data 210 to perform a statistical shape modeling process to generate the supraspinatus model 602. The presentation of the supraspinatus model 602 may be a type of virtual guidance. In some examples, the virtual modeling unit 224 may refine the supraspinatus model 602 based on ultrasound data 214 generated by the ultrasound probe 106, for example, as described elsewhere in this disclosure. In this manner, the virtual guidance unit 222 may cause the MR visualization device 104 to output the bone model and the soft tissue structure model such that the clinician sees the bone model and the soft tissue structure model superimposed on the patient.

[0065]

[0070] The MR visualization device 104 may display a virtual directional element 604 that indicates how the clinician 110 should move the ultrasound probe 106. For example, the virtual directional element 604 may indicate how the clinician 110 should move the ultrasound probe 106 to generate ultrasound data that provides more information about the supraspinatus muscle of the patient 112. Specifically, in the example of FIG. 6 , the virtual directional element 604 indicates that the clinician 110 should move the ultrasound probe 106 medially. Further, as shown in the example of FIG. 6 , the MR visualization device 104 may display the virtual directional element 604 (or other virtual guidance) such that the clinician 110 sees the virtual directional element 604 as if it were superimposed on the patient 112. Displaying the virtual directional element 604 (and / or other virtual guidance) superimposed on the patient 112 may make it easier for the clinician 110 to understand how to move the ultrasound probe 106. In other examples, the MR visualization device 104 may display the virtual direction element 604 (or other virtual guidance) at another location within the clinician's 110 field of view.

[0066]

[0071] In some examples, the virtual guidance instructing the clinician 110 how to move the ultrasound probe 106 may instruct the clinician 110 to move the ultrasound probe 106 laterally across the skin of the patient 112, for example, as shown in the example of FIG. 6 . In some examples, the virtual guidance instructing the clinician 110 how to move the ultrasound probe 106 may instruct the clinician 110 to rotate the ultrasound probe 106. In some examples, the virtual guidance instructing the clinician 110 how to move the ultrasound probe 106 may instruct the clinician 110 to change the angle at which the ultrasound probe 106 contacts the skin of the patient 112. Changing the rotation angle or skin contact angle of the ultrasound probe 106 may enable the ultrasound probe 106 to gather more information about the internal structure of the patient 112.

[0067]

[0072] The following is a non-limiting set of examples according to one or more techniques of this disclosure.

[0068]

[0073] Aspect 1: A method comprising: acquiring reference data depicting a patient's bones; determining a physical position of an ultrasound probe; generating alignment data based on first ultrasound data generated by the ultrasound probe, the alignment data aligning virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones; generating virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe; wherein the virtual guidance provides guidance to a clinician regarding how the ultrasound probe is positioned relative to a target position at which the ultrasound probe can generate second ultrasound data that provides information about the patient's soft tissue structure; and causing a head-mounted mixed reality (MR) visualization device to output the virtual guidance to the clinician.

[0069]

[0074] Aspect 2: The method of Aspect 1, wherein the physical position of the ultrasonic probe is the physical position at which the ultrasonic probe generated the first ultrasonic data, and generating the alignment data comprises: acquiring an ultrasound image based on the first ultrasonic data; determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data; generating displacement data describing a spatial displacement between the ultrasonic probe and the portion of the bone depicted in the ultrasound image; and generating the alignment data based on the physical position of the ultrasonic probe and the displacement data.

[0070]

[0075] Aspect 3: The method of aspect 2, wherein determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data comprises generating curve data characterizing a curve of the bone depicted in the ultrasound image, and searching the bone depicted in the reference data for a curve that matches the curve of the bone depicted in the ultrasound image.

[0071]

[0076] Aspect 4: A method according to any of Aspects 1 to 3, wherein determining the physical position of the ultrasound probe comprises determining the physical position of the ultrasound probe based on data from one or more sensors of the MR visualization device.

[0072]

[0077] Aspect 5: The method of any of Aspects 1-4, wherein the reference data comprises a plurality of computed tomography (CT) images of the bone.

[0073]

[0078] Aspect 6: The method of any one of Aspects 1 to 5, wherein the reference data comprises a three-dimensional model of the bone.

[0074]

[0079] Aspect 7: The method of any of Aspects 1 to 6, wherein generating the virtual guidance comprises generating a virtual directional element that indicates a direction in which the clinician should move the ultrasound probe.

[0075]

[0080] Aspect 8: A method according to any one of aspects 1 to 7, wherein causing the MR visualization device to output the virtual guidance to the clinician comprises causing the MR visualization device to output the virtual guidance so that the virtual guidance appears to the clinician as being superimposed on the patient.

[0076]

[0081] Aspect 9: A method according to any one of aspects 1 to 8, wherein the alignment data is first alignment data, and generating the virtual guidance comprises: obtaining an estimated model of the soft tissue structure based on the reference data; generating second alignment data based on the first alignment data that aligns a virtual position on the estimated model of the soft tissue structure with a corresponding physical position of the bone; and determining a direction in which to move the ultrasonic probe so that the ultrasonic probe is at the target position based on the second alignment data and the physical position of the ultrasonic probe.

[0077]

[0082] Aspect 10: The method of aspect 9, wherein obtaining the estimated model of the soft tissue structure comprises generating the estimated model of the soft tissue structure as a statistical shape model of the soft tissue structure based on the reference data.

[0078]

[0083] Aspect 11: The method of any one of Aspects 1 to 10, wherein the soft tissue structure is one of a tendon, a ligament, a muscle, a cartilage, or a blood vessel.

[0079]

[0084] Aspect 12: The method of any of Aspects 1 to 11, wherein the target position is a first target position and the virtual guidance is first virtual guidance, the method further comprising acquiring the second ultrasound data and determining second virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the second virtual guidance instructs the clinician on how to move the ultrasound probe so that it is at a second target position to generate third ultrasound data that provides additional information about the soft tissue structure of the patient, and causing the MR visualization device to output the second virtual guidance to the clinician.

[0080]

[0085] Aspect 13: A method described in any of aspects 1 to 12, further comprising causing the MR visualization device to output the model of the bone and the model of the soft tissue structure so that the clinician sees the model of the bone and the model of the soft tissue structure as being superimposed on the patient.

[0081]

[0086] Aspect 14: The method of aspect 13, wherein the virtual guidance indicates how to adjust the angle of the ultrasound probe relative to the patient so that the ultrasound probe is in the target position for generating the second ultrasound data.

[0082]

[0087] Aspect 15: A system including a memory configured to store reference data depicting a patient's bones; and a processing circuit, wherein the processing circuit is configured to: determine a physical position of an ultrasound probe; generate alignment data based on first ultrasound data generated by the ultrasound probe, which aligns virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones; generate virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the virtual guidance instructs a clinician how to move the ultrasound probe to a target position for generating second ultrasound data that provides information about the patient's soft tissue structure; and cause a head-mounted mixed reality (MR) visualization device to output the virtual guidance to the clinician.

[0083]

[0088] Aspect 16: The system described in Aspect 15, wherein the physical position of the ultrasonic probe is the physical position at which the ultrasonic probe generated the first ultrasonic data, and the processing circuit is configured to, as part of generating the alignment data, acquire an ultrasound image based on the first ultrasonic data, determine a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data, generate displacement data describing a spatial displacement between the ultrasonic probe and the portion of the bone depicted in the ultrasound image, and generate the alignment data based on the physical position of the ultrasonic probe and the displacement data.

[0084]

[0089] Aspect 17: The system described in Aspect 16, wherein the processing circuit is configured to, as part of determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data, generate curve data characterizing a curve of the bone depicted in the ultrasound image, and search the bone depicted in the reference data for a curve that matches the curve of the bone depicted in the ultrasound image.

[0085]

[0090] Aspect 18: A system described in any of Aspects 15 to 17, wherein the processing circuitry is configured to determine the physical position of the ultrasound probe based on data from one or more sensors of the MR visualization device as part of determining the physical position of the ultrasound probe.

[0086]

[0091] Aspect 19: The system of any of aspects 15 to 18, wherein the reference data comprises a plurality of computed tomography (CT) images of the bone.

[0087]

[0092] Aspect 20: The system of any of Aspects 15 to 19, wherein the reference data comprises a three-dimensional model of the bone.

[0088]

[0093] Aspect 21: A system described in any of aspects 15 to 20, wherein the processing circuitry is configured to generate, as part of generating the virtual guidance, a virtual directional element indicating a direction in which the clinician should move the ultrasound probe.

[0089]

[0094] Aspect 22: The system described in Aspect 21, wherein the processing circuitry is configured to cause the MR visualization device to output the virtual guidance to the clinician as part of causing the MR visualization device to output the virtual guidance so that the virtual guidance appears to the clinician as being superimposed on the patient.

[0090]

[0095] Aspect 23: A system described in any of Aspects 15 to 22, wherein the alignment data is first alignment data, and the processing circuit is configured to, as part of generating the virtual guidance, obtain an estimated model of the soft tissue structure based on the reference data, generate second alignment data based on the first alignment data that aligns a virtual position on the estimated model of the soft tissue structure with a corresponding physical position of the bone, and determine a direction in which to move the ultrasonic probe so that the ultrasonic probe is at the target position based on the second alignment data and the physical position of the ultrasonic probe.

[0091]

[0096] Aspect 24: The system described in Aspect 23, wherein the processing circuitry is configured to generate the estimated model of the soft tissue structure as a statistical shape model of the soft tissue structure based on the reference data as part of obtaining the estimated model of the soft tissue structure.

[0092]

[0097] Aspect 25: The system described in any of Aspects 15 to 24, wherein the soft tissue structure is one of a tendon, a ligament, a muscle, cartilage, or a blood vessel.

[0093]

[0098] Aspect 26: The system described in any of Aspects 15 to 25, wherein the target position is a first target position and the virtual guidance is first virtual guidance, and the processing circuit is further configured to acquire the second ultrasound data, determine second virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the second virtual guidance instructs the clinician on how to move the ultrasound probe so that it is at a second target position to generate third ultrasound data that provides additional information about the soft tissue structure of the patient, and cause the MR visualization device to output the second virtual guidance to the clinician.

[0094]

[0099] Aspect 27: A system described in any of Aspects 15 to 26, wherein the processing circuitry is further configured to cause the MR visualization device to output the model of the bone and the model of the soft tissue structure so that the model of the bone and the model of the soft tissue structure appear to the clinician as being superimposed on the patient.

[0095]

[0100] Aspect 28: The system described in Aspect 27, wherein the virtual guidance indicates how to adjust the angle of the ultrasound probe relative to the patient so that the ultrasound probe is in the target position for generating the second ultrasound data.

[0096]

[0101] Aspect 29: A computer-readable medium having stored thereon instructions that, when executed, cause a processing circuit to perform the method of any of aspects 1-14.

[0097]

[0102] Embodiment 30: A system comprising means for carrying out the method according to any one of embodiments 1 to 14.

[0098]

[0103] While the present technique has been disclosed with respect to a limited number of examples, those skilled in the art, having the benefit of this disclosure, will appreciate numerous modifications and variations therefrom. For example, it is contemplated that any reasonable combination of the described examples may be implemented. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of the invention.

[0099]

[0104] It should be recognized that, depending on the example, certain operations or events of any of the techniques described herein may occur in a different order, or may be added, combined, or entirely eliminated (e.g., not all operations or events described are required to implement the techniques). Furthermore, in certain examples, operations or events may occur not sequentially, but rather simultaneously, for example, via multi-threading, interrupt processing, or multiple processors.

[0100]

[0105] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or communication media, including any medium that facilitates transfer of a computer program from one place to another, for example according to a communications protocol. In this manner, computer-readable media may generally correspond to (1) tangible computer-readable storage media that is non-transitory, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include computer-readable media.

[0101]

[0106] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that may be used to store desired program code in the form of data structures or instructions and that may be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but instead cover non-transitory tangible storage media. Disk and disc, as used herein, include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, although disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above are also intended to be included within the scope of computer-readable media.

[0102]

[0107] The operations described in this disclosure may be performed by one or more processors, which may be implemented as fixed-function processing circuitry, programmable circuitry, or a combination thereof, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. A fixed-function circuit refers to a circuit that provides a specific function and is preconfigured with respect to the operations it can perform. A programmable circuit refers to a circuit that can be programmed to perform various tasks and provide flexibility in the operations it can perform. For example, a programmable circuit may execute instructions specified by software or firmware that cause the programmable circuit to operate in a manner defined by the software or firmware instructions. A fixed-function circuit may execute software instructions (e.g., to receive or output parameters), but the types of operations it performs are generally unchanged. Thus, the terms “processor” and “processing circuit,” as used herein, may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. The following is a summary of the claims as originally filed: [1] obtaining reference data depicting the patient's bones; Determining the physical location of an ultrasound probe; generating registration data based on first ultrasound data generated by the ultrasound probe, the registration data registering virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones; generating virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the virtual guidance provides guidance to a clinician regarding how the ultrasound probe should be positioned relative to a target position at which the ultrasound probe can generate second ultrasound data that provides information about the patient's soft tissue structure; causing a head-mounted mixed reality (MR) visualization device to output the virtual guidance to the clinician; A method comprising: [2] the physical location of the ultrasound probe is a physical location where the ultrasound probe generated the first ultrasound data; generating the alignment data includes: acquiring an ultrasound image based on the first ultrasound data; determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data; generating displacement data describing a spatial displacement between the ultrasound probe and a portion of the bone depicted in the ultrasound image; generating the alignment data based on the physical position of the ultrasound probe and the displacement data; The method according to [1], comprising: [3] Determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data includes: generating curve data characterizing a curve of the bone depicted in the ultrasound image; searching the bone depicted in the reference data for a curve that matches a curve of the bone depicted in the ultrasound image; The method according to [2], comprising: [4] The method according to any one of [1] to [3], wherein determining the physical position of the ultrasound probe comprises determining the physical position of the ultrasound probe based on data from one or more sensors of the MR visualization device. [5] The method of any one of [1] to [4], wherein the reference data comprises a plurality of computed tomography (CT) images of the bone. [6] The method according to any one of [1] to [5], wherein the reference data comprises a three-dimensional model of the bone. [7] The method according to any one of [1] to [6], wherein generating the virtual guidance comprises generating a virtual directional element indicating the direction in which the clinician should move the ultrasound probe. [8] A method according to any one of [1] to [7], wherein causing the MR visualization device to output the virtual guidance to the clinician comprises causing the MR visualization device to output the virtual guidance so that the virtual guidance appears to the clinician as being superimposed on the patient. [9] The alignment data is first alignment data, generating the virtual guidance obtaining an estimated model of the soft tissue structure based on the reference data; generating second registration data based on the first registration data, which registers virtual positions on the estimated model of the soft tissue structure with corresponding physical positions of the bones; determining a direction to move the ultrasonic probe based on the second alignment data and the physical position of the ultrasonic probe so that the ultrasonic probe is at the target position; The method according to any one of [1] to [8], comprising:

[10] The method described in [9], wherein obtaining the estimated model of the soft tissue structure comprises generating the estimated model of the soft tissue structure as a statistical shape model of the soft tissue structure based on the reference data.

[11] The method according to any one of [1] to

[10] , wherein the soft tissue structure is one of tendons, ligaments, muscles, cartilage, and blood vessels.

[12] The target position is a first target position, The virtual guidance is a first virtual guidance, and the method includes: acquiring the second ultrasound data; determining a second virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the second virtual guidance instructs the clinician on how to move the ultrasound probe to a second target position to generate third ultrasound data that provides additional information about the soft tissue structure of the patient; causing the MR visualization device to output the second virtual guidance to the clinician; The method according to any one of [1] to

[11] , further comprising:

[13] The method of any one of [1] to

[12] , further comprising causing the MR visualization device to output the model of the bone and the model of the soft tissue structure so that the clinician sees the model of the bone and the model of the soft tissue structure as being superimposed on the patient.

[14] The method described in

[13] , wherein the virtual guidance indicates how to adjust the angle of the ultrasound probe relative to the patient so that the ultrasound probe is in the target position for generating the second ultrasound data.

[15] a memory configured to store reference data depicting a bone of a patient; a processing circuit; The processing circuitry comprises: Determining the physical location of an ultrasound probe; generating registration data based on first ultrasound data generated by the ultrasound probe, the registration data registering virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones; generating virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the virtual guidance instructs a clinician on how to move the ultrasound probe to a target position for generating second ultrasound data that provides information about the patient's soft tissue structure; causing a head-mounted mixed reality (MR) visualization device to output the virtual guidance to the clinician; A system that is configured to:

[16] The physical location of the ultrasound probe is a physical location where the ultrasound probe generated the first ultrasound data; As part of generating the alignment data, the processing circuitry: acquiring an ultrasound image based on the first ultrasound data; determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data; generating displacement data describing a spatial displacement between the ultrasound probe and a portion of the bone depicted in the ultrasound image; generating the alignment data based on the physical position of the ultrasound probe and the displacement data; The system according to

[15] , configured to perform the following:

[17] As part of determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data, the processing circuitry: generating curve data characterizing a curve of the bone depicted in the ultrasound image; searching the bone depicted in the reference data for a curve that matches a curve of the bone depicted in the ultrasound image; The system according to

[16] , configured to perform the following:

[18] The system described in any one of

[15] to

[17] , wherein the processing circuitry is configured to determine the physical position of the ultrasound probe based on data from one or more sensors of the MR visualization device as part of determining the physical position of the ultrasound probe.

[19] The system of any one of

[15] to

[18] , wherein the reference data comprises a plurality of computed tomography (CT) images of the bone.

[20] The system described in any one of

[15] to

[19] , wherein the reference data comprises a three-dimensional model of the bone.

[21] The system described in any one of

[15] to

[20] , wherein the processing circuitry is configured to generate, as part of generating the virtual guidance, a virtual directional element indicating the direction in which the clinician should move the ultrasound probe.

[22] The system described in

[21] , wherein the processing circuitry is configured to cause the MR visualization device to output the virtual guidance to the clinician as part of causing the MR visualization device to output the virtual guidance so that the virtual guidance appears to the clinician as being superimposed on the patient.

[23] The alignment data is first alignment data; As part of generating the virtual guidance, the processing circuitry: obtaining an estimated model of the soft tissue structure based on the reference data; generating second registration data based on the first registration data, which registers virtual positions on the estimated model of the soft tissue structure with corresponding physical positions of the bones; determining a direction to move the ultrasonic probe based on the second alignment data and the physical position of the ultrasonic probe so that the ultrasonic probe is at the target position; The system according to any one of

[15] to

[22] , configured to perform the following.

[24] The system described in

[23] , wherein the processing circuitry is configured to generate the estimated model of the soft tissue structure as a statistical shape model of the soft tissue structure based on the reference data as part of obtaining the estimated model of the soft tissue structure.

[25] The system described in any one of

[15] to

[24] , wherein the soft tissue structure is one of tendons, ligaments, muscles, cartilage, or blood vessels.

[26] The target position is a first target position; The virtual guidance is a first virtual guidance, and the processing circuitry acquiring the second ultrasound data; determining a second virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the second virtual guidance instructs the clinician on how to move the ultrasound probe to a second target position to generate third ultrasound data that provides additional information about the soft tissue structure of the patient; causing the MR visualization device to output the second virtual guidance to the clinician; The system according to any one of

[15] to

[25] , further configured to perform the following.

[27] The system described in any one of

[15] to

[26] , wherein the processing circuitry is further configured to cause the MR visualization device to output the model of the bone and the model of the soft tissue structure so that the clinician sees the model of the bone and the model of the soft tissue structure as being superimposed on the patient.

[28] The system described in

[27] , wherein the virtual guidance indicates how to adjust the angle of the ultrasound probe relative to the patient so that the ultrasound probe is in the target position for generating the second ultrasound data.

[29] A computer-readable medium storing instructions that, when executed, cause a processing circuit to perform the method described in any one of [1] to

[14] .

[30] A system comprising means for implementing the method according to any one of [1] to

[14] .

Claims

1. obtaining reference data depicting a bone of a patient; determining a physical position of the ultrasound probe with a registration unit; generating registration data based on first ultrasound data generated by the ultrasound probe, the registration data registering virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones; generating virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the virtual guidance provides guidance to a clinician regarding how the ultrasound probe should be positioned relative to a target position at which the ultrasound probe can generate second ultrasound data that provides information about the patient's soft tissue structure; and causing a head-mounted mixed reality (MR) visualization device to output the virtual guidance to the clinician; the physical location of the ultrasound probe is a physical location where the ultrasound probe generated the first ultrasound data; generating the alignment data includes: acquiring an ultrasound image based on the first ultrasound data; determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data; generating displacement data describing a spatial displacement between the ultrasound probe and a portion of the bone depicted in the ultrasound image; generating the alignment data based on the physical position of the ultrasound probe and the displacement data; A method comprising:

2. Determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data includes: generating curve data characterizing a curve of the bone depicted in the ultrasound image; searching the bone depicted in the reference data for a curve that matches a curve of the bone depicted in the ultrasound image; The method of claim 1 , comprising:

3. 3. The method of claim 1, wherein determining the physical location of the ultrasound probe comprises determining the physical location of the ultrasound probe based on data from one or more sensors of the MR visualization device.

4. The method of claim 1 or 2, wherein the reference data comprises a plurality of computed tomography (CT) images of the bone.

5. The method of claim 1 or 2, wherein the reference data comprises a three-dimensional model of the bone.

6. The method of claim 1 or 2, wherein generating the virtual guidance comprises generating a virtual directional element that indicates a direction in which the clinician should move the ultrasound probe.

7. 3. The method of claim 1, wherein causing the MR visualization device to output the virtual guidance to the clinician comprises causing the MR visualization device to output the virtual guidance such that the virtual guidance appears to the clinician as superimposed on the patient.

8. the alignment data is first alignment data; generating the virtual guidance obtaining an estimated model of the soft tissue structure based on the reference data; generating second registration data based on the first registration data, which registers virtual positions on the estimated model of the soft tissue structure with corresponding physical positions of the bones; determining a direction to move the ultrasonic probe based on the second alignment data and the physical position of the ultrasonic probe so that the ultrasonic probe is at the target position; The method of claim 1 or 2, comprising:

9. 9. The method of claim 8, wherein obtaining the estimated model of the soft tissue structure comprises generating the estimated model of the soft tissue structure as a statistical shape model of the soft tissue structure based on the reference data.

10. The method of claim 1 or 2, wherein the soft tissue structure is one of a tendon, a ligament, a muscle, a cartilage, or a blood vessel.

11. the target position is a first target position, The virtual guidance is a first virtual guidance, and the method includes: acquiring the second ultrasound data; determining a second virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the second virtual guidance instructs the clinician on how to move the ultrasound probe to a second target position to generate third ultrasound data that provides additional information about the soft tissue structure of the patient; causing the MR visualization device to output the second virtual guidance to the clinician; The method of claim 1 or 2, further comprising:

12. 3. The method of claim 1 or 2, further comprising causing the MR visualization device to output the model of the bones and the model of the soft tissue structures such that the model of the bones and the model of the soft tissue structures appear to the clinician as superimposed on the patient.

13. 13. The method of claim 12, wherein the virtual guidance indicates how to adjust the angle of the ultrasound probe relative to the patient so that the ultrasound probe is in the target position for generating the second ultrasound data.

14. a memory configured to store reference data depicting bones of the patient; a processing circuit; The processing circuitry comprises: determining a physical position of the ultrasound probe with a registration unit; generating registration data based on first ultrasound data generated by the ultrasound probe, the registration data registering virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones; generating virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the virtual guidance instructs a clinician on how to move the ultrasound probe to a target position for generating second ultrasound data that provides information about the patient's soft tissue structure; causing a head-mounted mixed reality (MR) visualization device to output the virtual guidance to the clinician; and the physical location of the ultrasound probe is a physical location where the ultrasound probe generated the first ultrasound data; As part of generating the alignment data, the processing circuitry: acquiring an ultrasound image based on the first ultrasound data; determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data; generating displacement data describing a spatial displacement between the ultrasound probe and a portion of the bone depicted in the ultrasound image; generating the alignment data based on the physical position of the ultrasound probe and the displacement data; A system that is configured to:

15. As part of determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data, the processing circuitry: generating curve data characterizing a curve of the bone depicted in the ultrasound image; searching the bone depicted in the reference data for a curve that matches a curve of the bone depicted in the ultrasound image; The system of claim 14 configured to:

16. 16. The system of claim 14 or 15, wherein the processing circuitry is configured to determine the physical location of the ultrasound probe based on data from one or more sensors of the MR visualization device as part of determining the physical location of the ultrasound probe.

17. 16. The system of claim 14 or 15, wherein the reference data comprises a plurality of computed tomography (CT) images of the bone.

18. 16. The system of claim 14 or 15, wherein the reference data comprises a three-dimensional model of the bone.

19. 16. The system of claim 14 or 15, wherein the processing circuitry is configured to generate, as part of generating the virtual guidance, a virtual directional element that indicates a direction in which the clinician should move the ultrasound probe.

20. 20. The system of claim 19, wherein the processing circuitry is configured, as part of causing the MR visualization device to output the virtual guidance to the clinician, to cause the MR visualization device to output the virtual guidance such that the virtual guidance appears to the clinician as superimposed on the patient.

21. the alignment data is first alignment data; As part of generating the virtual guidance, the processing circuitry: obtaining an estimated model of the soft tissue structure based on the reference data; generating second registration data based on the first registration data, which registers virtual positions on the estimated model of the soft tissue structure with corresponding physical positions of the bones; determining a direction to move the ultrasonic probe based on the second alignment data and the physical position of the ultrasonic probe so that the ultrasonic probe is at the target position; 16. The system of claim 14 or 15, configured to:

22. 22. The system of claim 21, wherein the processing circuitry is configured, as part of obtaining the estimated model of the soft tissue structure, to generate the estimated model of the soft tissue structure as a statistical shape model of the soft tissue structure based on the reference data.

23. 16. The system of claim 14 or 15, wherein the soft tissue structure is one of a tendon, a ligament, a muscle, a cartilage, or a blood vessel.

24. the target position is a first target position, The virtual guidance is a first virtual guidance, and the processing circuitry acquiring the second ultrasound data; determining a second virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the second virtual guidance instructs the clinician on how to move the ultrasound probe to a second target position to generate third ultrasound data that provides additional information about the soft tissue structure of the patient; causing the MR visualization device to output the second virtual guidance to the clinician; 16. The system of claim 14 or 15, further configured to:

25. 16. The system of claim 14 or 15, wherein the processing circuitry is further configured to cause the MR visualization device to output the model of the bones and the model of the soft tissue structures such that the model of the bones and the model of the soft tissue structures appear to the clinician as superimposed on the patient.

26. 26. The system of claim 25, wherein the virtual guidance indicates how to adjust the angle of the ultrasound probe relative to the patient so that the ultrasound probe is in the target position for generating the second ultrasound data.

27. When executed, the processing circuitry obtaining reference data depicting a bone of a patient; determining a physical position of the ultrasound probe with a registration unit; generating registration data based on first ultrasound data generated by the ultrasound probe, the registration data registering virtual positions of the patient's bones depicted in the reference data with corresponding physical positions of the patient's bones; generating virtual guidance based on the reference data, the alignment data, and the physical position of the ultrasound probe, wherein the virtual guidance provides guidance to a clinician regarding how the ultrasound probe should be positioned relative to a target position at which the ultrasound probe can generate second ultrasound data that provides information about the patient's soft tissue structure; causing a head-mounted mixed reality (MR) visualization device to output the virtual guidance to the clinician; A computer-readable medium having stored thereon instructions for causing a the physical location of the ultrasound probe is a physical location where the ultrasound probe generated the first ultrasound data; As part of generating the alignment data, the processing circuitry: acquiring an ultrasound image based on the first ultrasound data; determining a portion of the bone depicted in the ultrasound image that corresponds to a portion of the bone depicted in the reference data; generating displacement data describing a spatial displacement between the ultrasound probe and a portion of the bone depicted in the ultrasound image; generating the alignment data based on the physical position of the ultrasound probe and the displacement data; 10. A computer-readable medium configured to:

Images