Systems and methods for multi-source medical imaging reconstruction

Abstract

Medical imaging reconstruction of an anatomy of a subject may include monitoring first and second medical imaging data of the anatomy of the subject, wherein: the first medical imaging data is captured from within the anatomy of the subject by at least one first medical imaging device; the second medical imaging data is captured by at least one second medical imaging device from outside the anatomy of the subject; and at least one of the first medical imaging device or the second medical imaging device is operable to selectively aim at different regions of the subject's anatomy; and in response to detecting, based on the monitoring, that a deviation in the first medical imaging data and the second medical imaging data is above a predetermined threshold, updating a medical imaging reconstruction of an anatomical structure of the subject based on the first medical imaging data and the second medical imaging data.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 368,840, filed on July 19, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0002] Various embodiments of the present invention relate generally to multi-source medical imaging reconstruction, and more particularly to systems and methods that account for data recency of medical imaging acquisition and / or reconstruction. Background Art

[0003] In certain medical procedures, it may be beneficial, or even necessary, to have current and accurate knowledge of the subject's anatomy and / or the location of a medical device that interacts with the patient's internal anatomy. For example, organs or tissues may shift during a surgical procedure, an implantable device may require a very high level of accuracy in positioning, a medical device may require a high level of aiming accuracy (e.g., for an ablation procedure), a practitioner may need to revisit previously imaged structures, etc. Medical imaging has been used to model or map portions of a subject's anatomy, for example, prior to and / or during a medical procedure. In some cases, the field of view of a single medical imaging device may be too narrow to capture the full extent of the relevant anatomy in some types of surgery. In order to expand the field of view, internal and external imaging has been used, in which a first medical imaging source internal to the subject is fused with a second medical imaging source captured external to the subject to generate a combined reconstruction that models the subject's anatomy.

[0004] However, an inherent challenge with fusing disparate medical imaging sources with internal and external imaging is that some of the information used for reconstruction is older than other information. In situations where tissue or organs may have shifted during surgery, the older information may be a less valid representation of reality. The operator may not be able to discern which parts of the reconstruction are valid and which are not, and therefore may not be able to tell which parts of the subject's anatomy need to be re-imaged or when to do so. This type of loss in validity, fidelity, or certainty may increase the time required for surgery, reduce the accuracy of the medical device, increase the risk of harm to the subject, increase the cost or complexity of the procedure, etc.

[0005] The present invention is directed to solving the challenges mentioned above. The background description provided herein is intended to generally present the background of the present invention. Unless otherwise indicated herein, by inclusion in this section, the materials described in this section are not prior art for the claims in this application and are not admitted to be prior art or suggestions of prior art. Summary of the invention

[0006] According to certain aspects of the present invention, methods and systems for in vivo navigation of a medical device are disclosed.

[0007] In one aspect, a computer-implemented method for performing medical imaging reconstruction of an anatomical structure of a subject may include: monitoring first medical imaging data of the anatomical structure of the subject and second medical imaging data of the anatomical structure of the subject, wherein: the first medical imaging data is captured from within the anatomical structure of the subject by at least one first medical imaging device; the second medical imaging data is captured from outside the anatomical structure of the subject by at least one second medical imaging device; and at least one of the first medical imaging device or the second medical imaging device is operable to selectively target different areas of the anatomical structure of the subject; and in response to detecting a deviation in the first medical imaging data and the second medical imaging data above a predetermined threshold based on the monitoring, updating the medical imaging reconstruction of the anatomical structure of the subject based on the first medical imaging data and the second medical imaging data.

[0008] In one aspect, updating a medical imaging reconstruction of an anatomical structure of a subject may include: causing at least one of a first medical imaging device or a second medical imaging device to acquire first medical imaging data or second medical imaging data from at least one region of the anatomical structure of the subject, respectively, such that the acquisition results in a reduction in deviation; and modifying the medical imaging reconstruction of the anatomical structure of the subject based on the acquired first medical imaging data or second medical imaging data.

[0009] In one aspect, causing at least one of the first medical imaging device or the second medical imaging device to reacquire first medical imaging data or second medical imaging data, respectively, may include operating an actuator configured to one or more of reposition or reorient at least one of the first medical imaging device or the second medical imaging device so that at least one of the first medical imaging device or the second medical imaging device is aimed at at least one region of the anatomical structure of the subject.

[0010] In one aspect, at least one region of the subject's anatomical structure may include multiple regions; and the actuator may be operated so that first medical imaging data or second medical imaging data for the multiple regions is reacquired based on a signal from a force transducer associated with the actuator according to one or more of a random or pseudo-random pattern, a linear pattern, a tilt pattern, a rotational pattern, or an obstacle avoidance protocol.

[0011] In one aspect, at least one of a first medical imaging device or a second medical imaging device operable to selectively target different regions of an anatomical structure of a subject may include multiple sensors or sensor arrays, changing one or more of the positions or orientations of each of the multiple sensors or sensor arrays so as to target different regions of the anatomical structure of the subject; and causing at least one of the first medical imaging device and the second medical imaging device to reacquire first medical imaging data or second medical imaging data, respectively, may include changing the targeting of the multiple sensors or sensor arrays.

[0012] In one aspect, the deviation in the first medical imaging data and the second medical imaging data can be based on one or more of: a distance between a first region of the subject's anatomy targeted by the first medical imaging device and a second region of the subject's anatomy targeted by the second medical imaging device; or a distance between a first position of the first medical imaging device and a second position of the second medical imaging device.

[0013] In one aspect, the deviation in the first medical imaging data and the second medical imaging data may be based on a moving average of the distances.

[0014] In one aspect, the method may also include one or more of the following: determining a first position of a first medical imaging device or one or more of a first region of an anatomical structure of a subject targeted by the first medical imaging device based on one or more of electromagnetic position tracking, fiber optic shape sensing, position tracking via a second medical imaging device, or impedance-based tracking; or determining a second position of a second medical imaging device or one or more of a first region of an anatomical structure of a subject targeted by the second medical imaging device based on one or more of electromagnetic position tracking, optical tracking, or internal position tracking of the second medical imaging device.

[0015] In one aspect, the deviation in the first medical imaging data and the second medical imaging data can be based on a time range that has passed since one or more of (a) the first medical imaging data was captured by the first medical imaging device or (b) the second medical imaging data was captured by the second medical imaging device.

[0016] In one aspect, the method may further include generating a medical imaging reconstruction of the anatomical structure of the subject based on the first medical imaging data and the second medical imaging data; and causing a display to output a visual depiction of at least a portion of the medical imaging reconstruction.

[0017] In one aspect, updating a medical imaging reconstruction of an anatomical structure of a subject may include modifying a visual depiction of at least a portion of the medical imaging reconstruction so that the at least a portion of the medical imaging reconstruction has a visual appearance that indicates a range of time that has elapsed since medical imaging data on which the at least a portion is based was captured.

[0018] In one aspect, a visual appearance indicating a range of time that has elapsed since at least a portion of the medical imaging data upon which the image is based was captured may include one or more of: a decrease in opacity, contrast, saturation, sharpness, resolution, brightness, or a combination thereof, wherein the magnitude of the decrease increases with increasing time elapsed; a transition in color, wherein the magnitude of the transition increases with increasing time elapsed; a flickering effect; or an indication of a boundary region or selection of at least a portion of the medical imaging reconstruction.

[0019] In one aspect, the method may also include: obtaining a previous medical imaging reconstruction of the subject's anatomical structure based on third medical imaging data captured by a third medical imaging device; determining an alignment of the medical imaging reconstruction with the previous medical imaging reconstruction; and causing a display to output the previous medical imaging reconstruction together with a visual depiction of at least a portion of the medical imaging reconstruction based on the determined alignment.

[0020] In one aspect, the method may also include: receiving a user selection of the relative visibility of a previous medical imaging reconstruction and at least a portion of the medical imaging reconstruction in the visual depiction; and based on the user selection, selectively increasing or decreasing the visibility of the previous medical imaging reconstruction and the visibility of at least a portion of the medical imaging reconstruction in the visual depiction, respectively.

[0021] In one aspect, the first medical imaging device may include one or more of an ultrasound transducer, a parallel or phased array transducer, an optical coherence tomography device, or an optical sensor.

[0022] In one aspect, the second medical imaging device may include one or more of an ultrasound transducer, an array of ultrasound transducers, an X-ray device, or a computed tomography device.

[0023] In one aspect, a system for performing medical imaging reconstruction of an anatomical structure of a subject may include: at least one memory storing instructions; a first medical imaging device configured to capture first medical imaging data from within the anatomical structure of the subject; a second medical imaging device configured to capture second medical imaging data from outside the anatomical structure of the subject, wherein at least one of the first medical imaging device or the second medical imaging device is operable to selectively target different regions of the anatomical structure of the subject; and a processor operably connected to the memory, the first medical imaging device, and the second medical imaging device, and configured to execute the instructions to perform operations. The operations may include: monitoring the first medical imaging data of the anatomical structure of the subject and the second medical imaging data of the anatomical structure of the subject; and in response to detecting a deviation in the first medical imaging data and the second medical imaging data above a predetermined threshold based on the monitoring, updating the medical imaging reconstruction of the anatomical structure of the subject based on the first medical imaging data and the second medical imaging data.

[0024] In one aspect, updating a medical imaging reconstruction of an anatomical structure of a subject may include: causing at least one of a first medical imaging device or a second medical imaging device to acquire first medical imaging data or second medical imaging data, respectively, from at least one region of the anatomical structure of the subject resulting in a reduced deviation; and modifying the medical imaging reconstruction of the anatomical structure of the subject based on the acquired first medical imaging data or second medical imaging data.

[0025] In one aspect, the operations may also include: generating a medical imaging reconstruction of the anatomical structure of the subject based on the first medical imaging data and the second medical imaging data; and causing a display to output a visual depiction of at least a portion of the medical imaging reconstruction; wherein updating the medical imaging reconstruction of the anatomical structure of the subject includes modifying the visual depiction of at least a portion of the medical imaging reconstruction so that at least a portion of the medical imaging reconstruction has a visual appearance indicating a time range that has passed since the medical imaging data on which the at least a portion is based was captured.

[0026] In one aspect, a computer-implemented method for medical imaging reconstruction of an anatomical structure of a subject may include: monitoring first medical imaging data of the anatomical structure of the subject and second medical imaging data of the anatomical structure of the subject, wherein: the first medical imaging data is captured from within the anatomical structure of the subject by at least one first medical imaging device; the second medical imaging data is captured from outside the anatomical structure of the subject by at least one second medical imaging device; and at least one of the first medical imaging device or the second medical imaging device is operable to selectively target different regions of the anatomical structure of the subject; and in response to detecting a deviation in the first medical imaging data and the second medical imaging data above a predetermined threshold based on the monitoring, updating A medical imaging reconstruction of an anatomical structure of a subject based on first medical imaging data and second medical imaging data, wherein updating the medical imaging reconstruction of the anatomical structure of the subject comprises: causing at least one of the first medical imaging device or the second medical imaging device to acquire the first medical imaging data or the second medical imaging data, respectively, from at least one region of the anatomical structure of the subject such that the acquisition results in a reduction in deviation; modifying the medical imaging reconstruction of the anatomical structure of the subject based on the acquired first medical imaging data or the second medical imaging data; and modifying a visual depiction of at least a portion of the medical imaging reconstruction such that at least a portion of the medical imaging reconstruction has a visual appearance indicating a range of time that has elapsed since the medical imaging data on which the at least portion is based was captured.

[0027] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

[0029] Figure 1 An exemplary environment for generating medical imaging reconstructions in accordance with one or more embodiments is depicted.

[0030] Figure 2 An exemplary embodiment of a first medical imaging device integrated into a medical device in accordance with one or more embodiments is depicted.

[0031] Figure 3 An exemplary embodiment of a second medical imaging apparatus is depicted.

[0032] Figure 4 Depicted is a flow chart of an exemplary process for generating a medical imaging reconstruction in accordance with one or more embodiments.

[0033] Figure 5Exemplary embodiments of medical imaging reconstruction are depicted.

[0034] Fig. 6A and Figure 6B Depicted is a schematic illustration of the use of a medical imaging device within the anatomy of a subject.

[0035] Figure 6C Depicted in Figure 6B An exemplary embodiment of a visual representation of medical imaging data captured in the illustration of FIG.

[0036] Figure 7 Depicted is an exemplary embodiment of a graphical user interface for visual depiction of medical imaging reconstruction in accordance with one or more embodiments.

[0037] Figure 8 Describes the use of Figure 7 An exemplary embodiment of a graphical user interface for visually depicting an example of medical imaging reconstruction of medical imaging data in a graphical user interface.

[0038] Fig. 9 Another embodiment of a graphical user interface for visual depiction of medical imaging reconstruction in accordance with one or more embodiments is depicted.

[0039] Fig.10 Another exemplary embodiment of medical imaging reconstruction is depicted.

[0040] Fig.11 An exemplary graphical user interface for adjusting visibility of different portions of a medical imaging reconstruction is depicted.

[0041] Fig.12 An example of a computing device is depicted in accordance with one or more embodiments. DETAILED DESCRIPTION

[0042] According to certain aspects of the present invention, methods and systems for multi-source medical imaging reconstruction are disclosed, and more particularly, systems and methods for considering the data recency of medical imaging acquisition and / or reconstruction. As used herein, considering "data recency" generally includes considering how much time has passed since the data was captured, and can be applied to various activities, such as, further procedures for acquiring data, how to display or model the data, etc.

[0043] In certain medical procedures, it may be desirable or necessary to have an accurate representation of the subject's anatomy. For example, in an ablation procedure, accurate knowledge of the position and orientation of an ablation device within the subject's anatomy, as well as accurate knowledge of the location of various structures of the subject's anatomy, particularly the location of the tissue to be ablated, may facilitate a successful outcome. However, conventional navigation techniques may not be appropriate. For example, conventional techniques may not take into account data recency, and therefore may not provide an accurate representation of the location of various structures of the subject's anatomy or the location of a medical device within the subject. As will be discussed in more detail below, in various embodiments, systems and methods for medical imaging reconstruction that take into account data recency are described.

[0044] Reference to any particular surgery is provided in the present invention only for convenience and is not intended to limit the present invention. Examples of surgeries that may be relevant to the present invention include, for example, excision or removal of lesions, expansion, retrieval, biopsy, implantation, ablation, delivery of therapeutic agents, etc. One of ordinary skill in the art will recognize that the underlying concepts of the disclosed devices and methods can be used in any suitable surgery. The present invention can be understood with reference to the following description and accompanying drawings, in which the same elements are referred to by the same reference numerals.

[0045] The terminology used below is to be interpreted in the broadest reasonable manner, even though it is used in conjunction with the detailed description of certain specific examples of the invention. Indeed, certain terms may even be emphasized below; however, any term intended to be interpreted in any limiting manner will be openly and specifically defined in the detailed description. The foregoing general description and the following detailed description are merely exemplary and illustrative of the features as claimed, and are not limiting descriptions.

[0046] For ease of description, portions of the device and / or its components are referred to as proximal and distal portions. It should be noted that the term "proximal" is intended to refer to a portion closer to a user of the device, and the term "distal" is used herein to refer to a portion further away from the user. Similarly, extending "distally" indicates that a component extends in a distal direction, and extending "proximally" indicates that a component extends in a proximal direction.

[0047] In the present invention, the term "based on" means "based at least in part on". Unless the context dictates otherwise, the singular forms "a", "an" and "the" include plural referents. The term "exemplary" is used in the sense of "example" rather than "exemplary". The terms "include", "including", "comprising", "including" or other variations thereof are intended to cover non-exclusive inclusions, such that a process, method or product that includes a list of elements does not necessarily include those elements, but may include other elements that are not explicitly listed or inherent to such process, method, article or apparatus. The term "or" is used disjunctive, so "at least one of A or B" includes (A), (B), (A and A), (A and B), (B and B), etc. Relative terms, such as, "substantially" and "approximately" are used to indicate possible variations of ±10% of the values ​​described or understood.

[0048] As used herein, terms such as "medical imaging data" generally encompass data associated with and / or indicative of the geometry and / or physiological function of a subject, for example, that can be generated via medical imaging. Medical imaging generally encompasses techniques whereby a signal (light, electromagnetic energy, radiation, etc.) is generated and a measurement is made that indicates how the signal interacts with and / or is affected by a subject, is transmitted through a subject, etc. Examples of medical imaging techniques include ultrasound scanning, magnetic resonance imaging (MRI) scanning, computed tomography (CT), X-ray scanning, visual imaging (e.g., performed via an optical channel or camera, etc.), or any other suitable modality, which can be used, for example, to visualize the interior of at least a portion of an anatomical structure of a subject. Medical imaging data may include, for example, two-dimensional data and / or images, three-dimensional data and / or images, voxel or pixel data, and / or any other suitable data associated with a subject and / or medical imaging.

[0049] As used herein, "medical imaging reconstruction" generally encompasses models, images, visual representations, etc. generated using medical imaging data. For example, a reconstruction may include a visual depiction of the portion of the anatomy of a subject based on medical imaging data, a video, images over time, a solid model of the portion of the anatomy of a subject, a grid of nodes or points representing the portion of the anatomy, and / or any other characteristics of the portion of the anatomy that can be visually represented as a mechanical or physiological model, etc.

[0050] As used herein, the terms "user" and the like generally encompass any person or entity that can interact with or operate a device or component. An "operator" generally encompasses any person involved in providing medical care, such as, for example, a physician, surgeon, nurse, radiologist, technician, hospital, physician's office or outpatient facility, etc. A "subject" generally encompasses a person receiving such medical care.

[0051] For many interventional procedures, e.g., procedures involving interaction with or navigation of a medical device within the subject's anatomy, the subject may undergo preoperative imaging, e.g., MRI scans, CT scans, X-ray scans, etc., which may be used by a medical provider, such as a physician, surgeon, etc., to plan the procedure. In some cases, prior imaging results and / or reconstructions generated therefrom may be obtained during surgery, e.g., for use as a reference or guide.

[0052] However, the previous imaging may not be a valid reflection of the reality of the subject's anatomical structure. For example, from the time the previous imaging was taken to the time the surgery was performed, or even during the surgery itself, an organ or tissue may metastasize, an area of ​​diseased tissue may develop, etc. Various other factors, such as the subject's hydration, diet, etc., may also cause the subject's anatomical structure to change. Therefore, in some cases, the previous imaging can be supplemented with further imaging captured during the surgery. For example, a mirror device, such as a ureteroscope, a duodenoscope, a gastroscope, an endoscopic ultrasound examination ("EUS") mirror, a colonoscope, a bronchoscope, a laparoscope, an arthroscope, a cystoscope, a suction mirror, a sheath, a catheter, etc., may include one or more optical channels, one or more image sensors, one or more ultrasonic transducers, etc., which can provide medical imaging during interventional surgery. However, medical images captured using a device within the subject's anatomical structure may have a narrow field of view and may not be suitable for a comprehensive understanding of that portion of the subject's anatomical structure associated with the surgery.

[0053] Medical images captured from outside the subject have also been used during surgery, for example, X-ray, percutaneous ultrasound, CT, etc. However, these techniques, while generally having a larger field of view than internal imaging, may not be accurate enough for certain procedures.

[0054] And, while both internal and external medical imaging, e.g., a combination of internal and external imaging, have been used during medical procedures, such combinations also often present challenges with respect to accuracy and validity. For example, an inherent challenge with fusing disparate medical imaging sources with internal and external imaging is that some of the information used for reconstruction is older than other information. Because, as discussed above, tissues or organs may shift between the captured imaging and the surgery performed or during surgery, the older information may be a less valid representation of reality. In other words, some parts of the reconstruction may not be as valid as others, and the medical provider may not be able to distinguish which parts of the reconstruction are reliable and which parts are not. The operator may also not know the appropriate time to reacquire medical imaging data. As a result, internal and external imaging may lead to an inherent risk that the operator may rely on invalid data.

[0055] For example, the above difficulties are exacerbated when the procedure and / or medical device being used is sensitive to subtle variations in the subject's anatomy. Without clear indications regarding the validity of any particular medical imaging relative to the subject's anatomy, medical providers may rely on inaccurate medical imaging data, which may negatively impact the outcome of the procedure or the subject. Therefore, there is a need for improved techniques related to accurately reflecting the validity of medical imaging reconstructions.

[0056] In the following description, embodiments will be described with reference to the accompanying drawings.As will be discussed in more detail below, in various embodiments, systems and methods for providing in vivo navigation of a medical device are described.

[0057] In an exemplary use case, a subject may receive preoperative medical imaging. During surgery, a subject may receive internal and external imaging. For example, a first medical imaging device, such as an ultrasound transducer, may be included on a medical device (e.g., an endoscope) inserted into the subject's anatomy, and a second medical imaging device, such as a transabdominal or transthoracic ultrasound sensor, may be used for external imaging. One or more of the first medical imaging data from the first medical imaging device, the second medical imaging data from the second medical imaging device, or the preoperative medical imaging data may be registered with each other and / or used to generate a medical imaging reconstruction of the subject's anatomy, for example, via a medical imaging reconstruction system or the like.

[0058] While performing a procedure, various circumstances may change with respect to the first medical imaging data and the second medical imaging data captured by the first medical imaging device and the second medical imaging device. For example, over time, a particular instance of medical imaging data captured by the medical imaging device may become less valid. In another example, one of the medical imaging devices may be moved to a different location and / or may be aimed at a different area within the anatomy of the subject. Such circumstances may result in deviations, for example, a change in state compared to the time period during which the medical imaging data was captured. The reconstruction system may monitor the first medical imaging data and the second medical imaging data captured by the first medical imaging device and the second medical imaging device, for example to determine deviations in the first medical imaging data and the second medical imaging data.

[0059] The deviation may be based on or associated with, for example, a time elapsed since the medical imaging data was captured, a distance or change in distance between the first medical device and the second medical device or between regions of the subject's anatomy targeted by the first medical device and the second medical imaging device, respectively, a registration error between the first medical imaging data and the second medical imaging data, etc. In another example, the deviation may be based on a user selection of a region of the subject's anatomy for imaging that is different than a region currently or previously imaged.

[0060] In response to the deviation being above a predetermined threshold, the reconstruction system can cause an update of the medical imaging reconstruction of the subject. Such an update can include, for example, causing one or more of the first medical imaging device or the second medical imaging device to reacquire medical imaging data and modify the medical imaging reconstruction based on the reacquired data, or modifying a visual depiction of the medical imaging reconstruction such that at least a portion of the medical imaging reconstruction has a visual appearance that indicates a time range that has elapsed since the capture of the medical imaging data on which the at least a portion is based.

[0061] In one example, a first medical imaging device can be moved within the anatomical structure of a subject. As a result of such movement, the first medical imaging device can target different regions of the anatomical structure of the subject and can move relative to the second medical imaging device. As the first medical imaging device moves, the captured first medical imaging data can be used to update the medical imaging reconstruction. In addition, as time passes after targeting a previous region of the anatomical structure of the subject, corresponding portions of the medical imaging reconstruction can become gray, blurred, etc., to visually indicate the age of the data used as the basis for those portions. Additionally, the data age of the previously targeted region and / or the increasing distance between the first medical imaging device and the second medical imaging device may cause the registration system to trigger the reacquisition of data by the second medical imaging device.

[0062] The above examples are intended to be illustrative only, but it should be understood that the technology according to the present invention can be applied to any suitable type of surgery or activity, and further aspects of the present invention will be discussed in further detail below. In addition, although some of the above examples involve ultrasound imaging, it should be understood that any suitable type of medical imaging can also be used.

[0063] Figure 1An exemplary environment that can be used with the techniques described herein is depicted. One or more user devices 105, one or more preoperative medical imaging devices 110, one or more medical devices 115, one or more first medical imaging devices 120 for internal imaging, one or more second medical imaging devices 125 for external imaging, one or more actuators 127, and one or more data storage systems 130 can communicate across an electronic network 135. One or more users 140 can interact with the user device 105, for example, to interact with another component of the environment and / or access information therefrom. One or more operators 145 can use, access, or direct the use of one or more of the components in the environment, for example, to provide medical services to a subject 150, such as medical intervention, medical surgery, diagnosis, etc. As will be discussed in further detail below, the environment can also include one or more medical imaging reconstruction systems 155, for example, which take into account the recency of data when generating a medical imaging reconstruction of the anatomical structure of the subject 150.

[0064] In some embodiments, the components of the environment are associated with a common entity, such as a hospital, a facility, etc. In some embodiments, one or more of the components of the environment are associated with an entity that is different from another. The systems and devices of the environment can communicate in any arrangement. As will be discussed herein, the systems and / or devices of the environment can communicate and / or operate to perform one or more of the following: perform a medical procedure on subject 150, or generate other activities such as a medical imaging reconstruction of the anatomical structure of subject 150 that takes into account the recency of the data.

[0065] The user device 105 may be a computer system, such as, for example, a desktop computer, a mobile device, a tablet computer, etc. In some embodiments, the user device 105 may include one or more electronic applications installed on the memory of the user device 105, such as programs, plug-ins, browser extensions, etc. In some embodiments, the electronic application may be associated with one or more of the other components in the environment. For example, the electronic application may include software or applications for accessing, controlling and / or operating a medical device, a medical imaging device, for retrieving or uploading data to a data storage system, for outputting data, such as medical imaging reconstruction, etc. The user device 105 may include any suitable input and output devices (not shown), such as a mouse, a keyboard, a microphone, a touch screen, a display, a speaker, etc.

[0066] The preoperative medical imaging device 110 can be any suitable type of medical imaging device, and can include a medical imaging modality similar to or different from the first medical imaging device 120 and the second medical imaging device 125. In one example, the preoperative medical imaging device 110 can include a modality of a medical imaging device that may not be easy or possible to use during a medical surgical procedure. In a non-limiting example, an MRI scanner can be used to generate a relatively detailed medical imaging reconstruction of the subject's anatomy and can have an expanded field of view relative to other medical imaging modalities. However, an MRI scanner may be difficult to use efficiently or effectively during a medical surgical procedure. Not only may the MRI scanner itself hinder access to the subject 150, but the medical procedure itself, for example, movement of the subject 150, the presence of the operator 145 and / or the medical device 115, etc., may also interfere with or negatively affect the ability of the MRI scanner to produce accurate or even usable results.

[0067] As will be discussed in further detail below, the preoperative medical imaging device 110 can be used, for example, to generate an initial medical imaging reconstruction of the subject's anatomy, which can serve as a guide and / or be combined with additional medical imaging data captured during the medical procedure. The medical imaging data and / or data based thereon, such as the initial medical imaging reconstruction, can be stored on the data storage system 130.

[0068] The medical device 115 can be any suitable device that can be used during a medical procedure, for example, for purposes of investigation, diagnosis, treatment, resection, etc. In one example, the medical device 115 includes a mirror device, for example, an endoscope, etc., which can have a functionalized end, for example, an ablation tool, a biopsy tool, an optical channel, etc. and / or a working channel for receiving a medical device, such as the above-mentioned tool. In another example, the medical device 115 includes a syringe with a needle, which is configured to deliver and / or obtain fluid within the anatomical structure of the subject. In an alternative, the medical device 115 may include another type of medical device, such as a laparoscopic instrument or an interventional radiology tool. In various embodiments, the medical device 115 can be operated by an operator 145, and / or can be at least partially autonomous. In some embodiments, the medical device 115 may also include a position sensor (not shown) for tracking the position of the medical device 115.

[0069] The first medical imaging device 120 is configured to capture first medical imaging data from within the anatomical structure of the subject 150. The first medical imaging data may be stored by the data storage system 130. In some embodiments, Figure 2As shown and described below, the first medical imaging device 120 can be at least partially integrated into the medical device 115. For example, the first medical imaging device 120 can include an ultrasound transducer located at the distal end of a shaft of a mirror device of the medical device 115. In other examples, the first medical imaging device 120 and the medical device 115 can be separate devices.

[0070] The first medical imaging device 120 can be selectively navigated or oriented within the anatomy of the subject. For example, in an example where the first medical imaging device 120 is integrated into the medical device 115, as discussed below, the handle portion 245 ( Figure 2 ) can control or steer the distal portion of the shaft 215, and the first medical imaging device 120 can be disposed at the distal portion of the shaft 215. In other examples where the first medical imaging device 120 is separated from the medical device 115, the first medical imaging device 120 can include an insertion portion for insertion into an anatomical structure of a subject (e.g., a body cavity of a subject). The first medical imaging device 120 can include an ultrasonic transducer at the distal end of the insertion portion. The first medical imaging device 120 can also include a control portion (e.g., a handle) proximal to the insertion portion, which can include an actuator or other mechanism external to the subject 150, and the control portion can be operated to control or guide the position and / or orientation of the ultrasonic transducer within the anatomical structure of the subject. By adjusting the position and / or orientation of the first medical imaging device 120 within the anatomical structure of the subject, the first medical imaging device 120 can selectively target different areas of the anatomical structure of the subject.

[0071] In some embodiments, the first medical imaging device 120 may include one or more position sensors 160. The position sensor 160 may be configured to generate a signal indicating the position or orientation of the first medical imaging device 120. Such a signal may also be used to determine what area of ​​the anatomical structure of the subject 150 the first medical imaging device 120 is aimed at. Any suitable type of position sensor may be used. For example, in one exemplary embodiment, the position sensor 160 may include an electromagnetic position sensor positioned at the distal end of the first medical imaging device 120. In another exemplary embodiment, the first medical imaging device 120 may include a fiber optic channel extending from the distal end to the proximal end, and the position sensor 160 may be configured to use fiber optic shape sensing to determine the orientation and position of the distal end. In yet another embodiment, the second medical imaging device 125 may be used to determine the position and / or orientation of the first medical imaging device 120, and may therefore act as a position sensor.

[0072] In another exemplary embodiment, the first medical imaging device 120 may include multiple sensors or multiple sensor arrays. For example, the first medical imaging device 120 may include an ultrasound array transducer, whereby operating different portions of the ultrasound array transducer enables targeting different regions of the anatomy of the subject 150. In yet another exemplary embodiment, the first medical imaging device 120 may not include a portion external to the anatomy of the subject 150, and may take the form of an implanted device, a swallowable sensor (e.g., a pill), etc.

[0073] Medical imaging data from an ultrasonic transducer, such as ultrasound data, typically includes data associated with the internal structure of a portion of the subject's anatomical structure, which is generated via the application of ultrasound to the subject's anatomical structure, thereby using a probe, such as an ultrasonic transducer, to transmit high-frequency vibration pulses into the tissue. The vibrations are at least partially reflected from a surface representing a change in acoustic impedance in the body, such as the geometry of a structure or tissue. The reflected vibrations returned to the transducer can be transmitted to, for example, a data storage system 130. The use of ultrasound data to generate image data and / or medical imaging reconstruction is typically based on the time it takes for the reflection to return to the transducer after the vibration is applied and the intensity of the returned reflection. Conventional transducers are typically configured to receive changes in signal response across only one dimension. In other words, for a static position of the transducer, only one column of pixel data for an ultrasound image can be received. Therefore, in order to generate an image or model of an anatomical structure, a single transducer is typically swept within a field of view, for example, rotating back and forth, so as to continuously add and / or refresh a column of values ​​to the data.

[0074] As described above, in some embodiments, the first medical imaging device 120 may include a transducer array. The transducer array may include, for example, a plurality of transducers arranged in parallel with each other, for example, distributed on at least a portion of the outer circumference of the distal end of the medical device 115 and / or the first medical imaging device 120, or in an approximately linear shape along the distal end of the medical device 115 and / or the first medical imaging device 120. As a result, multiple columns of data can be sensed at once without rotation. In various embodiments, any suitable number and / or type of parallel transducers may be used. Additional transducers effectively increase the static field of view.

[0075] Although some of the above examples involve ultrasound, it will be appreciated that any suitable modality or multiple modalities of medical imaging may be used by or in conjunction with the first medical imaging device 120, such as, for example, an optical sensor, an optical coherence tomography device, an infrared sensor, a spectrometer, etc.

[0076] Figure 2 An exemplary embodiment of a first medical imaging device integrated into medical device 115 is depicted. However, it should be understood that Figure 2The embodiments in are merely illustrative, and any suitable medical device may be used, for example, medical devices that do not include medical imaging devices. Figure 2 The medical device 115 in may include a distal end 205 connected to a proximal end 210 via a shaft 215 .

[0077] The distal end 205 can include one or more portions 220 configured to perform one or more of the following: receive a component or communicate with a lumen (e.g., a working channel) disposed in the shaft 215. For example, the first medical imaging device 120 can be disposed in one of the portions 220. In another example, a tool having an end effector 230 can be disposed in another of the portions 220, such as an ablation device, a needle, a knife, forceps, a snare, a balloon, a delivery device, a guidewire, a mesh, an orifice for aspirating or outputting fluids and / or materials, etc. The distal end 205 can also include, for example, one or more lights, an image sensor, an opening for air / water / suction, an elevator, or any other suitable feature.

[0078] In some embodiments, the shaft 215 may include a flexible material. The shaft 215 may include one or more lumens (not shown) that communicate between the distal end 205 and the proximal end 210. In some embodiments, the shaft 215 may also include and / or house other elements, such as components configured to transmit data between the distal end 205, e.g., a position sensor 225, an imaging device 120, a camera, or a light, and the proximal end 210.

[0079] The proximal end 210 may include, for example, a handle portion 245 that enables an operator to manipulate, advance, retract, and / or orient the distal end 205. For example, the handle portion 245 may include one or more actuators (e.g., knobs) 255 for articulating / steering the distal end 205 and the actuator to deliver fluid / suction or perform other controls. The proximal end 210 may also include one or more interfaces 250, such as an umbilical tube, to output data, send or receive electrical signals, and / or transfer fluids or materials to or from the medical device 115. The interface for data may include one or more of a wired or wireless connection. The interface 250 may also be configured to receive power to operate the first medical imaging device 120, the position sensor 225, the end effector 230, a camera, a light, or another feature of the distal end 205. In some embodiments, the medical device 115 or at least a portion thereof is configured as a disposable, e.g., a single-use device.

[0080] In another exemplary embodiment (not shown), the proximal end of the medical device 115 may include or be configured to engage with an actuator (e.g., Figure 1The medical device 115 may be operated with an actuator 127 in the distal end 205, which actuator may, for example, manipulate the distal end 205 via electronic control. For example, the proximal end of the medical device 115 may include features that allow the proximal side to be received by the actuator 127, or may include the actuator 127 integrated therein. In other words, in some embodiments, the distal end 205 may be remotely controlled and / or robotically controlled. The medical device 115 may include any features suitable for remote and / or robotic control.

[0081] return Figure 1 , the second medical imaging device 125 can be configured to capture second medical imaging data from outside the subject. In various embodiments, the second medical imaging device 125 can include, for example, a transcutaneous ultrasound device, an X-ray scanner, a CT scanner, etc. In some embodiments, the second medical imaging device 125 can be at least partially controlled by electronics. In some embodiments, the sensor of the second medical imaging device 125 can be movable and / or orientable, for example, via an operator 145 or via electronic control. For example, the sensor of the second medical imaging device 125 can be mounted on a movable arm (not shown). The movable arm can, for example, be manually manipulated, and / or can include one or more of the actuators 127, which can enable electronic control of the position and / or orientation of the sensor of the second medical imaging device 125, for example, to target different areas of the anatomical structure of the subject 150. In some embodiments, the second medical imaging device 125 may include a position sensor 165 , similar to the position sensor 160 used for the first medical imaging device 120 , which can be used to determine the position or orientation of the second medical imaging device 125 and / or the area of ​​the subject's anatomy targeted by the second medical imaging device 125 .

[0082] In some embodiments, the second medical imaging device 125 can have a configurable field of view, which enables the user 140 to select one or more of the targeting area or orientation. Any suitable external medical imaging device can be used. For example, in some embodiments, the second medical imaging device 125 can include multiple sensors or sensor arrays.

[0083] Figure 3 A schematic diagram of a second medical imaging device 125 including a plurality of sensor arrays 300 is depicted. Each array 305 in the plurality of arrays 300 includes a plurality of sensors 310, such as ultrasound transducers. In an exemplary embodiment, the plurality of sensor arrays 300 may be disposed above a subject 150. By activating and / or deactivating different arrays 305 in the plurality of arrays 300, different portions of the anatomy of the subject 150 may be imaged, for example, without moving or reorienting the second medical imaging device. Figure 3In the illustrated embodiment, the plurality of arrays 300 are arranged in a cross shape. However, any suitable arrangement may be used, such as, for example, any polygonal or irregular shape. In some embodiments, the plurality of arrays 300 may be shaped in a manner corresponding to at least a portion of a human anatomy, for example, shaped to correspond to a human torso shape, etc. Additionally, the array 305 may have any suitable shape (rectangular, triangular, circular, irregular, etc.). Although in Figure 3 305 may be at least partially overlapping. In addition, in some embodiments, individual sensors 310 may be individually addressable, e.g., such that the division of the sensors into different arrays 305 is electronically controlled. In some embodiments, different arrays 305 may be tilted, e.g., toward or away from each other or in irregular directions. Such tilting may reduce crosstalk between sensor signals from different arrays, e.g., when multiple arrays 305 are used simultaneously, and / or may enable different orientations for imaging anatomical structures of subject 150. Although some of the above embodiments are described in which multiple arrays 300 are maintained in a static position relative to subject 150, in some embodiments, multiple arrays 300 may be movable, e.g., manually or via electronically controlled actuators, e.g., Figure 1 The plurality of arrays 300 may be translatable, rotatable, and / or movable in a vibrating or oscillating motion, for example.

[0084] return Figure 1 , the data storage system 130 may include a medical data exchange system, etc., or may be accessed via it. The data storage system 130 may include a server system, an electronic medical data system, a computer-readable memory such as a hard drive, a flash drive, a disk, etc. In some embodiments, the data storage system 130 includes an application programming interface for exchanging data with other systems, for example, one or more of the other components of the environment and / or interacting with it. The data storage system 130 may include a repository or source for medical imaging data and / or act as a repository or source for medical imaging data. For example, medical imaging data generated from a medical imaging device may be stored by the data storage system 130 and / or provided by the data storage system 130 to the reconstruction system 155, as discussed in more detail below.

[0085] In various embodiments, the electronic network 135 may be a wired connection between devices, a wireless connection between devices, a wide area network ("WAN"), a local area network ("LAN"), a personal area network ("PAN"), etc. In some embodiments, the electronic network 135 includes the Internet, and the information and data provided between various systems occurs online. "Online" may mean connecting to or accessing source data or information from a location away from other devices or networks connected to the Internet. Alternatively, "online" may refer to connecting to or accessing an electronic network (wired or wireless) via a mobile communication network or device. The Internet is a global system of computer networks - a network of networks, in which a party at a computer or other device connected to the network can obtain information from any other computer and communicate with parties at other computers or devices. The most widely used part of the Internet is the World Wide Web (often abbreviated as "WWW" or referred to as "Web"). "Website pages" generally cover locations, data storage devices, etc., which, for example, are hosted and / or operated by a computer system so as to be accessible online, and which may include data configured to cause a program, such as a web browser, to perform operations, such as sending, receiving or processing data, generating visual displays and / or interactive interfaces, etc.

[0086] As discussed in further detail below, the reconstruction system 155 can perform operations related to generating and / or updating medical imaging reconstructions based on medical imaging data, monitoring medical imaging data captured by a medical imaging device, e.g., to determine deviations, guiding and / or operating a medical imaging device and / or a medical device, and other activities. The reconstruction system 155 can include, for example, a server system, a desktop computer, a laptop computer, a tablet computer, etc. In an exemplary embodiment, the reconstruction system 155 includes a cloud-based architecture, e.g., for performing operations including data processing, and can also include a display and / or interface, e.g., accessible to the operator 145 during a medical procedure.

[0087] The reconstruction system 155 may include software and / or hardware that facilitates operations such as those discussed above. For example, the reconstruction system 155 may include one or more algorithms, models, etc., for generating medical imaging reconstructions based on medical imaging data. In various embodiments, the reconstruction system 155 may facilitate generating medical imaging reconstructions in a manual (e.g., user-driven) manner, based on one or more baseline models, using a machine learning driven model, e.g., a deep learning model, etc. Any suitable technique or combination of techniques may be used to generate the medical imaging reconstructions.

[0088] In one exemplary embodiment, the reconstruction system 155 may include a first reconstruction algorithm, model, or the like, configured to access medical imaging data captured by the preoperative medical imaging device 110, e.g., from the data storage system 130, and generate an initial medical imaging reconstruction based on the preoperative medical imaging data. In yet another exemplary embodiment, the reconstruction system 155 may include one or more registration algorithms configured to determine registration between different instances of medical imaging data. For example, medical imaging data received from the first medical imaging device 120 or the second medical imaging device 125 may be registered with each other and / or with an existing medical imaging reconstruction, such as the initial medical imaging reconstruction discussed above.

[0089] The reconstruction system 155 may also include one or more monitoring algorithms configured to monitor the medical imaging data received from the first medical imaging device 120 or the second medical imaging device 125. For example, the one or more algorithms may be used to determine one or more characteristics of the acquired medical imaging data, such as, for example, what region of the anatomy of the subject 150 was targeted, the relative position of the targeted region to other regions targeted by other medical imaging, the age of the medical imaging data, the position of the medical imaging device used to capture the acquired medical imaging data, the relative position of the medical imaging device with respect to other medical imaging devices, the position of the medical device, etc. The reconstruction system 155 may include one or more algorithms for determining deviations in the medical data, for example, based on monitoring.

[0090] The reconstruction system 155 may include one or more algorithms for guiding or operating the medical imaging device, for example, by identifying one or more regions of the anatomical structure to be imaged, operating the actuator 127, etc. to reposition or reorient the medical imaging device, etc. In one example, the reconstruction system 155 may be configured to move the medical imaging device according to a predetermined route or pattern. The reconstruction system 155 may include one or more algorithms for updating and / or outputting information related to the medical imaging reconstruction, for example, taking into account the data recency of the medical imaging data received by the reconstruction system 155. Further aspects of the foregoing will be discussed in further detail below.

[0091] Despite Figure 1 Although depicted as separate components in the environment, it should be understood that in some embodiments, components or portions of components in the environment can be integrated with or incorporated into one or more other components. For example, a portion of the reconstruction system 155 can be integrated into the user device 105 or the data storage system 130. In some embodiments, operations or aspects of one or more of the components discussed above can be distributed among one or more other components. Any suitable arrangement and / or integration of the various systems and devices of the environment can be used.

[0092] Further aspects of the registration system 155 and / or how it may be used in conjunction with the medical device 115 and / or a medical procedure to provide information about the data recency of captured medical imaging data will be discussed in more detail in the following method. In the following method, various actions may be described as being performed by Figure 1 The instructions may be executed or performed by a component of the environment, such as the user device 105, the first medical imaging device 120 or the second medical imaging device 125, the registration system 155, or a component thereof. However, it should be understood that in various embodiments, various components of the environment discussed above may implement instructions or perform actions including the actions discussed below. The actions performed by the device may be considered to be performed by a processor, actuator, etc. associated with the device. Additionally, in various embodiments, various operations may include or enable user control, for example, via the operator 145. Furthermore, it should be understood that in various embodiments, various steps may be added, omitted, and / or rearranged in any suitable manner.

[0093] Figure 4 An exemplary process for providing a medical imaging reconstruction of a subject's anatomy that takes into account the data recency of the medical imaging data used for reconstruction is shown. At step 405, a preoperative medical imaging device 110 may be used to capture preoperative medical imaging data targeting at least a portion of the anatomy of a subject 150. For example, an operator 145 may use an MRI scanner to target a portion of the anatomy of the subject 150 and capture MRI data. The preoperative medical imaging data may be stored on a data storage system 130, for example, accessible by the preoperative medical imaging device 110 via an electronic network 135. At step 410, a reconstruction system 155 may generate an initial medical imaging reconstruction based on the preoperative medical imaging data. Any suitable medical imaging reconstruction technique may be used.

[0094] At step 415 , the reconstruction system 155 may cause a display to output a visual representation of at least a portion of the initial medical imaging reconstruction. For example, the display may be a display accessible to an operator 145 performing a medical procedure on the subject 150 . Figure 5 An exemplary embodiment of a visual representation 500 of a segmented CT reconstruction is shown, for example, a medical imaging reconstruction generated based on segments or slices of a subject's anatomy captured via a CT scanner.

[0095] return Figure 4At step 420, one or more of the first medical imaging device 120 or the second medical imaging device 125 may capture first medical imaging data or second medical imaging data, respectively, e.g., during a surgical procedure. The medical imaging data may be captured continuously, intermittently, upon request or action by the operator 145, automatically (e.g., via electronic control of the reconstruction system 155, etc.), etc. The captured medical imaging data may be stored on the data storage system 130.

[0096] In some embodiments, one or more of the operator 145 or the reconstruction system 155 may reposition or reorient one or more of the first medical imaging modality 120 or the second medical imaging modality 125, for example, via manual manipulation, re-aiming of focus, or via electronic control of one or more actuators 127. For example, if a procedure being performed involves a specific region of the anatomy of the subject 150, one or more of the first medical imaging modality or the second medical imaging modality may be repositioned and / or reoriented to target the specific region.

[0097] At step 425, the reconstruction system 155 may determine a registration of the captured medical imaging data, e.g., a spatial and / or geometric mapping, relationship, or transformation between the data. In some embodiments, the captured medical imaging data may be registered relative to the initial reconstruction. In some embodiments, for example, in embodiments without preoperative medical imaging, the captured medical imaging data may be registered relative to a baseline model, may be manually registered, e.g., by an operator 145, etc. In some embodiments, the first medical imaging data may be registered relative to the second medical imaging data. Registration may be used to determine one or more characteristics of the captured medical imaging data, such as, for example, one or more regions of the anatomical structure of the subject targeted by the medical imaging device, the position of the medical imaging device relative to each other and / or the anatomical structure of the subject, etc. In some embodiments, the registration system 155 may obtain or record additional data, as well as the captured medical image data. Such additional data may include, for example, metadata describing one or more of the time when the data was captured, one or more characteristics of the medical imaging device used to capture the data (e.g., position or aiming data based on feedback from actuator 127), one or more characteristics of the subject's anatomy (e.g., determined via an analysis algorithm based on the captured data), and the like.

[0098] In one example, the anatomy of subject 150 may include one or more structures of interest, such as lesions, wounds, foreign bodies, diseased tissue, and the like. Fig. 6ADepicted is a schematic diagram of a portion of the anatomy 600 of a subject 150 including a cavity 610 through which the first medical imaging device 120 may be navigated, and several lesions 620 to be investigated during surgery.

[0099] like Figure 6B As shown, an exemplary embodiment of a first medical imaging device 120 separate from the medical device 115 can be navigated through the cavity 610. It should be understood that in various embodiments, any suitable first medical imaging device can be used. In some embodiments, the first medical imaging data can be captured while the first medical imaging device 120 is navigated. In some embodiments, the first medical imaging data can be captured continuously, periodically, on request, etc. In this example, the first medical imaging device 120 can include an ultrasound transducer 630, which is configured to image a portion of a tissue 640 that defines the boundaries of the cavity 610. Figure 6C An exemplary embodiment of an “ultrasound fan” is depicted, for example, a visual depiction 650 of ultrasound data that can be captured via an ultrasound transducer 630, for example, by sweeping the ultrasound transducer 630 through a rotational angle and / or activating a parallel array of transducers, etc. when the first medical imaging device 120 is positioned at a region of interest.

[0100] In some embodiments, the captured first medical imaging data may be analyzed, for example, to identify lesions, wounds, foreign bodies, diseased tissue, the location of the medical device 115, etc. present therein. For example, in some embodiments, the reconstruction system 155, a separate classification system, an operator 145, etc. may perform one or more analyses on the first medical imaging data. Any suitable analysis techniques may be used, for example, analysis performed by the operator 145, analysis performed using automated techniques such as machine learning, etc. Any identified structures of interest may be associated with the registration, for example, to identify the location of the identified structure in the reconstruction.

[0101] Figure 6C The visual depiction in represents a single instance of capturing first medical imaging data using the first medical imaging device 120, for example, a single instance of sweeping the ultrasound transducer 630 across a region of interest, a single instance of operating a phased array of the transducer, etc. In addition, although the above description of Fig. 6A1 through C discuss the first medical imaging device 120, but it should be understood that the second medical imaging device can capture second medical imaging data of the anatomical structure of the subject 150 in place of and / or in addition to the first medical imaging data captured by the first medical imaging device 120. In various embodiments, the first medical imaging device 120 and / or the second medical imaging device 125 can capture multiple instances of medical imaging data. For example, as the first medical imaging device 120 is navigated through the cavity 610, the first medical imaging data of the first medical imaging device 120 can be captured at various locations. In various embodiments, the medical imaging data can be captured continuously, in discrete samples, or any combination thereof.

[0102] Return again Figure 4 At step 430, the reconstruction system 155 may generate a medical imaging reconstruction of the anatomical structure of the subject based on the first medical imaging data and / or the second medical imaging data, and may cause a display to output the generated reconstruction. In one example, the reconstruction system 155 may generate a new medical imaging reconstruction based on the first medical imaging data and / or the second medical imaging data. Figure 7 An exemplary embodiment of a graphical user interface 700 that may be output by the reconstruction system 155 is depicted. The interface 700 shows that multiple instances of data 705 have been captured and used to generate a medical imaging reconstruction 710 based thereon. Figure 7 As shown, instances 705 may be shown with an orientation that shows how they relate to each other in three-dimensional space.

[0103] The interface 700 may also illustrate a current position 715 of the first medical imaging device 120 within the reconstruction 710, e.g., based on the determined registration, the position sensor 160, based on tracking of the second medical imaging device 125, etc. The interface 700 may also depict, for example, structures of interest 720 identified by the analysis algorithm, such as lesions 1 through 5. In various embodiments, the interface 700 may also include other data or views of the reconstruction 710. For example, in one embodiment, the interface 700 may include an axial view (e.g., a cross-sectional view) of the cavity 610 and a direction indicator indicating the current orientation of the first medical imaging device 120 (e.g., which direction the sensor of the first medical imaging device 120 is facing in the plane of the cross-sectional view).

[0104] In some embodiments, interface 700 may include user-selectable options, such as interface item 735 for displaying or hiding a selected structure of interest 720. In some embodiments, one or more of instances 705 or structures of interest 720 may be user-selectable. Figure 8An exemplary embodiment of a display 800 of the interface 700 is depicted after the user 140 selects a particular instance 802 from the medical imaging data instances 705. The display 800 may include a visual representation 805 of the selected medical imaging data instance 705 (e.g., an ultrasound fan when the instance is ultrasound data), a display 815 of structures of interest 720 visible in the particular instance 705, and a visual highlighting 820 of the visible structures of interest 720 visible in the visual representation 805. In some embodiments, the display 800 may enable the user 140 to select portions of the visual representation, for example, to add, remove, or edit the structures of interest 720.

[0105] As further medical imaging data is captured, the reconstruction system 155 may update the interface 700 with additional instances 705 and with updates to the reconstruction 710 . Fig. 9 An exemplary embodiment of an interface 700 is shown in which the first medical imaging device 120 has navigated back to a previously imaged area, for example, in order to capture additional information about a structure of interest 720 so that the medical imaging reconstruction 710 includes the additional information in the vicinity of the structure of interest 720. Fig. 9 In the illustrated interface 700 , a visual depiction 805 of a current instance of the first medical imaging data is also shown.

[0106] In some embodiments, for example, embodiments in which preoperative medical imaging data is available, the reconstruction system 155 may include information from the preoperative medical imaging data in the reconstruction 710. The preoperative imaging data may be used, for example, to generate a reconstruction with a wider field of view or a more encompassing imaging of an anatomical region relevant to the surgery. Such a reconstruction may serve as a general mapping guide, for example, providing the operator 145 with a general understanding of the anatomical structures relevant to the surgery, which is supplemented with imaging data from the first medical imaging device 120 and the second medical imaging device 125. Fig.10 An exemplary embodiment of a visual depiction of a reconstruction 710 is depicted that includes a first portion 1005 based on the first medical imaging data and / or the second medical imaging data and a second portion 1010 based on pre-operative medical imaging data.

[0107] In some embodiments, the reconstruction system 155 may be configured to change the relative visibility of the first medical image data and / or the second medical image data with respect to the pre-operative medical image data. Fig.11An exemplary embodiment of an interface 1100 is depicted that includes a user-settable slider 1105 operable to select a relative visibility of the first medical image data and / or the second medical image data relative to the preoperative medical image data. For example, moving the slider 1105 to the left may increase the visibility of the first medical image data and / or the second medical image data and / or decrease the visibility of the preoperative medical image data such that one or the other or both may be displayed. In another example, moving the slider 1105 to the right may decrease the visibility of the first medical image data and / or the second medical image data and / or increase the visibility of the preoperative medical image data such that one or the other or both may be displayed.

[0108] Return to Figure 4 At step 435, reconstruction system 155 may monitor the first medical imaging data of the anatomical structure of subject 150. And / or the second medical imaging data. In one example, reconstruction system 155 may monitor the position of first medical imaging device 120 and / or second medical imaging device 125, may monitor one or more positions aimed by first medical imaging device 120 and / or second medical imaging device 125, and / or may monitor the time range that has passed since different instances of the first medical imaging data and / or second medical imaging data were captured. In some embodiments, the reconstruction system may include additional data in the monitoring, such as status or feedback signals from one or more actuators 127 and / or position sensors 160 or 165, the position of medical device 115, etc.

[0109] At step 440, the reconstruction system 155 may determine a deviation in the first medical imaging data and the second medical imaging data based on the monitoring. The deviation may be based on, for example, associated with, one or more factors, such as the following examples.

[0110] In some embodiments, the deviation may be determined based on a distance and / or a change in distance between a first region of the anatomy of the subject 150 targeted by the first medical imaging device 120 and a second region of the anatomy of the subject 150 targeted by the second medical imaging device 125. For example, in an exemplary use case, the second medical imaging device 125 may be used to externally image a region of the anatomy of the subject 150 that is considered relevant to the surgery. During the surgical procedure, the operator 145 may navigate the first medical imaging device 120 and / or the medical device 115 toward a portion of the anatomy that is outside of the region. Thus, the distance and / or a change in distance between the regions targeted by the first medical imaging device 120 and the second medical imaging device 125 may indicate a deviation in the first medical imaging data and the second medical imaging data.

[0111] In some embodiments, the deviation may be determined based on a distance or a change in distance between a first position of the first medical imaging device 120 and a second position of the second medical imaging device 125. For example, in an exemplary use case, it may be desirable to have the areas targeted by the first medical imaging device 120 and the second medical imaging device 125 be immediately adjacent to and / or overlap each other. Thus, the distance and / or a change in distance between the positions of the first medical imaging device 120 and the second medical imaging device 125 may indicate a deviation in the first and second medical imaging data.

[0112] In some embodiments, the deviation may be determined based on a distance or change in distance between the location of a portion of the medical imaging reconstruction (e.g., a portion having high confidence in validity as discussed further below) and the location of the medical device 115. For example, the operator 145 may navigate the medical device 115 to an area of ​​lower confidence in validity within the reconstruction 710 and / or outside the bounds of the reconstruction 710.

[0113] In some embodiments, the deviation can be based on a moving average of the distance between the positions of the first medical imaging device 120 and the second medical imaging device 125, a moving average of the distance between the areas targeted by the first medical imaging device 120 and the second medical imaging device 125, and / or a moving average of the distance between the medical device 115 and the portion of the medical imaging reconstruction 710. The use of an average term, such as a moving average, can make the deviation less susceptible to noise and / or rapid or abnormal movement generated by one of the first medical imaging device 120 and the second medical imaging device 125. When determining the distance, such as in the above example, any other suitable smoothing or noise attenuation technique can also be used. As will be discussed in further detail below, such smoothing can reduce jitter in the operation of the medical imaging device, promote the safety of the subject, and reduce the risk of imaging unwanted or unnecessary portions of the anatomy of the subject 150.

[0114] In some embodiments, when determining the distance, such as in one or more of the foregoing examples, the reconstruction system 155 can use signals received from one or more position sensors 160 or 165. For example, the first medical imaging device 120 may include a position sensor 160 in the form of an electromagnetic position sensor. In other examples, such as described above with respect to Figure 2 As is the case in the discussed embodiments, the first medical imaging device 120 may be integrated into a medical device 115 including a shaft 215 including a fiber optic channel, and the reconstruction system 155 may employ fiber optic shape sensing to position and / or orient the first medical imaging device 120 .

[0115] In some embodiments, the reconstruction system 155 can use the second medical imaging data to determine the position of the first medical imaging device 120. For example, the reconstruction system 155 and / or another system can employ an analysis algorithm on the second medical imaging data to identify the first medical imaging device 120 in the area targeted by the second medical imaging device 125, and determine the position and / or orientation of the first medical imaging device 120 based on the identification and / or registration of the second medical imaging device 125 with the anatomical structure of the subject 150. In some embodiments, feedback from an actuator 127 can be used, which is used to control the position or orientation of one or more of the first medical imaging device 120 or the second medical imaging device 125. For example, feedback from the actuator 127 and / or associated sensors can track how the medical imaging device is manipulated. In some embodiments, impedance-based position tracking can be used. For example, one or more electrodes can be placed at different locations on the body of the subject 150, currents can be generated by the electrodes included in the first medical imaging device 120, and the reconstruction system 155 can determine the position of the first medical imaging device 120 based on the different currents exhibited by the electrodes on the body of the subject 150. In some embodiments, the position of the second medical imaging modality 125 may be determined based on signals from the position sensor 165 in a manner similar to the example provided above for the first medical imaging modality 120. In some embodiments, the position of the second medical imaging modality 125 may be determined based on optical tracking, for example via signals from an optical sensor, such as a camera disposed proximate to the second medical imaging modality 125.

[0116] In some embodiments, the deviation can be based on the time range that has passed since one or more instances of medical imaging data were captured. In other words, in some embodiments, the deviation can be based on the time range that has passed since one or more of the first medical imaging data was captured by the first medical imaging device or the second medical imaging data was captured by the second medical imaging device. As described above, the state of the anatomical structure of the subject 150 may change over time for various reasons. The longer the time period since a specific instance of medical imaging data was captured, the more likely the specific instance is to represent an at least partially inaccurate reflection of the reality about the subject's anatomical structure. In other words, the validity and / or confidence of a specific instance of medical imaging data may decrease over time. Therefore, the reconstruction system 155 can monitor how much time has passed since each instance of medical imaging data was captured.

[0117] Although various examples of bases for deviations are discussed above, it should be understood that any suitable basis or combination thereof may also be used. In addition, in various embodiments, deviations may be determined for each instance of medical imaging data, on an anatomical structure basis (e.g., for an area imaged by multiple instances), individually for each medical imaging device, or in combination thereof. In some embodiments, the deviation may be represented by a value (e.g., an amplitude corresponding to a distance or time period, etc.). In some embodiments, the deviation may be based on a combination of these factors, and different quantities may be weighted or normalized via any suitable technique. For example, in addition to one deviation point per inch of separation between medical imaging devices, one deviation point may be generated per second. However, it should be understood that the foregoing examples are illustrative only, and any suitable scheme may be used that uses one or more of the factors discussed above to determine the deviation.

[0118] In some embodiments, the determined deviation can be compared to a predetermined threshold. Such comparison can occur, for example, in real time or near real time, periodically, such as once per second, once every 5 seconds, once per minute, once every two minutes, etc., or at any other suitable time, in response to a request by the operator 145, in response to detecting a characteristic of the anatomical structure of the subject 150, such as detecting a structure of interest, in response to the reconstruction system 155 receiving a new instance of medical imaging data, etc. Any suitable predetermined threshold can be used. In some embodiments, such as those discussed in further detail below, in which the display of the medical imaging reconstruction 710 indicates the time elapsed relative to each instance of the medical imaging data, the predetermined threshold can be a non-zero amount. In some embodiments, different deviations associated with different factors and / or different predetermined thresholds can be determined for different purposes. For example, for the purpose of updating a visual depiction of a medical imaging reconstruction, a deviation based on a threshold of elapsed time greater than zero can be determined, and in order to cause a medical imaging device to reacquire medical imaging data, a deviation based on a threshold of elapsed time greater than 5 minutes can be determined, as discussed in further detail below.

[0119] At step 445 , in response to detecting, based on the monitoring, that a deviation in one or more of the first medical image data and the second medical image data is above one or more predetermined thresholds, the reconstruction system 155 may cause the medical imaging reconstruction 710 to update.

[0120] In some embodiments, updating the medical imaging reconstruction 710 includes updating the medical imaging data used to form the medical imaging reconstruction. For example, in some embodiments, the reconstruction system 155 may cause at least one of the first medical imaging device and the second medical imaging device to reacquire the first medical imaging data or the second medical imaging data from one or more regions of the anatomical structure of the subject 150, and then modify the medical imaging reconstruction 710 using the newly acquired data. In some embodiments, the first medical imaging device 120 may be operated by the operator 145, and thus the updating of the medical imaging reconstruction 710 may include causing the second medical imaging device 125 to reacquire the medical imaging data, which may be autonomously electronically controlled by the reconstruction system 155. For example, the reconstruction system 155 may operate one or more actuators 127 to reposition and / or reorient one of the medical imaging devices. In some embodiments, the reconstruction system 155 may output a message or visual prompt to the operator 145 and via a display, which includes instructions for operating one of the medical imaging devices to reacquire the medical imaging data. In various embodiments, the first medical imaging device 120 may be electronically controlled, or both or neither of the first medical imaging device 120 and the second medical imaging device 125 may be electronically controlled.

[0121] In some embodiments, the one or more areas to be imaged and / or re-imaged for updating may include areas that will reduce deviations. For example, an area corresponding to an instance of medical imaging data captured some time ago may be re-imaged to reduce the time that has passed since the area was imaged. In another example, the second medical imaging device 125 may be moved or re-oriented to image an area that is closer to the location of the first medical imaging device 120 or closer to the area targeted by the medical imaging device, or vice versa.

[0122] In some embodiments, the one or more regions may include a plurality of regions, and the reacquisition of medical imaging data may include targeting the plurality of regions according to a sequence and / or pattern. For example, in an exemplary embodiment, the second medical imaging device 125 may include a transabdominal, transthoracic, or conventional ultrasound sensor, etc., and a robotic arm controlled by one or more actuators 127. The reconstruction system 155 may cause the second medical imaging device 125 to reacquire second medical imaging data for a plurality of regions, for example, regions associated with surgery, and may operate the one or more actuators 127 to manipulate the second medical imaging device 125 according to one or more patterns that cause the second medical imaging device 125 to continuously target each of the plurality of regions.

[0123] Any suitable pattern may be used, such as a random or pseudo-random pattern, a linear pattern, a spiral pattern, a movement scheme predetermined by a decision algorithm, etc. In some embodiments, the pattern may include one or more of tilting, rotating, sweeping, or pivoting the second medical imaging device 125. In some embodiments, instead of or in addition to applying a pattern such as described above, the reconstruction system 155 may apply an obstacle avoidance protocol to the medical imaging device 125. For example, one or more actuators 127 may include or be associated with one or more force sensors. In response to a signal from the force sensor indicating that the force exceeds a predetermined threshold, the reconstruction system 155 may perform one or more of the following: stop the manipulation of the medical imaging device, identify an obstacle and manipulate the medical imaging device around the obstacle, modify the pattern, etc.

[0124] In another exemplary embodiment, as discussed above, the second medical imaging device 125 may include a plurality of sensors or a sensor array 300 ( Figure 3 ), and causing the second medical imaging device 125 to reacquire medical imaging data from one or more regions of the anatomical structure of the subject 150 may include one or more of the following: changing which sensors or arrays 305 among the multiple sensors or arrays 300 are activated, changing the position or orientation of the multiple sensors or arrays 300, etc., so as to target one or more regions of the anatomical structure of the subject 150.

[0125] In some embodiments, updating the medical imaging reconstruction 710 includes modifying the visual depiction of the medical imaging reconstruction 710. For example, in some embodiments, updating may include modifying the medical imaging reconstruction to include and / or take into account newly acquired medical imaging data, and then updating the visual depiction of the medical imaging reconstruction 710 accordingly. In some embodiments, updating may include modifying the visual depiction of at least a portion of the medical imaging reconstruction so that at least a portion of the medical imaging reconstruction has a visual appearance that indicates a range of time that has passed since the medical imaging data on which the at least a portion is based was captured. For example, the first medical imaging device 120 and the second medical imaging device 125 may capture the first medical imaging data and the second medical imaging data from various regions at various times. The visual depiction of the medical imaging reconstruction 710 may gradually change over time the visual appearance of each region depending on how much time has passed since the particular region was captured.

[0126] The visual appearance indicating the elapsed time range may include, for example, a decrease in opacity, contrast, saturation, and / or brightness, wherein the magnitude of the decrease increases as the elapsed time range increases. Fig. 9In one embodiment, a more recently reconstructed area is depicted with one or more of different colors or saturations relative to an area that has been imaged less recently. In another example, the visual appearance may include one or more of a reduction in sharpness or resolution, wherein the magnitude of the reduction increases as the time range elapsed increases. In the illustrated embodiment, the magnitude of the reduction in resolution or sharpness may be configured to be related to the amount by which confidence in the validity of the imaging of the area is reduced, for example, so that a high level of confidence may be maintained for depicting the area. In another example, the visual appearance may include a transition in color, wherein the magnitude of the transition increases as the time range elapsed increases. In another example, the visual appearance may include a flickering effect, which may be initiated when the time range since capture reaches a predetermined threshold, and / or may be accelerated / decelerated over time. In another example, the visual appearance may include a boundary area or selection, for example, visually indicating that the area or areas have a lower confidence or validity.

[0127] In another example, the visual appearance can include hiding an area where a period of time exceeding a predetermined threshold has passed since the area was imaged. In other words, the reconstruction system 155 can limit the medical imaging reconstruction 710 to only be based on data captured within a predetermined threshold amount of time. In some embodiments, such hiding may only apply to data captured by the first medical imaging device 120 and the second medical imaging device 125, for example, so that preoperative medical imaging remains visible.

[0128] In some embodiments, the reconstruction system 155 may, for example, determine that the medical device 115 is located in an area where the deviation is above a predetermined threshold, such as an area with low confidence of effectiveness, based on the determined deviation. In some embodiments, the reconstruction system 155 may output a notification to the operator 145, such as a visual or auditory message, such as via a display, or a tactile response, such as via an actuator connected to the medical device 115. In some embodiments, the reconstruction system 155 may do one or more of the following: output instructions or prompts for the operator 145 to guide the medical device 115 to an area with higher confidence of effectiveness, output further notifications when the medical device 115 is in an area with higher confidence of effectiveness, aim one or more of the first medical imaging device 120 or the second medical imaging device 125 at an area closer to the medical device 115, etc.

[0129] It should be understood that the embodiments of the present invention are exemplary only, and other embodiments may include various combinations of features from other embodiments, as well as additional or fewer features. For example, while some of the above embodiments involve ablation of tissue within the periphery of the lung, any suitable procedure may be used. In addition, while some of the above embodiments involve ultrasound, any suitable imaging modality or technique may be used.

[0130] Generally, it is understood that any process or operation discussed in the present invention can be implemented by a computer, such as Figure 4 The processes shown in can be executed by one or more processors of a computer system, such as Figure 1 The process or process steps performed by one or more processors may also be referred to as operations. One or more processors may be configured to perform these processes by accessing instructions (e.g., software or computer readable code) that, when executed by one or more processors, cause the one or more processors to perform the process. The instructions may be stored in a memory of a computer system. The processor may be a central processing unit (CPU), a graphics processing unit (GPU), or any suitable type of processing unit.

[0131] A computer system, such as a system or device implementing a process or operation in the above examples, may include one or more computing devices, such as Figure 1 The computer system may include one or more of the systems or devices in the computer system. The one or more processors of the computer system may be included in a single computing device or distributed in multiple computing devices. The memory of the computer system may include a corresponding memory of each computing device in the multiple computing devices.

[0132] Fig.12 is an exemplary embodiment of the present invention may be configured to perform Figure 41200 is a simplified functional block diagram of a computer 1200 for an apparatus of a method of performing the present invention. For example, according to an exemplary embodiment of the present invention, the computer 1200 can be configured to reconstruct the system 155 and / or another system. In various embodiments, any of the systems herein can be a computer 1200, which includes, for example, a data communication interface 1220 for packet data communication. The computer 1200 can also include a processor 1202 in the form of one or more processors or central processing units ("CPU") for executing program instructions. Although the computer 1200 can receive programming and data via network communications, the computer 1200 can include an internal communication bus 1208 and a storage unit 1206 (such as ROM, HDD, SDD, etc.), which can store data on a computer readable medium 1222. Although the instructions 1224 can be temporarily or permanently stored in other modules of the computer 1200 (e.g., the processor 1202 and / or the computer readable medium 1222), the computer 1200 can also have a memory 1204 (such as RAM) storing instructions 1224 for performing the techniques presented herein. The computer 1200 may also include input and output ports 1212 and / or a display 1210 to connect to input and output devices such as a keyboard, a mouse, a touch screen, a monitor, a display, etc. Various system functions may be implemented in a distributed manner on multiple similar platforms to distribute the processing load. Alternatively, the system may be implemented by appropriate programming of one computer hardware platform.

[0133] The program aspects of the present technology can be considered as "products" or "manufactures", which typically use executable code and / or associated data carried or embodied on or in a type of machine-readable medium. "Storage" type media include any or all of computers, processors, etc. or their associated modules, such as various semiconductor memories, tape drives, disk drives, etc. (which can provide non-temporary storage for software programming at any time). Sometimes, all or part of the software can be communicated through the Internet or various other telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another, for example, from a management server or host of a mobile communication network to a computer platform of a server and / or from a server to a mobile device. Therefore, another type of medium that can carry software elements includes light waves, radio waves, and electromagnetic waves, such as across physical interfaces between local devices, through wired and optical fixed telephone line networks and through various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be regarded as media that carry software. As used herein, unless restricted to non-transitory, tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0134] Although the disclosed methods, apparatuses, and systems are described with exemplary reference to transmitting data, it should be understood that the disclosed embodiments may be applicable to any environment, such as a desktop or laptop computer, a car entertainment system, a home entertainment system, etc. In addition, the disclosed embodiments may be applicable to any type of Internet protocol.

[0135] It should be understood that in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes combined together in a single embodiment, figure, or description thereof to simplify the present invention and to aid in understanding one or more of the various inventive aspects. However, this approach to the present invention should not be interpreted as reflecting an intention that the claimed invention requires more features than those explicitly recited in each claim. On the contrary, as reflected in the following claims, inventive aspects exist in features that are less than all of the features in a single, previously disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim existing independently as a separate embodiment of the present invention.

[0136] In addition, although some embodiments described herein include some features included in other embodiments, but not other features, the combination of features of different embodiments should be within the scope of the present invention and form different embodiments, as will be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0137] Therefore, although certain embodiments have been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the invention, and it is intended that all such changes and modifications fall within the scope of the invention. For example, functions may be added or deleted from the block diagrams, and operations may be interchanged between functional blocks. Steps may be added or deleted from the method within the scope of the invention.

[0138] The above disclosed subject matter should be considered illustrative, not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments, which fall within the true spirit and scope of the present invention. Therefore, to the maximum extent permitted by law, the scope of the present invention should be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be restricted or limited by the preceding detailed description. Although various embodiments of the present invention have been described, it is apparent to those of ordinary skill in the art that more embodiments may be within the scope of the present invention. Therefore, the present invention is not limited except according to the appended claims and their equivalents.

Claims

1. A computer-implemented method for medical imaging reconstruction of an anatomical structure of a subject, comprising: monitoring first medical imaging data of an anatomical structure of the subject and second medical imaging data of an anatomical structure of the subject, wherein: The first medical imaging data is captured from within the anatomy of the subject by at least one first medical imaging device; The second medical imaging data is captured by at least one second medical imaging device from outside the anatomy of the subject; and At least one of the first medical imaging device or the second medical imaging device is operable to selectively target different regions of the subject's anatomy; and In response to detecting, based on the monitoring, that a deviation in the first medical imaging data and the second medical imaging data is above a predetermined threshold, a medical imaging reconstruction of the anatomical structure of the subject based on the first medical imaging data and the second medical imaging data is updated.

2. The computer-implemented method of claim 1 , wherein updating the medical imaging reconstruction of the subject's anatomical structure comprises: causing the at least one of the first medical imaging device or the second medical imaging device to acquire first medical imaging data or second medical imaging data, respectively, from at least one region of an anatomical structure of the subject such that the acquiring results in a reduction in the deviation; as well as The medical imaging reconstruction of the anatomical structure of the subject is modified based on the acquired first medical imaging data or second medical imaging data.

3. The computer-implemented method of claim 2, wherein causing the at least one of the first medical imaging device or the second medical imaging device to reacquire first medical imaging data or second medical imaging data, respectively, comprises operating an actuator configured to one or more of reposition or reorient the at least one of the first medical imaging device or the second medical imaging device such that the at least one of the first medical imaging device or the second medical imaging device is aimed at the at least one region of the anatomical structure of the subject.

4. The computer-implemented method of claim 3, wherein: The at least one region of the subject's anatomy comprises a plurality of regions; and The actuator is operated so that the first medical imaging data or the second medical imaging data for the multiple regions is reacquired based on a signal from a force transducer associated with the actuator according to one or more of a random or pseudo-random pattern, a linear pattern, a tilt pattern, a rotational pattern, or an obstacle avoidance protocol.

5. A computer-implemented method according to any one of claims 2 to 4, wherein: the at least one of the first medical imaging device or the second medical imaging device operable to selectively target different regions of the subject's anatomy comprises a plurality of sensors or sensor arrays, one or more of a position or orientation of each of the plurality of sensors or sensor arrays being changed so as to target different regions of the subject's anatomy; as well as Causing the at least one of the first medical imaging modality and the second medical imaging modality to reacquire first medical imaging data or second medical imaging data, respectively, includes changing a pointing of the plurality of sensors or sensor array.

6. The computer-implemented method of any one of the preceding claims, wherein the deviation in the first medical imaging data and the second medical imaging data is based on one or more of: a distance between a first region of the subject's anatomy targeted by the first medical imaging device, and a second region of the subject's anatomy targeted by the second medical imaging device; or A distance between a first position of the first medical imaging modality and a second position of the second medical imaging modality. 7 . The computer-implemented method of claim 6 , wherein the deviation in the first medical imaging data and the second medical imaging data is based on a moving average of the distances.

8. The computer-implemented method of claim 6 or 7, further comprising one or more of the following: determining one or more of the first position of the first medical imaging device or the first region of the subject's anatomy targeted by the first medical imaging device based on one or more of electromagnetic position tracking, fiber optic shape sensing, position tracking via the second medical imaging device, or impedance-based tracking; or One or more of the second position of the second medical imaging device or the first region of the subject's anatomical structure targeted by the second medical imaging device are determined based on one or more of electromagnetic position tracking, optical tracking, or internal position tracking of the second medical imaging device.

9. A computer-implemented method according to any of the preceding claims, wherein the deviation in the first medical imaging data and the second medical imaging data is based on a time range that has passed since one or more of (a) the first medical imaging data was captured by the first medical imaging device or (b) the second medical imaging data was captured by the second medical imaging device.

10. The computer-implemented method of any preceding claim, further comprising: generating the medical imaging reconstruction of the anatomical structure of the subject based on the first medical imaging data and the second medical imaging data; as well as A display is caused to output a visual depiction of at least a portion of the medical imaging reconstruction.

11. A computer-implemented method according to claim 10, wherein updating the medical imaging reconstruction of the subject's anatomical structure includes modifying the visual depiction of at least a portion of the medical imaging reconstruction so that the at least a portion of the medical imaging reconstruction has a visual appearance that indicates a time range that has elapsed since the medical imaging data on which the at least a portion is based was captured.

12. The computer-implemented method of claim 11, wherein the visual appearance indicative of a range of time that has elapsed since the medical imaging data upon which the at least a portion is based was captured comprises one or more of: a decrease in opacity, contrast, saturation, sharpness, resolution, brightness, or a combination thereof, wherein the magnitude of the decrease increases as the elapsed time range increases; a color transition, wherein the magnitude of the transition increases as the elapsed time range increases; Flashing effects; or A boundary region or selection of the at least a portion of the medical imaging reconstruction is indicated.

13. The computer-implemented method of any one of claims 10 to 12, further comprising: obtaining a prior medical imaging reconstruction of the anatomical structure of the subject based on third medical imaging data captured by a third medical imaging device; determining a registration of the medical imaging reconstruction with the previous medical imaging reconstruction; as well as Based on the determined registration, the display is caused to output a visual depiction of the previous medical imaging reconstruction together with the at least a portion of the medical imaging reconstruction.

14. The computer-implemented method of claim 13, further comprising: receiving a user selection of relative visibility of the previous medical imaging reconstruction and the at least a portion of the medical imaging reconstruction in the visual depiction; as well as Based on the user selection, visibility of the previous medical imaging reconstruction and visibility of the at least a portion of the medical imaging reconstruction in the visual depiction are selectively increased or decreased, respectively.

15. A computer-implemented method according to any one of the preceding claims, wherein: The at least one first medical imaging device comprises one or more of an ultrasound transducer, a parallel or phased array transducer, an optical coherence tomography device, or an optical sensor; and The at least one second medical imaging device comprises one or more of an ultrasound transducer, an array of ultrasound transducers, an X-ray device, or a computed tomography device.

Images