System and method for multi-source medical image reconstruction

Abstract

Medical image reconstruction of the anatomical structure of a subject comprises monitoring first medical image data of the anatomical structure of the subject and second medical image data, wherein in the monitoring, the first medical image data is captured from inside the anatomical structure of the subject by at least one first medical imaging device, the second medical image data is captured by at least one second medical imaging device, at least one of the first medical imaging device and the second medical imaging device is operable to selectively target different regions of the anatomical structure of the subject, and in response to detecting based on the monitoring that a deviation between the first medical image data and the second medical image data exceeds a preset threshold, updating the reconstruction of the medical image of the anatomical structure of the subject based on the first medical image data and the second medical image data.

Description

Technical Field

[0001] Various embodiments of the present disclosure generally relate to multi-source medical image reconstruction, and more specifically, to systems and methods that take into account data currency for medical image acquisition and / or reconstruction.

Background Art

[0002] In certain medical procedures, it may be beneficial or even necessary to know the current accurate state of the anatomical structure of the subject and / or the position of a medical device that interacts with the internal anatomical structure of the patient. For example, organs or tissues may move during the procedure, a very high level of accurate positioning of an implanted device may be required, the medical device may require a high level of accuracy for targeting (e.g., for ablation procedures), and the operator may need to review previously imaged structures, etc. Medical images have been used, for example, to model or map portions of the anatomical structure of a subject before and / or during a medical procedure. In some cases, the field of view of a single medical imaging device may be too narrow to image the entire range of relevant anatomical structures for some types of procedures. Internal and external imaging have been used to expand the field of view, where the internal and external imaging fuse a first medical image source inside the subject with a second medical image source imaged outside the subject for combined reconstruction to model the anatomical structure of the subject.

[0003] However, as an inherent problem associated with fusing different medical imaging sources that use internal and external imaging, some of the information used in the reconstruction may be older than other information. In situations where tissues or organs may shift during a procedure, the older information is of poor validity as representing the current state. The operator may not be able to identify which parts of the reconstructed image are valid and which are not, and thus may not be able to determine which parts of the subject's anatomical structure, or when, need to be re-imaged. This type of loss in effectiveness, fidelity, or certainty can increase the time required for the procedure, reduce the accuracy of the medical device, increase the risk of harm to the subject, and increase the cost or complexity of the procedure, among other things.

[0004] The present disclosure is directed to addressing the above problems. The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. Unless otherwise indicated herein, the content described in this section is not prior art to the claims of the present application and is not admitted to be prior art or an indication of prior art by including it in this section. SUMMARY OF THE INVENTION

[0005] According to certain aspects of the present disclosure, methods and systems for providing in-vivo navigation of a medical device are disclosed. In one aspect, a computer-implemented method for reconstructing a medical image of a subject's anatomical structure comprises monitoring a first medical image data of the subject's anatomical structure and a second medical image data of the subject's anatomical structure, wherein in the monitoring, the first medical image data is captured from inside the subject's anatomical structure by at least one first medical imaging device, the second medical image data is captured by at least one second medical imaging device, at least one of the first medical imaging device and the second medical imaging device is operable to selectively target different regions of the subject's anatomical structure, and in response to detecting based on the monitoring that the deviation between the first medical image data and the second medical image data exceeds a preset threshold, updating the reconstruction of the medical image of the subject's anatomical structure based on the first medical image data and the second medical image data.

[0006] In one aspect, updating the reconstruction of the medical image of the subject's anatomical structure may include causing at least one of the first medical imaging device or the second medical imaging device to obtain the first medical image data or the second medical image data respectively from at least one region of the subject's anatomical structure so as to cause a reduction in the deviation by the obtaining, and modifying the reconstruction of the medical image of the subject's anatomical structure based on the obtained first medical image data or the second medical image data.

[0007] In one aspect, causing at least one of the first medical imaging device or the second medical imaging device to re-acquire the first medical image data or the second medical image data respectively may include operating an actuator configured to perform one or more of re-positioning or re-orienting at least one of the first medical imaging device or the second medical imaging device so that at least one of the first medical imaging device or the second medical imaging device targets at least one region of the subject's anatomical structure.

[0008] In one aspect, at least one region of the anatomical structure of the subject may include a plurality of regions, and the actuator is based on a signal from a force transducer associated with the actuator such that the first medical image data or the second medical image data is reacquired for the plurality of regions, and may be operated according to one or more of a random pattern or a pseudo-random pattern, a linear pattern, a tilt pattern, a rotation pattern, or an obstacle avoidance protocol.

[0009] In one aspect, at least one of the first medical imaging device or the second medical imaging device that is operable to selectively target different regions of the anatomical structure of the subject may be a plurality of sensors or a sensor array, each of which may include a plurality of sensors or a sensor array that are in one or more positions or are oriented to target different regions of the anatomical structure of the subject, and causing at least one of the first medical imaging device and the second medical imaging device to reacquire the first medical image data or the second medical image data, respectively, may include changing the target of the plurality of sensors or the sensor array.

[0010] In one aspect, the deviation between the first medical image data and the second medical image data is based on at least one distance between a first region of the anatomical structure of the subject targeted by the first medical imaging device and a second region of the anatomical structure of the subject targeted by the second medical imaging device, or between a first position of the first medical imaging device and a second position of the second medical imaging device.

[0011] In one aspect, the deviation between the first medical image data and the second medical image data may be based on a moving average of the distance. In one aspect, the method may further include determining one or more of a first position of a first medical imaging device or a first region of an anatomical structure of a subject targeted by the first medical imaging device based on one or more of electromagnetic position tracking, optical fiber shape sensing, position tracking by a second medical imaging device, or impedance-based tracking; and determining one or more of a second position of the second medical imaging device or a first region of an anatomical structure of a subject targeted by the second medical imaging device based on one or more of electromagnetic position tracking, optical tracking, or internal position tracking of the second medical imaging device.

[0012] In one aspect, the deviation between the first medical image data and the second medical image data may be based on (a) the elapsed time since one or more of the first medical image data were captured by the first medical imaging device, or (b) the elapsed time since the second medical image data were captured by the second medical imaging device.

[0013] In one aspect, the method may further include reconstructing a medical image of the anatomical structure of the subject based on the first medical image data and the second medical image data; and outputting a visual depiction of at least a portion of the reconstruction of the medical image to a display.

[0014] In one aspect, updating the reconstruction of the medical image of the anatomical structure of the subject may include modifying the visual depiction of at least a portion of the reconstruction of the medical image such that at least a portion of the reconstruction of the medical image has a visual appearance indicating the elapsed time since the medical image data on which the at least a portion is based were captured.

[0015] In one aspect, a visual appearance indicating the elapsed time since the medical image data, on which at least a part thereof is based, was captured is a reduction in opacity, contrast, saturation, sharpness, resolution, luminance, or a combination thereof, where the magnitude of the reduction increases with the increase in the elapsed time, a reduction, a color transition where the magnitude of the transition increases with the increase in the elapsed time, a color transition, a blinking effect, or a boundary region or selection indicating at least a part of the reconstruction of the medical image, and may include one or more of them.

[0016] In one aspect, this method may further include obtaining a reconstruction of a previous medical image of the subject's anatomical structure based on third medical image data captured by a third medical imaging device, determining a registration of the reconstruction of this medical image with the reconstruction of the previous medical image, and causing the display to output a visual depiction of the reconstruction of the previous medical image based on the determined registration, together with at least a part of the reconstruction of the medical image.

[0017] In one aspect, this method may further include receiving a user selection regarding the relative visibility between the reconstruction of the previous medical image and at least a part of the reconstruction of this medical image in the visual depiction, and selectively increasing or decreasing the visibility of the reconstruction of the previous medical image and the visibility of at least a part of the reconstruction of this medical image in the visual depiction based on the user selection.

[0018] In one aspect, the first medical imaging device may include one or more of an ultrasonic transducer, a parallel array or phased array transducer, an optical coherence tomography device, or an optical sensor. In one aspect, the second medical imaging device may include one or more of an ultrasonic transducer, an array of ultrasonic transducers, an X-ray device, or a computed tomography device.

[0019] In one aspect, a system for reconstructing a medical image of a subject's anatomical structure may include a memory that stores instructions, a first medical imaging device configured to capture first medical image data from within the subject's anatomical structure, a second medical imaging device configured to capture second medical image data from outside the subject's anatomical structure, and a processor operably connected to the memory, the first medical imaging device, and the second medical imaging device and configured to execute instructions to perform operations. At least one of the first medical imaging device and the second medical imaging device may be operable to selectively target different regions of the subject's anatomical structure. The operations may include monitoring the first medical image data of the subject's anatomical structure and the second medical image data of the subject's anatomical structure, and updating the reconstruction of the medical image of the subject's anatomical structure based on the first medical image data and the second medical image data in response to detecting, based on the monitoring, that a deviation between the first medical image data and the second medical image data exceeds a preset threshold.

[0020] In one aspect, updating the reconstruction of the medical image of the subject's anatomical structure may include causing at least one of the first medical imaging device and the second medical imaging device to obtain, respectively, the first medical image data or the second medical image data from at least one region of the subject's anatomical structure, the obtaining causing a reduction in the deviation, and modifying the reconstruction of the medical image of the subject's anatomical structure based on the obtained first medical image data or second medical image data.

[0021] In one aspect, the operation may further include reconstructing a medical image of the anatomical structure of a subject based on first medical image data and second medical image data, and outputting a visual depiction of at least a portion of the reconstruction of the medical image to a display. Updating the reconstruction of the medical image of the anatomical structure of the subject includes modifying the visual depiction of the at least a portion of the reconstruction of the medical image such that the at least a portion has a visual appearance indicating the elapsed time since the medical image data on which the at least a portion is based was captured.

[0022] In one aspect, a computer-implemented method for reconstructing a medical image of a subject's anatomical structure comprises monitoring a first medical image data of the subject's anatomical structure and a second medical image data of the subject's anatomical structure, wherein, in the monitoring, the first medical image data is captured from inside the subject's anatomical structure by at least one first medical imaging device, the second medical image data is captured from outside the subject's anatomical structure by at least one second medical imaging device, at least one of the first medical imaging device and the second medical imaging device is operable to selectively target different regions of the subject's anatomical structure, and in response to detecting, based on the monitoring, that the deviation between the first medical image data and the second medical image data exceeds a preset threshold, updating the reconstruction of the medical image of the subject's anatomical structure based on the first medical image data and the second medical image data. Updating the reconstruction of the medical image of the subject's anatomical structure may include causing at least one of the first medical imaging device or the second medical imaging device to acquire, respectively, the first medical image data or the second medical image data from at least one region of the subject's anatomical structure so as to reduce the deviation by the acquisition, modifying the reconstruction of the medical image of the subject's anatomical structure based on the acquired first medical image data or second medical image data, and modifying the visual depiction of at least a portion of the reconstruction of the medical image such that the at least a portion of the reconstruction has a visual appearance indicating the elapsed time since the medical image data on which the at least a portion is based was captured.

[0023] It should be understood that both the foregoing general description and the following "Detailed Description of the Invention" are exemplary and explanatory only and are not restrictive of the disclosed embodiments as claimed.

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Best Mode for Carrying Out the Invention

[0026] According to certain aspects of the present disclosure, a method and system for multi-source medical image reconstruction are disclosed, and more specifically, a system and method that take into account data currency for medical image acquisition and / or reconstruction are disclosed. As used herein, taking into account "data currency" generally includes taking into account how much time has elapsed since the data was imaged, and can be applied to activities such as further data acquisition procedures, how the data is displayed or modeled, and the like.

[0027] In certain medical procedures, it may be desirable or necessary to have an accurate representation of the anatomical structure of the subject. For example, in an excision procedure, accurate knowledge of the position and orientation of the ablation device within the anatomical structure of the subject, as well as accurate knowledge of the various structures of the subject's anatomical structure, particularly the position of the tissue to be excised, can facilitate a successful outcome. However, conventional navigation techniques may not be appropriate. For example, since conventional techniques may not take into account data currency, an accurate representation of the position of the subject's anatomical structure or the position of a medical device within the subject may not be obtained. As will be discussed in more detail below, in various embodiments, a system and method for medical image reconstruction that take into account data currency are described.

[0028] Any reference to a particular procedure is provided in this disclosure for convenience only and is not intended to limit the disclosure. Examples of procedures that may be relevant to the present disclosure include, for example, excision or removal of a lesion, dilation, retrieval, biopsy, transplantation, ablation, delivery of a therapeutic agent, and the like. Those skilled in the art will recognize that the concepts underlying the disclosed apparatus and methods can be utilized in any suitable procedure. The present disclosure can be understood with reference to the following description and the accompanying drawings, and like elements are referred to by the same reference numerals.

[0029] The terms used below can be interpreted in the broadest reasonable manner even when used in conjunction with the detailed description of specific embodiments of the present disclosure. In fact, although certain terms may even be emphasized below, any term that is intended to be interpreted in any limited manner is so clearly and specifically defined in the "Mode for Carrying Out the Invention" section. Both the foregoing general description and the following detailed description are merely illustrative and explanatory and do not limit the claimed features.

[0030] For ease of explanation, a part of the device and / or a part of its components are referred to as a proximal part and a distal part. Note that the term "proximal" is intended to refer to the part closer to the user of the device, and the term "distal" is used herein to refer to the part farther away from the user. Similarly, extending "distally" indicates that the component extends in the distal direction, and extending "proximally" indicates that the component extends in the proximal direction.

[0031] In the present disclosure, the term "based on" means "at least partially based on". The singular forms "a", "an", and "the" include plural referents unless the context indicates otherwise. The term "exemplary" is used in the sense of "an example" rather than "ideal". The terms "comprises", "comprising", "includes", "including", or other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or product that comprises a list of elements does not necessarily include only those elements, but may include other elements not expressly listed or other elements inherent to such a process, method, article, or apparatus. The term "or" is used disjunctively such that "at least one of A or B" includes (A), (B), (A and A), (A and B), etc. Relative terms such as "substantially" and "generally" are used to indicate that variations of ± 10% of the stated or understood value are contemplated.

[0032] As used herein, terms such as "medical image data" generally include data associated with and / or indicative of the geometric shape and / or physiological function of a subject that can be generated via, for example, medical images. Medical images generally include techniques by which a signal (such as light, electromagnetic energy, radiation, etc.) is generated, and measurements are obtained indicating how the signal interacts with and / or is affected by the subject, and how it passes through the subject. Examples of medical imaging techniques include ultrasound scans, magnetic resonance imaging (MRI) scans, computed tomography (CT) scans, X-ray scans, visual imaging (such as via an optical channel or a camera, etc.), or any other suitable modality that can be used, for example, to visualize at least a portion of the internal anatomy of a subject. Medical image data can include, for example, two-dimensional data and / or images, three-dimensional data and / or images, voxel or pixel data, and / or any other appropriate data associated with the subject and / or the medical image.

[0033] As used herein, "reconstruction of a medical image" generally includes models, images, visual displays, etc. generated using medical image data. For example, the reconstruction can include a visual depiction of a two-dimensional or three-dimensional geometric model of a portion of the subject's anatomy generated based on any other suitable characteristics of that portion of the anatomy that can be visually represented as, for example, medical image data, video, time-varying images, a solid model of at least a portion of the subject's anatomy, a mesh of nodes or points representing that portion of the anatomy, and / or a mechanical or physiological model, etc.

[0034] As used herein, terms such as "user" generally include any person or entity that can interact with or operate an apparatus or component. "Operator" generally includes any personnel involved in providing medical care, such as, for example, physicians, surgeons, nurses, radiologists, technicians, hospitals, clinics, or outpatient facilities. "Subject" generally includes a person who receives such medical care.

[0035] In many interventional therapies, such as procedures involving interacting with or navigating a medical device within a subject's anatomical structure, the subject may undergo pre-procedure imaging, such as an MRI scan, a CT scan, an X-ray scan, etc., that can be used by a medical provider, such as a physician or a surgeon, in order to plan the procedure. In some cases, previous imaging results and / or reconstructions generated therefrom may be available during the procedure to function, for example, as a reference or a guide.

[0036] However, previous imaging may not effectively reflect the current state of the subject's anatomical structure. For example, between the time when the previous imaging was performed and the time when the procedure is performed, or even during the procedure, there may be migration of organs or tissues, progression of the area of diseased tissue, etc. Also, various other factors, such as the subject's hydration, diet, etc., may cause changes in the subject's anatomical structure. Thus, in some cases, previous imaging can be complemented by additional imaging performed during the procedure. For example, scope devices, such as ureteroscopes, duodenoscopes, gastroscopes, endoscopic ultrasound ("EUS") scopes, colonoscopes, bronchoscopes, laparoscopes, arthroscopes, cystoscopes, suction mirrors, sheaths, catheters, etc., may include one or more optical channels, one or more image sensors, one or more ultrasonic transducers, etc., that can provide medical images during interventional therapy. However, the field of view of medical images obtained using a device within the subject's anatomical structure may be narrow and may not be suitable for achieving a complete understanding of the portion of the subject's anatomical structure relevant to the procedure.

[0037] For example, medical images taken from outside a subject, such as X-rays, transcutaneous ultrasound, CT, etc., have also been used during a procedure. However, while such techniques generally have a wider field of view than internal imaging, they may not have sufficient accuracy for certain procedures.

[0038] Also, combinations of internal and external medical images, such as internal imaging and external imaging for example, have been used during medical procedures, but such combinations also generally involve issues regarding accuracy and effectiveness. For example, one fundamental issue associated with the fusion of different medical image sources using internal imaging and external imaging is that some of the information used in the reconstruction is older than other information. As described above, tissues or organs may shift between when the image is being taken and when the procedure is being performed, or during the procedure, so the older information may be of poor effectiveness as a representation of the current state. In other words, some parts of the reconstruction may be less effective than other parts, and it may be difficult for a healthcare provider to distinguish which parts of the reconstruction are reliable and which are not. Also, the operator may not have any means of knowing the appropriate time to re-acquire the medical image data. As a result, internal imaging and external imaging can lead to an inherent risk that the operator relies on invalid data.

[0039] The aforementioned problems are exacerbated, for example, when the procedure and / or medical device used is sensitive to minor changes in the subject's anatomical structure. In the absence of clear indication of the effectiveness of a particular medical image regarding the current state of the subject's anatomical structure, a healthcare provider may rely on inaccurate medical image data, which can have an adverse effect on the procedure or the subject's outcome. Therefore, there is a need for improvement in techniques regarding accurately reflecting the effectiveness of the reconstruction of medical images.

[0040] In the following description, embodiments will be described with reference to the accompanying drawings. As will be considered in more detail below, in various embodiments, systems and methods for providing in-vivo navigation of a medical device are considered.

[0041] In an exemplary use case, a subject may receive a pre-treatment medical image. During the treatment, the subject may receive internal and external image formation. For example, in a first medical imaging device, a medical device (e.g., an endoscope) inserted into the anatomical structure of the subject may include an ultrasonic transducer, etc., and in a second medical imaging device, for external image formation, for example, a transabdominal or transthoracic ultrasonic sensor, etc. may be used. One or more of the first medical image data from the first medical imaging device, the second medical image data from the second medical imaging device, or the pre-treatment medical image data may be registered (aligned) with each other, and / or may be used, for example, via a medical image reconstruction system, etc., to reconstruct a medical image of the anatomical structure of the subject.

[0042] When the treatment is performed, various situations may change with respect to the first and second medical image data captured by the first and second medical imaging devices. For example, as time passes, the effectiveness of a particular instance of the medical image data captured by the medical imaging device may decrease. In another example, one of the medical imaging devices may be moved to a different position, and / or may target different regions within the anatomical structure of the subject. As a result of such situations, for example, a divergence such as a change in state from the period when the medical image data was captured may occur. The reconstruction system may monitor the first and second medical image data captured by the first and second medical imaging devices to determine, for example, the divergence between the first medical image data and the second medical image data.

[0043] The deviation can be based on, or associated with, for example, the time elapsed since the medical image data was captured, the distance or change in distance between the first and second medical devices or between regions of the anatomical structure of the subject targeted by the first and second medical imaging devices, the registration error (alignment error) between the first and second medical image data, etc. In another example, the deviation can be based on the case where the imaging region of the anatomical structure of the subject, which is different from the region currently being imaged or previously imaged, is selected by the user.

[0044] In response to the deviation exceeding a preset threshold, the reconstruction system can cause an update to the reconstruction of the medical image of the subject. Such an update can include, for example, causing one or more of the first or second medical imaging devices to reacquire the medical image data and modifying the reconstruction of the medical image based on the reacquired data, or modifying the visual depiction of the reconstruction of the medical image such that at least a portion of the reconstruction of the medical image has a visual appearance indicating the time elapsed since the medical image data on which at least a portion thereof is based was captured.

[0045] In one example, the first medical imaging device can be moved within the anatomical structure of the subject. As a result of such movement, the first medical imaging device can target different regions of the anatomical structure of the subject and can also move relative to the second medical imaging device. When the first medical imaging device moves, the reconstruction of the medical image may be updated using the captured first medical image data. Further, as time elapses after a previous region of the anatomical structure of the subject has been targeted, the corresponding portions of the reconstruction of the medical image may be grayed out, blurred, etc. to visually indicate the elapsed time of the data used as the basis for those portions. Additionally, the elapsed time of the data of the previously targeted region and / or the increase in the distance between the first and second medical imaging devices can cause the registration system to trigger the reacquisition of data by the second medical imaging device.

[0046] The above examples are merely illustrative, and it should be understood that the techniques according to the present disclosure can be adapted to any suitable type of procedure or activity, and that further aspects of the present disclosure will be considered in more detail below. Further, although some of the above examples related to ultrasonic imaging, it should be understood that any suitable type of medical image can be used.

[0047] FIG. 1 shows an exemplary environment that can be utilized with the techniques presented herein. One or more user devices 105, one or more pre-treatment medical imaging devices 110, one or more medical devices 115, one or more first medical imaging devices 120 for internal image formation, one or more second medical imaging devices 125 for external image formation, one or more actuators 127, and one or more data storage systems 130 can communicate via an electronic network 135. One or more users 140 can interact with the user device 105, for example, to interact with and / or access information from other components of the environment. One or more operators 145 can use, access, or direct the use of one or more of the components within the environment, for example, to provide medical services such as medical interventions, medical procedures, diagnoses, etc. to a subject 150. As will be considered in more detail below, the environment can further include one or more medical image reconstruction systems 155 that consider data currency when reconstructing medical images of the anatomical structure of the subject 150, for example.

[0048] In some embodiments, the components of the environment are associated with a common entity, such as a hospital, a facility, etc. In some embodiments, one or more of the components of the environment are associated with an entity different from another entity. The systems and devices of the environment can communicate in any arrangement. As considered herein, the systems and / or devices of the environment can communicate and / or operate for one or more of, among other activities, performing a medical procedure on the subject 150 or reconstructing a medical image of the anatomical structure of the subject 150 considering data currency.

[0049] The user device 105 can be a computer system such as, for example, a desktop computer, a mobile device, a tablet, etc. In some embodiments, the user device 105 can include one or more electronic applications such as, for example, programs, plugins, browser extensions, etc. installed on the memory of the user device 105. In some embodiments, those electronic application(s) can be associated with one or more of the other components in the environment. For example, the electronic application(s) can include software or applications for accessing, controlling, and / or operating medical devices, medical imaging devices, for taking in data or uploading data to a data storage system, outputting data such as the reconstruction of medical images, etc. The user device 105 can include appropriate input / output devices (not shown) such as, for example, a mouse, a keyboard, a microphone, a touch screen, a display, a speaker, etc.

[0050] The pre-treatment medical imaging device 110 can be any suitable type of medical imaging device and can include medical imaging modalities similar to or different from the first medical imaging device 120 and the second medical imaging device 125. In one example, the pre-treatment medical imaging device 110 can include medical imaging modalities that may be difficult or impossible to use during a medical procedure. As a non-limiting example, an MRI scanner may be used to reconstruct relatively detailed medical images of a subject's anatomical structure and may have an extended field of view compared to other medical imaging modalities. However, an MRI scanner may be difficult to use efficiently or effectively during the course of a medical procedure. Not only can the MRI scanner itself interfere with access to the subject 150, but the medical procedure itself, such as, for example, the movement of the subject 150, the presence of the operator 145 and / or the medical device 115, etc., can interfere with or adversely affect the ability of the MRI scanner to produce accurate or usable results.

[0051] As will be discussed in more detail below, the pre-treatment medical imaging device 110 can be used to reconstruct, for example, an initial medical image of the anatomical structure of a subject, and this initial reconstruction can also function as a guide and / or be combined with additional medical image data captured during a medical procedure. Data based on medical image data, such as the medical image data and / or the reconstruction of the initial medical image, can be stored in the data storage system 130.

[0052] The medical device 115 can be any suitable device that can be used during a medical procedure, for example, for purposes such as examination, diagnosis, treatment, resection, etc. In one example, the medical device 115 includes a scope device such as an endoscope, which can have a working channel for receiving functional ends such as a cautery tool, a biopsy tool, an optical channel, and / or medical instruments such as the above tools. In another example, the medical device 115 includes a syringe having a needle configured to deliver fluid into or acquire fluid from the anatomical structure of the subject. As an alternative example, the medical device 115 can include another type of medical device such as a laparoscopic instrument or an interventional radiology tool. In various embodiments, the medical device 115 may be operated by the operator 145 and / or may be at least partially autonomous. In some embodiments, the medical device 115 can further include a position sensor (not shown) for tracking the position of the medical device 115.

[0053] The first medical imaging device 120 is configured to capture first medical image data from inside the anatomical structure of the subject 150. The first medical image data may be stored by the data storage system 130. In some embodiments, as shown in FIG. 2 and described below, the first medical imaging device 120 can be at least partially integrated with the medical device 115. For example, the first medical imaging device 120 can include an ultrasonic transducer at the distal end of the shaft of the scope device of the medical device 115. In other examples, the first medical imaging device 120 and the medical device 115 can be separate devices.

[0054] The first medical imaging device 120 may be selectively navigable or orientable within the anatomical structure of a subject. For example, in an example where the first medical imaging device 120 is integrated with the medical device 115, as will be described later, the handle portion 245 (FIG. 2) can control or operate the distal portion of the shaft 215, and the first medical imaging device 120 can be disposed at the distal end portion of the shaft 215. In another example where the first medical imaging device 120 is separate from the medical device 115, the first medical imaging device 120 may include an insertion portion for insertion into the anatomical structure of a subject (e.g., the body cavity of a subject). The first medical imaging device 120 may include an ultrasonic transducer at the distal end of the insertion portion. The first medical imaging device 120 may further include a control portion (e.g., a handle) proximal to the insertion portion, and this portion may include an actuator or other mechanism external to the subject 150, and the control portion may be operable to control or direct the position and / or orientation of the ultrasonic transducer within the anatomical structure of the subject. By adjusting the position and / or orientation of the first medical imaging device 120 within the anatomical structure of the subject, the first medical imaging device 120 may be able to selectively target different regions of the anatomical structure of the subject 150.

[0055] In some embodiments, the first medical imaging device 120 may include one or more position sensors 160. The position sensor 160 may be configured to generate a signal indicating the position or orientation of the first medical imaging device 120. Such a signal may also be used to determine which region of the anatomical structure of the subject 150 the first medical imaging device 120 is targeting. Any suitable type of position sensor may be used. For example, in an exemplary embodiment, the position sensor 160 may include an electromagnetic position sensor disposed at the distal end of the first medical imaging device 120. In another exemplary embodiment, the first medical imaging device 120 may include an optical fiber channel extending from the distal end to the proximal end, and the position sensor 160 may be configured to use optical fiber shape sensing to determine the orientation and position of the distal end. In a further embodiment, the second medical imaging device 125 may be used to determine the position and / or orientation of the first medical imaging device 120 and thus may function as a position sensor.

[0056] In another exemplary embodiment, the first medical imaging device 120 may include a plurality of sensors or a plurality of sensor arrays. For example, the first medical imaging device 120 may include an ultrasonic array transducer, whereby it is possible to target different regions of the anatomical structure of the subject 150 by operating different portions of the ultrasonic array transducer. In a further exemplary embodiment, the first medical imaging device 120 may not include portions outside the anatomical structure of the subject 150 and may be in the form of an implant device, a swallowable sensor (e.g., a pill), or the like.

[0057] Medical image data from an ultrasonic transducer, such as ultrasonic data for example, is generally generated by applying ultrasonic waves to the anatomical structure of a subject, whereby a pulse of high-frequency vibrations is transmitted into the tissue using a probe such as, for example, an ultrasonic transducer, and includes data associated with the internal structure of a part of the anatomical structure of the subject. The vibrations are at least partially reflected from a surface representing a change in acoustic impedance within the body, such as, for example, the geometry of a structure or tissue. The reflected vibrations returning to the transducer may be transmitted, for example, to the data storage system 130. Generation of image data using ultrasonic data and / or reconstruction of a medical image is generally based on the time required for the reflection to return to the transducer after application of the vibrations and the intensity of the returned reflection. Conventional transducers are generally configured to receive variations in the signal response only over one dimension. In other words, for a static position of the transducer, only one column of the pixel data of the ultrasonic image can be received. Thus, to generate an image or model of the anatomical structure, a single transducer is generally swept across the field of view, for example rotated back and forth, to successively add and / or refresh columns of values to the data.

[0058] As described above, in some embodiments, the first medical imaging device 120 may include a transducer array. The transducer array may include, for example, a plurality of transducers arranged in parallel with each other, for example, distributed over at least a portion of the outer periphery of the distal end of the medical device 115 and / or the first medical imaging device 120, or arranged in a substantially linear shape along the distal end of the medical device 115 and / or the first medical imaging device 120. As a result, multiple columns of data can be sensed at once without rotation. In various embodiments, any suitable number and / or type of parallel transducers may be used. With additional transducers, the static field of view is effectively enlarged.

[0059] Some of the above examples relate to ultrasound, but it should be understood that any suitable one or more medical imaging modalities, such as optical sensors, optical coherence tomography devices, infrared sensors, spectrometers, etc., may be used by or with the first medical imaging device 120.

[0060] FIG. 2 shows an exemplary embodiment of a first medical imaging device integrated into a medical device 115. However, it should be understood that the embodiment of FIG. 2 is merely exemplary, and any suitable medical device, such as a medical device that does not include a medical imaging device, may be used. The medical device 115 of FIG. 2 may include a distal end 205 connected to a proximal end 210 via a shaft 215.

[0061] The distal end 205 may include one or more portions 220 configured to perform one or more of receiving components or communicating with a lumen (e.g., a working channel) provided within the shaft 215. For example, the first medical imaging device 120 may be disposed in one of the plurality of portions 220. In another example, a tool having an end effector 230, such as a cautery device, a needle, a knife, forceps, a snare, a balloon, a delivery device, a guidewire, a net, an orifice for taking in or out fluid and / or material, etc., may be disposed in another of the plurality of portions 220. The distal end 205 may also include, for example, one or more lights, image sensors, openings for air / water / suction, elevators, or any other suitable features.

[0062] In some embodiments, the shaft 215 may include a flexible material. The shaft 215 may include one or more lumens (not shown) that communicate between the distal end 205 and the proximal end 210. In some embodiments, the shaft 215 may further include and / or house other elements, such as a wire connector configured to communicate data between components at the distal end 205, such as a position sensor 225, an imaging device 120, a camera, or a light, and the proximal end 210.

[0063] The proximal end 210 may include, for example, a handle portion 245 that enables an operator to manipulate, advance, retract, and / or orient the distal end 205. For example, the handle portion 245 may include one or more actuators (e.g., knobs) 255 used to articulate / manipulate the distal end 205, as well as actuators for delivering / aspirating fluid or performing other controls. The proximal end 210 may further include one or more interfaces 250, such as an umbilicus to output data, for transmitting or receiving electrical signals and / or for transmitting fluid or materials in and out of the medical device 115. The interface for data may include one or more of a wired connection or a wireless connection. The interface 250 may also be configured to receive power for operating the first medical imaging device 120, the position sensor 225, the end effector 230, the camera, the light, or another feature of the distal end 205. In some embodiments, the medical device 115 or at least a portion thereof is configured to be disposable, such as, for example, a single-use device.

[0064] In another exemplary embodiment (not shown), the proximal end of the medical device 115 may include or be configured to operate with an actuator (e.g., actuator 127 of FIG. 1) operable to manipulate the distal end 205, for example, via electronic control. For example, the proximal end of the medical device 115 may include features such that the proximal end may be received by the actuator 127, or may include the actuator 127 integrated therein. In other words, in some embodiments, the distal end 205 may be remotely controlled and / or robotically controlled. The medical device 115 may include any features suitable for remote control and / or robotic control.

[0065] Returning to FIG. 1, the second medical imaging device 125 can be configured to capture second medical image data from outside the subject. In various embodiments, the second medical imaging device 125 can include, for example, a transcutaneous ultrasound device, an X-ray scanner, a CT scanner, and the like. In some embodiments, the second medical imaging device 125 may be at least partially electronically controlled. In some embodiments, the sensors of the second medical imaging device 125 can be movable and / or orientable, for example, via an operator 145 or via electronic control. For example, the sensors of the second medical imaging device 125 can be attached to a movable arm (not shown). The movable arm may be manually operated, for example, and / or may include one or more of an actuator 127 that enables electronic control of the position and / or orientation of the sensors of the second medical imaging device 125 to target different regions of the anatomical structure of the subject 150. In some embodiments, the second medical imaging device 125 can include a position sensor 165 that can be used to determine the position or orientation of the second medical imaging device 125 and / or the region of the anatomical structure of the subject targeted by the second medical imaging device 125, similar to the position sensor 160 of the first medical imaging device 120.

[0066] In some embodiments, the second medical imaging device 125 can have a configurable field of view that enables the user 140 to select one or more of the target region or orientation. Any suitable external medical imaging device may be used. For example, in some embodiments, the second medical imaging device 125 can include a plurality of sensors or an array of sensors.

[0067] FIG. 3 shows an exemplary schematic diagram of a second medical imaging device 125 that includes a plurality of sensor arrays 300. Each array 305 among the plurality of arrays 300 includes a plurality of sensors 310, such as, for example, ultrasonic transducers. In an exemplary embodiment, the plurality of sensor arrays 300 may be placed over a subject 150. By activating and / or deactivating a plurality of different arrays 305 among the plurality of arrays 300, a plurality of different portions of the anatomical structure of the subject 150 can be imaged, for example, without the need to move or reorient the second medical imaging device. In the embodiment shown in FIG. 3, the plurality of arrays 300 are arranged in a cross shape. However, any suitable arrangement may be used, such as, for example, any polygon or irregular shape. In some embodiments, the plurality of arrays 300 may be shaped in a manner corresponding to at least a portion of the human anatomical structure, for example, shaped to correspond to the shape of the human torso. Further, the array 305 may have any suitable shape (rectangular, triangular, circular, irregular, etc.). Although shown as separate in FIG. 3, in some embodiments, one or more arrays 305 may at least partially overlap. Further, in some embodiments, the individual sensors 310 may be separately addressable such that, for example, the distribution of sensors to the plurality of different arrays 305 is electronically controlled. In some embodiments, the plurality of different arrays 305 may be tilted, for example, facing each other, or back to back, or in an irregular direction. Such tilting may, for example, reduce crosstalk between the sensor signals of different arrays when the plurality of arrays 305 are used simultaneously and / or may enable different orientations for imaging the anatomical structure of the subject 150. While some of the above embodiments in which the plurality of arrays 300 remain in a stationary position with respect to the subject 150 have been described, in some embodiments, the plurality of arrays 300 may be movable, for example, manually or via an electronically controlled actuator such as, for example, the actuator 127 of FIG. 1. For example, the plurality of arrays 300 may be movable in a parallel, rotatable, and / or vibratory or oscillatory motion.

[0068] Returning to FIG. 1, data storage system 130 may include, or be accessible through, a medical data exchange system or the like. Data storage system 130 may include computer-readable memory such as a server system, an electronic medical data system, a hard drive, a flash drive, a disk, and the like. In some embodiments, data storage system 130 includes and / or interacts with an application programming interface for exchanging data with other systems, such as one or more of the other components of the environment. Data storage system 130 may include, and / or function as, a repository or source for medical image data. For example, medical image data obtained from a medical imaging device may be stored by data storage system 130 and / or provided by data storage system 130 to reconstruction system 155, as will be discussed in more detail below.

[0069] In various embodiments, the electronic network 135 can be a wired connection between devices, a wireless connection between devices, a wide area network ("WAN"), a local area network ("LAN"), a personal area network ("PAN"), and the like. In some embodiments, the electronic network 135 includes the Internet, and the information and data provided between various systems occur online. "Online" can mean connecting to or accessing source data or information from a location remote from other devices or networks coupled to the Internet. Alternatively, "online" can refer to connecting to or accessing an electronic network (wired or wireless) via a mobile communication network or device. The Internet is a worldwide system of computer networks, a network of multiple networks in which a party at one computer or other device connected to the network can obtain information from any other computer and communicate with parties at other computers or devices. The most widely used part of the Internet is the World Wide Web (often abbreviated as "WWW" or called "the Web"). A "website page" generally includes, for example, data hosted and / or operated by a computer system so as to be accessible online, configured to cause a program such as a web browser to perform operations such as sending, receiving, or processing data, and generating a visual display and / or an interactive interface, and can include locations, data stores, and the like.

[0070] As will be discussed in more detail below, the reconstruction system 155 may perform operations related to reconstructing and / or updating medical images based on medical image data, monitoring medical image data captured by a medical imaging device to determine, for example, misalignments, guiding and / or operating the medical imaging device and / or the medical imaging device, and other activities. The reconstruction system 155 may include, for example, a server system, a desktop computer, a laptop, a tablet, and the like. In an exemplary embodiment, the reconstruction system 155 includes a cloud-based architecture for performing operations such as, for example, data processing, and may further include a display and / or interface that is accessible by the operator 145, for example, during a medical procedure.

[0071] The reconstruction system 155 may include software and / or hardware that facilitates operations such as those described above. For example, the reconstruction system 155 may include one or more algorithms, models, etc. for reconstructing medical images based on medical image data. In various embodiments, the reconstruction system 155 may facilitate the reconstruction of medical images using a machine learning-driven model, such as a deep learning model, based on one or more reference models, in one or more manual (e.g., user-driven) manners. Any suitable combination of techniques or methodologies for reconstructing medical images may be used.

[0072] In an exemplary embodiment, the reconstruction system 155 may include a first reconstruction algorithm, model, etc. configured to access medical image data captured by the pre-treatment medical imaging device 110, e.g., from the data storage system 130, and reconstruct an initial medical image based on the pre-treatment medical image data. In a further exemplary embodiment, the reconstruction system 155 may include one or more registration algorithms configured to determine registration between different instances of medical image data. For example, the medical image data received from the first medical imaging device 120 or the second medical imaging device 125 may be registered with each other and / or with an existing medical image reconstruction such as the reconstruction of the initial medical image described above.

[0073] The reconstruction system 155 may further include one or more monitoring algorithms configured to monitor the medical image data received from the first medical imaging device 120 or the second medical imaging device 125. For example, one or more characteristics of the acquired medical image data, such as which region of the subject 150's anatomical structure is targeted, the relative position of the targeted region with respect to other regions targeted by other medical images, the elapsed time of the medical image data, the position of the medical imaging device used to capture the medical image data to be acquired, the relative position of the medical imaging device with respect to other medical imaging devices, the position of the medical device, etc., may be determined using, for example, one or more algorithms. The reconstruction system 155 may include one or more algorithms for determining misalignment of medical data, for example, based on the monitoring.

[0074] The reconfiguration system 155 may include one or more algorithms for guiding or operating a medical imaging device, such as by identifying one or more regions of an anatomical structure to be imaged and operating an actuator 127 or the like to reposition or reorient the medical imaging device. In one example, the reconfiguration system 155 may be configured to move the medical imaging device according to a predetermined path or pattern. The reconfiguration system 155 may include one or more algorithms for updating and / or outputting information related to medical image reconstruction, taking into account, for example, the data currency of medical image data received by the reconfiguration system 155. Further aspects of the foregoing are discussed in more detail below.

[0075] Although shown as separate components in FIG. 1, it should be understood that a component or portion of a component within the environment may in some embodiments be integrated with or incorporated into one or more other components. For example, a portion of the reconfiguration system 155 may be integrated with the user device 105 or the data storage system 130. In some embodiments, one or more operations or aspects of the components described above may be distributed among one or more other components. Any suitable arrangement and / or integration of the various systems and devices of the environment may be used.

[0076] For a further aspect of the reconfiguration system 155 and / or how it can be utilized in conjunction with the medical device 115 and / or a medical procedure to provide information regarding the data currency of the captured medical image data, it will be discussed in more detail in the following manner. In the following manner, various actions are described as being performed or being able to be performed by the components of FIG. 1, such as the user device 105, the first medical imaging device 120 or the second medical imaging device 125, the reconfiguration system 155, or their components. However, in various embodiments, it should be understood that the various components of the environment described above can execute instructions or perform actions including those considered below. The actions performed by the device can be considered to be performed by a processor, actuator, etc. associated with the device. Additionally, in various embodiments, various operations can include or enable user control, for example, via the operator 145. Further, in various embodiments, it should be understood that the various steps may be added, omitted, and / or rearranged in any suitable manner.

[0077] FIG. 4 shows an exemplary process for providing a reconstruction of a medical image of a subject's anatomical structure considering the data currency of the medical image data used for the reconstruction. In step 405, pre-procedure medical image data targeting at least a portion of the anatomical structure of the subject 150 can be captured using the pre-procedure medical imaging device 110. For example, the operator 145 can use an MRI scanner to target the portion of the anatomical structure of the subject 150 and capture MRI data. The pre-procedure medical image data may be stored in the data storage system 130, which is accessible to the pre-procedure medical imaging device 110, for example, via the electronic network 135. In step 410, the reconfiguration system 155 can reconstruct an initial medical image based on the pre-procedure medical image data. Any suitable medical image reconstruction technique may be used.

[0078] In step 415, the reconstruction system 155 can cause the display to output a visual representation of at least a portion of the reconstruction of the initial medical image. For example, the display can be a display accessible to the operator 145 who performs a medical procedure on the subject 150. FIG. 5 shows an exemplary embodiment of a visual representation 500 of a segmented reconstructed CT, such as a reconstruction of a medical image generated based on a segment or slice of an imaging of the anatomical structure of a subject taken via, for example, a CT scanner.

[0079] Returning to FIG. 4, in step 420, one or more of the first medical imaging device 120 or the second medical imaging device 125 can capture the first medical image data or the second medical image data, respectively, for example, during the course of the procedure. The capture of the medical image data can be continuous, intermittent, or automatic (e.g., via electronic control such as the reconstruction system 155) in response to a request or action of the operator 145. The captured medical image data may be stored in the data storage system 130.

[0080] In some embodiments, one or more of the operator 145 or the reconstruction system 155 can reposition or reorient one or more of the first or second medical imaging devices 120 or 125, for example, by manual operation, re-focusing, or electronic control via one or more actuators 127. For example, if the procedure being performed is related to a particular region of the anatomical structure of the subject 150, one or more of the first or second medical imaging devices may be repositioned and / or reoriented to target the particular region.

[0081] In step 425, the reconstruction system 155 may determine the registration of the captured medical image data, such as spatial and / or geometric mapping, relationships, or transformations between the data. In some embodiments, the captured medical image data may be registered relative to the initial reconstruction. For example, in some embodiments, such as those without pre-treatment medical images, the captured medical image data may be registered relative to a reference model and may be manually registered, for example, by an operator 145. In some embodiments, the first medical image data may be registered relative to the second medical image data. Registration may be used, for example, to determine one or more characteristics of the captured medical image data, either of one or more regions of the anatomical structure of the subject targeted by the medical imaging device(s), relative to each other, and / or relative to the anatomical structure of the subject, and / or the position of the medical imaging device(s). In some embodiments, the reconstruction system 155 may acquire or record additional data along with the captured medical image data. Such additional data may include metadata that describes one or more of, for example, the time at which the data was captured, one or more characteristics of the medical imaging device(s) used to capture the data (such as position or target data based on feedback from the actuator 127), one or more characteristics of the anatomical structure of the subject (such as determined based on data captured via an analysis algorithm), etc.

[0082] In one example, the anatomical structure of the subject 150 may include one or more target structures, such as, for example, a lesion, a wound, a foreign body, diseased tissue, etc. FIG. 6A shows a schematic view of a portion of the anatomical structure 600 of the subject 150, which includes a lumen 610 through which the first medical imaging device 120 may be navigated and several lesions 620 that are investigated during the procedure.

[0083] As shown in FIG. 6B, an exemplary embodiment of a first medical imaging device 120 that is separate from the medical device 115 may be navigated through the lumen 610. It should be understood that in various embodiments, any suitable first medical imaging device may be used. In some embodiments, first medical image data may be captured when the first medical imaging device 120 is navigated. In some embodiments, the capture of the first medical image data may be continuous, periodic, on demand, etc. In this example, the first medical imaging device 120 may include an ultrasonic transducer 630 configured to image a portion of the tissue 640 in contact with the lumen 610. FIG. 6C shows an "ultrasound fan", which is, for example, when the first medical imaging device 120 is positioned in the target area, by sweeping the ultrasonic transducer 630 over a rotation angle and / or operating a parallel array of transducers, etc., a visual depiction 650 of an exemplary embodiment of ultrasonic data that can be captured via the ultrasonic transducer 630.

[0084] In some embodiments, the captured first medical image data may be analyzed, for example, to identify lesions, wounds, foreign objects, diseased tissue, the location of the medical device 115, etc. present therein. For example, in some embodiments, a reconstruction system 155, a separate classification system, an operator 145, etc. may perform one or more analyses on the first medical image data. Any suitable analysis technique may be used, such as analysis by an operator 145, automated techniques such as machine learning, etc. Any identified target structure may be associated with registration, for example, to identify its position in the reconstruction of the identified structure.

[0085] The visual depiction of FIG. 6C represents a single instance of capturing first medical image data using the first medical imaging device 120, such as a single instance of sweeping the ultrasonic transducer 630 across the target region, a single instance of operating the phased array of the transducer, etc. Also, although the first medical imaging device 120 was discussed above with respect to FIGS. 6A - 6C, it should be understood that instead of and / or in addition to the first medical image data captured by the first medical imaging device 120, a second medical imaging device may capture second medical image data of the anatomical structure of the subject 150. In various embodiments, the first medical imaging device 120 and / or the second medical imaging device 125 may capture one or more instances of medical image data. For example, the first medical image data of the first medical imaging device 120 may be captured at various positions as the first medical imaging device 120 is navigated through the lumen 610. In various embodiments, the medical image data may be captured continuously, in discrete samples, or in any combination thereof.

[0086] Returning again to FIG. 4, at step 430, the reconstruction system 155 can reconstruct a medical image of the anatomical structure of the subject based on the first and / or second medical image data and can cause the generated reconstruction to be displayed on a display. In one example, the reconstruction system 155 can reconstruct a new medical image based on the first and / or second medical image data. FIG. 7 shows an exemplary embodiment of a graphical user interface 700 that can be output by the reconstruction system 155. The interface 700 shows that one or more instances 705 of data have been captured and used to generate a reconstruction 710 of a medical image based thereon. As shown in FIG. 7, the one or more instances 705 may be shown in an orientation that indicates how they are related to each other within a three - dimensional space.

[0087] The interface 700 may also indicate the current position 715 of the first medical imaging device 120 within the reconstruction 710, for example, based on a determined registration, a position sensor 160, tracking based on the second medical imaging device 125, and the like. The interface 700 may also depict a target structure 720 identified by an analysis algorithm, such as lesions 1-5. In various embodiments, the interface 700 may further include other data or views of the reconstruction 710. For example, in one embodiment, the interface 700 may include an axial view (e.g., a cross-sectional view) of the lumen 610, along with a direction indicator indicating the current orientation of the first medical imaging device 120 (e.g., which direction the sensor of the first medical imaging device 120 is facing within the plane of the cross-sectional view).

[0088] In some embodiments, the interface 700 may include user-selectable options, such as an interface item 735 for displaying or hiding a selected target structure 720. In some embodiments, one or more of the instances 705 or target structures 720 may be user-selectable. FIG. 8 shows an exemplary embodiment of a display 800 of the interface 700 after the user 140 has selected a particular instance 802 from among the instances 705 of the medical imaging data. The display 800 may include a visual display 805 of the selected instance 705 of the medical imaging data (e.g., an ultrasound fan if the instance is ultrasound data), a display 815 of which target structures 720 are visible within that particular instance 705, and a visual emphasis display 820 of the visible target structures 720 that are visible within the visual display 805. In some embodiments, the display 800 may enable the user 140 to select some portions of the visual display, for example, to add, remove, or edit a target structure 720.

[0089] When additional medical image data is captured, the reconstruction system 155 may update the interface 700 using updates to the additional instance 705 and the reconstruction 710. FIG. 9 shows an exemplary embodiment of the interface 700 navigated to return to an area previously imaged by the first medical imaging device 120 to capture additional information regarding the target structure 720, for example, to include additional information near the target structure 720 in the reconstruction 710 of a medical image. The interface 700 shown in FIG. 9 also shows a visual depiction 805 of the current instance of the first medical image data.

[0090] In some embodiments, such as embodiments where pre-treatment medical image data is available, the reconstruction system 155 may include information from the pre-treatment medical image data in the reconstruction 710. The pre-treatment image data may be used, for example, for reconstruction having a wider field of view or more comprehensive imaging of an area of the anatomical structure related to the treatment. Such a reconstruction may function as a general mapping guide to provide the operator 145 with a general understanding of the anatomical structure related to the treatment, supplemented, for example, using image data from the first medical imaging device 120 and the second medical imaging device 125. FIG. 10 shows an exemplary embodiment of a visual depiction of the reconstruction 710 including a first portion 1005 based on the first and / or second medical image data and a second portion 1010 based on the pre-treatment medical image data.

[0091] In some embodiments, the reconstruction system 155 may be configured to change the relative visibility of the first and / or second medical image data with respect to the pre-treatment medical image data. FIG. 11 shows an exemplary embodiment of an interface 1100 including a user-configurable slider 1105 operable to select the relative visibility of the first and / or second medical image data with respect to the pre-treatment medical image data. For example, moving the slider 1105 to the left increases the visibility of the first and / or second medical image data and / or decreases the visibility of the pre-treatment medical image data so that one or both can be displayed. In another example, moving the slider 1105 to the right decreases the visibility of the first and / or second medical image data and / or increases the visibility of the pre-treatment medical image data so that one or both can be displayed.

[0092] Returning to FIG. 4, at step 435, the reconstruction system 155 may monitor the first and / or second medical image data of the anatomical structure of the subject 150. In one example, the reconstruction system 155 can monitor the positions of the first medical imaging device 120 and / or the second medical imaging device 125, monitor one or more positions targeted by the first medical imaging device 120 and / or the second medical imaging device 125, and / or monitor the elapsed time since different instances of the first and / or second medical image data were captured. In some embodiments, the reconstruction system may include additional data such as status or feedback signals from one or more actuators 127 and / or position sensors 160 or 165, the position of the medical device 115, and the like.

[0093] At step 440, the reconstruction system 155 may determine a shift between the first medical image data and the second medical image data based on the monitoring. The shift may be based on one or more factors such as the following examples and may be associated, for example.

[0094] In some embodiments, the deviation may be determined based on the distance and / or change in distance between a first region of the anatomical structure of the subject 150 targeted by the first medical imaging device 120 and a second region of the second medical imaging anatomical structure of the subject 150 targeted by the device 125. For example, in an exemplary use case, the second medical imaging device 125 may be used to perform external imaging of a region of the anatomical structure of the subject 150 that is considered to be related to the treatment. During the treatment process, the operator 145 may navigate the first medical imaging device 120 and / or the medical device 115 towards a part of the anatomical structure outside that region. Therefore, the distance and / or change in distance between the region targeted by the first medical imaging device 120 and the region targeted by the second medical imaging device 125 may indicate the deviation between the first medical image data and the second medical image data.

[0095] In some embodiments, the deviation may be determined based on the distance or change in distance between a first position of the first medical imaging device 120 and a second position of the second medical imaging device 125. For example, in an exemplary use case, it may be desirable to bring the regions targeted by the first medical imaging device 120 and the second medical imaging device 125 close to each other and / or overlap. Therefore, the distance and / or change in distance between the position of the first medical imaging device 120 and the position of the second medical imaging device 125 may indicate the deviation between the first medical image data and the second medical image data.

[0096] In some embodiments, the deviation may be determined based on the distance or change in distance between the position of a part of the medical image reconstruction 710 (for example, a part with a high reliability of effectiveness as further discussed below) and the position of the medical device 115. For example, the operator 145 may navigate the medical device 115 into a region with low reliability of effectiveness within the reconstruction 710 and / or outside the boundary of the reconstruction 710.

[0097] In some embodiments, the shift can be based on a moving average of the distance between the position of the first medical imaging device 120 and the position of the second medical imaging device 125, a moving average of the distance between the target region of the first medical imaging device 120 and the target region of the second medical imaging device 125, and / or a moving average of the distance between the medical device 115 and a portion of the medical image reconstruction 710. By using an averaging term such as a moving average, the shift can be less susceptible to noise and / or the effects of rapid or abnormal movement by one of the first medical imaging device 120 and the second medical imaging device 125. When determining the distance, as in the example above, any other suitable smoothing or noise attenuation technique may also be used. As will be discussed in more detail below, such smoothing can reduce jitter in the operation of the medical imaging device, promote the safety of the subject, and reduce the risk of imaging undesirable or unnecessary portions of the anatomy of the subject 150.

[0098] In some embodiments, when determining the distance, such as in one or more of the examples above, the reconstruction system 155 can use signals received from one or more position sensors 160 or 165. For example, the first medical imaging device 120 can include a position sensor 160 in the form of an electromagnetic position sensor. In another example, such as the embodiment described above with respect to FIG. 2, the first medical imaging device 120 may be integrated into the medical device 115 that includes a shaft 215 that includes an optical fiber channel, and the reconstruction system 155 may use optical fiber shape sensing to position and / or orient the first medical imaging device 120.

[0099] In some embodiments, the reconstruction system 155 may determine the position of the first medical imaging device 120 using the second medical image data. For example, the reconstruction system 155 and / or another system may use an analysis algorithm on the second medical image data to identify the first medical imaging device 120 within the area targeted by the second medical imaging device 125, and determine the position and / or orientation of the first medical imaging device 120 based on the identification result and / or the registration of the second medical imaging device 125 with the anatomical structure of the subject 150. In some embodiments, feedback from the actuator 127 used to control the positioning or orientation of one or more of the first medical imaging device 120 or the second medical imaging device 125 may be used. For example, feedback from the actuator 127 and / or associated sensors may track how the medical imaging device was operated. In some embodiments, impedance-based position tracking may be used. For example, one or more electrodes may be placed at different positions on the body of the subject 150, a current may be generated by the electrodes included in the first medical imaging device 120, and the reconstruction system 155 may determine the position of the first medical imaging device 120 based on the different currents indicated by the electrodes on the body of the subject 150. In some embodiments, similar to the example described above for the first medical imaging device 120, the position of the second medical imaging device 125 may be determined based on a signal from the position sensor 165. In some embodiments, the position of the second medical imaging device 125 may be determined based on optical tracking, for example, via a signal from an optical sensor such as a camera placed near the second medical imaging device 125.

[0100] In some embodiments, the deviation may be based on the elapsed time since one or more instances of the medical image data were captured. In other words, in some embodiments, the deviation may be based on the elapsed time since one or more of the first medical image data were captured by the first medical imaging device or since the second medical image data were captured by the second medical imaging device. As described above, the state of the anatomical structure of the subject 150 can change over time for various reasons. The longer the period since a particular instance of the medical image data was captured, the more likely it is that that particular instance is at least a partially inaccurate representation of the current state of the subject's anatomical structure. In other words, the validity and / or reliability of a particular instance of the medical image data may decrease over time. Thus, the reconstruction system 155 may monitor how much time has elapsed since each instance of the medical image data was captured.

[0101] Although various examples of the basis for the deviation have been discussed above, it should be understood that any suitable basis or combination thereof may be used. Further, in various embodiments, the deviation may be determined separately for each medical imaging device, or for any combination thereof, based on the basis for each region of the anatomical structure (e.g., for one region imaged by a plurality of instances) for each instance of the medical image data. In some embodiments, the deviation may be represented by a value (e.g., a value corresponding to the magnitude of a distance or period). In some embodiments, the deviation may be based on a combination of such factors, and multiple different quantities may be weighted or normalized via any suitable technique. For example, in addition to one deviation point occurring for every one inch separation between medical imaging devices, one deviation point may occur every second. However, the foregoing examples are merely illustrative, and it should be understood that any suitable manner may be used to determine the deviation using one or more of the factors described above.

[0102] In some embodiments, the determined deviation may be compared to a preset threshold. Such a comparison may be made, for example, in real-time or near real-time, periodically such as once per second, every 5 seconds, every minute, every 2 minutes, etc., or at any other appropriate timing, in response to a request by the operator 145, in response to the detection of anatomical features of the subject 150 such as the detection of the target structure, in response to the reconstruction system 155 receiving a new instance of medical image data, etc. Any appropriate preset threshold may be used. In some embodiments, such as embodiments to be discussed in more detail below where the display of the reconstruction 710 of the medical image indicates the time elapsed for each instance of the medical image data, the preset threshold may be any non-zero amount. In some embodiments, different deviations associated with different factors and / or different preset thresholds may be determined for different purposes. For example, a deviation based on an elapsed time having a threshold greater than 0 may be determined for the purpose of updating the visual depiction of the reconstruction of the medical image, and a deviation based on an elapsed time having a threshold greater than 5 minutes may be determined to cause the medical imaging device to reacquire the medical image data, as will be discussed in more detail below.

[0103] In step 445, in response to detecting based on monitoring that a deviation in one or more of the first medical image data and the second medical image data exceeds one or more preset thresholds, the reconstruction system 155 can cause an update of the reconstruction 710 of the medical image.

[0104] In some embodiments, updating the medical image reconstruction 710 includes updating the medical image data used to form the medical image reconstruction. For example, in some embodiments, the reconstruction system 155 causes at least one of the first medical imaging device and the second medical imaging device to re-acquire the first medical image data or the second medical image data from one or more regions of the anatomical structure of the subject 150, and then modifies the medical image reconstruction 710 using the newly acquired data. In some embodiments, the first medical imaging device 120 may be operated by the operator 145, and thus, updating the medical image reconstruction 710 may include causing the second medical imaging device 125, which can be electronically and autonomously controlled by the reconstruction system 155, to re-acquire the medical image data. For example, the reconstruction system 155 may operate one or more actuators 127 to relocate and / or reorient one of the medical imaging devices. In some embodiments, the reconstruction system 155 outputs a message or visual prompt, including instructions for operating one of the medical imaging devices to re-acquire the medical image data, to the operator 145 via a display, for example. In various embodiments, the first medical imaging device 120 may be electronically controlled, both the first medical imaging device 120 and the second medical imaging device 125 may be electronically controlled, or neither may be electronically controlled.

[0105] In some embodiments, one or more regions imaged and / or re-imaged for updating may include regions that would reduce misalignment. For example, a region corresponding to an instance of medical image data captured some time ago may be re-imaged to reduce the time elapsed since that region was imaged. In another example, the second medical imaging device 125 may be moved or reoriented to image a region closer to the position of the first medical imaging device 120, or a region closer to the region targeted by that medical imaging device, and vice versa.

[0106] In some embodiments, one or more of those regions may include a plurality of regions, and the reacquisition of medical image data may include targeting the plurality of regions according to a certain sequence and / or pattern. For example, in an exemplary embodiment, the second medical imaging device 125 may include a transabdominal, a transthoracic, or a conventional ultrasonic sensor, etc., and a robotic arm controlled by one or more actuators 127. The reconstruction system 155 may cause the second medical imaging device 125 to reacquire second medical image data for a plurality of regions (e.g., regions related to a treatment), and may operate one or more actuators 127 so as to operate the second medical imaging device 125 according to one or more patterns that successively target each of the plurality of regions.

[0107] Any suitable pattern may be used, such as a random or pseudo-random pattern, a linear pattern, a spiral pattern, a movement scheme predetermined by a decision algorithm, etc. In some embodiments, the pattern may include one or more of tilting, rotating, sweeping, or pivoting the second medical imaging device 125. In some embodiments, the reconstruction system 155 may apply, instead of or in addition to, a pattern as described above, an obstacle avoidance protocol to the medical imaging device 125. For example, one or more of the actuators 127 may include or be associated with one or more force transducers. In response to a signal from the force transducer indicating a force exceeding a preset threshold, the reconstruction system 155 may stop the operation of the medical imaging device, identify the obstacle, avoid the obstacle so that the medical imaging device is operated around the obstacle, modify the pattern, etc.

[0108] In another exemplary embodiment, as described above, the second medical imaging device 125 may include a plurality of sensors or an array of sensors 300 (FIG. 3), and causing the second medical imaging device 125 to reacquire medical image data from one or more regions of the anatomy of the subject 150 may include changing which of the plurality of sensors or arrays 305 of sensors 300 are activated so as to target one or more regions of the anatomy of the subject 150, changing the positioning or orientation of the plurality of sensors or array 300, and the like.

[0109] In some embodiments, updating the medical image reconstruction 710 includes modifying the visual depiction of the medical image reconstruction 710. For example, in some embodiments, updating may include modifying the medical image reconstruction to include and / or account for newly acquired medical image data, and correspondingly updating the visual depiction 710 of the medical image reconstruction. In some embodiments, updating may include modifying the visual depiction of at least a portion of the medical image reconstruction such that at least a portion of the medical image reconstruction has a visual appearance indicative of the elapsed time since the medical image data on which at least that portion is based was captured. For example, the first medical imaging device 120 and the second medical imaging device 125 may capture first and second medical image data from various regions at various times. The visual depiction of the medical image reconstruction 710 may gradually change the visual appearance of each region over time in accordance with how much time has elapsed since that particular region was imaged.

[0110] The visual appearance indicating the degree of elapsed time may include, for example, a reduction in opacity, contrast, saturation, and / or luminance, and the magnitude of the reduction increases as the elapsed time increases. For example, in FIG. 9, the most recent region of the reconstruction is indicated by one or more of different colors or saturations compared to regions that were not imaged most recently. In another example, the visual appearance may include one or more of a reduction in sharpness or resolution, and the magnitude of the reduction increases as the elapsed time increases. In an exemplary embodiment, the magnitude by which the resolution or sharpness is reduced is configured to correlate with the amount by which the reliability of the effectiveness of imaging of that region is reduced, such that, for example, a high level of reliability may be maintained for the depiction of that region. In another embodiment, the visual appearance may include a color transition, and the magnitude of the transition increases as the degree of elapsed time increases. In another example, the visual appearance may include a blinking effect, which can start when the elapsed time from capture reaches a preset threshold and / or can accelerate / decelerate over time. In another example, the visual appearance may include, for example, a boundary region or selection to visually indicate that one or more of its regions has lower reliability or effectiveness.

[0111] In another example, the visual appearance may include hiding regions for which a period exceeding a preset threshold has elapsed since imaging. In other words, the reconstruction system 155 may limit the reconstruction 710 of the medical image to be based only on data captured within a preset threshold time. In some embodiments, such hiding may be applied only to data captured by the first medical imaging device 120 and the second medical imaging device 125, for example, so that the pre-treatment medical image remains visible.

[0112] In some embodiments, the reconfiguration system 155 may determine that the medical device 115 is placed in an area where the deviation exceeds a preset threshold, such as an area with a low reliability of effectiveness, based on, for example, the determined deviation. In some embodiments, the reconfiguration system 155 may output a notification to the operator 145, such as a visual or auditory message via a display, or a tactile response via an actuator connected to the medical device 115, for example. In some embodiments, the reconfiguration system 155 outputs an instruction or prompt to the operator 145 to guide the medical device 115 to an area with a higher reliability of effectiveness, outputs a further notification when the medical device 115 is in an area with a higher reliability of effectiveness, and may perform one or more of the following, such as targeting an area closer to the medical device 115 in one or more of the first medical imaging device 120 or the second medical imaging device 125.

[0113] It should be understood that the embodiments in the present disclosure are merely exemplary, and other embodiments may include various combinations of features from other embodiments, as well as additional or fewer features. For example, some of the above embodiments relate to ablation of tissue within the periphery of the lung, but any suitable treatment may be used. Further, some of the above embodiments relate to ultrasound, but any suitable imaging modality or technique may be used.

[0114] Generally, any process that is understood to be computer-implementable, such as the process shown in FIG. 4, among others considered in the present disclosure, can be performed by one or more processors of a computer system, such as any of the systems or apparatuses within the environment of FIG. 1 as described above. A process or process step performed by one or more processors may also be referred to as an operation. One or more processors may be configured to perform such a process by having access to instructions (e.g., software or computer-readable code) that cause the one or more processors to perform the process when executed by the one or more processors. The instructions may be stored in the memory of the computer system. The processor may be a central processing unit (CPU), a graphics processing unit (GPU), or any suitable type of processing unit.

[0115] A computer system, such as a system or apparatus implementing the process or operation in the above example, may include one or more computing devices, such as one or more of the systems or apparatuses in FIG. 1. One or more processors of the computer system may be included within a single computing device or may be distributed among multiple computing devices. One memory of the computer system may include the respective memories of each of the multiple computing devices.

[0116] FIG. 12 is a simplified functional block diagram of a computer 1200 that can be configured as an apparatus for executing the method of FIG. 4 according to an exemplary embodiment of the present disclosure. For example, computer 1200 can be configured as a reconfiguration system 155 and / or as another system according to an exemplary embodiment of the present disclosure. In various embodiments, any of the systems herein can be, for example, a computer 1200 that includes a data communication interface 1220 for packet data communication. Computer 1200 can also include a processor 1202 in the form of one or more processors or central processing units (“CPUs”) for executing program instructions. Computer 1200 can include an internal communication bus 1208 and a storage unit 1206 (such as ROM, HDD, SDD, etc.) that can store data on a computer-readable medium 1222, although computer 1200 can receive programming and data via network communication. Computer 1200 can also have a memory 1204 (such as RAM, etc.) that stores instructions 1224 for executing the techniques presented herein, although instructions 1224 may be stored temporarily or permanently within other modules of computer 1200 (such as processor 1202 and / or computer-readable medium 1222). Computer 1200 can also include an input / output port 1212 and / or a display 1210 for connecting to input / output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Various system functions may be implemented in a distributed manner on several similar platforms to distribute the processing load. Alternatively, the system may be implemented by appropriate programming of a single computer hardware platform.

[0117] The various program aspects of the present technology can typically be considered as a "product" or "manufactured article" in the form of executable code and / or associated data carried on or embodied in a type of machine-readable medium. Media of the "memory area" type include any or all of various semiconductor memories, tape drives, disk drives, etc., tangible memories such as computers, processors that can provide a non-transitory memory area for software programming at any time, or their associated modules. All or some parts of the software may sometimes be communicated via the Internet or various other electrical communication networks. Such communication can enable the loading of software, for example, from a management server or host computer of a mobile communication network to a server's computer platform, and / or from the server to a mobile device, etc., i.e., from one computer or processor to another computer or processor. Thus, another type of medium that can carry software elements includes light, electrical, and electromagnetic waves as used over physical interfaces between local devices, through wired and optical terrestrial communication line networks, and also via various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be regarded as media that carry software. As used herein, unless limited to non-transitory tangible "memory" media, terms such as computer or machine "readable media" also mean any medium involved in providing instructions to a processor for execution.

[0118] The methods, apparatuses, and systems of the present disclosure are illustratively described with reference to sending data, but it should be understood that the embodiments of the present disclosure can be applicable to any environment, such as a desktop or laptop computer, an automotive entertainment system, a home entertainment system, etc. Also, the embodiments of the present disclosure can be applicable to any type of Internet protocol.

[0119] In the foregoing description of exemplary embodiments of the present invention, for the purpose of simplifying the present disclosure and facilitating the understanding of one or more of the various aspects of the invention, it should be understood that the various features of the present invention may be grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, aspects of the invention lie in less than all of the features of a single foregoing disclosed embodiment. Accordingly, the claims following "DETAILED DESCRIPTION OF THE INVENTION" are hereby expressly incorporated by this reference into this "DETAILED DESCRIPTION OF THE INVENTION", and each claim stands on its own as a separate embodiment of the present invention.

[0120] Furthermore, some of the embodiments described herein include some features and not other features included in other embodiments, but as will be understood by those skilled in the art, combinations of features of different embodiments are intended to be within the scope of the present invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0121] Accordingly, while particular embodiments have been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the present disclosure, and all such changes and modifications are intended to be claimed as being within the scope of the present disclosure. For example, functions may be added to or deleted from a block diagram, and operations may be exchanged between functional blocks. Steps may be added to or deleted from the methods described within the scope of the present invention.

[0122] The subject matter disclosed above should be regarded as illustrative, not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other implementations that fall within the true spirit and scope of this disclosure. Accordingly, to the maximum extent permitted by law, the scope of this disclosure should be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be limited or restricted by the foregoing detailed description. Although various implementations of this disclosure have been described, it will be apparent to those skilled in the art that many more implementations are possible within the scope of this disclosure. Accordingly, this disclosure should not be limited except as taking into account the appended claims and their equivalents.

Claims

1. A computer-implemented method for medical image reconstruction of the anatomical structure of a subject, The reconstruction system includes monitoring a first medical image data of the anatomical structure of the subject and a second medical image data of the anatomical structure of the subject, and in monitoring, The first medical image data is captured from within the anatomical structure of the subject by at least one first medical imaging device. The second medical image data is captured from outside the anatomical structure of the subject by at least one second medical imaging device. At least one of the first medical imaging device or the second medical imaging device is capable of operating to selectively target different regions of the anatomical structure of the subject, In response to the reconstruction system detecting, based on the monitoring, that the discrepancy between the first medical image data and the second medical image data exceeds a preset threshold, it updates the reconstruction of the medical image of the subject's anatomical structure based on the first and second medical image data. A computer-implemented method.

2. Updating the reconstruction of the medical images of the anatomical structure of the subject, The reconstruction system causes at least one of the first or second medical imaging devices to acquire first medical image data or second medical image data from at least one region of the anatomical structure of the subject, thereby reducing the misalignment. The reconstruction system modifies the reconstruction of the medical image of the anatomical structure of the subject based on the acquired first medical image data or second medical image data. The computer-implemented method according to claim 1, including the method described in claim 1.

3. The computer-implemented method according to claim 2, wherein causing at least one of the first medical imaging device or the second medical imaging device to reacquire the first medical image data or the second medical image data, respectively, includes the reconstruction system operating an actuator configured to reposition or reorient at least one of the first medical imaging device or the second medical imaging device so that at least one of the first medical imaging device or the second medical imaging device targets at least one region of the anatomical structure of the subject.

4. The at least one region of the anatomical structure of the subject includes multiple regions, The actuator is operated according to one or more of the following random or pseudo-random patterns, linear patterns, inclination patterns, rotation patterns, or obstacle avoidance protocols, based on signals from a force transducer associated with the actuator, so that the first or second medical image data is reacquired for the plurality of regions. The computer-implemented method according to claim 3.

5. At least one of the first medical imaging apparatus or the second medical imaging apparatus includes a plurality of sensors or sensor arrays that are operable to selectively target different regions of the anatomical structure of the subject, each of the plurality of sensors or sensor arrays is positioned or oriented to target one or more different regions of the anatomical structure of the subject. The reconstruction system includes changing the target settings of the plurality of sensors or sensor arrays, such that at least one of the first medical imaging device and the second medical imaging device reacquires the first medical image data or the second medical image data, respectively. The computer-implemented method according to any one of claims 2 to 4.

6. The discrepancy between the first medical image data and the second medical image data is, Between the first region of the anatomical structure of the subject targeted by the first medical imaging device and the second region of the anatomical structure of the subject targeted by the second medical imaging device, Between the first position of the first medical imaging device and the second position of the second medical imaging device, A computer-implemented method according to any one of claims 1 to 4, based on at least one of the distances.

7. The computer-implemented method according to claim 6, wherein the discrepancy between the first medical image data and the second medical image data is based on a moving average of the distance.

8. The reconstruction system determines one or more of the first position of the first medical imaging device or the first region of the anatomical structure of the subject targeted by the first medical imaging device, based on one or more of the following: electromagnetic position tracking, optical fiber shape detection, position tracking by the second medical imaging device, or impedance-based tracking. The reconstruction system further includes determining one or more of the second position of the second medical imaging device or the first region of the anatomical structure of the subject targeted by the second medical imaging device, based on one or more of the electromagnetic position tracking, optical tracking, or internal position tracking of the second medical imaging device, The computer-implemented method according to claim 6.

9. The computer-implemented method according to any one of claims 1 to 4, wherein the discrepancy between the first medical image data and the second medical image data is based on (a) the elapsed time since one or more of the first medical image data were captured by the first medical imaging device, or (b) the elapsed time since the second medical image data was captured by the second medical imaging device.

10. The reconstruction system reconstructs a medical image of the anatomical structure of the subject based on the first medical image data and the second medical image data, The reconstruction system further comprises outputting a visual representation of at least a portion of the reconstruction of the medical image to a display. A computer-implemented method according to any one of claims 1 to 4.

11. The computer-implemented method according to claim 10, wherein the reconstruction system updates the reconstruction of the medical image of the anatomical structure of the subject, which includes modifying the visual depiction of at least a portion of the reconstruction of the medical image such that at least a portion of the reconstruction of the medical image has a visual appearance indicating the time elapsed since the medical image data on which the at least portion is based was captured.

12. The visual appearance indicating the elapsed time since the medical image data on which at least a portion of the above was based was captured, A reduction in opacity, contrast, saturation, sharpness, resolution, brightness, or a combination thereof, wherein the magnitude of the reduction increases with increasing elapsed time. A color transition, wherein the magnitude of the transition increases with increasing elapsed time. Flashing effect, or, A boundary region or selection representing at least a portion of the reconstruction of the medical image, Including one or more of the following: The computer-implemented method according to claim 11.

13. The reconstruction system obtains a reconstruction of a previous medical image of the anatomical structure of the subject based on a third medical image data captured by a third medical imaging device, The reconstruction system determines the registration of the reconstruction of the medical image with respect to the previous reconstruction of the medical image, The reconstruction system causes the display to output a visual representation of the previous medical image reconstruction based on the determined registration, along with at least a portion of the reconstruction of the medical image. The computer-implemented method according to claim 10, further comprising:

14. The reconstruction system receives a user selection regarding the relative visibility of the reconstruction of the previous medical image in the visual representation and the reconstruction of the medical image, The reconstruction system selectively increases or decreases the visibility of the reconstruction of the previous medical image in the visual depiction and the visibility of at least a portion of the reconstruction of the medical image, based on the user selection. The computer-implemented method according to claim 13, further comprising:

15. The at least one first medical imaging device includes one or more of the following: an ultrasonic transducer, a parallel array or phased array transducer, an optical coherence tomography device, or an optical sensor. The at least one second medical imaging device includes one or more of the following: an ultrasonic transducer, an array of ultrasonic transducers, an X-ray device, or a computed tomography device. A computer-implemented method according to any one of claims 1 to 4.

Images