System and method for volume reconstruction using priori patient data

Abstract

A system according to at least one embodiment of the present disclosure, the system comprising: a processor; and a memory having stored thereon instructions that, when executed by the processor, cause the processor to: identify a boundary region corresponding to a shape of an object based on first imaging data associated with the object; identifying at least one voxel included in second imaging data associated with the object, wherein the at least one voxel is located outside the boundary region; and generating a multi-dimensional image volume corresponding to the object using the second imaging data, wherein generating the multi-dimensional image volume is relative to one or more criteria associated with voxels located outside the boundary region.

Description

Background Art

[0001] The present disclosure generally relates to surgical data models and, more particularly, to volume reconstruction processes.

[0002] Imaging devices can be used in the context of a surgical procedure or surgical protocol. The imaging device can include a transmitter and a detector to capture an image of an object placed therebetween. The captured image can assist a surgeon in performing the surgical procedure or surgical protocol better. Summary of the Invention

[0003] Example aspects of the present disclosure include:

[0004] A system according to at least one embodiment of the present disclosure includes: a processor; and a memory storing instructions that, when executed by the processor, cause the processor to: identify a boundary region corresponding to the shape of an object based on first imaging data associated with the object; identify at least one voxel included in second imaging data associated with the object, wherein the at least one voxel is located outside the boundary region; and generate a multi-dimensional image volume corresponding to the object using the second imaging data, wherein generating the multi-dimensional image volume is relative to one or more criteria associated with voxels located outside the boundary region.

[0005] Any of the features herein, wherein the instructions are further executable by the processor to: regenerate the first imaging data associated with the object based on failure to meet the one or more criteria; identify a second boundary region corresponding to the shape of the object based on the regenerated first imaging data associated with the object; identify at least one second voxel included in third imaging data, wherein the at least one second voxel is located outside the second boundary region; and generate a second multi-dimensional image volume corresponding to the object using the third imaging data, wherein generating the second multi-dimensional image volume includes meeting one or more criteria associated with voxels located outside the second boundary region.

[0006] Any of the features herein, wherein the one or more criteria include a threshold attenuation value associated with voxels located outside the boundary region.

[0007] Any of the features herein, wherein the one or more criteria include a target ratio of the attenuation of voxels located outside the boundary region to the attenuation of voxels located inside the boundary region.

[0008] Any of the features herein, wherein the instructions are further executable by the processor to: identify second voxels located inside the boundary region based on the second imaging data associated with the object, wherein the one or more criteria include meeting a threshold difference between a first attenuation value of the at least one voxel and a second attenuation value of the second voxels.

[0009] Any one of the features of the present disclosure, wherein the first imaging data includes one or more panoramic sagittal images of the object.

[0010] Any one of the features of the present disclosure, wherein: the first imaging data includes one or more images of the object; an upper portion of the one or more images includes a first air margin corresponding to a boundary of the object; and a lower portion of the one or more images includes a second air margin corresponding to another boundary of the object.

[0011] Any one of the features of the present disclosure, wherein the instructions can be further executed by the processor to: capture a set of points associated with the object using one or more light-based ranging operations; and generate the first imaging data associated with the object based on the set of points.

[0012] Any one of the features of the present disclosure, wherein the first imaging data includes one or more x-ray images, one or more optical images, one or more depth images, one or more magnetic resonance imaging (MRI) images, one or more computed tomography (CT) images, or a combination thereof.

[0013] Any one of the features of the present disclosure, wherein the instructions can be further executed by the processor to capture the first imaging data associated with the object, wherein capturing the first imaging data includes: capturing a first image of the object, wherein capturing the first image is associated with first pose information of the imaging device relative to the object; and capturing a second image of the object, wherein capturing the second image is associated with second pose information of the imaging device relative to the object, and wherein identifying the boundary region corresponding to the shape of the object is based on the first image, the first pose information, the second image, and the second pose information.

[0014] Any one of the features of the present disclosure, wherein the size of the boundary region corresponds to at least one of: a size of the object in a first direction relative to a plane; and a second size of the object in a second direction relative to the plane, wherein the second direction is orthogonal to the first direction.

[0015] Any one of the features of the present disclosure, wherein the instructions can be further executed by the processor to: dynamically capture the second imaging data using one or more imaging devices.

[0016] A system according to at least one embodiment of the present disclosure includes: one or more imaging devices; a processor; and a memory that stores data thereon, the data causing the processor, when processed by the processor, to: identify a boundary region corresponding to the shape of an object based on first imaging data generated using the one or more imaging devices; identify at least one voxel included in second imaging data generated using the one or more imaging devices, wherein the at least one voxel is located outside the boundary region; and generate a multi-dimensional image volume corresponding to the object using the second imaging data, wherein generating the multi-dimensional image volume is relative to one or more criteria associated with voxels located outside the boundary region.

[0017] Any of the features herein, wherein the data can further be executed by the processor to: regenerate the first imaging data in response to failure to meet the one or more criteria; identify a second boundary region corresponding to the shape of the object based on the regenerated first imaging data; identify at least one second voxel included in third imaging data, wherein the at least one second voxel is located outside the second boundary region; and generate a second multi-dimensional image volume corresponding to the object using the third imaging data, wherein generating the second multi-dimensional image volume includes meeting one or more criteria associated with voxels located outside the second boundary region.

[0018] Any of the features herein, wherein the one or more criteria include a threshold attenuation value associated with voxels located outside the boundary region.

[0019] Any of the features herein, wherein the one or more criteria include a target ratio of the attenuation of voxels located outside the boundary region to the attenuation of voxels located inside the boundary region.

[0020] Any of the features herein, wherein the data can further be executed by the processor to: identify second voxels located inside the boundary region based on the second imaging data associated with the object, wherein the one or more criteria include a threshold difference between a first attenuation value of the at least one voxel and a second attenuation value of the second voxels.

[0021] Any of the features herein, wherein the first imaging data includes one or more panoramic sagittal images of the object.

[0022] Any of the features herein, wherein: the first imaging data includes one or more images including the object; an upper portion of the one or more images includes a first air edge corresponding to a boundary of the object; and a lower portion of the one or more images includes a second air edge corresponding to another boundary of the object.

[0023] A method according to at least one embodiment of the present disclosure includes: identifying a boundary region corresponding to the shape of an object based on first imaging data associated with the object; identifying at least one voxel included in second imaging data associated with the object, wherein the at least one voxel is located outside the boundary region; and generating a volume construct corresponding to the object using the second imaging data, wherein generating the volume construct is with respect to meeting one or more criteria associated with the voxel located outside the boundary region.

[0024] Any one aspect in combination with any one or more other aspects.

[0025] Any one or more of the features disclosed herein.

[0026] Any one or more of the features generally disclosed herein.

[0027] Any one or more of the features generally disclosed herein in combination with any one or more other features generally disclosed herein.

[0028] Any one of an aspect / feature / embodiment in combination with any one or more other aspects / features / embodiments.

[0029] Use any one or more of the aspects or features disclosed herein.

[0030] It should be understood that any feature described herein may be claimed in combination with any other feature described herein, regardless of whether the features are from the same described embodiment.

[0031] Details of one or more aspects of the present disclosure are set forth in the following drawings and description. Other features, objects, and advantages of the technology described in the present disclosure will be apparent from the description, drawings, and claims.

[0032] The foregoing is a simplified summary of the present disclosure to provide an understanding of some aspects of the present disclosure. This Summary of the Invention is neither an extensive nor an exhaustive overview of the present disclosure and its various aspects, embodiments, and configurations. It is neither intended to identify key or important elements of the present disclosure nor to delineate the scope of the present disclosure, but rather to present selected concepts of the present disclosure in a simplified form as an introduction to the more detailed description presented below. As should be understood, other aspects, embodiments, and configurations of the present disclosure may utilize one or more of the features set forth above or described in detail below, either alone or in combination.

[0033] Many additional features and advantages of the present disclosure will become apparent to those skilled in the art upon consideration of the following detailed description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are incorporated in and form a part of this specification to illustrate several examples of the present disclosure. These drawings, together with the description, explain the principles of the present disclosure. The drawings merely illustrate how to implement and use the preferred and alternative examples of the present disclosure, and these examples should not be construed as limiting the present disclosure only to the illustrated and described examples. Additional features and advantages will become apparent from the following more detailed description of the various aspects, specific implementations, and configurations of the present disclosure, as illustrated by the accompanying drawings referred to below.

[0035] Figure 1 is a block diagram of a system according to at least one embodiment of the present disclosure;

[0036] Figure 2A depicts a first imaging technique of a patient according to at least one embodiment of the present disclosure;

[0037] Figure 2B is a first image generated using a first imaging technique according to at least one embodiment of the present disclosure;

[0038] Figure 2C depicts a second imaging technique of a patient according to at least one embodiment of the present disclosure;

[0039] Figure 2D is a second image generated using a second imaging technique according to at least one embodiment of the present disclosure;

[0040] Figure 2E depicts a third imaging technique of a patient according to at least one embodiment of the present disclosure;

[0041] Figure 2F depicts a grid according to at least one embodiment of the present disclosure;

[0042] Figure 3A depicts a first set of voxels in a reconstructed image according to at least one embodiment of the present disclosure;

[0043] Figure 3B depicts a second set of voxels in a reconstructed image according to at least one embodiment of the present disclosure;

[0044] Figure 4 is a flowchart according to at least one embodiment of the present disclosure; and

[0045] Figure 5 is a flowchart according to at least one embodiment of the present disclosure. Detailed Description

[0046] It should be understood that the various aspects disclosed herein can be combined in combinations different from those specifically presented in the specification and the drawings. It should also be understood that depending on the example or specific implementation, certain actions or events of any of the processes or methods described herein can be performed in a different order, and / or can be added, combined, or completely omitted (e.g., depending on different specific implementations of the present disclosure, not all of the described actions or events may be required to implement the disclosed technology). Additionally, although some aspects of the present disclosure are described for clarity as being performed by a single module or unit, it should be understood that the technology of the present disclosure can be performed by a combination of units or modules associated with, for example, a computing device and / or a medical device).

[0047] In one or more examples, the methods, processes, and techniques described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Alternatively or additionally, the functions can be implemented using a machine learning model, a neural network, an artificial neural network, or a combination thereof (either alone or in combination with instructions). Alternatively or additionally, the functions can be implemented using a machine learning model, a neural network, an artificial neural network, or a combination thereof (either alone or in combination with instructions). The computer-readable medium can include a non-transitory computer-readable medium, which corresponds to a tangible medium, such as a data storage medium (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0048] The instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple A11, A12, A12X, A12Z, or A13 Bionic processors; or any other general-purpose microprocessor), graphics processing units (e.g., Nvidia GeForce RTX 2000 series processors, Nvidia GeForce RTX 3000 series processors, AMD Radeon RX 5000 series processors, AMD Radeon RX 6000 series processors, or any other graphics processing unit), application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, as used herein, the term "processor" can refer to any one of the foregoing structures or any other physical structure suitable for implementing the described techniques. Additionally, these techniques can be implemented entirely in one or more circuits or logic elements.

[0049] Before explaining any specific embodiments of the present disclosure in detail, it should be understood that the present disclosure is not limited in its application to the construction details and component arrangements set forth in the following description or illustrated in the drawings. The present disclosure is capable of other specific embodiments and can be practiced or carried out in various ways. Additionally, it should be understood that the terminology and terms used herein are for the purpose of description and should not be regarded as limiting. The use of "comprising," "including," or "having" and variations thereof herein is intended to cover the items listed thereafter and their equivalents, as well as additional items. Furthermore, the present disclosure may use examples to illustrate one or more of its aspects. Unless otherwise expressly stated, the use or listing of one or more examples (which may be indicated by "for example," "by way of example," "such as," or similar language) is not intended and does not limit the scope of the present disclosure.

[0050] The terms proximal and distal are used in their conventional medical meanings in the present disclosure, where proximal is closer to the operator or user of the system and farther from the surgical area of concern within or on the patient's body, while distal is closer to the surgical area of concern within or on the patient's body and farther from the operator or user of the system.

[0051] Reconstruction processes, such as X-ray three-dimensional (3D) volume reconstruction or single 2D image reconstruction, can be used to generate an accurate 3D model or 2D model (e.g., 2D slices) of a scanned object or patient. Inaccurate reconstructions can result in incorrect dimensions, incorrect values, or other image deformations. Inaccurate reconstructions can occur due to sub-optimal reconstruction parameters, reconstruction algorithms, or conditions. Inaccurate reconstructions can lead to incorrect diagnoses and prognoses.

[0052] When using an iterative method to adjust a single 2D image or volume reconstruction, one potential problem is determining how to improve the reconstruction output (volume or 2D image) from one iteration to the next. The reconstruction output from an iterative method can include a 3D matrix composed of voxels, where each voxel maps a value to a small portion of 3D data in 3D space (e.g., a “cube”). In the case of a single 2D image, instead of a 3D cube, 2D pixels are used as the small portions of data. In embodiments implementing X-ray imaging, these values can be attenuation values. In some iterative reconstruction methods, such as algebraic reconstruction technique (ART), maximum likelihood expectation maximization (MLEM), or ordered subset expectation maximization (OSEM), voxels with attenuation values can be shifted to different voxels while keeping the sum of all voxels fixed. In other words, the values associated with one or more voxels can be decreased while the values associated with one or more other voxels can be increased such that there is no overall change in the sum of all voxels.

[0053] Another potential problem with volume reconstruction is the use of small detectors, which can result in less information about the peripheral portions of the patient (since the body is too large to be fully seen by the detector). One potential remedy is to use a computed tomography (CT) scanner that includes a large detector or to take many additional images with a small detector. However, such a solution also exposes the patient to an increased dose of radiation since the source and detector must be large enough to capture the peripheral portions of the patient.

[0054] Another potential problem with volume or single 2D image reconstruction is the dissipation of attenuation values from the body contour of the patient to the surrounding air. This can result in reduced contrast in the image, lack of consistency in gray values, and other image artifacts.

[0055] According to at least one embodiment of the present disclosure, a priori body shape estimation of a patient's volume can beneficially improve reconstruction, such as by improving the quality of the reconstruction, the accuracy of the reconstruction, and / or the time required to perform the reconstruction. For example, instead of a random initial guess, the body shape estimation can be used as an initial guess in an algorithm. The use of body shape estimation can provide a more accurate starting point for an algorithm used to optimize the reconstruction. Additionally, using body shape estimation can result in better quality images. For example, by providing boundaries, the algorithm can be constrained by the boundaries to reconstruct the image using the same general aspects of the body shape estimation, resulting in a more accurate reconstructed image. The a priori body shape estimation will help keep the air volume around the patient from being attenuated while limiting the attenuation to the patient (or more generally, to the object itself).

[0056] According to at least one embodiment of the present disclosure, different methods and information sources are used to achieve a priori body shape estimation. For example, information associated with previous CT scans, magnetic resonance imaging (MRI) scans, etc. can be retrieved. As another example, one or more devices (e.g., an optical camera, a depth camera, a light detection and ranging (LIDAR) camera, or any other imaging modality or measurement device) can be used to capture information for generating a volume grid around the patient. As yet another example, a priori body shape information can be generated based on a planned scan (e.g., a scan of the patient during a preoperative planning phase), which can be used to find the boundaries of the patient's body (e.g., by estimating the width and height of the patient). The body shape estimation can be integrated into the reconstruction algorithm in the form of conditions that limit the voxel values outside the body. For example, the conditions on the voxel values can be such that voxels located outside the boundary cannot have values that are not attributable to air (or more generally, values not attributable to the medium around the object). Such conditions on the voxels outside the boundary can beneficially enable the algorithm to better shift the attenuation values between voxels while iteratively reconstructing the volume by, for example, capping the values of the voxels located outside the boundary, resulting in an improvement in the quality of the final image. The improved final image can also beneficially improve the procedure planning, reduce the patient's exposure to potentially dangerous radiation (e.g., excessive X-ray imaging), and improve the clinical outcome.

[0057] Specific embodiments of the present disclosure provide a technical solution to one or more of the following problems: (1) generating an incorrect or inaccurate reconstruction, (2) attenuation in the reconstructed image leaking into the surrounding non-patient areas, and (3) incorrect or inaccurate gray value shifting during iterative volume reconstruction.

[0058] Figure 1 An example of a system 100 that supports aspects of the present disclosure is illustrated.

[0059] System 100 includes computing device 102, one or more imaging devices 112, robot 114, navigation system 118, database 130, and / or cloud network 134 (or other network). Systems according to other specific implementations of the present disclosure may include more or fewer components than system 100. For example, system 100 may omit and / or include additional instances of one or more components of computing device 102, imaging device 112, robot 114, navigation system 118, database 130, and / or cloud network 134. For example, system 100 may omit robot 114 and navigation system 118. System 100 may support the implementation of one or more other aspects of one or more of the methods disclosed herein.

[0060] Computing device 102 includes processor 104, memory 106, communication interface 108, and user interface 110. Computing devices according to other specific implementations of the present disclosure may include more or fewer components than computing device 102.

[0061] Processor 104 of computing device 102 may be any processor described herein or any similar processor. Processor 104 may be configured to execute instructions stored in memory 106 that may cause processor 104 to perform one or more computational steps using or based on data received from imaging device 112, robot 114, navigation system 118, database 130, and / or cloud network 134.

[0062] Memory 106 may be or include RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible non-transitory memory for storing computer-readable data and / or instructions. Memory 106 may store information or data associated with completing any steps of, for example, methods 400 and 500 described herein or any other method. Memory 106 may store instructions and / or machine learning models that support, for example, one or more functions of imaging device 112, robot 114, and navigation system 118. For example, memory 106 may store content (e.g., instructions and / or machine learning models) that, when executed by processor 104, enables image processing 120, segmentation 122, transformation 124, registration 128, and / or filtering 136. In some specific implementations, if provided as instructions, such content may be organized into one or more applications, modules, packages, layers, or engines.

[0063] Alternatively or additionally, the memory 106 may store other types of content or data (e.g., machine learning models, artificial neural networks, deep neural networks, etc.) that can be processed by the processor 104 to perform the various methods and features described herein. Thus, although the various contents of the memory 106 may be described as instructions, it should be understood that the functions described herein may be implemented by using instructions, algorithms, and / or machine learning models. The data, algorithms, and / or instructions may cause the processor 104 to manipulate the data stored in the memory 106 and / or received from or via the imaging device 112, the robot 114, the navigation system 118, the database 130, and / or the cloud network 134.

[0064] The computing device 102 may also include a communication interface 108. The communication interface 108 can be used to receive data or other information from external sources (e.g., the imaging device 112, the robot 114, the navigation system 118, the database 130, the cloud network 134, and / or any other system or component separate from the system 100), and / or to send instructions, data (e.g., image data, measurement results, etc.), or other information to external systems or devices (e.g., another computing device 102, the imaging device 112, the robot 114, the navigation system 118, the database 130, the cloud network 134, and / or any other system or component that is not part of the system 100). The communication interface 108 may include one or more wired interfaces (e.g., USB ports, Ethernet ports, FireWire ports) and / or one or more wireless transceivers or interfaces (configured to send and / or receive information via, for example, one or more wireless communication protocols such as 802.11a / b / g / n, Bluetooth, NFC, ZigBee, etc.). In some specific implementations, the communication interface 108 may support communication between the device 102 and one or more other processors 104 or computing devices 102, either to reduce the time required to complete computationally intensive tasks or for any other reason.

[0065] The computing device 102 may also include one or more user interfaces 110. The user interface 110 may be or include a keyboard, a mouse, a trackball, a monitor, a television, a screen, a touch screen, and / or any other device for receiving information from a user and / or for providing information to a user. The user interface 110 may be used, for example, to receive user selections or other user inputs regarding any step of any method described herein. Nevertheless, any required input for any step of any method described herein may be automatically generated by the system 100 (e.g., by the processor 104 or another component of the system 100) or received by the system 100 from a source external to the system 100. In some specific embodiments, the user interface 110 may support user modification (e.g., by a surgeon, medical staff, patient, etc.) of instructions to be executed by the processor 104 in accordance with one or more specific embodiments of the present disclosure, and / or support user modification or adjustment of settings of other information displayed on or corresponding to the user interface 110.

[0066] In some specific embodiments, the computing device 102 may utilize a user interface 110 that is separately housed from one or more remaining components of the computing device 102. In some specific embodiments, the user interface 110 may be located near one or more other components of the computing device 102, while in other specific embodiments, the user interface 110 may be located remotely from one or more other components of the computing device 102.

[0067] The imaging device 112 can be used to image anatomical features (e.g., bones, veins, tissues, etc.) and / or other aspects of a patient's anatomy to generate image data (e.g., image data depicting or corresponding to bones, veins, tissues, etc.). As used herein, "image data" or "imaging data" refers to data generated or captured by the imaging device 112, including data in machine-readable form, graphical / visual form, and in any other form. In various examples, the image data can include data corresponding to an anatomical feature portion of a patient or a part thereof. The image data can be or include preoperative images, intraoperative images, postoperative images, or images taken independent of any surgical procedure. In some embodiments, the first imaging device 112 can be used to obtain first image data (e.g., a first image) at a first time, and the second imaging device 112 can be used to obtain second image data (e.g., a second image) at a second time after the first time. The imaging device 112 may be capable of taking 2D images or 3D images to generate image data. The imaging device 112 can be or include, for example, an ultrasound scanner (which can include, for example, physically separate transducers and receivers, or a single ultrasound transceiver), an O-arm, a C-arm, a G-arm, or any other device that uses X-ray-based imaging (e.g., a fluoroscope, a CT scanner, or other X-ray machine), a magnetic resonance imaging (MRI) scanner, an optical coherence tomography (OCT) scanner, an endoscope, a microscope, an optical camera, a thermal imaging camera (e.g., an infrared camera), a radar system (which can include, for example, a transmitter, a receiver, a processor, and one or more antennas), or any other imaging device 112 suitable for obtaining an image of an anatomical feature portion of a patient. The imaging device 112 can be fully contained within a single housing, or can include a transmitter / transmitter and a receiver / detector that are in separate housings or otherwise physically separated.

[0068] In some embodiments, the imaging device 112 can include more than one imaging device 112. For example, a first imaging device can provide first image data and / or a first image, and a second imaging device can provide second image data and / or a second image. In still other embodiments, the same imaging device can be used to provide both first image data and second image data and / or any other image data described herein. The imaging device 112 can be used to generate an image data stream. For example, the imaging device 112 can be configured to operate using an open shutter, or using a shutter that continuously alternates between open and closed, in order to capture consecutive images. For the purposes of this disclosure, unless otherwise specified, image data can be considered continuous and / or provided as an image data stream if the image data represents two or more frames per second.

[0069] The robot 114 can be any surgical robot or surgical robot system. The robot 114 can be or include, for example, a Mazor X TMStealth version robot guidance system. The robot 114 can be configured to position the imaging device 112 at one or more precise positions and orientations, and / or return the imaging device 112 to the same position and orientation at a later time point. The robot 114 can additionally or alternatively be configured to manipulate a surgical tool (whether or not based on guidance from the navigation system 118) to complete or assist with a surgical task. In some specific implementations, the robot 114 can be configured to hold and / or manipulate anatomical elements during or in conjunction with a surgical procedure. The robot 114 can include one or more robotic arms 116. In some specific implementations, the robotic arm 116 can include a first robotic arm and a second robotic arm, but the robot 114 can include more than two robotic arms. In some specific implementations, one or more of the robotic arms 116 can be used to hold and / or manipulate the imaging device 112. In specific implementations where the imaging device 112 includes two or more physically separate components (e.g., a transmitter and a receiver), one robotic arm 116 can hold one such component, and another robotic arm 116 can hold another such component. Each robotic arm 116 can be positioned independently of the other robotic arms. The robotic arms 116 can be controlled in a single shared coordinate space or in separate coordinate spaces.

[0070] The robot 114 together with the robotic arms 116 can have, for example, one, two, three, four, five, six, seven or more degrees of freedom. Additionally, the robotic arms 116 can be positioned or locatable in any pose, plane and / or focus. This pose includes position and orientation. Thus, the imaging device 112, surgical tool or other object held by the robot 114 (or more specifically, by the robotic arms 116) can be precisely positioned at one or more desired and specific positions and orientations.

[0071] The robotic arm 116 can include one or more sensors that enable the processor 104 (or the processor of the robot 114) to determine the precise pose of the robotic arm (and any object or element held or fixed to the robotic arm) in space.

[0072] In some specific implementations, reference markers (i.e., navigation markers) may be placed on the robot 114 (including, for example, on the robot arm 116), the imaging device 112, or any other object in the surgical space. The reference markers may be tracked by the navigation system 118, and the results of the tracking may be used by the robot 114 and / or by an operator of the system 100 or any of its components. In some specific implementations, the navigation system 118 may be used to track other components of the system (e.g., the imaging device 112), and the system may operate without using the robot 114 (e.g., a surgeon may manually manipulate the imaging device 112 and / or one or more surgical tools based on information and / or instructions generated by the navigation system 118).

[0073] During operation, the navigation system 118 may provide navigation for the surgeon and / or the surgical robot. The navigation system 118 may be any known or future-developed navigation system, including, for example, the Medtronic StealthStation TM S8 surgical navigation system or any of its successors. The navigation system 118 may include one or more cameras or other sensors for tracking one or more reference markers, navigation trackers, or other objects within the operating room or other room in which part or all of the system 100 is located. The one or more cameras may be optical cameras, infrared cameras, or other cameras. In some specific implementations, the navigation system 118 may include one or more electromagnetic sensors. In various specific implementations, the navigation system 118 may be used to track the position and orientation (e.g., pose) of the imaging device 112, the robot 114, and / or the robot arm 116 and / or one or more surgical tools (or more specifically, to track the pose of a navigation tracker directly or indirectly attached to one or more of the foregoing in a fixed relationship). The navigation system 118 may include a display for displaying one or more images from an external source (e.g., the computing device 102, the imaging device 112, or other source) or for displaying images and / or video streams from one or more cameras or other sensors of the navigation system 118. In some specific implementations, the system 100 may operate without using the navigation system 118. The navigation system 118 may be configured to provide guidance to the surgeon or other users of the system 100 or its components, to the robot 114, or to any other element of the system 100 regarding, for example, the pose of one or more anatomical elements, whether a tool is on the proper trajectory, and / or how to move the tool to the proper trajectory to perform a surgical task according to a preoperative or other surgical plan.

[0074] The processor 104 may utilize data stored in the memory 106 as a neural network. The neural network may include a machine learning architecture. In some aspects, the neural network may be or include one or more classifiers. In some other aspects, the neural network may be or include any machine learning network, such as, for example, a deep learning network, a convolutional neural network, a reconstruction neural network, a generative adversarial neural network, or any other neural network capable of implementing the functions of the computing device 102 described herein. Some elements stored in the memory 106 may be described as or referred to as instructions or instruction sets, and some functions of the computing device 102 may be implemented using machine learning techniques.

[0075] For example, the processor 104 may support a machine learning model in the form of a reconstruction 136 that may be trained and / or updated based on training data provided or accessed by any one of the computing device 102, the imaging device 112, the robot 114, the navigation system 118, the database 130, and / or the cloud network 134.

[0076] In some embodiments, one or more training sets included in the training data 146 may be utilized to train the reconstruction. In some aspects, the training data 146 may include multiple training sets. In an example, the training data 146 may include a first training set that includes depth data and / or motion data associated with one or more medical conditions described herein. In an example, the depth data and / or motion data included in the training set may indicate a change in breathing or body movement that indicates one or more medical conditions. In some aspects, the depth data and / or motion data included in the training set may be associated with a confirmed instance of one or more medical conditions (e.g., by a healthcare provider, a patient, etc.).

[0077] In other embodiments, the reconstruction 136 may provide one or more algorithms, such as ART, MLEM, OSEM, filtered backprojection (FBP), combinations thereof, etc., for reconstructing a multi-dimensional image representing the patient (e.g., a 3D image representing the volume of the patient). In such embodiments, and as further discussed below, the algorithms may utilize prior patient body shape estimation data (e.g., based on previous scans retrieved from the database 130, based on boundaries determined from a preliminary scan, based on a mesh, etc.) to provide criteria for the reconstruction algorithms. In some embodiments, the criteria may be used as a constraint that is used to influence the optimization of the multi-dimensional image by the algorithm. For example, based on the patient body shape estimation, the algorithm may evaluate and optimize voxels classified by the algorithm as being outside the patient (e.g., voxels associated with the air surrounding the patient) differently from voxels classified by the algorithm as being inside the patient (e.g., voxels associated with patient tissue). In some embodiments, the reconstruction 136 may receive multiple different types of image data and use these image types as patient body shape estimates when generating the multi-dimensional image. For example, the reconstruction 136 may receive LIDAR data and a CT scan and may convert the LIDAR image data and the CT scan into a format that can be processed by the reconstruction 136 (e.g., 3D coordinates, 3D mesh, etc.). In some embodiments, the image data types may be converted into a common imaging plane (e.g., using the registration 128) before being passed to the reconstruction 136.

[0078] The database 130 may store information that associates one coordinate system to another (e.g., associates one or more robotic coordinate systems to the patient coordinate system and / or the navigation coordinate system). The database 130 may additionally or alternatively store, for example, one or more surgical plans (including, for example, pose information about the target and / or image information about the anatomy of the patient at and / or near the surgical site, for use by the robot 114, the navigation system 118, and / or the user of the computing device 102 or the system 100); one or more images that may be used in conjunction with a surgical procedure performed by or with the assistance of one or more other components of the system 100; and / or any other useful information. In some embodiments, the database 130 may include patient information that may be used to estimate the patient body shape. For example, the database 130 may store patient data (e.g., patient height, patient weight, etc.) and / or imaging data associated with the patient (e.g., previous scans or images of the patient), and the patient data and / or the imaging data associated with the patient may be used to estimate the patient body shape.

[0079] The database 130 can be configured to provide any such information to the computing device 102 or any other device within the system 100 or any other device external to the system 100, either directly or via the cloud network 134. In some embodiments, the database 130 can be or include a part of a hospital image storage system, such as a Picture Archiving and Communication System (PACS), a Health Information System (HIS), and / or another system for collecting, storing, managing, and / or transmitting electronic medical records including image data.

[0080] In some aspects, the computing device 102 can communicate directly or indirectly with a server and / or a database (e.g., the database 130) via a communication network (e.g., the cloud network 134). The communication network can include any type of known communication medium or a collection of communication media, and can use any type of protocol to transmit data between endpoints. The communication network can include wired communication technologies, wireless communication technologies, or any combination thereof.

[0081] Wired communication technologies can include, for example, Ethernet-based wired local area network (LAN) connections using physical transmission media (e.g., coaxial cables, copper cables / wires, fiber optic cables, etc.). Wireless communication technologies can include, for example, cellular or cellular data connections and protocols (e.g., digital cellular, Personal Communication Service (PCS), Cellular Digital Packet Data (CDPD), General Packet Radio Service (GPRS), Enhanced Data Rates for GSM Evolution (EDGE), Code Division Multiple Access (CDMA), Single Carrier Radio Transmission Technology (1×RTT), Evolution-Data Optimized (EVDO), High-Speed Packet Access (HSPA), Universal Mobile Telecommunications Service (UMTS), 3G, Long-Term Evolution (LTE), 4G, and / or 5G, etc.), low-power, Wi-Fi, radio, satellite, infrared connections, and / or communication protocols.

[0082] The Internet is an example of a communication network that forms an Internet Protocol (IP) network, which is composed of multiple computers, computing networks, and other communication devices located at multiple locations, and components in the communication network (e.g., computers, computing networks, communication devices) can be connected through one or more telephone systems and other means. Other examples of communication networks may include, but are not limited to, standard Plain Old Telephone System (POTS), Integrated Services Digital Network (ISDN), Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Wireless LAN (WLAN), Session Initiation Protocol (SIP) network, Voice over Internet Protocol (VoIP) network, cellular network, and any other type of packet-switched or circuit-switched network known in the art. In some cases, the communication network may include any combination of networks or network types. In some aspects, the communication network may include any combination of communication media, such as coaxial cables, copper cables / wires, fiber optic cables, or antennas for conveying data (e.g., sending / receiving data).

[0083] The computing device 102 can be connected to the cloud network 134 via the communication interface 108 using a wired connection, a wireless connection, or both. In some specific implementations, the computing device 102 can communicate with the database 130 and / or external devices (e.g., computing devices) via the cloud network 134.

[0084] The system 100 or a similar system can be used, for example, to perform one or more aspects of the methods 400 and 500 described herein. The system 100 or a similar system can also be used for other purposes.

[0085] In some embodiments, the system 100 can be used to generate a patient body shape estimate to improve the quality of the patient's reconstructed volume model. The reconstructed volume model (also referred to herein as a reconstructed image, a reconstructed multi-dimensional image, or a reconstructed volume) can be a 2D or 3D model that is at least partially based on the captured images (e.g., images generated by the imaging device 112), and can provide information associated with the volume occupied by the patient, the pose of the patient (e.g., position and / or orientation), their combination, etc. The reconstructed volume model can be generated by the computing device 102 using the captured images. The reconstructed volume model can model the entire patient or, in some cases, a part of the patient (e.g., one or more vertebrae of the patient, other anatomical elements or tissues of the patient, etc.). In some embodiments, the reconstructed volume model can be a 2D model (e.g., a region) and / or can depict 2D slices of the overall reconstructed volume model. The computing device 102 can access one or more algorithms or data models (e.g., ART, FBP, trained data models, etc.) from the memory 106 when reconstructing a multi-dimensional image based on information associated with the captured images of the patient.

[0086] In some specific implementations, prior data related to the estimation of the patient's body size, shape, volume, and / or boundaries associated with the patient can be passed into an algorithm or data model to provide one or more criteria (e.g., constraints) that the reconstruction must satisfy. For example, the boundaries associated with the patient's body can be passed into the algorithm or data model, and the criterion for reconstruction can be that voxels located outside the patient's boundaries cannot increase their attenuation values. As another example, voxels located outside the boundaries cannot have an attenuation value higher than a threshold. The patient body size estimation can be generated based on prior images of the patient (e.g., previously captured CT scans, MRI scans, etc.), a mesh based on the patient's anatomy, and / or images captured during a pre-operative scanning protocol, as further discussed with reference to Figures 2A to 2E which is further discussed.

[0087] Turning first to Figures 2A to 2B , aspects of a first imaging technique according to at least one embodiment of the present disclosure are shown. Figure 2A One or more imaging devices 112 are illustrated, which are arranged to be close to the patient 208 (e.g., on either side, adjacent, etc.), and each of the one or more imaging devices is configured to emit energy (e.g., X-rays, non-ionizing radiation, etc.) that can be collected by the detector 204. Although three imaging devices 112 and three detectors 204 are illustrated, additional or alternative numbers of imaging devices 112 and / or detectors 204 can be used. Additionally, the patient 208 can be positioned in a variety of different orientations, and the patient does not need to be in the supine position as shown in Figure 2A . For example, the patient 208 can be in the prone position, where the imaging device 112 is on the first side of the patient, and the detector 204 is on the second side of the patient opposite the first side. As another example, the patient can be standing, where the imaging device 112 is on the first side of the patient, and the detector 204 is on the second side of the patient opposite the first side. The energy measured by the detector 204 can be used by the processor 104 (e.g., using image processing 120) to generate an image 212 of the patient 208, as illustrated in Figure 2B . The image 212 can depict one or more anatomical elements (e.g., vertebrae) of the patient 208. Based on the type of imaging device used when capturing the image 212, the image 212 can be or include various types of images. The image 212 can be or include, for example, a CT image (e.g., when the imaging device 112 includes an X-ray emitter), an optical image (e.g., when the imaging device 112 emits non-ionizing radiation), a depth image (e.g., when the imaging device 112 includes a LIDAR camera), and / or an MRI image (e.g., when the imaging device 112 generates a magnetic field).

[0088] Processor 104 may use one or more algorithms or data models (e.g., image processing 120, segmentation 122) to identify boundaries 216A, 216B between a patient (or a portion thereof, such as a patient anatomy) and the environment surrounding the patient as depicted in image 212. For example, segmentation 122 may be or include an edge detection algorithm that identifies changes in pixel values above a threshold to identify the boundary between a patient anatomy and the environment surrounding the patient anatomy. As Figure 2B illustrated, boundaries 216A and 216B may be the boundaries identified by the edge detection algorithm. In other embodiments, segmentation 122 may be or include a neural network or other data model that is trained on images similar to image 212 and is configured to receive image 212 and identify boundaries 216A and 216B between the patient and the patient environment (e.g., the air surrounding the patient).

[0089] Processor 104 may also calculate a thickness 220 of patient 208 based on the distance between boundaries 216A, 216B. Although the term "thickness" is used herein, it should be understood that thickness is the general distance in any spatial direction between one side of the patient and the other side of the patient and may alternatively be described as height, width, depth, etc. In some embodiments, processor 104 may use a transformation 124 to map the relative distance between the image data (e.g., pixel values) associated with boundaries 216A, 216B to the relative thickness 220 of patient 208. In other words, transformation 124 may convert the distance between pixel values in image 212 to a real-world distance measurement corresponding to thickness 220. When computing device 102 generates a reconstructed volume model of the patient, the determined thickness 220 may be used in one or more criteria.

[0090] Figures 2C to 2D Aspects of a second imaging technique in accordance with at least one embodiment of the present disclosure are illustrated. The second imaging technique may include positioning imaging device 112 such that energy (e.g., X-rays, non-ionizing radiation, etc.) emitted from imaging device 112 is emitted along the length of patient 208 and captured by a detector (not shown). In some embodiments, multiple images of the patient may be captured when the imaging device is positioned in different poses. The energy measured by the detector may be used by processor 104 (e.g., using image processing 120) to generate an image 224 of patient 208, as Figure 2D depicted. Image 224 may depict one or more anatomical elements (e.g., vertebrae) of patient 208. In some embodiments, image 224 may be similar to image 212 but may depict the anatomical elements of patient 208 from a different angle or direction than image 212. In some embodiments, imaging device 112 may be perpendicular to being positioned Figure 2AThe imaging device 112 in [reference] is aligned such that image 212 depicts the patient along or relative to a first direction, while image 224 depicts the patient along a second direction perpendicular to the first direction. Image 224 may be or include a CT image (e.g., when the imaging device 112 includes an X-ray emitter), an optical image (e.g., when the imaging device 112 emits non-ionizing radiation), a depth image (e.g., when the imaging device 112 includes a LIDAR camera), and / or an MRI image (e.g., when the imaging device 112 generates a magnetic field).

[0091] The processor 104 may use one or more algorithms or data models (e.g., image processing 120, segmentation 122) to identify boundaries 228A, 228B between the patient (or a part thereof, such as the patient's anatomy) and the environment surrounding the patient depicted in image 224. For example, segmentation 122 may be or include an edge detection algorithm that identifies changes in pixel values above a threshold to identify the boundary between the patient's anatomy and the environment surrounding the patient's anatomy. As Figure 2D shown, boundaries 228A and 228B may be the boundaries identified by the edge detection algorithm. In other embodiments, segmentation 122 may be or include a neural network or other data model that is trained on images similar to image 224 and is configured to receive image 224 and identify boundaries 228A and 228B between the patient and the patient's environment (e.g., the air surrounding the patient).

[0092] The processor 104 may also calculate a width 232 of the patient 208 based on the distance between the boundaries 228A, 228B. Although the term "width" is used herein, it should be understood that the term "width" is used to refer to the physical distance or dimension in any spatial direction between one side of the patient and the other side of the patient, and may alternatively be described as height, thickness, depth, etc. In some embodiments, the processor 104 may use the transformation 124 to map the relative distance between the image data associated with the boundaries 228A, 228B (e.g., the distance between pixel values) to the relative width 232 of the patient 208. In other words, the transformation 124 may convert the distance between pixel values in the image 224 into a real-world distance measurement corresponding to the width 232. In embodiments where the patient is imaged in multiple different poses (or, similarly, when the patient remains fixed but the imaging device 112 captures image data while in different poses relative to the patient), the processor 104 may use the registration 128 to map the image data from different poses into a common coordinate system before determining the width 232. The registration 128 may be or include an algorithm that receives pose information for each depicted angle in the image 224 associated with the imaging device and uses the pose information to map the pixel values into a common coordinate system such that both the boundaries 228A, 228B reflect pixel values in the common coordinate system. When the computing device 102 generates a reconstructed volume model of the patient, the determined width 232 may be used in one or more criteria.

[0093] In some embodiments, both the thickness 220 and the width 232 may be used to determine the boundary conditions of the patient in the reconstructed volume. For example, the thickness 220 may provide a first dimension of the patient relative to a plane, and the width 232 may provide a second orthogonal (e.g., perpendicular) dimension of the patient relative to the plane, thereby creating an estimated shape (e.g., rectangle, square, cube, cuboid, etc.) that can be used to represent the space occupied by the patient. Additionally or alternatively, the thickness 220 and / or the width 232 may each separately include a plurality of individual thickness and / or width measurements such that the shape created based on the thickness 220 and the width 232 forms a shell representing the outer boundary of the patient 208. The boundaries forming the shape may be used as one or more criteria for the reconstructed volume model of the patient, as discussed further below.

[0094] Figures 2E to 2F Illustrates various aspects of a technique for generating a mesh 236 of a patient's anatomy in accordance with at least one embodiment of the present disclosure. The mesh 236 may depict the overall shape and / or outer boundary of the patient 208. As Figure 2EAs shown, the imaging device 112 can be moved relative to the patient 208 to capture image data of the patient at different angles or poses. The imaging device 112 can translate and / or orbit around the patient 208, and can generate a panoramic image, a panoramic sagittal image, etc. of the patient 208 based on the captured image data. In one embodiment, the imaging device 112 can use LIDAR scanning, where the imaging device 112 acts as both a transmitter and a detector to capture a set of points associated with the patient. The processor 104 can use the captured data and one or more algorithms or data models (e.g., image processing 120) to generate the mesh 236. For example, the imaging process 120 can be an algorithm that calculates the relative distance between the patient and the imaging device 112 at each point of the captured set of points based on the data (e.g., based on the captured set of points) to construct the mesh 236. In another example, the processor 104 can use a neural network or other data model trained on similar image data to generate the mesh 236.

[0095] When reconstructing the multi-dimensional volume of the patient, the mesh 236 can be used for patient body shape estimation. In some embodiments, the data associated with the mesh (e.g., the data associated with the boundaries of the mesh) can be overlaid or otherwise combined with the surgical scan data used to perform the reconstruction of the multi-dimensional image, where the mesh data serves as one or more criteria for the reconstruction. For example, the outer boundary formed by the mesh 236 can be used to determine what constraints (if any) should be imposed on the voxels for the volume reconstruction.

[0096] Figures 3A to 3BIllustrates various aspects of a reconstructed image 300 in accordance with at least one embodiment of the present disclosure. The reconstructed image 300 may be a reconstruction of a multi-dimensional image (e.g., 2D image, 3D image, etc.) of a patient, patient anatomy, etc. In some embodiments, the reconstructed image 300 may be based on pre-operative and / or intra-operative scans or images of the patient 208. For example, the patient 208 may undergo a surgical procedure or surgical protocol where pre-operative images of the patient are captured and a volumetric reconstruction of the patient 208 is created. In some embodiments, images of the patient may be dynamically captured by the robotic arm 116, where one robotic arm holds the emitter of the imaging device 112 and the other robotic arm holds the detector of the imaging device 112. The volumetric reconstruction may be based on the images of the patient and may be used for surgical navigation purposes (e.g., navigating the robot 114 using the navigation system 118) to assist a surgeon in planning and / or performing a surgical procedure, or for any other reason. In some embodiments, the reconstructed image 300 may be initiated by the processor 104 using one or more algorithms or data models (e.g., reconstruction 136). The reconstruction 136 may use ART, FBP, combinations thereof, etc. to generate the reconstructed image 300. In some embodiments, the reconstruction 136 may iteratively change (e.g., increase or decrease) the values of one or more voxels (e.g., values in 3D space) that form the multi-dimensional image according to one or more criteria. Figure 3A Depicts the reconstructed image 300 after the first iteration of the reconstruction 136 (e.g., before any iterative change to the voxels), while Figure 3B Illustrates the reconstructed image 300 after the reconstruction 136 has iteratively changed the values of one or more voxels (e.g., after one iteration, after two iterations, after three iterations, etc.).

[0097] The reconstructed image 300 includes a plurality of voxels 308A to 308J, including a plurality of internal voxels 308A to 308E disposed inside the boundary 304 (also referred to herein as the boundary region) and a plurality of external voxels 308A to 308E disposed outside the boundary 304. It should be understood that although the reconstructed image 300 is depicted as 2D in the figures, in embodiments where the multi-dimensional image is three-dimensional, the reconstructed image 300 provides a view of a 2D slice of the overall volumetric reconstruction.

[0098] The boundary 304 may represent an estimated patient body shape based on, for example, the thickness 220, width 232, and / or grid 236. In some embodiments, the dimensions of the boundary 304 may correspond to the estimated body shape of the patient 208. For example, Figure 3A the height of the boundary 304 in Figure 3AThe length of the boundary 304 in [description] can have the same value as the width 232. As another example, the dimensions of the boundary 304 can correspond to the dimensions of the grid 236. In other words, Figure 3A depicts an estimated patient body shape covered with a plurality of voxels 308A through 308J, the values of which can be varied during the process of reconstructing the multi-dimensional image. Although the boundary 304 is depicted as a quadrilateral in Figures 3A to 3B [description], it should be understood that the shape used to estimate the patient body shape is in no way limiting, and the boundary 304 based on the estimated patient body shape can take on any size, shape, or form. In some embodiments, the estimated patient body shape can be based on other patient records (e.g., previously recorded patient measurements, previous images and / or scans of patient 208, combinations thereof, etc.), where the data associated with the patient records is input into the reconstruction 136.

[0099] Each of the voxels can include an attenuation value. The attenuation value can reflect the tendency of the region (or volume) represented by the voxel to be penetrated by energy (e.g., radiation from an X-ray). In some embodiments, the attenuation value can be based on Hounsfield units (HU). The Hounsfield unit is a dimensionless unit commonly used in CT scans to express CT numbers in a standardized and convenient form. The Hounsfield unit is obtained by a linear transformation of the measured attenuation coefficient. This transformation is based on arbitrarily assigned densities of air and pure water. For example, the radiation density of distilled water at standard temperature of zero degrees Celsius and a pressure of 105 Pascals (STP) is 0 HU; the radiation density of air at STP is -1000 HU. Although the attenuation values of the voxels are discussed herein qualitatively (e.g., low attenuation, medium attenuation, high attenuation, etc.) and / or quantitatively (e.g., based on values in HU), it should be understood that the values of the voxels discussed herein are in no way limiting.

[0100] In Figures 3A to 3B [description], an increase in the darkness of the voxel indicates less attenuation (which can also indicate a greater tendency for energy penetration); in other words, a white voxel can have a high attenuation value (e.g., an attenuation value with HU greater than 700), a gray voxel can have a medium attenuation value (e.g., an attenuation value with HU between 100 and 700), and a black voxel can have a low attenuation value (e.g., an attenuation value with HU below 0 HU).

[0101] The first voxel 308A, the second voxel 308B, the third voxel 308C, the fourth voxel 308D, and the fifth voxel 308E may be located within the boundary 304; while the sixth voxel 308F, the seventh voxel 308G, the eighth voxel 308H, the ninth voxel 308I, and the tenth voxel 308J may be located outside the boundary 304. Although the voxels 308A to 308J are illustrated as not contacting the boundary 304, in some cases, one or more edges of the voxels may be adjacent to the boundary 304. In some embodiments, the reconstruction 136 may be able to identify which voxels are located within the boundary 304 and which voxels are located outside the boundary 304, and may label the voxels accordingly. In such embodiments, the reconstruction 136 may perform such identification on the voxels in the dynamically captured image data prior to the first iteration of generating the reconstructed image 300.

[0102] The second voxel 308B, the fifth voxel 308E, the seventh voxel 308G, and the eighth voxel 308H may each have a high attenuation value; the first voxel 308A, the third voxel 308C, the fourth voxel 308D, and the tenth voxel 308J may each have a medium attenuation value; and the sixth voxel 308F and the ninth voxel 308I may each have a low attenuation value.

[0103] As Figure 3B shown, the attenuation value of one or more of the voxels 308A to 308J may be adjusted according to one or more criteria during one or more iterations of the reconstruction 136. For example, the reconstruction 136 may reduce the attenuation value of the second voxel 308B from high attenuation to medium attenuation, and may also increase the attenuation value of the first voxel 308A from medium attenuation to high attenuation. In some embodiments, the reconstruction 136 may keep the sum of all attenuation values the same (e.g., fixed) while reducing or increasing the attenuation value of a voxel.

[0104] In some embodiments, one or more voxels can be located on the boundary 304 (e.g., a voxel is located both inside and outside the boundary 304), such as the eleventh voxel 308K. In such embodiments, the algorithm can classify the eleventh voxel 308K as being inside the boundary 304 or as being outside the boundary 304 according to one or more parameters or other settings associated with the algorithm. For example, the algorithm can be configured such that all voxels on the boundary 304 are classified as being included within the boundary 304. In other cases, the algorithm can classify all voxels on the boundary as being outside the boundary 304. In some embodiments, sub-voxel (or sub-pixel when the multi-dimensional image being reconstructed is 2D) resolution interpolation can be performed to estimate what percentage of the voxel (or pixel) is located within the boundary 304. When more than half of the voxels are within the boundary 304 (e.g., the percentage is higher than 50%), the algorithm can classify the voxel as being within the boundary 304, and when more than half of the voxels are outside the boundary 304, the algorithm can classify the voxel as being outside the boundary 304.

[0105] One or more criteria can be based on patient body size estimation, the attenuation values of one or more voxels, their combination, etc. In one embodiment, one or more criteria can be based on the boundary 304. For example, one criterion can include that voxels located outside the boundary 304 cannot have an attenuation value that increases beyond a threshold (e.g., a threshold associated with the attenuation value of the skin). As another example, one criterion can include that the difference between the attenuation value of a voxel outside the boundary 304 and the attenuation value of a voxel inside the boundary 304 must be higher than a threshold. As yet another example, one criterion can include that the attenuation of voxels that must be towards the outside of the boundary 304 must be adjusted relative to the target ratio of the attenuation of voxels inside the boundary 304 (and vice versa).

[0106] In some embodiments, the reconstruction 136 may fail to meet one or more criteria or may fail to generate the reconstructed image 300. For example, the criteria may impose constraints on the voxels outside the boundary 304, which results in no possible solution to adjust the attenuation values of the voxels while also keeping the sum of the attenuation values constant. In another example, the image data used by the reconstruction 136 may contain insufficient image data for the reconstruction 136 to generate the reconstructed image 300 (e.g., due to corrupted image data). As yet another example, the boundary 304 (estimated based on the patient's body shape) may be inaccurate (e.g., the boundary is too large, the boundary is too small, etc.), resulting in too few voxels outside or inside the boundary 304. In such cases, the reconstruction 136 may return an error message or other indicator that one or more criteria are not met. Accordingly, additional images of the patient may be taken and / or additional information associated with the patient may be retrieved (e.g., retrieved from the database 130) and used to create a new patient body shape estimate. In other embodiments, the original image data used to generate the initial patient body shape estimate may be reused, but the parameters of the algorithms and / or data models (e.g., mesh generation algorithms / data models, edge detection algorithms / data models, etc.) used to generate the body shape estimate may be changed, adjusted, or otherwise tuned to produce a new patient body shape estimate. The new patient body shape estimate (e.g., in the form of a new boundary 304) may then be used in the reconstruction 136 to create a multi-dimensional image.

[0107] As Figure 3B illustrated, one or more criteria may result in a decrease in the attenuation values of the voxels outside the boundary 304. In other words, the reconstruction 136 may treat the voxels outside the boundary 304 as not associated with the patient 208 or, in other words, associated with the environment surrounding the patient 208. The environment (e.g., air) has an attenuation value different from that of the patient's anatomy. By introducing the boundary 304, which corresponds to the estimated patient body shape, the reconstruction 136 can more effectively and accurately reconstruct the patient volume. In other words, by applying one or more criteria to the voxels outside the boundary 304 (and / or in some embodiments, to the voxels inside the boundary 304), the attenuation values of the voxels not associated with the patient's anatomy are decreased or otherwise distinguished from the patient's anatomy. This allows the reconstruction 136 and, when the reconstructed image 300 is rendered on a display, makes it easier for the surgeon to distinguish the patient's anatomy from the non-patient anatomy. Similarly, compared to the case where the reconstructed image 300 is reconstructed without the boundary 304 (and by extending the prior data related to the patient's body shape), the reconstructed image 300 may provide a more accurate representation of the space occupied by the patient 208, thereby enabling improved navigation of the robot 114 and / or the robotic arm 116 relative to the patient 208.

[0108] Figure 4Depicts method 400, which can be used, for example, to reconstruct an image volume that includes a prior patient body shape estimate.

[0109] Method 400 (and / or one or more of its steps) can be implemented or otherwise executed, for example, by at least one processor. The at least one processor can be the same as or similar to processor 104 of computing device 102 described above. The at least one processor can be part of a robot (such as robot 114) or part of a navigation system (such as navigation system 118). Processors other than any of the processors described herein can also be used to execute method 400. The at least one processor can execute method 400 by executing elements stored in a memory (such as memory 106). The elements stored in the memory and executed by the processor can cause the processor to execute one or more steps of the functions as shown in method 400. One or more parts of method 400 can be executed by a processor that executes any of the contents in the memory (such as image processing 120, segmentation 122, transformation 124, registration 128, and / or reconstruction 136).

[0110] Method 400 includes identifying a boundary region corresponding to the shape of an object based on first imaging data associated with the object (step 404). The first imaging data can be obtained from one or more images captured using one or more imaging devices (e.g., imaging device 112). In one embodiment, the imaging data can be identified from one or more CT scans, MRI scans, fluoroscopy scans, LIDAR scans, combinations thereof, etc. The object can be or include a patient or one or more parts of a patient (e.g., anatomical elements). The boundary region can be determined based on, for example, a mesh generated based on the imaging data, the estimated thickness and width of the patient (which can be based on the width and thickness of the patient depicted in one or more images), combinations thereof, etc. In some embodiments, the boundary region can be determined based on one or more algorithms and / or data models that use, for example, edge detection to identify the boundary region in the first imaging data. Edge detection can identify different parts of the first imaging data (e.g., upper and lower), and can determine an air edge (e.g., the edge where the image changes from depicting patient anatomy to depicting the air or other environment around the patient). In some embodiments, the air edge can be used to define the boundary of the patient.

[0111] Method 400 further includes identifying at least one voxel included in second imaging data associated with an object, where the at least one voxel is located outside a boundary region (step 408). The second imaging data may be associated with scans or other images generated by one or more imaging devices during a surgical procedure or surgical protocol. The second imaging data may be based on, for example, data generated by a transmitter and detector attached to a robotic arm that dynamically navigates around a patient. The second imaging data may be used in an iterative algorithm or data model (e.g., as a basis or initial guess) to generate a volume reconstruction representing the volume occupied by the patient. The second imaging data may include a plurality of voxels, the plurality of voxels including attenuation values (e.g., HU values), and the boundary region may be combined with the second imaging data (e.g., the boundary may be overlaid on the second imaging data) such that the voxels are divided into two groups of voxels: voxels located within the boundary region and voxels located outside the boundary region. Step 408 may identify each voxel among the plurality of voxels, the plurality of voxels including voxels within the boundary region and voxels outside the boundary region. In some embodiments, the combined imaging data (e.g., the second imaging data and the overlaid boundary region) and the identified voxels may be rendered to a display.

[0112] Method 400 further includes generating a multi-dimensional image volume corresponding to the object using the second imaging data, where the multi-dimensional image volume is generated relative to one or more criteria associated with voxels located outside the boundary region (step 412). In some embodiments, the multi-dimensional image volume may be similar to or the same as the reconstructed image 300. The multi-dimensional image volume may depict a 3D model of the volume occupied by the object (e.g., the patient). Step 412 may generate the multi-dimensional image volume based on one or more criteria using, for example, one or more iterative algorithms and / or data models (e.g., ART, MLEM, OSEM, FBP, combinations thereof, etc.).

[0113] In some embodiments, the one or more criteria may include a threshold attenuation value associated with one or more voxels located outside the boundary region. For example, the one or more criteria may include limiting the sum of the attenuation values of all voxels located outside the boundary region to a threshold. As another example, the one or more criteria may include that voxels located outside the boundary region cannot have an attenuation value that increases to exceed a threshold (e.g., a threshold associated with the attenuation value of the skin).

[0114] In some embodiments, one or more criteria may be based on the attenuation values of voxels outside the boundary region relative to the attenuation values of voxels inside the boundary region. For example, one or more criteria may include that the difference between the attenuation values of voxels outside the boundary region and the attenuation values of voxels inside the boundary region must be higher than a threshold. As another example, one or more criteria may include that the voxels outside the boundary region must be adjusted towards a target ratio of the attenuation of the voxels outside the boundary region relative to the attenuation of the voxels inside the boundary region (and vice versa). In other words, when the attenuation values of the voxels outside the boundary region are summed and divided by the sum of the attenuation values of the voxels inside the boundary region (or alternatively, the reciprocal of such a value), the attenuation values of the voxels outside the boundary region must be lower (or alternatively higher) than a threshold.

[0115] In some embodiments, method 400 may not include step 504. In some embodiments, method 400 may not include step 412.

[0116] The present disclosure encompasses embodiments of method 400 that include more or fewer steps and / or one or more steps different from those described above.

[0117] Figure 5 Method 500 is depicted that can be used to adjust a boundary region, for example, when generating a multi-dimensional image volume.

[0118] Method 500 (and / or one or more of its steps) may be implemented or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to processor 104 of computing device 102 described above. The at least one processor may be part of a robot (such as robot 114) or part of a navigation system (such as navigation system 118). A processor other than any of the processors described herein may also be used to perform method 500. The at least one processor may perform method 500 by executing elements stored in a memory (such as memory 106). The elements stored in the memory and executed by the processor may cause the processor to perform one or more steps of the functions as shown in method 500. One or more portions of method 500 may be performed by a processor that executes any of the contents in the memory (such as image processing 120, segmentation 122, transformation 124, registration 128, and / or reconstruction 136).

[0119] Method 500 includes regenerating first image data associated with an object (step 504) based on failure to meet one or more criteria. Step 504 may start from step 412, where a boundary region of the object is determined and a multi-dimensional image volume is generated based on one or more criteria.

[0120] In step 504, when generating a multi-dimensional image volume, one or more criteria may not be met. For example, the criteria may impose constraints on voxels outside the boundary region, which results in no possible solution to adjust the attenuation values of the voxels while also keeping the sum of the attenuation values constant. In another example, the image data may contain insufficient image data to accurately generate the multi-dimensional image volume (e.g., due to corrupted image data). As yet another example, the boundary region (estimated based on the patient's body shape) may be inaccurate (e.g., the boundary region is too large, the boundary region is too small, etc.), resulting in too few voxels outside or inside the boundary region.

[0121] Step 504 can regenerate the first imaging data by causing additional images of the patient to be taken and / or additional information associated with the patient to be retrieved (e.g., retrieved from database 130) and used to create a new patient body shape estimate. In other embodiments, the original image data used to generate the initial patient body shape estimate can be reused, but the parameters of the algorithms and / or data models (e.g., mesh generation algorithms / data models, edge detection algorithms / data models, etc.) used to generate the body shape estimate can be changed, adjusted, or otherwise tuned to produce a new patient body shape estimate. Then, the new patient body shape estimate can be used as the second boundary region to create the multi-dimensional image.

[0122] Method 500 further includes identifying a second boundary region corresponding to the shape of the object based on the regenerated first imaging data associated with the object (step 508). In some embodiments, step 508 may be similar to or the same as step 404. Step 508 can use one or more algorithms and / or data models (e.g., reconstruction 136) to identify the second boundary region in the regenerated first imaging data.

[0123] Method 500 further includes identifying at least one second voxel included in the third imaging data, where at least one second voxel is located outside the second boundary region (step 512). In some embodiments, step 512 may be similar to step 408. When the second boundary region is combined with the third imaging data, step 512 can use one or more algorithms and / or data models (e.g., reconstruction 136) to identify the voxels inside and outside the second boundary region. In some embodiments, the third imaging data can be or include the first imaging data, while in other embodiments, the third imaging data can be based on dynamically captured images and / or scans of the patient taken preoperatively and / or intraoperatively.

[0124] Method 500 also includes generating a second multi-dimensional image volume corresponding to the object using third imaging data, wherein generating the second multi-dimensional image volume includes meeting one or more criteria associated with voxels located outside a second boundary region (step 516). In some embodiments, step 516 may be similar to or the same as step 412. Step 516 may use one or more algorithms and / or data models (e.g., reconstruction 136) to generate the second multi-dimensional image volume. In some embodiments, the second boundary region may be different from the initial boundary region such that the reconstruction 136 can meet one or more criteria when generating the second multi-dimensional image volume.

[0125] The present disclosure encompasses embodiments of method 500 that include fewer steps and / or one or more steps different from those described above than those described above.

[0126] As described above, the present disclosure encompasses methods having fewer steps than all of the steps identified in Figure 4 and Figure 5 (and the corresponding descriptions of methods 400 and 500), as well as methods that include additional steps beyond the steps identified in Figure 4 and Figure 5 (and the corresponding descriptions of methods 400 and 500). The present disclosure also encompasses methods that include one or more steps from one method described herein and one or more steps from another method described herein. Any correlation described herein may be or include registration or any other correlation.

[0127] The foregoing is not intended to limit the present disclosure to one or more forms disclosed herein. In the foregoing detailed description, for example, for purposes of simplifying the present disclosure, various features of the present disclosure are grouped together in one or more aspects, specific implementations, and / or configurations. Features of the aspects, specific implementations, and / or configurations of the present disclosure may be combined in alternative aspects, specific implementations, and / or configurations other than those discussed above. The methods of the present disclosure should not be construed as reflecting an intention that the claims require more features than those expressly recited in each claim. Rather, as reflected in the following claims, aspects of the present invention lie in less than all of the features of a single foregoing disclosed aspect, specific implementation, and / or configuration. Accordingly, the following claims are hereby incorporated into this detailed description, where each claim stands on its own as a separate preferred specific implementation of the present disclosure.

[0128] In addition, although the foregoing has included a description of one or more aspects, specific implementations, and / or configurations, as well as certain variations and modifications, other variations, combinations, and modifications are within the scope of the present disclosure, for example, within the skills and knowledge of those of ordinary skill in the art. It is intended to obtain rights to include alternative aspects, specific implementations, and / or configurations within the scope of what is permitted, including alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps of those claimed, whether or not such alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps are disclosed herein, and it is not intended to disclose dedicated to any patentable subject matter.

[0129] It should be understood that any feature described herein can be combined with any other feature described herein to claim protection, regardless of whether the features are from the same described specific implementation.

[0130] The phrases “at least one,” “one or more,” and “and / or” are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” “A, B, and / or C,” and “A, B, or C” means only A, only B, only C, A and B together, A and C together, B and C together, or A, B, and C together.

[0131] The term “a” entity means one or more of that entity. Thus, the terms “a,” “one or more,” and “at least one” may be used interchangeably herein. It should also be noted that the terms “comprising,” “including,” and “having” may be used interchangeably.

[0132] As used herein, the term “automatically” and variations thereof refer to any process or operation that is typically continuous or semi-continuous and is completed without substantial human input when the process or operation is performed. However, even if the execution of a process or operation uses substantial or significant human input, the process or operation may still be automatic if the input is received prior to the execution of the process or operation. An input is considered substantial if it affects the manner in which the process or operation is executed. Human input that consents to the execution of a process or operation is not considered “substantial.”

[0133] Aspects of the present disclosure may take the form of a specific implementation that is entirely hardware, a specific implementation that is entirely software (including firmware, resident software, microcode, etc.), or a specific implementation that combines software and hardware aspects, which may all be generally referred to herein as “circuitry,” “module,” or “system.” Any combination of one or more computer-readable media may be utilized. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium.

[0134] A computer-readable storage medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0135] A computer-readable signal medium may include a propagated data signal embodied in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code embodied on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0136] As used herein, the terms "determine", "calculate", "estimate", and variations thereof may be used interchangeably and include any type of method, process, mathematical operation, or technique.

Claims

1. A system, the system comprising: a processor; and a memory storing instructions which, when executed by the processor, cause the processor to: identify a boundary region corresponding to the shape of an object based on first imaging data associated with the object; identify at least one voxel included in second imaging data associated with the object, wherein the at least one voxel is located outside the boundary region; and generate a multi-dimensional image volume corresponding to the object using the second imaging data, wherein generating the multi-dimensional image volume is relative to one or more criteria associated with voxels located outside the boundary region.

2. The system according to claim 1, wherein the instructions are further executable by the processor to: regenerate the first imaging data associated with the object based on a failure to meet the one or more criteria; identify a second boundary region corresponding to the shape of the object based on the regenerated first imaging data associated with the object; identify at least one second voxel included in third imaging data, wherein the at least one second voxel is located outside the second boundary region; and generate a second multi-dimensional image volume corresponding to the object using the third imaging data, wherein generating the second multi-dimensional image volume includes: meeting one or more criteria associated with voxels located outside the second boundary region.

3. The system according to claim 1, wherein the one or more criteria include a threshold attenuation value associated with voxels located outside the boundary region.

4. The system according to claim 1, wherein the one or more criteria include a target ratio of the attenuation of voxels located outside the boundary region to the attenuation of voxels located inside the boundary region.

5. The system according to claim 1, wherein the instructions are further executable by the processor to: identify second voxels located inside the boundary region based on the second imaging data associated with the object, wherein the one or more criteria include meeting a threshold difference between a first attenuation value of the at least one voxel and a second attenuation value of the second voxels.

6. The system according to claim 1, wherein the first imaging data includes one or more panoramic sagittal images of the object.

7. The system according to claim 1, wherein: the first imaging data includes one or more images including the object; an upper portion of the one or more images includes a first air edge corresponding to a boundary of the object; and a lower portion of the one or more images includes a second air edge corresponding to another boundary of the object.

8. The system according to claim 1, wherein the instructions are further executable by the processor to: capture a set of points associated with the object using one or more light-based ranging operations; and generate the first imaging data associated with the object based on the set of points.

9. The system according to claim 1, wherein the first imaging data includes one or more x-ray images, one or more optical images, one or more depth images, one or more magnetic resonance imaging (MRI) images, one or more computed tomography (CT) images, or a combination thereof.

10. The system according to claim 1, wherein the instructions are further executable by the processor to capture the first imaging data associated with the object, wherein capturing the first imaging data includes: capturing a first image of the object, wherein capturing the first image is associated with first pose information of the imaging device relative to the object; and capturing a second image of the object, wherein capturing the second image is associated with second pose information of the imaging device relative to the object, and wherein identifying the boundary region corresponding to the shape of the object is based on the first image, the first pose information, the second image, and the second pose information.

11. The system according to claim 1, wherein the size of the boundary region corresponds to at least one of the following: the size of the object in a first direction relative to a plane; and a second size of the object in a second direction relative to the plane, wherein the second direction is orthogonal to the first direction.

12. The system according to claim 1, wherein the instructions are further executable by the processor to: dynamically capture the second imaging data using one or more imaging devices.

13. A system, the system comprising: one or more imaging devices; a processor; and a memory storing data thereon, the data when processed by the processor causes the processor to: identify a boundary region corresponding to the shape of an object based on first imaging data generated using the one or more imaging devices; identify at least one voxel included in second imaging data generated using the one or more imaging devices, wherein the at least one voxel is outside the boundary region; and generate a multi-dimensional image volume corresponding to the object using the second imaging data, wherein generating the multi-dimensional image volume is relative to one or more criteria associated with voxels located outside the boundary region.

14. The system according to claim 13, wherein the data is further executable by the processor to: regenerate the first imaging data in response to failure to meet the one or more criteria; identify a second boundary region corresponding to the shape of the object based on the regenerated first imaging data; identify at least one second voxel included in third imaging data, wherein the at least one second voxel is outside the second boundary region; and generate a second multi-dimensional image volume corresponding to the object using the third imaging data, wherein generating the second multi-dimensional image volume includes: meeting one or more criteria associated with voxels located outside the second boundary region.

15. The system according to claim 13, wherein the one or more criteria include a threshold attenuation value associated with a voxel located outside the boundary region.

16. The system according to claim 13, wherein the one or more criteria include a target ratio of the attenuation of a voxel located outside the boundary region to the attenuation of a voxel located inside the boundary region.

17. The system according to claim 13, wherein the data can be further executed by the processor to: Based on the second imaging data associated with the object, identify a second voxel located inside the boundary region, wherein the one or more criteria include a threshold difference between a first attenuation value of the at least one voxel and a second attenuation value of the second voxel.

18. The system according to claim 13, wherein the first imaging data includes one or more panoramic sagittal images of the object.

19. The system according to claim 13, wherein: The first imaging data includes one or more images including the object; An upper portion of the one or more images includes a first air edge corresponding to a boundary of the object; and A lower portion of the one or more images includes a second air edge corresponding to another boundary of the object.

20. A method, the method comprising: Based on first imaging data associated with an object, identify a boundary region corresponding to the shape of the object; Identify at least one voxel included in second imaging data associated with the object, wherein the at least one voxel is located outside the boundary region; And Generate a volume construct corresponding to the object using the second imaging data, wherein generating the volume construct is relative to satisfying one or more criteria associated with voxels located outside the boundary region.

Images