System and method for positioning of a patient's body part during operative intervention

Abstract

The present invention relates to a method to position a human body part for imaging during operative intervention in an operating room. A radio opaque marker is placed on a skin area near an area of interest. Imaging is performed using imaging ray emitters and images produced therefrom are used to adjust the position of the imaging ray emitters. A Quick Response code is placed in the operating room and a mixed virtual reality headset is used to ascertain the relative position of the area of interest and the operating room. The relative position is used by a computing programming product to compute an optimal position for the human body part. The position of the human body part is adjusted based on the computed result.

Description

SYSTEM AND METHOD FOR POSITIONING OF A PATIENT’S BODY PARTDURING OPERATIVE INTERVENTIONBACKGROUND

[0001] An aneurysm is a bulging, weakened area in the wall of a blood vessel, usually an artery. Aneurysms can expand like a balloon as the walls of the artery become thinner. The larger an aneurysm becomes, the greater the risk for rupture (bursting), which can result in lifethreatening bleeding. The most common location of an aneurysm is the aorta, the largest artery, which carries oxygenated blood from the heart to the body. Aortic aneurysms may be thoracic - located in the segment of the aorta in the chest cavity, or abdominal - in the part of the aorta that runs through the abdomen. An aneurysm may also be located in the blood vessels of the brain (cerebral aneurysm). Aneurysms can also develop in the blood vessels of the neck, the intestines, the kidney, the spleen or the legs.

[0002] Open surgery and endovascular repair are the two main types of procedures to repair an aneurysm. Imaging by radiology procedure is a key requirement to visualize aneurysm during endovascular procedures. A basic form of 3D imaging is one option for radiology procedures in certain aneurysm treatment.

[0003] These methods have historically exposed patients to excessive radiation through the need to take a large number of X-ray images. And since these images are inherently static in nature and the intracerebral vasculature of each patient is unique, these imaging techniques are inherently flawed and of limited utility as they result in only approximations of patient and intracerebral vasculature positioning to guide interventions.

[0004] The present invention addresses this and other related needs in the art.SUMMARY

[0005] These and other embodiments, features, and advantages will become apparent to those skilled in the art when taken with reference to the following more detailed description of various exemplary embodiments of the present disclosure in conjunction with the accompanying drawings.

[0006] According to frequently included embodiments, there is provided a method to adjust position of a body part for operative intervention, the method comprises: positioning a body part of with an area of interest near an imaging device positioned in an operating room;placing a device carrying a QR code adjacent to the area of interest, the device having radio-opaque markers; emitting imaging rays from the imaging ray emitter and receiving imaging rays on an imaging ray receiver; obtaining at least one image of the area of interest; displaying the at least one image of the area of interest on a user interface display; placing virtual 3-dimensinal boundaries surrounding a target area in the at least one image of the area of interest; computing a position of the imaging ray emitter using geometric parameters extracted from the target area in the at least one image of the area of interest; assigning a confidence score of the virtual 3 -dimensional boundaries surrounding a target are in the area of interest; segmenting the target area in the at least one image of the area of interest to delineate the target area in the area of interest; computing an optimal position of the imaging ray emitter to change the confidence score of the virtual 3 -dimensional boundaries surrounding the target area in the area of interest; and changing the position of the imaging ray emitter and / or position of the body part to the optimal position, wherein the imaging ray emitter and the imaging ray receiver are positioned on each end of a C-shape mechanical arm.

[0007] Often according to embodiments described herein, there is provided a method as above, wherein segmenting the target area in the at least one image of the area of interest is by deep learning, support vector machine, heuristic, linear regression, or random forest strategy.

[0008] Frequently according to embodiments described herein, there is provided a method as above, wherein segmenting the target area in the at least one image of the area of interest is initialized by user input.

[0009] Often according to embodiments described herein, there is provided a method as above, wherein the virtual 3-dimensinal boundaries are in a box shape.

[0010] Frequently according to embodiments described herein, there is provided a method as above, wherein the virtual 3-dimensinal boundaries are cuboid in shape.

[0011] Often according to embodiments described herein there is provided a method as above, wherein the C-shape mechanical arm further comprises a track along the arm, the track configured for the imaging ray emitter and / or the imaging ray receiver to move along the track.

[0012] Frequently according to embodiments described herein, there is provided a method as above, wherein there are two C-shape mechanical arms, two imaging ray emitters and two imaging ray receivers, and wherein each pair of imaging ray emitter and corresponding imaging ray receiver is located on opposing ends of each C-shape mechanical arm.

[0013] Often according to embodiments described herein, there is provided a method as above, wherein the two C-shape mechanical arms are each attached to an extension at a pivot point and are configured to rotate around the pivot point.

[0014] Frequently according to embodiments described herein, there is provided a method as above, wherein the extension is attached to a frame at a pivot point and is configured to rotate around the pivot point.

[0015] Often according to embodiments described herein, there is provided a method as above, wherein the extension is attached to a frame and is configured to slide along the frame.

[0016] Frequently according to embodiments described herein, there is provided a method as above, wherein the body part is a human head.

[0017] Often according to embodiments described herein, there is provided a method as above, wherein the target area is an aneurysm in an artery in a human brain.

[0018] Frequently according to embodiments described herein, there is provided a method as above, further comprising segmenting at least one additional target area in the area of interest to delineate the at least one additional target area in the area of interest in the at least one image of the area of interest.

[0019] Often according to embodiments described herein, there is provided a method as above, further comprising segmenting at least one additional target area in the area of interest to delineate the at least one additional target area in the area of interest in the at least one image of the area of interest.

[0020] Frequently according to embodiments described herein, there is provided a method as above, the method further comprises: obtaining relative position between the target area in the area of interest and the radio opaque markers adjacent the area of interest using rigid transformation step T1 ; obtaining relative position between the QR code adjacent the area of interest and a mixed reality headset camera using rigid transformation step T2; obtaining relative position between the mixed reality headset camera and a reference QR code in the operating room using rigid transformation step T3;obtaining relative position between the target area in the area of interest and the mixed reality headset camera using reverse rigid transformation step IT1and rigid transformation step T2; obtaining relative position between the target area in the area of interest and the operating room using reverse rigid transformation step IT1, rigid transformation step T2, and rigid transformation step T3.

[0021] Often according to embodiments described herein, there is provided a method as above, the method further comprises computing a position and orientation of the body part using the relative position between the target area in the area of interest and the operating room and the optimal position of the imaging ray emitter.

[0022] Frequently according to embodiments described herein, there is provided a method as above, the method further comprises moving the body part to the computed position and orientation.

[0023] Often according to embodiments described herein, a system to adjust position of a body part for operative intervention is provided, the system comprises: a room suitable for medical operation on a subject; a table for a subject to rest on; at least one imaging ray emitter and at least one imaging ray receiver, the at least one imaging ray emitter and the at least one imaging ray receiver is each attached to an end of at least one C-shape mechanical arm; a device carrying at least one QR code and having radio opaque markers for placement adjacent an area of interest on the subject; a computing article operatively connected to at least one user interface display; a computer executable code adapted to direct and / or perform the steps of: receiving and displaying at least one image of an area of interest from the imaging ray emitter and imaging ray receiver; placing virtual 3-dimensinal boundaries surrounding a target area in the at least one image of the area of interest; computing a position of the imaging ray emitter and imaging ray receiver using geometric parameters extracted from the target area in the at least one image of the area of interest; assigning a confidence score to the target area in the at least one image of the area of interest;segmenting the target area in the at least one image of the area of interest to delineate the target area in the area of interest; computing an optimal position of the imaging ray emitter and imaging ray receiver to change the confidence score of the target are in the area of interest; and changing the position of the imaging ray emitter to the optimal position.

[0024] Frequently according to embodiments described herein, there is provided an apparatus as above, wherein there are two C-shape mechanical arms, with each C-shape mechanical arm having one imaging ray emitter and one imaging ray receiver attached to each end of the C-shape mechanical arm.

[0025] Often according to embodiments described herein, there is provided an apparatus as above, the apparatus further comprises a reference QR code attached to the operating room and a mixed reality headset equipped with an RGB camera.

[0026] Frequently according to embodiments described herein, there is provided an apparatus as above, wherein the computer executable code is further configured to perform the steps of: obtaining relative position between the target area in the area of interest and the radio opaque markers using rigid transformation step Tl; obtaining relative position between the QR code and a mixed reality headset camera using rigid transformation step T2; obtaining relative position between the mixed reality headset camera and a reference QR code in the operating room using rigid transformation step T3; obtaining relative position between the target area in the area of interest and the mixed reality headset camera using reverse rigid transformation step Tl'1and rigid transformation step T2; obtaining relative position between the target area in the area of interest and the operating room using reverse rigid transformation step Tl'1, rigid transformation step T2, and rigid transformation step T3.

[0027] Often according to embodiments described herein, there is provided an apparatus as above, wherein the computer executable code is further configured to perform the step of computing a position and orientation of the body part using the relative position between the target area in the area of interest and the operating room and the optimal position of the imaging ray emitters.

[0028] Frequently according to embodiments described herein, there is provided an apparatus as above, wherein the computer executable code is further configured to perform thestep of segmenting at least one additional target area in the area of interest to delineate the at least one additional target area in the at least one image of the area of interest.

[0029] Often according to embodiments described herein, there is provided an apparatus as above, wherein the body part is a human head.

[0030] Frequently according to embodiments described herein, there is provided an apparatus as above, wherein the at least one area of interest is an aneurysm.

[0031] Often according to embodiments described herein, there is provided an apparatus as above, wherein the at least one additional area of interest is a vascular tree in a human head.

[0032] Frequently according to embodiments described herein, a method to obtain relative position between a target area and a reference point is provided, the method comprises: placing a device carrying a QR code adjacent to an area of interest having a target area, the device having radio-opaque markers; obtaining relative position between the target area in the area of interest and the radio opaque markers adjacent the area of interest using rigid transformation step T1 ; obtaining relative position between the QR code adjacent the area of interest and a mixed reality headset camera using rigid transformation step T2; obtaining relative position between the mixed reality headset camera and a reference QR code using rigid transformation step T3; relative position between the target area in the area of interest and the mixed reality headset camera using reverse rigid transformation step Tl’1and rigid transformation step T2; obtaining relative position between the target area in the area of interest and reference point using reverse rigid transformation step Tl'1, rigid transformation step T2, and rigid transformation step T3.BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The skilled person in the art will understand that the drawings, described below, are for illustration purposes only.

[0034] FIG. 1 depicts an example of an angio-suite as provided by Siemens Healthineers.

[0035] FIG. 2 depicts a drawing of an exemplary C-arm setup, including an X-ray emitter, an X-ray sensor, a human body part, and a table, and exemplary motions directions for the exemplary C-arm.

[0036] FIG. 3 depicts an exemplary view angle for multiple X-ray devices of an aneurysm for coiling.

[0037] FIGS. 4A-4C depict exemplary C-arm angles, including a depiction of C-arm at CRA 30° (FIG. 4A), C-arm at RAO 60° (FIG. 4B), and conventional directions of RAO, CRA, CAU, LAO, and AP as used in imaging (FIG. 4C).

[0038] FIGS. 5A-B depict aneurysm detection resulting in 3D bounding shape (FIG. 5A; Left: box. Right: ellipsoid) and aneurysm segmentation (FIG. 5B).

[0039] FIG. 6 depicts an example of defining a camera by its position (Pc), direction (Zc), and orientation (Yc).

[0040] FIG. 7 depicts an exemplary aneurysm 3D reconstruction according to the methods described herein.

[0041] FIG. 8 depicts an exemplary optimization of the camera position and orientation to avoid occultation of the aneurysm by other arteries.

[0042] FIG. 9 depicts exemplary relative rigid transformations between absolute reference frame, aneurysm reference frame, patient reference frame and Head Mount Display (HMD) reference frame.

[0043] FIG. 10 depicts an exemplary Head Mount Display with a camera attached to the HMD.

[0044] FIG. 11 depicts the rigid transformation process between the aneurysm reference frame and a patient reference frame defined by a device bearing a QR code attached to the patient.

[0045] FIG. 12 depicts an operation block diagram of a radiology assistance system.

[0046] FIG. 13 A depicts an MR scene with recommended repositioning steps by rotation for a patient in an angio-suite.

[0047] FIG. 13B depicts an MR scene with recommended repositioning steps by table lift for a patient in an angio-suite.

[0048] FIG. 13C depicts an MR scene showing a radio opaque device being detected.

[0049] FIG. 14 depicts the step of detecting a device with radio opaque marker to display a Mixed Reality scene.

[0050] FIG. 15 depicts the step of detecting QR markers to display a Mixed Reality scene.

[0051] FIG. 16 depicts the step of determining best possible view to display a Mixed Reality scene for adjusting of patient and equipment position.DETAILED DESCRIPTION

[0052] For clarity of disclosure, and not by way of limitation, the detailed description of the invention is divided into the subsections that follow.

[0053] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as is commonly understood by one of ordinary skill in the art to which this invention belongs. All patents, applications, published applications and other publications referred to herein are incorporated by reference in their entirety. If a definition set forth in this section is contrary to or otherwise inconsistent with a definition set forth in the patents, applications, published applications and other publications that are herein incorporated by reference, the definition set forth in this section prevails over the definition that is incorporated herein by reference.

[0054] As used herein, “a” or “an” means “at least one” or “one or more.”

[0055] As used herein, the term “and / or” may mean “and,” it may mean “or,” it may mean “exclusive-or,” it may mean “one,” it may mean “some, but not all,” it may mean “neither,” and / or it may mean “both.”

[0056] As used herein, “spatial awareness” refers to the capability of understanding and interacting with the physical space around them.

[0057] As used herein, “angio-suite” refers to an operating room equipped for endovascular procedures, including X-ray equipment with computerized movement and real time screen viewing.

[0058] As used herein, “C-arm” refers to a mechanical device shaped like the letter C, upon which X-ray emitting devices are mounted.

[0059] As used herein, “contrast agent” refers to a composition safe to be taken into a human body and capable of being viewed on an X-ray image.

[0060] As used herein, “fluoroscopy” refers to an imaging technique that makes a realtime video inside the body by passing X-rays through the body over a period of time. Fluoroscopy is made by an imaging device called a C-arm.

[0061] As used herein, “imaging ray” refers to radiating wave (e.g., X-ray, radio waves, light waves, etc.) based imaging such as X-ray, MRI, CT, laser or other based imaging.

[0062] As used herein, “micro-catheter” refers to a small diameter catheter that is used in minimally invasive procedures for delivering devices. Its small build makes it ideal for navigating complex vasculatures within the human body.

[0063] As used herein, “radio-opaque” refers to matter that is opaque to X-rays or similar tradition.

[0064] As used herein, “registration” refers to refers to the process of aligning and overlaying multiple images of the same or different modalities to create a single, integrated image or to establish a spatial relationship between them in the context of medical imaging.

[0065] As used herein, “surface registration,” refers to the process of aligning and matching the surfaces or geometries of objects or structures represented in two or more three- dimensional (3D) surface models or point clouds in the context of computer science and medical imaging.

[0066] As used herein, “QR code” refers to a series of geometric patterns that store a variety of information and are machine-readable. “QR code” and “pattern marker” are used interchangeably.

[0067] As used herein, “mixed reality” or “MR” refers to the presentation of virtual objects such that a user sees images that include both real, physical objects and virtual objects.

[0068] As used herein, “MR scene” refers to a scene perceived by a user that includes one or more virtual objects and may include one or more physical objects.

[0069] An angio-suite is the operating room where interventional radiology procedures take place. Key devices in this room include the X-ray imaging device and the table for positioning the patient. The table can be horizontally and vertically moved, or titled at various angle. In advanced angio-suites the X-ray imaging device is provided in a C-arm or a double C-arm. Each C-arm consists of an X-ray emitter, an X-ray sensor, and a C-shaped track along which the emitter / sensor glides. To limit the patient exposure to radiation, the patient has to be placed as close as possible to the X-ray emitter. Since the X-ray emitter and receiver can only move along the C-arm, moving the table also help with moving the patient closer to the X-ray emitter.

[0070] One of the complications encountered using complex imaging systems like this involves determining the proper viewpoint for X-ray imaging for a particular procedure or aspect of a procedure. X-ray images have a small set of available resolutions, the areas sought to be imaged are often at variable depth levels within the skull of the patient, every patient has an individualized vasculature formation and positioning, and aneurysm location is variable. The X-ray emitter and sensor are positioned on the track according to a viewing angle to view the relevant vasculature or part thereof, and the table is raised, lowered, or rotated, and as there are so many variables involved in this process there is generally a trial-and-error approach to patient positioning, even for medical practitioners that have reviewed pre-operative 3D images. And, it is not uncommon for the patient to move or be moved during the procedure in a manner that requires the positioning to be done again.

[0071] At the beginning of INR procedure, the medical team sets up the patient in the angio-suite, where they lie on a table. Before acquiring any images, the medical team will adjust the position of the head to accommodate both the range of motions of the C-arms and the best views of the region of interest. The head is generally initially positioned facing the ceiling. Then the patient is repeatedly positioned and repeatedly X-Ray imaged to reach two canonical views of the skull. With no formal assessment of the optimal head position and no possibility to measure the actual position of the head or any of its displacement, the positioning is an approximation, and the overall positioning process exposes the patient to a large amount of dangerous radiation. It is very often that a first failed attempt at imaging the region of interest triggers a repositioning of the head and a second round of imaging: this is costly in time, in radiation exposure, and injection of contrast agent.

[0072] According to the presently contemplated procedures, for each step of the interventional neural radiology (INR) procedure, two angles of view are provided for the physician, one angle for each of the two C-arms. For example, to treat an aneurysm with coiling, one C-arm may be positioned to image the aneurysm laterally whilst the second C-arm may be positioned to image the main artery carrying the aneurysm.

[0073] In certain embodiments described herein, a system and method to position a human body part for imaging of vascular aneurysm in the body part is provided. The method is performed using a system comprising imaging ray emitter, imaging ray sensor, and a moveable path (e.g., C-shaped track) along which the imaging ray emitter and imaging ray sensor can be individually or together situated, and a computer assisted method to calculate an optimum position for the body part based on certain variables and information provided herein. Other imaging methods are also contemplated. Magnetic Resonance Imaging (MRI) and Computer Tomography (CT) scan are other imaging methods that can be used in the process described herein. For simplicity, the description herein discusses X-ray imaging as the exemplary method of imaging for embodiments included herein but this is intended to be nonlimiting. For example, it is specifically contemplated that other, non-X-ray, manners of imaging such as MRI, CT and other imaging methods are utilized in systems and methods described herein in connection with the specifically exemplified X-ray imaging. As X-ray imaging systems have a small set of available resolutions, the positioning, direction, and orientation of the X-ray emitter / receiver define the angle of imaging. The computer assisted methods described herein use reference frames present in an angio-suite and on the skin of a patient in the form of QR codes to adjust the position of the body part to be imaged using acamera attached to a Head Mount Display, ultimately determining a position of the body part for optimal imaging during operative intervention.

[0074] In embodiments described herein, exemplary methods are conducted using commercially available equipment such as equipment depicted in FIG. 1, which shows an angio-suite as envisaged by Siemens Healthineers. In this setting a table is provided for a patient to lie on. A device shaped like the letter C, termed the C-arm, is provided for carrying an X-ray emitter A and an X-ray receiver B. The C-arm is attached to an extension and can rotate around pivot point on the extension. The extension is attached to a frame and can also rotate around a pivot point. Alternatively, the extension can also slide along the frame. This is a fairly generic depiction of the type of arrangement contemplated herein with reference to a C-arm angio-suite arrangement.

[0075] FIG. 2 depicts a side view of one exemplary C-arm showing positioning of an X-ray emitter 201 and an X-ray receiver 202. The X-ray emitter 201 is shown in FIG. 2 at the bottom of the C-arm 200, and the X-ray receiver 202 is shown in FIG. 2 at the top of the C- arm. A human body part 208 is shown positioned on table 207, and table 207 is positioned between the X-ray emitter and X-ray receiver 201, 202.

[0076] In FIG. 2, the exemplary C-arm 200 with an X-ray emitter 201 and an X-ray receiver 202 and an extension 205 upon which the C-arm 200 is movably attached. Extension 205 attaches to a rotating portion 206 at pivot Ry. Rotating portion 206 is attached to an anchor area in the angio-suite, for example, above, below, or laterally to the patient. In the depicted example, rotation portion 206 can rotate around pivot point Rz, thereby extending the movement range of the C-arm 200. The X-ray emitter and the X-ray receiver 201, 202 can move along a track Rxprovided on the C-arm 200. In the depicted example, the X-ray emitter and the X-ray receiver 201, 202 are fixedly positioned on the C-arm 200, and C-arm 200 travels on a track positioned on extension 205 to provide for, for example, up to 180° of rotational positioning. The X-ray emitter and X-ray receiver 201, 202 are X-ray imaging equipment capable of taking X-ray images. Two C-arms configured as shown in FIG. 2 are typically provided in an angio-suite, as shown in FIG. 1. A patient can lie on table 207 during the imaging procedure. Table 207 can move horizontally, vertically, or tilt at various angles.

[0077] In embodiments described herein, a method to position a human body part for imaging during operative intervention is provided. According to the procedure, the vasculature of a patient, e.g., with an aneurysm, is visualized using a 3D imaging modality. As blood vessels do not show up on typical X-ray imaging, a contrast agent that will show up on an X- ray image is generally administered to the patient in advance of such visualization. The timingfor this administration is variable based on the specific procedure and location sought to be imaged. FIG. 3 illustrates a step where two X-ray emitter / receiver pairs are utilized in different positions relative to an area of interest in the patent, for example an area with an arterial aneurysm. Notably, due to the large number of variables involved in the process, this is not the positioning initially achieved without existing imaging available to guide the X-ray system positioning. The X-ray system first obtains one or more 3D images of an area of interest of the body of the patient, such as a portion of the vasculature within a specific part of the body for example, the head. On a C-arm, X-ray emitter 201 emits X-rays through a main artery 101 laterally to the aneurysm 102, the X-rays are received in an X-ray receiver 202 positioned opposite X-ray emitter 201 on the other side of the body part. In practice, the images taken during this step are 2D in nature, but a number of images are taken from a variety of different angles. A 3D image is compiled based on these 2D images. An aneurysm 102 is illustrated in FIG. 4 as protruding from the main artery 101.

[0078] FIGS. 4A-C illustrate various imaging positions according to the angle from which the camera is oriented. FIG. 4A illustrates CRA 30° (cranial direction, camera is oriented at 30° towards the cranium). FIG. 4B illustrates RAO 60° (right side of the body, camera is at 60° towards the right of the patient’s body). FIG. 4C depicts the CRA, CAU, RAO, LAO, and AP angles illustrated on a 360° plane with a patient on a table at the center of the plane. The CRA, CAU, RAO, and LAO angles are utilized according to the presently described methods to aid in the positioning of the patient for optimal imaging. According to such methods CRA, CAU, RAO, and / or LAO angles / positions relative to the position of the area of interest will have definite values for each subject. While positioning using degrees as a metric are exemplified, the present disclosure is not intended to be so limited. Radians and other units may be employed for identifying the designated imaging positions.

[0079] To assist in producing images allowing visualization of the blood vessels and aneurysm and positioning of the patient, a device 405 capable of being uniquely identified, positioned, and oriented in a medical 3D image (e.g., Computed Tomography Angiography, etc.) and in RGB (red, green, blue) image is often used. In frequent embodiments, the device 405 is, for example, a 3D distinguishable device. In one example embodiment, device 405 is provided as a multi-sided dice. The device 405 is adapted such that its position and orientation is distinguishable during imaging and can be computed. One manner of doing this includes adapting device 405 to have radiopaque surfaces capable of being imaged, and its position and orientation able to be discerned in three dimensions. 3D beads encapsulating a contrast agent or comprised of a surface that will be shown in the selected imaging modality are used.

[0080] The device 405 is a rigid object, such as a dice (or other fiducial object, indicator or marker), with planar printed pattern markers (QR code) 402, typically black and white. For example, a QR code may be provided on at least one or each side of device 405. When there are multiple sides with printed QR codes, each side displays a QR code that may or may not be different. An RGB camera or a black and white camera can detect one or several pattern markers 402 and track them while the pattern markers 402 are within the view of the camera. Each pattern marker is unique and encodes a unique identifier. A camera captures images or videos in real time. A computer algorithm processes the camera’s images or videos and analyzes pixel data to determine if one or several markers are visible in its view. If a marker is visible in its view, the algorithm identifies and locates the marker in its view. Once a marker is detected, the software calculates its position and orientation in the camera's coordinate system. By knowing the marker's size and the camera's intrinsic parameters (such as focal length and principal point), the software can determine the marker's 3D position and orientation relative to the camera. This process is known as pose estimation.

[0081] Since a marker is provided, e.g., printed, on 405, its 3D position and orientation define uniquely the position and orientation of 405. When several sides of device 405 are provided with pattern markers, it ensures that at least one pattern marker is visible from the camera regardless of the orientation of the device 405. If several pattern markers are visible, the position and orientation of each marker can also lead to the same position and orientation determination of device 405. The position and orientation derived from each pattern marker are combined to result in the actual position and orientation of device 405.

[0082] Device 405 is positioned on, adjacent to, or affixed / attached to the skin or clothing of the patient at or adjacent to an area of interest for imaging. 3D medical images (Magnetic Resonance, Computer Tomography) are obtained of the area of interest including the device 405. A camera, such as a camera integrated into a Head Mount Display, detects the markers 402 on device 405 to determine the device’s 405 3D position and orientation relative to the camera. FIG. 13C illustrates one exemplary embodiment of the MR scene visualized by an operator when device 405 is detected, with a virtual circle 907 shown around device 405. Extracting the radio-opaque markers 406 within device 405 allows device 405 to be positioned and oriented relatively to the aneurysm 102. This is the rigid transformation T2.

[0083] FIG. 11 illustrates an example of device 405 having 3D beads 406. On the top left corner of FIG. 11 is device 405 showing as a cube with beads 406 embedded into device 405. Reference frame 408 as an xyz axis system is shown next to device 405. On the top right comer of FIG. 11, the first image shows the lateral view of device 405 along the y axis ofreference frame 408. The second image shows the lateral view of device 405 along the x axis of reference frame 408. The third image shows the lateral view of device 405 along the z axis of the absolute reference frame 408. Beads 406 are shown on these views. On the bottom of FIG. 11 is device 405 shown with pattern markers 402 on several sides of device 405. Reference frame 408 is also shown next to device 405.

[0084] The system automatically identifies the 3D location of the markers within a 3D image using any suitable technique. For example, the system may use a machine learned model such as one or more deep learning algorithms to process the image data and identify the location of the marker. An algorithm may use known relative 3D locations of the markers to refine the 3D location of each marker. The location is expressed either in voxels or sub-voxel units and / or mm (or any other length unit). A result of this process may also be a 3D binary mask (the list of voxels belonging to the marker(s) in the 3D image) or a 3D probability map (the probability of each voxel belonging to the marker). This algorithm may involve a preprocessing step to enhance the quality of the image. This may involve tasks such as noise reduction, smoothing, and normalization.

[0085] In exemplary embodiments, 3D beads contained within device 405 are radioopaque spheres (or other 3D shapes) located either within or on device 405. The 3D beads are stationary against each other and against device 405, and thus together they define a rigid 3- dimensional body for device 405. Each of the 3D beads are designed such that they can be uniquely identified in a 3D image. Alternatively, the 3D beads together describe a specific 3D pattern which allows identification of each of them in a 3D image. When enough 3D beads are uniquely identified, determining the position and orientation of the rigid body, or device 405, relative to the 3D image is possible, and reciprocally the position and orientation of the 3D image relative to the rigid body is possible. The process results in rigid transformations T1 and Tl'1. FIG. 9 shows an example of this process, where a 3D image is taken of aneurysm 102 and device 405 located on the patient with aneurysm 102. Using rigid transformations Tl and Tl'1, the position and orientation of device 405 relative to the aneurysm 102 can be determined.

[0086] Rigid transformation as used herein refers to a mathematical representation of the position and orientation of an object relative to a frame of reference. The frame of reference is a fixed set of 3 orthogonal normalized vectors. In FIG. 9, the absolute reference frame 401 is defined by a set of 3 orthogonal normalized vectors, which is depicted above the reference planar printed pattern markers (QR code) 407. Reference frame 401 represents the position and orientation of reference marker 407. Reference frame 401 relative to the camera 501 of HMD 103 is calculated. This is rigid transformation T3. The position and orientation of thereference marker 407 and its reference frame 401 relative to HMD 103 is the same T3, because the camera 501 is a rigid part of the HMD 103. The position and orientation of HMD 103 relative to reference marker 407 is rigid transformation T3'1. From the origin of the 3 orthogonal normalized vectors representing the reference marker 407, T3 would move the HMD 103 from the reference marker 405 to the HMD’s position.

[0087] After initial imaging of the blood vessel and aneurysm, optimal view angles are computed using one or more preoperative 3D images, which can be images obtained from the earlier step described herein. To locate the aneurysm in the input image, aneurysm 102 is first detected using one or more deep learning algorithm, then a 3D shape fitting the aneurysm 102 is defined. The system may automatically identify the location of the aneurysm within the 3D image using any suitable technique. For example, the system may use a machine learned model such as a deep learning algorithm to process the image data and identify the location of the aneurysm. A machine learning algorithm may learn from an existing image database about how an aneurysm presents in an image and how a 3D shape could be fitted to an aneurysm in the image. The shape is either expressed mathematically (for example a sphere is described by its center and its radius, a box by two of its corners), as a 3D binary mask (the list of voxels belonging to the shape in the 3D image), or as a 3D probability map (the probability of each voxel to belong to the shape). Exemplary embodiments of this algorithm involve a preprocessing step to enhance the quality of the image. This involves, for example, tasks such as noise reduction, smoothing, and normalization. In some exemplary embodiments, changes made during the manual correction step are used as training data to refine the machine learning techniques. The system can apply machine learning over many aneurysm procedures and utilize decisions and changes made in past aneurysm procedures to drive machine learning and thereby to help fitting a shape to an aneurysm. In some cases, machine learning is based on past surgical procedures, decisions and changes made by a specific radiologist. In other words, the past surgical procedures, decisions and changes made by a specific radiologist are stored in memory and used as data for driving a machine learning algorithm that can guide fitting a shape to an aneurysm. This way, machine learning can predict how a shape should be fitted to a particular aneurysm for a given procedure, and the predictions may be based on historical data, such as past aneurysm procedures, decisions and changes made by a specific radiologist. In some cases, historical data of a variety of radiologists is used for this machine learning, e.g., whereby past aneurysm procedures, decisions and changes made by many radiologists are used to drive machine learning algorithm that can help fitting a shape to an aneurysm.

[0088] FIG. 5A illustrates this process, where a virtual bounding box 301a in a box shape or an ellipsoid shape 301b is fitted around the 3D image of the aneurysm. Other 3- dimensional shapes, geometric or non-geometric in shape, such as icosahedron or another shape, may also be used, depending on the shape of the aneurysm - and in any case for efficiency purposes all such shapes are intended to be included in the reference to box / ellipsoid 301a / 301b as those terms are used herein. The 3D bounding box / ellipsoid 301a / 301b can be automatically computed, semi-automatically computed, or manually drawn and defined by a human operator. When the 3D bounding box / ellipsoid 301a / 301b is automatically computed, a computer programming product detects the 3D image of the aneurysm 102 and computes a 3D bounding box / ellipsoid 301a / 301b using an embedded algorithm. When the 3D bounding box / ellipsoid 301a / 301b is semi-automatically computed, a human operator clicks on different points of the aneurysm 102 or draws lines to approximate and surround the aneurysm 102 and a computer programming product computes the 3D bounding box / ellipsoid 301a / 301b using the embedded algorithm. When the 3D bounding box / ellipsoid 301a / 301b is manually computed, a human operator draws lines to surround the aneurysm 102 and the computer programming product computes the 3D bounding box / ellipsoid 301a / 301b based on the lines drawn by the human operator using the embedded algorithm.

[0089] In exemplary embodiments, the fitting step is run automatically as a default, and the result reviewed by an operator. If approved, the fitting box is approved by the operator. If not, the operator can click at several points on the aneurysm image to give guidance to the algorithm and increase the fitting score. The operator can also choose to define the 3D box fitting the aneurysm themselves.

[0090] After a 3D bounding box / ellipsoid 301a / 301b is formed, a confidence score is computed by the embedded algorithm or given by the operator during review of the 3D bounding box / ellipsoid 301a / 301b. A machine learning algorithm used for this bounding box generation process generates a confidence score as the last optimization step. A confidence score reflects the assessment of how well the 3D bounding box / ellipsoid 301a / 301b fits the aneurysm and is used to account for discrepancies between computed position and actual position of the body part. The operator can refine the shape of the 3D bounding box / ellipsoid 301a / 301b if the confidence score is not high. In manual or semi-automatic computation of the 3D bounding box / ellipsoid 301a / 301b, the refining process involves redrawing of the box, lines, or clicks on the aneurysm to better define the 3D bounding box / ellipsoid 301a / 301b. In automatic computation, the 3D bounding box / ellipsoid 301a / 301b can be refined by the operator, such as by clicking on certain points on the aneurysm to guide the algorithm.

[0091] In embodiments described herein, 3D segmentation of the aneurysm 102 is performed using the 3D bounding box / ellipsoid 301a / 301b to precisely delineate the aneurysm 102 and the blood vessel 103 having the aneurysm. The 3D bounding box / ellipsoid 301a / 301b can be used to initialize the precise delineation of the aneurysm. In some embodiments, one algorithm is used to generate both the 3D bounding box / ellipsoid 301a / 301b and the delineation process. The region to be segmented can be driven by the aneurysm detection step, such that the area with aneurysm 102 is segmented. In the delineation process, the aneurysm 102 is inside the 3D bounding box and it is the aneurysm that is being segmented. In some cases, the region being segmented may include regions outside of the bounding box / ellipsoid 301a / 301b, especially when the bounding box is not well defined. The underlying algorithm can be automated using different kinds of strategies, such as deep learning, support vector machine, heuristic, linear regression, random forests, among other strategies. The underlying algorithm can also provide a confidence score to the bounding box. The aneurysm can be further delineated by the operator, such as by 3D painting, graph cuts, or other techniques for delineating, using tools embedded in the computing programming product. Further delineating is also useful in assessing and / or planning for intervention of the aneurysm and may be used.

[0092] An algorithm to automatically delineate the aneurysm within the 3D image may use any suitable technique. For example, the system may use a machine learned model such as deep learning algorithm to process the image data. A machine learning algorithm may learn from an existing image database how an aneurysm looks like in an image and how to delineate aneurysm. The result may be expressed as a 3D binary mask (the list of voxels belonging to the shape in the 3D image), or as a 3D probability map (the probability of each voxel to belong to the marker), or as a 3D surface. This algorithm may involve a pre-processing step to enhance the quality of the image. This involves, for example, tasks such as noise reduction, smoothing, and normalization. This algorithm may or may not use the results of fitting a shape to an aneurysm as an initialization step. In some exemplary embodiments, changes made during the manual correction step can be used as training data to refine the machine learning techniques. The system can apply machine learning over many aneurysm procedures and utilize decisions and changes made in past aneurysm procedures to drive machine learning and thereby to help delineating the aneurysm. In some cases, machine learning may be based on past surgical procedures, decisions and changes made by a specific radiologist. In other words, the past surgical procedures, decisions and changes made by a specific radiologist are stored in memory and used as data for driving a machine learning algorithm that can guide delineating the aneurysm. This way, machine learning can predict the delineation of an aneurysm for a givenprocedure, and the predictions may be based on historical data, such as past aneurysm procedures, decisions and changes made by a specific radiologist. In some cases, historical data of a variety of radiologists may be used for this machine learning, e.g., whereby past aneurysm procedures, decisions and changes made by many radiologists are used to drive machine learning algorithm that can help delineating the aneurysm.

[0093] FIG. 6 illustrates the process of defining a camera position, direction, and orientation relative to a reference frame. On the left of FIG. 6, a reference frame is shown as a set of orthogonal axes xyz, the position of the camera 501 relative to a reference frame is noted as Pc. The camera 501 has a frame of reference defined as Xc¥cZc 601. The direction of the camera 501, or the direction of the rays emitting from camera 501, is defined by an axis Zc. A set of orthogonal axes Xc and Yc complete the camera reference frame. On the right of FIG. 6, an X-ray imaging device is illustrated, with X-ray emitter 201 placed opposite to X-ray receiver 202 on a C-arm. The X-ray imaging device shown here serves the function of a camera. The position and orientation of the X-ray device is determined in the same manner as the position and orientation of the camera.

[0094] In an exemplary embodiment, the reference marker 407 provides a reference frame. In this exemplary embodiment, the reference marker 407 serves as or defines the xyz reference frame as illustrated in FIG. 6, and the X-ray emitter serves as the camera 501 as illustrated in FIG. 6. Similar to a camera, an X-ray device emits X-rays as the rays to generate images to be captured. A detector is present in an X-ray device, which may comprise scintillators that convert X-rays into visible light, which is then detected by a sensor. Exposure time when an X-ray device is used is the duration of X-ray radiation used to obtain an image. The output images show the density of tissues and structures within the subject, with different shades of grey representing different levels of X-ray absorption. The position and orientation of the X-ray emitter relative to the reference marker 407 and its reference frame is defined. From this defined position, a set of imaging angles, including RAO / LAO and / or CAU / CRA can be computed, such that movement to a different location can be defined by a new set of values relative to the same xyz reference frame defining at least one of a new position, a new direction, and a new orientation.

[0095] FIG. 8 illustrates an exemplary aneurysm 102 reconstructed in 3D by the methods described herein. Aneurysm 102 protrudes from the main artery 101 and can be visualized as part of the blood vessel system attached to the main artery 101, or by itself with different views. The reconstructed aneurysm 102 can also be viewed as a separate object indifferent views. The ability to simulate the various views of the aneurysm increases a physician’s ability to plan an intervention procedure.

[0096] FIG. 8 illustrates an application of the present methods to optimize the patient and / or table positioning in a manner that avoids obstructing vasculature. As seen in the top image, the initial position of the camera, where the images taken from the camera 104 show a blood vessel 103 image obstructing the aneurysm 102 image. This obstruction would not be known prior to imaging and knowing how to position the patient to provide for an image that lacks this obstruction is not a straightforward process, requiring a trial-and-error approach rather than a predictable method, including multiple administrations of contrast agent, additional delays in a time-sensitive procedure and added cost. Positioning of the camera, or the X-ray emitter and corresponding X-ray receiver relative to patient positioning, and the table, plays an important role in the image produced in terms of resolution and lack of obstructions. The methods disclosed herein are used to adjust camera positioning relative to the patient, or vice versa. The xyz axis system shown in FIG. 9 represents a definition of the camera position and image path in 3 dimensions, with the x axis being the direction of the X- ray emitted from the camera during imaging. Using the camera positioning parameters provided and described in conjunction with FIG. 6, the relative patient / camera positioning can be manipulated for optimal imaging, for example as depicted in the lower image of FIG. 9. The bottom of FIG. 9 is an illustration of the optimized position of the camera, where the images taken from the camera show a blood vessel 103 image no longer obstructing the aneurysm 102 image. In one embodiment, the camera is attached to the C-arm 200 which is moved around a pivot point to a position that allows better viewing of aneurysm 102. Rotating portion 206 to which the C-arm attaches and extension 205 to which the rotating portion 206 attaches may also move around two other pivot points to adjust the camera positioning and orientation. In another embodiment, the patient or a body portion of the patient, such as the head of the patient, is re-positioned from the initial position exemplified in the top image of FIG. 8 to the optimal position exemplified in the bottom image of FIG. 8.

[0097] FIGS. 5A & 5B illustrate the aneurysm detection and validation process. Starting from a 3D image, an algorithm, which could be operated manually, or semi- automatically, delineates an aneurysm through segmentation. A 3D image of an area having an aneurysm is loaded into a computing article. A computer algorithm automatically detects the aneurysm then segments the aneurysm image and displays the result. If the aneurysm is validated by a human operator, the process ends. If it is not validated and a semi-automatic mode of operation is in use, the operator can assist the algorithm by clicking on certain pointson the aneurysm image, and the shape outlined by the operator is fed back into the algorithm to repeat the step of detecting the aneurysm. If the aneurysm is not validated and a manual mode of operation is in use, the operator can draw a 3D box or ellipsoid to assist the algorithm. The steps may continue until the aneurysm is validated.

[0098] With the camera position relative to the area of interest in the patient to be imaged being optimized, the aneurysm 102 is now delineated in 2D images by way of projecting the 3D delineation using the C-arm parameters.

[0099] In embodiments described herein, the present methods are also used to delineate the blood vessels and / or the “neck” of the aneurysm in the area of interest. The neck of an aneurysm is the narrow part connecting the aneurysm to the blood vessel. Typically, the neck is the least dilated part of the aneurysm. Delineation of the blood vessels is important for accessing and treatment of aneurysms. The region within the blood vessels to be segmented is chosen by the aneurysm step. Typically, the blood vessels and branches surrounding the aneurysm and affecting the view of the aneurysm are delineated. Blood vessels within the blood vessels leading to the aneurysm may also need to be delineated. Underlying algorithms for delineating blood vessels can use different types of strategies, including deep learning, support vector machine, heuristic, linear regression, random forests, among other strategies. The neck of the aneurysm can also be delineated with an algorithm using deep learning, support vector machine, heuristic, linear regression, random forests, among other strategies.

[0100] The algorithm to extract the neck of an aneurysm may use the aneurysm and artery image (as a 3D surface, 3D binary mask, or a 3D probability map), and possibly a 3D image covering the aneurysm as the input. The system may automatically reconstruct the pre- morbid artery using any suitable technique. For example, the system may use a machine learned model such as deep learning to process the image and / or surface data. A machine learning algorithm may learn from an existing image / surface database about how pre-morbid artery presents in an image or as a surface and how to reconstruct the artery. The neck is defined through a geometric relationship between the aneurysm and the pre-morbid artery. In some examples, changes made during the manual correction step may be used as training data to refine the machine learning techniques. The system may apply machine learning over many aneurysm procedures and may use decisions and changes made in past aneurysm procedures to drive machine learning and thereby to help extracting of the aneurysm neck. In some cases, machine learning may be based on past surgical procedures, decisions and changes made by a specific radiologist. In other words, the past surgical procedures, decisions and changes made by a specific radiologist may be stored in memory and used as data for driving a machinelearning algorithm that can help extracting of the aneurysm neck. This way, machine learning can help to predict the extracting of the aneurysm neck for a given procedure, and the predictions may be based on historical data, such as past aneurysm procedures, decisions and changes made by a specific radiologist. In some cases, historical data of a variety of radiologists may be used for this machine learning, e.g., whereby past aneurysm procedures, decisions and changes made by many radiologists can be used to drive machine learning algorithm that can help extracting of the aneurysm neck.

[0101] In embodiments described herein, as the camera position and orientation and the table position and orientation relative to the area of interest in the patient have been optimized, the C-arm carrying each camera can move the / each camera, such that the / each camera can move to the optimized position. Depending on the position, the cameras may move along the track on the C-arms to get to the computed position. Alternatively, or in concert, the patient may be re-positioned into the optimized position using the present methods. For each step of the procedure to optimize the position of the camera relative to an area of interest, an optimal view angle for both C-arms is also computed

[0102] The first estimate of the camera position parameters is derived from the initial steps of this method using geometric parameters extracted from the aneurysm and the arteries surrounding it. The relative positioning of the area of interest and the camera is then adjusted to the first estimated position. For each step of the procedure, the best relative position of the area of interest (e.g., based on C-arm positioning) and the camera is computed and recorded. Over many uses, a list of C-arm positions is recorded and can be used as training materials or to speed up the camera position optimization process. A database of 3D and 2D images of various C-arm positions as well as geometric parameters of the same can be collected and used in machine learning algorithms, which can optimize this process further. This process also takes into account the physical constraints of the C-arms and, as well as the best angle for relative body area of interest positioning.

[0103] In embodiments described herein, real time positioning of the area of interest is performed using mixed reality visualization and patient / imaging modality positioning in the angio-suite. A Head Mounted Display (HMD) is shown in FIG. 10, which comprises a housing with optic screen in the screen 701 and a strap 703 for attaching to a person’s head. While a strap is exemplified it is contemplated that any appropriate means manner of situating the HMD on the head of the user or in a manner that the HMD can be viewed by the user can be utilized without departing from the gist and scope of the present disclosure. A camera 501 is attached to the front of the HMD 700. A user of HMD 700 may view real -world objects via asee-through (e.g., transparent) screen, such as see-through holographic lenses, of HMD 700 and also see virtual guidance that appear to be projected on the screen or within the real -world scene, such that the MR guidance object(s) appear to be part of the real -world scene, e.g., with the virtual objects appearing to the user to be integrated with the actual, real-world scene. For example, the virtual guidance may be projected on the screen of an HMD 700, such that the virtual guidance is overlaid on, and appears to be placed within, an actual, observed view of the patient’s actual artery viewed by the user through the transparent screen, e.g., through see- through holographic lenses. Hence, the virtual guidance may be a virtual 3D object that appears to be part of the real -world environment, along with actual, real -world objects.

[0104] FIG. 10 illustrates an exemplary HMD 700 with its components. HMD 700 includes a transparent screen 701 that is positioned at eye level when HMD 700 is worn by a user. In some examples, screen 701 may include one or more liquid crystal displays (LCDs), organic light emitting diode (OLED) displays, or other types of display screens on which images are perceptible to a user who is wearing or otherwise using HMD 700. In some examples, HMD 700 can operate to project 3D images onto the user's retinas using techniques known in the art.

[0105] In some embodiments, screen 701 may include see-through holographic lenses, which are sometimes referred to as waveguides. The see-through holographic lenses permit a user to see real -world objects through (e.g., beyond) the see-through holographic lenses and also see holographic imagery projected into the see-through holographic lenses and from there onto the user’s retinas. The holographic imagery may be projected by displays, such as liquid crystal on silicon (LCoS) display devices, which are sometimes referred to as light engines or projectors, operating as a holographic projection system within HMD 700. Hence, in some embodiments, HMD 700 can project 3D images onto the user's retinas via screen 701. In this manner, HMD 700 may be configured to present a virtual image to a user within a real-world view observed through screen 701, e.g., such that the virtual image appears to form part of the real-world environment. In some embodiments, HMD 700 may be a Microsoft HOLOLENS™ headset, and Apple Vision Pro by Apple® or a similar device, such as, for example, a similar HMD that includes waveguides.

[0106] HMD 700 may generate a user interface (UI) 707 that is visible to the user, e.g., as holographic imagery projected into see-through holographic lenses as described above. UI 707 may include a variety of selectable widgets 708 that allow the user to interact with radiology assistance system 800. HMD 700 also may include other components. For example, HMD 700 may include one or more speakers or other sensory devices 709 that may bepositioned adjacent to the user's ears. Sensory devices 709 may convey audible information or other perceptible information (e.g., vibrations) to assist the user of HMD 700. HMD 700 can also include a transceiver 710 to connect HMD 700 to network 809, such as via a wired or wireless communication channel.

[0107] HMD 700 may also include a variety of sensors to collect sensor data, such as one or more optical camera(s) 501 (or other optical sensors) and one or more depth camera(s) 711 (or other depth sensors), mounted to, on or within frame 706. In some examples, the optical sensor(s) 501 are operable to scan the geometry of the physical environment in which the user of HMD 700 is located (e.g., an operating room) and collect two-dimensional (2D) optical image data (either monochrome or color). Depth sensor(s) 711 are operable to provide 3D image data, such as by employing time of flight, stereo or other known or future-developed techniques for determining depth and thereby generating image data in three dimensions. Other sensors of HMD 700 may include motion sensors 712 (e.g., Inertial Measurement Unit (IMU) sensors, accelerometers, gyroscopes, etc.) to assist with tracking movement.

[0108] HMD 700 may include one or more processors 704 and memory 705, e.g., within frame 706 of HMD 700. In some embodiments, one or more external computing resources 713 process and store information, such as sensor data, instead of or in addition to processor(s) 704 of HMD 700 and memory 705 of HMD 700. Computing system 802 may include external computing resources 713. For instance, external computing resources 713 may include processing circuitry 805 and / or memory 807 of computing system 802. In this way, processor(s) 704 and memory 705 of HMD 700 may perform data processing and storage and / or some of the processing and storage requirements may be offloaded from HMD 700. Hence, in some embodiments, operation of HMD 700 may be controlled in part by a combination one or more processors 704 within HMD 700 and processing circuitry 805 external to HMD 700. In some examples, processor(s) 704 and memory 705 of HMD 700 may provide sufficient computing resources to process the sensor data collected by cameras 501, 711 and motion sensors 712.

[0109] According to embodiments herein, radiological assistance system 800 may provide virtual guidance that may help a user, such as a radiologist, position the head of a patient. For instance, in accordance with some examples, HMD 700 presents an MR scene that includes a virtual angulation guide. In this example, the virtual angulation guide comprises 3D arrows and text. The 3D arrows correspond to the axis of the rotation to be applied to the head of the patient. The text corresponds to the amount of rotation to be applied to the head of the patient. The arrows and the text are close to the head of the patient. In this example, an arrowmay be visually distinguished (e.g., color-coded) based on how close the head is to its target position.

[0110] FIG. 12 depicts an operation block diagram of a radiology assistance system 800, which carries out the process according to embodiments herein. A computing system 802 is configured to perform one or more processes described herein. Computing system 802 may include various types of computing devices, such as server computers, personal computers, smartphones, wearable devices, laptop computers, and other types of computing devices. Computing system 802 includes processing circuitry 805, memory 807, a display 803, a user interface 804, and a communication interface 806. Display 803 may be optional, such as in examples where computing system 802 comprises a server computer.

[0111] Communication interface 806 allows computing system 802 to output data and instructions to and receive data and instructions from an HMD 700 and / or other devices via network 809. Communication interface 806 may comprise hardware circuitry that enables computing system 802 to communicate (e.g., wirelessly or using wires) to other computing systems and devices, such as HMD 700. Network 809 may include various types of communication networks including one or more wide-area networks, such as the Internet, local area networks, and so on. In some examples, network 809 may include wired and / or wireless communication links.

[0112] Radiology assistance system 800 may be configured to cause display 803 and / or HMD 700 to display virtual guidance including one or more virtual guides for performing work. For instance, radiology assistance system 800 may cause display 803 to display virtual guidance, such as 3-dimensional virtual models of arteries or aneurysm and other virtual objects, on display 803 during a preoperative planning phase of a radiology procedure. Radiology assistance system 800 may cause HMD 700 to present an MR scene that includes virtual guidance during an intraoperative phase (i.e., during performance of) the radiology procedure.

[0113] FIG. 9 illustrates one exemplary process, including certain variables that are involved in the exemplary process. According to one set of examples, a planar QR code 401 associated with an absolute frame of reference of the operating room with the C-arms in neutral position is input in the system using a mixed or virtual reality headset Head Mounted Display (HMD). Such an exemplary HMD is equipped with a RGB (red, green, blue) camera that can locate itself in 3D space of the operating room relative to the absolute reference frame by referencing itself to the QR code associated with the operating room 407. This process is denoted as T3 (rigid transformation T3) in FIG.9.

[0114] In FIG. 9, the rigid transformation of the HMD 103 to the reference QR code 401 is denoted as T3. The reverse rigid transformation, or the rigid transformation of the reference QR code 407 to the HMD 103 is denoted as T3'1.

[0115] In frequent embodiments, the HMD used is equipped with Simultaneous Localization and Mapping (SLAM) for three-dimensional space, often referred to as 3D SLAM. SLAM allows a device, typically a robot or a sensor-equipped device, to simultaneously create a map of its environment in three dimensions (3D) while also determining its own position within that environment. The display from HMD equipped with SLAM is often referred to as mixed reality display. In some embodiments, HMDs used are natively equipped with SLAM. In other embodiments, HMDs are equipped with sensors, such as gyroscopes and accelerometers to oriented HMDs in space. Examples of mixed reality display HMD are Apple Vision Pro by Apple® or HoloLens® by Microsoft®. HMD equipped with SLAM may replace pattern markers 407 in defining the absolute reference frame 401.

[0116] FIG. 14 depicts the step of detecting device 405 and displaying it in a MR scene. An HMD detects the device 405 using the process described herein to detect the radio opaque markers to display an MR scene that includes virtual highlighting of device 405. The MR scene displayed is depicted in FIG. 13C, with device 407 being surrounded by a circle 907 viewable in the MR scene.

[0117] FIG. 15 depicts the step of detecting the markers on device 405 and displaying it in a MR scene. An HMD detects the markers printed on device 405 using the process described herein to detect the markers printed on device 405 to display an MR scene that includes virtual highlighting of device 405

[0118] A device 405 carrying a QR code 402 associated with an area of interest, including the physical orientation of that area of interest is then provided within the operating room along with the patient. Device 405 provides a patient frame of reference that can be utilized in calculating and recalculating patient positioning and positioning of the area of interest (e.g., the known or suspected aneurysm). The HMD camera 103 detects the device 405, which can be visualized as a virtual circle around device 405, as shown in FIG. 13C, and the QR code(s) 402 on it and computes the position of the device 405 relative to the position of the HMD 103 and the absolute frame of reference 401. This process is denoted as T2 (rigid transformation T2) in FIG. 9. The HMD 103 thus can monitor changes in device 405 orientation and position in real time.

[0119] In the step illustrated in FIG. 7, the aneurysm 102 has a reference frame oriented on an xyz axis system, this aneurysm frame is denoted as 404 in FIG.9. The position andorientation of the aneurysm 102 relative device 405 was previously calculated in rigid transformation Tl. From there, the position and orientation of aneurysm 102 relative to the HMD 103 are computed. This process is denoted as T2 x Tl'1. Thereafter, the position and orientation of aneurysm 102 relative to the operating room are computed. This process is denoted as rigid transformation T3 x T2 x Tl'1.

[0120] FIG. 16 depicts the MR scene display process of determining the best possible view. After the best possible view is determined using the process described herein, an MR scene that includes virtual head position guides, table position guides, current estimated views and target views is display in the HMD optics. Examples of the MR scene displayed from this process are in FIGS. 13A & 13B.

[0121] Once the position and orientation of aneurysm 102 relative to the operating room and the best position and orientation for the X-ray emitter and receiver are determined, the optimal position and orientation of the body part is computed. The estimated view with the current body part position is displayed via MR, as well as the targeted body part position, as reference. FIG. 13 A depicts a result of this process. A MR display viewable by a user shows an arrow and recommended rotation direction and angle of the patient’s head. FIG. 13B depicts another result of this process. An MR display viewable by a user shows an arrow and recommended lifting direction and distance for the table on which the patient lies. Using sensory cues (3D arrows, sounds, voices), the operator can position the body part (e.g., head, abdomen, etc.) of the patient at the optimal position as computed by this method.

[0122] Other features and advantages of the invention will be apparent from the following detailed description, and from the claims.

[0123] The present invention is further described by the following examples. The examples are provided solely to illustrate the invention by reference to specific embodiments. These exemplifications, while illustrating certain specific aspects of the invention, do not portray the limitations or circumscribe the scope of the disclosed invention.EXAMPLESEXAMPLE 1

[0124] An angio-suite with a reference QR code, labeled as 407 in FIG. 9, is attached to a location in the angio-suite. A patient arrives in the angio-suite and is placed on the operating table. A healthcare professional positions the head of the patient. Registration device 405 is attached to the head of the patient. The healthcare professional wears an HMD and looksat the reference QR code 407 which is detected by the camera 501. The absolute reference frame is then computed. From this point, T3 and T3'1are computed as frequently as possible.

[0125] At least one 3D image of the cerebral vasculature of the patient is captured. The algorithms utilized as described herein are utilized to detect and segment the vascular tree within the 3D image. The algorithms utilized as described herein are utilized to detect the aneurysm within the 3D image using the bounding box and delineating process. The radioopaque marker detection algorithm is utilized to detect the beads 406 and computes the reference frame 408. T1 and Tl'1are computed and stored. The algorithms are utilized to compute the best possible views of the aneurysm for both C arms and stores them as angles. Based on these best possible views, the algorithms compute the optimal position of the aneurysm relative to the C arms in neutral position: this is Topt

[0126] The optimal position of the head relative to the C arms in neutral position is transferred to the HMD. The healthcare professional wearing the HMD looks at the head of the patient and the camera 501 detects and starts tracking device 405 QR codes; T2 and T2'1are computed as frequently as possible. From this point, the position and orientation of the aneurysm is known at any time (Tl^xT^TS’1).

[0127] The healthcare professional wearing the HMD moves the head of the patient while looking at it and device 405 until an optimal position is reached with the orientation of the aneurysm at Topt. This process is aided by visual and other sensitive cues given par the HMD such as, for example, color-coded arrows indicating in which direction to turn the head and by how much.

[0128] During the procedure, each time the head of the patient moves away from Topt, the healthcare professional wearing the HMD is aware of this and can then move the head of the patient until it reaches the optimal position and orientation of the aneurysm Topt as described before.

[0129] If in doubt, the healthcare professional wearing the HMD looks at the head of the patient Toptand device 405 and the algorithm checks the difference between the optimal position of the head of the patient and its actual position and orientation. If this difference is beyond a threshold, a warning is triggered by the HMD and the healthcare professional knows that the patient moved their head. The healthcare professional follows these exemplary procedures to reach the optimal position and orientation.

[0130] During the medical procedure, the healthcare professional wearing the HMD can recall the stored best angles. They can then set the C arms at these angles or have the C arms automatically set potion based on these angles.EXAMPLE 2

[0131] The angio-suite has only one C-arm, instead of two. Computation of the 3D image and recommended movement of the one C-arm and patient positioning are provided using the same system and method as described herein.EXAMPLE 3

[0132] Another body part with an aneurysm present therein can be imaged and adjusted according to the methods and systems described herein. Device 405 is rigidly attached to this body part at or adjacent to the location of the aneurysm, and optimal position and orientation of the body part are calculated using the methods and systems described herein. The position of this body part, the operating table, and the C-arm angle are adjusted accordingly.

[0133] The above examples are included for illustrative purposes only and are not intended to limit the scope of the invention. Many variations to those described above are possible. Since modifications and variations to the examples described above will be apparent to those of skill in this art, it is intended that this invention be limited only by the scope of the appended claims.

[0134] In a first embodiment, a method to adjust position of a body part for operative intervention, the method comprises: positioning a body part of with an area of interest near an imaging device positioned in an operating room; placing a device carrying a QR code adjacent to the area of interest, the device having radio-opaque markers; emitting imaging rays from the imaging ray emitter and receiving imaging rays on an imaging ray receiver; obtaining at least one image of the area of interest; displaying the at least one image of the area of interest on a user interface display; placing virtual 3-dimensinal boundaries surrounding a target area in the at least one image of the area of interest; computing a position of the imaging ray emitter using geometric parameters extracted from the target area in the at least one image of the area of interest; assigning a confidence score of the virtual 3 -dimensional boundaries surrounding a target are in the area of interest; segmenting the target area in the at least one image of the area of interest to delineate the target area in the area of interest;computing an optimal position of the imaging ray emitter to change the confidence score of the virtual 3 -dimensional boundaries surrounding the target area in the area of interest; and changing the position of the imaging ray emitter and / or position of the body part to the optimal position, wherein the imaging ray emitter and the imaging ray receiver are positioned on each end of a C-shape mechanical arm.

[0135] In a second embodiment, the first embodiment includes segmenting the target area in the at least one image of the area of interest is by deep learning, support vector machine, heuristic, linear regression, or random forest strategy.

[0136] In a third embodiment, the first and second embodiment includes segmenting the target area in the at least one image of the area of interest is initialized by user input.

[0137] In a fourth embodiment, the first through third embodiments include wherein the virtual 3-dimensinal boundaries are in a box shape.

[0138] In a fifth embodiment, the first through fourth embodiments include wherein the virtual 3-dimensinal boundaries are cuboid in shape.

[0139] In a sixth embodiment, the first through fifth embodiments include wherein the C-shape mechanical arm further comprises a track along the arm, the track configured for the imaging ray emitter and / or the imaging ray receiver to move along the track.

[0140] In a seventh embodiment, the first through sixth embodiments include wherein there are two C-shape mechanical arms, two imaging ray emitters and two imaging ray receivers, and wherein each pair of imaging ray emitter and corresponding imaging ray receiver is located on opposing ends of each C-shape mechanical arm.

[0141] In a eighth embodiment, the seventh embodiment includes wherein the two C- shape mechanical arms are each attached to an extension at a pivot point and are configured to rotate around the pivot point.

[0142] In a ninth embodiment, the eighth embodiment includes wherein the extension is attached to a frame at a pivot point and is configured to rotate around the pivot point.

[0143] In a tenth embodiment, the eighth embodiment includes wherein the extension is attached to a frame and is configured to slide along the frame.

[0144] In an eleventh embodiment, the first through tenth embodiments include wherein the body part is a human head.

[0145] In a twelfth embodiment, the first through eleventh embodiments include wherein the target area is an aneurysm in an artery in a human brain.

[0146] In a thirteenth embodiment, the first through twelfth embodiments include segmenting at least one additional target area in the area of interest to delineate the at least one additional target area in the area of interest in the at least one image of the area of interest.

[0147] In a fourteenth embodiment, the thirteenth embodiment comprises segmenting at least one additional target area in the area of interest to delineate the at least one additional target area in the area of interest in the at least one image of the area of interest.

[0148] In a fifteenth embodiment, the first through fourteenth embodiments further comprises: obtaining relative position between the target area in the area of interest and the radio opaque markers adjacent the area of interest using rigid transformation step T1 ; obtaining relative position between the QR code adjacent the area of interest and a mixed reality headset camera using rigid transformation step T2; obtaining relative position between the mixed reality headset camera and a reference QR code in the operating room using rigid transformation step T3; obtaining relative position between the target area in the area of interest and the mixed reality headset camera using reverse rigid transformation step IT1and rigid transformation step T2; obtaining relative position between the target area in the area of interest and the operating room using reverse rigid transformation step IT1, rigid transformation step T2, and rigid transformation step T3.

[0149] In a sixteenth embodiment, the fifteenth embodiment further comprises computing a position and orientation of the body part using the relative position between the target area in the area of interest and the operating room and the optimal position of the imaging ray emitter.

[0150] In a seventeenth embodiment, the sixteenth and seventeenth embodiments further comprises moving the body part to the computed position and orientation.

[0151] In an eighteenth embodiment, a system to adjust position of a body part for operative intervention is provided, the system comprises: a room suitable for medical operation on a subject; a table for a subject to rest on; at least one imaging ray emitter and at least one imaging ray receiver, the at least one imaging ray emitter and the at least one imaging ray receiver is each attached to an end of at least one C-shape mechanical arm; a device carrying at least one QR code and having radio opaque markers for placement adjacent an area of interest on the subject;a computing article operatively connected to at least one user interface display; a computer executable code adapted to direct and / or perform the steps of receiving and displaying at least one image of an area of interest from the imaging ray emitter and imaging ray receiver; placing virtual 3-dimensinal boundaries surrounding a target area in the at least one image of the area of interest; computing a position of the imaging ray emitter and imaging ray receiver using geometric parameters extracted from the target area in the at least one image of the area of interest; assigning a confidence score to the target area in the at least one image of the area of interest; segmenting the target area in the at least one image of the area of interest to delineate the target area in the area of interest; computing an optimal position of the imaging ray emitter and imaging ray receiver to change the confidence score of the target are in the area of interest; and changing the position of the imaging ray emitter to the optimal position.

[0152] In a nineteenth embodiment, the eighteenth embodiment includes wherein there are two C-shape mechanical arms, with each C-shape mechanical arm having one imaging ray emitter and one imaging ray receiver attached to each end of the C-shape mechanical arm

[0153] In a twentieth embodiment, the nineteenth embodiment further comprises a reference QR code attached to the operating room and a mixed reality headset equipped with a RGB camera.

[0154] In a twenty first embodiment, the nineteenth embodiment includes wherein the computer executable code is further configured to perform the steps of: obtaining relative position between the target area in the area of interest and the radio opaque markers using rigid transformation step Tl; obtaining relative position between the QR code and a mixed reality headset camera using rigid transformation step T2; obtaining relative position between the mixed reality headset camera and a reference QR code in the operating room using rigid transformation step T3; obtaining relative position between the target area in the area of interest and the mixed reality headset camera using reverse rigid transformation step Tl'1and rigid transformation step T2;obtaining relative position between the target area in the area of interest and the operating room using reverse rigid transformation step IT1, rigid transformation step T2, and rigid transformation step T3.

[0155] In a twenty second embodiment, the twenty first embodiment includes wherein the computer executable code is further configured to perform the step of computing a position and orientation of the body part using the relative position between the target area in the area of interest and the operating room and the optimal position of the imaging ray emitters.

[0156] In a twenty third embodiment, the twenty first embodiment includes wherein the computer executable code is further configured to perform the step of segmenting at least one additional target area in the area of interest to delineate the at least one additional target area in the at least one image of the area of interest.

[0157] In a twenty fourth embodiment, the eighteenth embodiment includes wherein the body part is a human head.

[0158] In a twenty fifth embodiment, the eighteenth embodiment includes wherein the at least one area of interest is an aneurysm.

[0159] In a twenty sixth embodiment, the twenty third embodiment includes wherein the at least one additional area of interest is a vascular tree in a human head.

[0160] In a twenty seventh embodiment, a method to obtain relative position between a target area and a reference point is provided, the method comprises: placing a device carrying a QR code adjacent to an area of interest having a target area, the device having radio-opaque markers; obtaining relative position between the target area in the area of interest and the radio opaque markers adjacent the area of interest using rigid transformation step T1 ; obtaining relative position between the QR code adjacent the area of interest and a mixed reality headset camera using rigid transformation step T2; obtaining relative position between the mixed reality headset camera and a reference QR code using rigid transformation step T3; relative position between the target area in the area of interest and the mixed reality headset camera using reverse rigid transformation step IT1and rigid transformation step T2; obtaining relative position between the target area in the area of interest and reference point using reverse rigid transformation step IT1, rigid transformation step T2, and rigid transformation step T3.

[0161] Citation of the above publications or documents is not intended as an admission that any of the foregoing is pertinent prior art, nor does it constitute any admission as to the contents or date of these publications or documents.

Claims

CLAIMSWe claim:

1. A method to adjust position of a body part for operative intervention, the method comprises: positioning a body part of with an area of interest near an imaging device positioned in an operating room; placing a device carrying a QR code adjacent to the area of interest, the device having radio-opaque markers; emitting imaging rays from the imaging ray emitter and receiving imaging rays on an imaging ray receiver; obtaining at least one image of the area of interest; displaying the at least one image of the area of interest on a user interface display; placing virtual 3-dimensinal boundaries surrounding a target area in the at least one image of the area of interest; computing a position of the imaging ray emitter using geometric parameters extracted from the target area in the at least one image of the area of interest; assigning a confidence score of the virtual 3 -dimensional boundaries surrounding a target are in the area of interest; segmenting the target area in the at least one image of the area of interest to delineate the target area in the area of interest; computing an optimal position of the imaging ray emitter to change the confidence score of the virtual 3 -dimensional boundaries surrounding the target area in the area of interest; and changing the position of the imaging ray emitter and / or position of the body part to the optimal position, wherein the imaging ray emitter and the imaging ray receiver are positioned on each end of a C-shape mechanical arm.

2. The method of claim 1, wherein segmenting the target area in the at least one image of the area of interest is by deep learning, support vector machine, heuristic, linear regression, or random forest strategy.

3. The method of claim 1, wherein segmenting the target area in the at least one image of the area of interest is initialized by user input.

4. The method of claim 1, wherein the virtual 3-dimensinal boundaries are in a box shape.

5. The method of claim 1, wherein the virtual 3-dimensinal boundaries are cuboid in shape.

6. The method of claim 1, wherein the C-shape mechanical arm further comprises a track along the arm, the track configured for the imaging ray emitter and / or the imaging ray receiver to move along the track.

7. The method of claim 1, wherein there are two C-shape mechanical arms, two imaging ray emitters and two imaging ray receivers, and wherein each pair of imaging ray emitter and corresponding imaging ray receiver is located on opposing ends of each C-shape mechanical arm.

8. The method of claim 7, wherein the two C-shape mechanical arms are each attached to an extension at a pivot point and are configured to rotate around the pivot point.

9. The method of claim 8, wherein the extension is attached to a frame at a pivot point and is configured to rotate around the pivot point.

10. The method of claim 8, wherein the extension is attached to a frame and is configured to slide along the frame.

11. The method of claim 1, wherein the body part is a human head.

12. The method of claim 1, wherein the target area is an aneurysm in an artery in a human brain.

13. The method of claim 1, further comprising segmenting at least one additional target area in the area of interest to delineate the at least one additional target area in the area of interest in the at least one image of the area of interest.

14. The method of claim 12, wherein the at least one additional target area is a vascular tree within a human brain.

15. The method of claim 1, further comprising: obtaining relative position between the target area in the area of interest and the radio opaque markers adjacent the area of interest using rigid transformation step Tl; obtaining relative position between the QR code adjacent the area of interest and a mixed reality headset camera using rigid transformation step T2; obtaining relative position between the mixed reality headset camera and a reference QR code in the operating room using rigid transformation step T3; obtaining relative position between the target area in the area of interest and the mixed reality headset camera using reverse rigid transformation step Tl'1and rigid transformation step T2; obtaining relative position between the target area in the area of interest and the operating room using reverse rigid transformation step Tl'1, rigid transformation step T2, and rigid transformation step T3.

16. The method of claim 15, further comprising computing a position and orientation of the body part using the relative position between the target area in the area of interest and the operating room and the optimal position of the imaging ray emitter.

17. The method of claim 16, further comprising moving the body part to the computed position and orientation.

18. A system to adjust position of a body part for operative intervention, the system comprises: a room suitable for medical operation on a subject; a table for a subject to rest on;at least one imaging ray emitter and at least one imaging ray receiver, the at least one imaging ray emitter and the at least one imaging ray receiver is each attached to an end of at least one C-shape mechanical arm; a device carrying at least one QR code and having radio opaque markers for placement adjacent an area of interest on the subject; a computing article operatively connected to at least one user interface display; a computer executable code adapted to direct and / or perform the steps of: receiving and displaying at least one image of an area of interest from the imaging ray emitter and imaging ray receiver; placing virtual 3-dimensinal boundaries surrounding a target area in the at least one image of the area of interest; computing a position of the imaging ray emitter and imaging ray receiver using geometric parameters extracted from the target area in the at least one image of the area of interest; assigning a confidence score to the target area in the at least one image of the area of interest; segmenting the target area in the at least one image of the area of interest to delineate the target area in the area of interest; computing an optimal position of the imaging ray emitter and imaging ray receiver to change the confidence score of the target are in the area of interest; and changing the position of the imaging ray emitter to the optimal position.

19. The system of claim 18, wherein there are two C-shape mechanical arms, with each C-shape mechanical arm having one imaging ray emitter and one imaging ray receiver attached to each end of the C-shape mechanical arm.

20. The system of claim 19, further comprising a reference QR code attached to the operating room and a mixed reality headset equipped with a RGB camera.

21. The system of claim 19, wherein the computer executable code is further configured to perform the steps of: obtaining relative position between the target area in the area of interest and the radio opaque markers using rigid transformation step Tl;obtaining relative position between the QR code and a mixed reality headset camera using rigid transformation step T2; obtaining relative position between the mixed reality headset camera and a reference QR code in the operating room using rigid transformation step T3; obtaining relative position between the target area in the area of interest and the mixed reality headset camera using reverse rigid transformation step IT1and rigid transformation step T2; obtaining relative position between the target area in the area of interest and the operating room using reverse rigid transformation step IT1, rigid transformation step T2, and rigid transformation step T3.

22. The system of claim 21, wherein the computer executable code is further configured to perform the step of computing a position and orientation of the body part using the relative position between the target area in the area of interest and the operating room and the optimal position of the imaging ray emitters.

23. The system of claim 21, wherein the computer executable code is further configured to perform the step of segmenting at least one additional target area in the area of interest to delineate the at least one additional target area in the at least one image of the area of interest.

24. The system of claim 18, wherein the body part is a human head.

25. The system of claim 18, wherein the at least one area of interest is an aneurysm.

26. The system of claim 23, wherein the at least one additional area of interest is a vascular tree in a human head.

27. The method to obtain relative position between a target area and a reference point, comprising: placing a device carrying a QR code adjacent to an area of interest having a target area, the device having radio-opaque markers; obtaining relative position between the target area in the area of interest and the radio opaque markers adjacent the area of interest using rigid transformation step Tl;obtaining relative position between the QR code adjacent the area of interest and a mixed reality headset camera using rigid transformation step T2; obtaining relative position between the mixed reality headset camera and a reference QR code using rigid transformation step T3; obtaining relative position between the target area in the area of interest and the mixed reality headset camera using reverse rigid transformation step IT1and rigid transformation step T2; obtaining relative position between the target area in the area of interest and reference point using reverse rigid transformation step IT1, rigid transformation step T2, and rigid transformation step T3.

Images