Medical procedures using augmented reality

Abstract

The present disclosure provides a method for performing a medical procedure using augmented reality that superimposes a medical image (e.g., CT or MRI) of a patient on a real-time camera view of the patient. Prior to the medical procedure, the medical image of the patient is processed to generate a 3D model representing the skin contours of the patient's body. The 3D model is further processed to generate skin markers that include only selected portions of the 3D model. During the medical procedure, a 3D image of the patient's body is captured using a camera, and the 3D image is then registered with the skin markers. The medical image of the patient can then be superimposed on the real-time camera view, which is presented to a person performing the medical procedure.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Patent Application No. 16 / 863,498, filed on April 30, 2020, U.S. Patent Application No. 17 / 067,509, filed on October 9, 2020, and Korean Patent Application No. 10-2019-0136031, filed on October 30, 2019, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present application relates to the medical industry, and more particularly to medical procedures using augmented reality. Background Art

[0004] The medical industry has a high interest in improving techniques for visualizing the inside of a patient during medical procedures. Today, various imaging techniques, such as computed tomography (CT) scans and magnetic resonance imaging (MRI), are available to allow physicians to see organs, bones, and other tissues inside a patient's body. Summary of the invention

[0005] The present disclosure provides a method for performing a medical procedure using augmented reality that superimposes a medical image (e.g., CT or MRI) of a patient on a real-time camera view of the patient. Prior to the medical procedure, the medical image of the patient is processed to generate a 3D model representing the skin contours of the patient's body. The 3D model is further processed to generate skin markings that include only selected portions of the 3D model. During the medical procedure, a camera is used to capture a photograph or video of the patient's body (also referred to herein, individually or collectively, as an "image"), and then the camera-captured image is aligned with the skin marking. Since the skin marking is already aligned with the patient's medical image, aligning the camera-captured image with the skin marking will also align the patient's medical image with the real-time camera view presented on a display device. The patient's medical image can then be superimposed on the real-time camera view.

[0006] According to one aspect of the present disclosure, a method for a medical procedure includes: before performing the medical procedure for treating or examining a target tissue of a patient, processing medical image data to generate a skin 3D model that defines the 3D contours of the patient's skin, the medical image data including multiple 2D cross-sectional images of the patient's body covering the target tissue; still before the medical procedure, processing the skin 3D model to generate skin markings, wherein the processing of the skin 3D model includes removing one or both of the following items: (1) a portion of the skin 3D model corresponding to a skin area of ​​the patient, the skin area will contact or directly face a support surface on which the patient will lie during the medical procedure, and (2) a portion of the skin 3D model corresponding to a skin area of ​​the patient adjacent to a joint of the patient's body, the skin The area will significantly change its position relative to the target tissue and / or significantly change its contour when the patient's body part moves about the joint, wherein the removal provides the skin marker; at or during the medical procedure, using a handheld computing device to capture an image of the patient lying on a hospital bed; still at or during the medical procedure, processing the captured image of the patient and the skin marker to align them to achieve augmented reality without using artificial physical markers fixed to or placed on the patient's body; and still at or during the medical procedure, presenting a real-time augmented reality view on the display screen of the handheld computing device, the real-time augmented reality view including at least one of the multiple 2D cross-sectional images of the patient's body superimposed on one or more additional real-time images of the patient captured after the alignment.

[0007] According to another aspect, the processing of the skin 3D model includes removing both: (1) a portion of the skin 3D model corresponding to a skin area of ​​the patient, which skin area will contact or directly face a support surface on which the patient will lie to perform the medical procedure, and (2) a portion of the skin 3D model corresponding to a skin area of ​​the patient adjacent to a joint of the patient's body, which skin area will significantly change its position relative to the target tissue and / or significantly change its contour when the patient's body part moves around the joint.

[0008] According to another aspect, the patient's body includes a target body part covering the target tissue and a non-target body part connected to the target body part via a joint, wherein the skin 3D model defines a 3D contour of the patient's skin at the target body part and a 3D contour of the patient's skin at the non-target body part, wherein the processing of the skin 3D model includes further removing the 3D contour of the patient's skin at the non-target body part.

[0009] According to another aspect, the patient's body includes a target body part covering the target tissue and a non-target body part connected to the target body part via a joint, wherein the skin 3D model defines a 3D contour of the patient's skin at the target body part, while not defining a 3D contour of the patient's skin at the non-target body part.

[0010] According to another aspect, the patient's body includes a target body part covering the target tissue and a non-target body part connected to the target body part via a joint, wherein the skin 3D model defines a 3D contour of the patient's skin at the target body part and a 3D contour of the patient's skin at the non-target body part, wherein the processing of the skin 3D model includes further removing the 3D contour of the patient's skin at the non-target body part.

[0011] According to another aspect, the medical procedure includes open surgery, wherein in the real-time augmented reality view, at least one of the multiple 2D cross-sectional images of the patient's body showing the target tissue is displayed on the display screen over the one or more additional real-time images of the patient for the surgeon to determine an incision location on the patient's body.

[0012] According to another aspect, the method further comprises, prior to the medical procedure, processing the medical image data to generate a tissue 3D model defining a 3D contour surrounding the target tissue, wherein the tissue 3D model is located within the skin 3D model, wherein the real-time augmented reality view further comprises the tissue 3D model superimposed on one or more additional captured images of the patient.

[0013] According to another aspect, the processing of the skin 3D model comprises further removing a portion of the skin 3D model corresponding to a skin area of ​​the patient which will be covered by a medical drape during the medical procedure and which will therefore not be exposed.

[0014] According to another aspect, the processing of the skin 3D model comprises further removing a portion of the skin 3D model corresponding to a skin area of ​​the patient which will significantly change its contour before and after the patient eats.

[0015] According to another aspect, the skin marking comprises at least two portions that are not adjacent to each other and correspond to skin areas of the patient.

[0016] According to another aspect, the method further comprises placing a medical cover on the patient lying on the hospital bed in a manner that exposes a skin portion of the patient's body, wherein the placing of the medical cover is performed before or after capturing the image of the patient.

[0017] According to another aspect, the method further includes placing a medical cover on the patient lying on the hospital bed while exposing a skin area, wherein the placing of the medical cover is performed after the capturing of the image of the patient, and wherein the skin marking does not include a portion of the skin 3D model corresponding to the exposed skin area.

[0018] According to another aspect, the skin marking does not include a portion of the 3D skin model that corresponds to a skin area of ​​the patient that is closest to the target tissue.

[0019] According to another aspect, the processing of the skin 3D model comprises further removing at least one portion of the skin 3D model corresponding to at least one skin area of ​​a body part of the patient selected from the group consisting of abdomen, axilla and a limb.

[0020] According to another aspect, the processing of the skin 3D model includes further removing at least one portion of the skin 3D model corresponding to at least one skin area of ​​a body part of the patient, the body part of the patient being selected from the group consisting of a pubic area, an area of ​​densely gathered hair, and a mouth.

[0021] According to another aspect, the processing of the skin 3D model includes retaining a portion of the skin 3D model corresponding to a cheekbone of the patient.

[0022] According to another aspect, the processing of the skin 3D model comprises retaining a portion of the skin 3D model corresponding to an ear of the patient.

[0023] According to another aspect, a non-transitory storage medium is provided, the non-transitory storage medium storing a plurality of instructions executable by a computer, wherein the plurality of instructions, when executed, cause the computer to: receive medical image data, the medical image data comprising a plurality of 2D cross-sectional images of a patient's body covering a target tissue; process the medical image data to generate a 3D skin model defining a 3D contour of the patient's skin; receive information related to the target tissue and information related to a proposed medical procedure for treating or examining the target tissue; identify one or both of the following: (1) a first skin area of ​​the patient that will contact or directly face a lying area on which the patient will lie during the proposed medical procedure; and a lying surface, and (2) a second skin area of ​​the patient adjacent to a joint of the patient's body, which second skin area will significantly change its position relative to the target tissue and / or significantly change its contour when the patient's body part moves about the joint; and removing one or both of the following items from the skin 3D model: (1) a first portion of the skin 3D model corresponding to the first skin area; and (2) a second portion of the skin 3D model corresponding to the second skin area, thereby providing skin markers for future real-time alignment with images of the patient captured during the proposed medical procedure without the use of artificial physical markers fixed to or placed on the patient's body.

[0024] According to another aspect, the computer is a handheld computing device, wherein the plurality of instructions, when executed, further cause the handheld computing device to: capture the image of the patient while the patient is lying on a hospital bed; process the captured image of the patient and the skin markers to align them to achieve augmented reality without using artificial physical markers fixed to the patient's body or worn by the patient; and present a real-time augmented reality view on the handheld computing device, the real-time augmented reality view including at least one of the plurality of 2D cross-sectional images of the patient's body superimposed on one or more additional images of the patient captured after the alignment.

[0025] According to another aspect, the plurality of instructions, when executed, further cause the handheld computing device to present on the handheld computing device one or both of: (a) a mesh of the skin 3D model; and (b) a mesh of the skin markings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The foregoing and other features, aspects and advantages of the present invention are described in detail below with reference to various implementations of the accompanying drawings, which are intended to illustrate rather than limit the present invention. The accompanying drawings include the following figures, in which:

[0027] Figure 1A illustrates operation of a computing device by a physician to view a medical image of a patient superimposed on a real-time camera view displayed on the computing device according to an example implementation;

[0028] Figure 1B shows a segmentation process for extracting skin contours from a medical image according to an example implementation;

[0029] Figure 1C shows how to combine according to an example implementation Figure 1B The skin contour in the image is used to generate a 3D model;

[0030] Figure 2A According to an example implementation, Figure 1B The set of pixels extracted during the segmentation process;

[0031] Figure 2B According to an example implementation, Figure 2A A downsampled version of

[0032] Figure 3A shows the use of Figure 2B A 3D model generated by a collection of pixels in;

[0033] Figure 3B According to an example implementation, Figure 3A A smoothed version of

[0034] Figure 4A According to an example implementation, Figure 3B The result of removing the patient's back from the 3D model;

[0035] Figure 4B According to an example implementation, Figure 4A The result of removing the patient's arm from the 3D model;

[0036] Figure 5A According to an example implementation, Figure 4B The result of removing the patient's abdomen from the 3D model;

[0037] Figure 5B According to an example implementation, Figure 5A The result of removing the left chest part of the patient from the 3D model;

[0038] Figure 5C shows an image of a patient taken on a surgical table according to an example implementation;

[0039] Figure 6A flow chart illustrating a method for preprocessing a medical image and generating skin markings according to an example implementation is shown;

[0040] Figure 7 A flow chart illustrating a method for processing an image of a patient and overlaying a medical image on a live camera view according to an example implementation is shown;

[0041] Figure 8 An implementation of a computing device according to an example implementation is shown. DETAILED DESCRIPTION

[0042] Implementations of the present invention will now be described with reference to the accompanying drawings. The terms used in the specification presented herein are not intended to be interpreted in any restrictive or limiting manner, simply because these terms are used in conjunction with the detailed description of certain specific implementations of the present invention. In addition, implementations of the present invention may include several novel features, no single feature of which is entirely responsible for its desired attributes, or no single feature is essential to practicing the invention described herein.

[0043] The use of augmented reality (AR) in the medical field

[0044] Augmented reality (AR) refers to the insertion of virtual objects into real scenes. There are many attempts to provide augmented reality in the medical field. For example, a medical image of a patient (such as a CT or MRI image) (virtual object) is superimposed with a camera view of the patient (real scene) to achieve AR, which neither the medical image alone nor the camera view alone would provide. Medical professionals can use this AR in the context of medical procedures to understand the location and condition of tissue of interest under the skin without opening the patient's body for the procedure or before it.

[0045] Registration

[0046] A key issue in AR is the correct alignment of virtual objects with the real scene, which is called "registration". To facilitate registration, artificial objects can be fixed to the patient's body so that the artificial objects are included in the CT or MRI image and are also included in the camera view. The artificial objects are then used as references called markers to align the two. Alternatively, in some AR algorithms, artificial objects are not used.

[0047] Registration without manual landmarks

[0048] When taking CT or MRI images and also when not using artificial objects during medical procedures, registration relies on medical images of the patient obtained before the medical procedure and the images captured by the camera during the medical procedure. This can require a lot of computing power and a lot of time. However, in reality, AR for medical procedures requires real-time performance.

[0049] Registration using pre-generated landmarks

[0050] According to an implementation of the present invention, registration of a patient's CT or MRI image with a camera view of the patient is performed without fixing an artificial object to the patient's body. Prior to the proposed medical procedure, the patient's CT or MRI image is processed to provide a body contour model (also referred to herein as a "3D model" or "skin 3D model"), which represents the skin contour of the patient's body or a portion thereof. The body contour model is processed to provide skin markers, which are one or more portions of the body contour model selected for effective registration. In the above process, the patient's CT or MRI image has been aligned with the body contour model and, therefore, also with the skin markers. During the medical procedure, a 3D image of the patient's body or a portion thereof is captured using a camera. The 3D image captured when the procedure is performed is then registered with the pre-prepared skin markers. Since the skin markers have been aligned with the patient's CT or MRI image, the registration between the 3D image and the skin model will correctly align the patient's CT or MRI image with the camera view presented on the camera. The patient's CT or MRI image (virtual object) can then be superimposed with the camera view (real scene).

[0051] After successful registration, the medical image can be displayed on the camera view

[0052] Figure 1A An example use case is shown after the registration described above has been completed using the techniques of the present disclosure, which will be referred to below. Figures 1B to 7 Describe in more detail. Figure 1A , physician 101 is operating a computing device 102 having a camera pointed at a patient 103 lying on a surgical table. Computing device 102 displays a real-time camera view 104 of patient 103 and displays a medical image 106 as superimposed on a particular section of the patient's body in real-time camera view 104. Computing device 102 also displays a plurality of user interface controls 108 that physician 101 can use to navigate to other medical images of the patient corresponding to other sections of the patient's body in real-time camera view 104. Although Figure 1ANot shown, but the medical image superimposed on the real-time camera view may indicate the presence and specific location of a lesion beneath the patient's skin and / or the distance from the patient's skin to the location of the lesion. Using the information provided by the medical image superimposed on the real-time camera view, physician 101 may more efficiently prescribe a medical procedure (e.g., make appropriate landmarks on the patient's body, decide where to make an incision, etc.) than if such medical image had to be viewed on a separate screen that did not also display a real-time view of the patient.

[0053] Medical procedures

[0054] As used herein, the phrase "medical procedure" refers to any procedure performed by a medical professional, such as a physician. The techniques described herein can be applied in a variety of different medical and surgical environments. For example, in some implementations, these techniques can be applied to medical procedures, such as surgical procedures in a patient's brain, liver, pancreas, abdominal organs and blood vessels, soft organs such as breasts, and other applications such as endoscopy, bronchoscopy, and the like.

[0055] Medical imaging

[0056] As used herein, the phrase "medical image" refers to any image of a patient generated prior to a medical procedure and referenced by a physician in preparation for or during a medical procedure. Although CT and MRI images are used herein as examples, in other implementations, other images may be used.

[0057] Computing Devices

[0058] Figure 1A The computing device 102 used by the physician 101 in the present invention may include any computing device having a camera and a display, including but not limited to: a smartphone, a tablet computer, a head-mounted display, a desktop computer, a laptop computer, a wearable device, etc. As described in more detail below, the pre-generated skin markers simplify the registration process, so that the registration and superposition can be performed in real time by a portable handheld computing device. In some implementations, the computing device 102 does not include one or both of a camera and a display, and communicates with such one or more components via a wired or wireless connection to implement the techniques described herein.

[0059] Registration involves aligning coordinate systems

[0060] The registration described herein involves aligning the coordinate system used by the 3D model (also referred to herein as the "3D model coordinate system") with the coordinate system used by computing device 102 (also referred to herein as the "computing device coordinate system"), such that specific locations within the real-time camera view provided by computing device 102 can be associated with corresponding locations in the 3D model and, therefore, with the corresponding medical image, and further such that changes in the camera position and angle will result in corresponding changes in the size and angle of the medical image superimposed on the camera view.

[0061] Improved registration involving both pre-processing and runtime processing

[0062] Registering the 3D model coordinate system to the computing device coordinate system and overlaying the medical image on the real-time camera view may include, for example, steps performed before preparing the patient for the medical procedure (pre-processing), such as generating a 3D model using the medical image of the patient and generating skin markers using the 3D model by removing specific portions of the 3D model; and steps performed while preparing the patient for the medical procedure or during the medical procedure (run-time processing), such as aligning the pre-generated skin markers with an image of the patient (e.g., an image of the patient lying on a surgical table) and displaying the medical image as overlaid on the real-time camera view of computing device 102. Because the 3D model and skin markers are generated in advance (rather than at runtime), using the techniques described herein, the time and computing resources required to perform the registration and overlay at runtime can be reduced, thereby achieving and improving the real-time experience provided to the operator.

[0063] Start preprocessing of medical images

[0064] One or more medical images (such as CT or MRI images) of a patient may be taken to visualize the interior of the patient's body. If the condition shown by the medical image requires a medical procedure that would benefit from superimposing the medical image on a real-time view of the patient, pre-processing of the medical image may be initiated to facilitate such superimposition of the medical image, an example of which is shown in Figure 1A middle.

[0065] Preprocessing of medical images

[0066] In some implementations, preprocessing of the medical images includes a segmentation process during which a skin contour of the patient's body (e.g., a contour of the outer surface of the patient shown in the medical image) is extracted from each of the medical images (e.g., Figure 1B ); and a 3D modeling process during which the extracted skin contour is used to generate a 3D point cloud (e.g., as Figure 2B 3D point cloud is also referred to as “3D model” in this article.

[0067] Segmentation process

[0068] like Figure 1B As shown, according to a certain implementation, the segmentation process may include filtering, representing the filtered data in a histogram, thresholding to identify the portion of the filtered data corresponding to the patient's body, refining the segmentation, and performing edge extraction. Although specific methods and algorithms are used as examples below, it will be appreciated by those of ordinary skill in the art that other techniques may be used. One of the multiple steps described herein may be omitted or exchanged with another step, and one or more additional steps not described herein may be added.

[0069] Filtering

[0070] The filtering step reduces noise in the data included in the medical image (also referred to herein as "medical image data"). Gaussian filtering, anisotropic diffusion filtering (ADF) and / or other techniques may be used.

[0071] Thresholding

[0072] The thresholding step uses a specific threshold value (e.g., pixel intensity level) to segment the image. For example, the thresholding step may include representing the filtered medical image data in a histogram using the Otsu method and distinguishing portions of the medical image data corresponding to the patient's body from portions of the medical image data corresponding to the background (e.g., areas outside the patient's body or empty space). Although the Otsu method is used as an example, other techniques may be used.

[0073] Refine segmentation

[0074] The refinement step refines the segmented regions. For example, the refinement step may include performing seeded region growing (SRG) segmentation and morphological segmentation. For example, the SRG segmentation and morphological segmentation may be performed sequentially. Although SRG and morphological segmentation are used as examples, other techniques may be used.

[0075] Edge extraction

[0076] The edge extraction step extracts the skin contour from the refined segmented area. Figure 1B As shown, from Figure 1B In the refined segmented region shown in the fifth image of FIG, an outer edge corresponding to the skin contour of the patient can be extracted for a specific cross section of the patient's body corresponding to the medical image. For example, any known edge detection and extraction algorithm can be used.

[0077] 3D Modeling

[0078] The 3D modeling process includes combining the skin contours extracted from the segmentation process described above (e.g., the outer edges extracted from each medical image) to generate a 3D model (e.g., Figure 2A and Figure 2B The following reference FIG. 2A to FIG. 3B Describe the 3D modeling process in more detail.

[0079] A collection of vertices

[0080] Figure 2A According to an example implementation, Figure 1B A point cloud is a collection of vertices extracted during the segmentation process of (e.g., corner points in the skin contour of each medical image). For example, the skin contour extracted from the segmentation process may include multiple vertices, and a point cloud may be generated by combining all vertices from all medical images that have been preprocessed as described herein. Figure 2A The point cloud is shown in .

[0081] Vertex coordinates

[0082] Each vertex in the point cloud may be associated with a set of coordinate values ​​in a 3D coordinate system (e.g., a set of x-axis, y-axis, and z-axis coordinate values ​​in a coordinate system associated with a medical image). In other implementations, each vertex is not associated with an explicit set of coordinate values, and the position of the vertex is determined based on its relative position in a data structure (e.g., a point cloud).

[0083] Vertex Downsampling

[0084] Vertices in the point cloud may be downsampled to reduce the amount of data points and computations, and to reduce the complexity of generating a 3D model based on the vertices in the point cloud. Figure 2B Shows Figure 2A A downsampled version of the point cloud is shown. For example, the number of vertices may be reduced by about one-tenth (eg, from about 100,000 vertices to about 10,000).

[0085] Normal vector

[0086] Figure 2A The normal vector of each vertex included in the illustrated point cloud may be calculated for use in generating a 3D model, for example, by filling in the gaps between vertices in the downsampled point cloud.

[0087] Meshing

[0088] Meshing facilitates visualization of 3D models by generating 3D geometric objects using vertices and / or vectors of the 3D model (e.g., 3D point cloud). Figure 1C, Figure 3A , Figure 3B , Figure 4A , Figure 4B , Figure 5A and Figure 5B As shown, the meshing process makes it easier to visualize the 3D model or the skin markers generated based on the 3D model.

[0089] Grid Instance

[0090] Figure 3A Shown based on Figure 2B For example, according to an example implementation, the mesh may be generated using Figure 2B The 3D model shown in FIG. 1 is generated by using a 3D discrete scalar field (also referred to herein as voxel data) of the 3D model. For example, a marching cube algorithm can be used to generate the mesh. In some implementations, a fast quadratic mesh simplification algorithm can be used to simply generate the mesh, thereby reducing the consumption of computing resources. In other implementations, one or more other algorithms (e.g., mesh extraction, mesh simplification, etc.) are used instead.

[0091] smooth

[0092] Smoothing is a technique used to capture salient patterns in a dataset without taking into account noise or other meso-scale properties. Figure 3B According to an example implementation, Figure 3A For example, a Gaussian filter can be used to smooth Figure 3A The grid shown. In other implementations, one or more other smoothing filters are used.

[0093] Using meshes to illustrate 3D models

[0094] The 3D models described herein may be illustrated using meshes generated based on the 3D models. For example, FIG. 3A to FIG. 5B Each shows a mesh corresponding to an underlying 3D model (or a portion thereof, which is also referred to herein as a "skin marker").

[0095] Generation of 3D tissue models

[0096] Using the same or similar process described above for generating a 3D model representing the patient's skin contour, another 3D model representing a tissue or region of interest inside the patient's body can be generated. This 3D model may be referred to herein as a "tissue 3D model".

[0097] Additional details on segmentation, meshing, smoothing, matching, and registration

[0098] Additional details related to segmentation, meshing, smoothing, matching, registration, and other techniques described herein are provided, for example, in the following: International Application Publication No. WO 2005 / 088539, entitled “AUGMENTED REALITY SYSTEM WITH COREGISTRATION OF VIRTUAL OBJE CTS ON IMAGES OF REAL OBJECTS”; U.S. Patent No. 9,498,132, entitled “VISUALIZATION OF ANATOMICAL DATA BY A UGMENTED REALITY”; U.S. Patent Application Publication No. 2018 / 0114325, entitled “SYSTEMS AND METHODS FOR DETERMINING QUALITY METRICS OF AN IMAGE OR IMAGES BASED ON AN EDGE GRADIENT PR OFILE AND CHARACTERIZING REGIONS OF INTEREST IN AN IMAGE OR IMAGES”; and U.S. Patent Application Publication No. 2018 / 0114325, entitled “KEYPOINT-BASED POINT-PAIR-FEATURE FOR SCALABLE AUTOMATIC GLOBAL REGISTRATION OF LARGE RGB-D SCANS”; and U.S. Patent No. 10,603,118, entitled “METHOD FOR RECOVERING PATIENT REGISTRATION”, the disclosures of which are incorporated herein by reference in their entirety for all purposes. In addition to the techniques described herein and the foregoing disclosure, other techniques known to those of ordinary skill in the art may also be used.

[0099] Generate skin markers from 3D models

[0100] Using the 3D model generated from the medical image via the process described above, computing device 102 generates skin markers before preparing the patient for the medical procedure. Pre-generated skin markers can speed up the registration process and improve the accuracy of the registration by removing portions of the 3D model that would interfere with or delay the registration process performed at runtime (e.g., during preparation for or during the medical procedure), thereby providing real-time performance suitable for medical procedures.

[0101] Which one generates the skin tag?

[0102] In some implementations, a computing device generates a skin marker based on input from a medical professional. For example, a 3D model may be displayed on a display screen of a computing device, and the medical professional may select portions of the 3D model to be removed via a touch screen control or a physical control. The portion of the 3D model remaining after all removals may be used as a skin marker. In other implementations, automated computing software generates the skin marker. For example, automated computing software may determine which portions of the 3D model should be removed based on a medical image and / or a 3D model and remove those portions, thereby generating a skin marker. The portions of the 3D model that a medical professional and / or automated computing software may decide to remove from the 3D model will be described in more detail below.

[0103] Skin tags are not artificial

[0104] In some implementations, the skin markers do not include markers artificially introduced into the medical image. For example, the skin markers are generated by retaining or removing portions of the 3D model that correspond to natural features of the patient shown in the medical image (e.g., joints, bones, hair, etc.), without resorting to artificial markers introduced into the medical image and / or 3D model to facilitate the registration process described herein (e.g., by introducing physical or virtual objects or landmarks on or into the patient's body prior to capturing the medical image). In other implementations, the skin markers are generated at least in part based on artificial markers introduced into the medical image.

[0105] Registration without manual landmarks is usually inaccurate

[0106] In a typical implementation that does not use artificial markers and relies solely on the patient's natural features, aligning the 3D model with the 3D image of the patient captured at runtime is less accurate due to the presence of noise that may interfere with the alignment process. For example, between the time the medical image is taken and the time the patient has been prepared for the medical procedure (e.g., positioning the patient on a surgical table for surgery), the position of various body parts of the patient may have changed (e.g., arms down vs. arms up, full stomach vs. empty stomach). As another example, some of the body parts of the patient shown in the medical image may not be displayed in the 3D image captured at runtime because such parts are covered (e.g., covered by a surgical drape) or otherwise hidden or facing away from the camera's field of view (e.g., in the case of thoracic surgery, the patient's back contacts the surgical table). Therefore, in such an implementation, the possibility of incorrect alignment and / or registration is increased.

[0107] Registration without manual labeling is usually more resource intensive

[0108] Additionally, in typical implementations that do not use artificial markers and rely solely on natural features of the patient, aligning the 3D model with the 3D image of the patient captured at runtime may consume a greater amount of time and / or computing resources because the unavailability of artificial markers may result in more guesswork on the part of the computing device performing the alignment. Thus, in such implementations, more time and computing resources are required to provide adequate alignment between the 3D model and the 3D image. As a result, real-time performance is difficult to achieve without artificial markers.

[0109] Improved skin marking

[0110] In implementations of the present disclosure, skin markers are generated by removing portions of the 3D model that could potentially interfere with or delay the registration process, such as movable or non-visible portions of the patient's body as described herein. By doing so, the likelihood of inaccurate alignment / registration may be reduced. Additionally or alternatively, the amount of time and computing resources consumed during the registration process may be reduced.

[0111] Noisy parts of the 3D model that may interfere with the registration

[0112] In some implementations, it is determined (e.g., by a human operator, a computing device 102, or another computing device different from the computing device used to provide the real-time view at runtime) that certain portions of the 3D model may interfere with or delay the alignment process performed at runtime. Such portions may include (i) one or more portions of the 3D model corresponding to portions of the patient's body that may be displayed in the medical image but cannot be displayed in an image of the patient's body at runtime (e.g., after the patient has been prepared for surgery and the patient is lying on the surgical bed), and (ii) one or more portions of the 3D model corresponding to portions of the patient's body that may move or be repositioned between the time the medical image is taken and the time the patient is positioned for the medical procedure. Such portions may also be referred to herein as "noise portions." hereinafter referenced FIG. 4A to FIG. 5C The noise part is described in more detail.

[0113] Identify the noisy part

[0114] In some implementations, a target body part for a medical procedure is identified, for example, by a human operator via a user interface provided by a computing device, or automatically by a computing device based on a medical image. The identified target body part may be a portion of a patient's body that includes an area of ​​interest for the medical procedure, such as a tissue or lesion that needs to be examined or treated. For the identified target body part, certain predetermined body parts that are known to potentially interfere with or delay registration (due to being potentially invisible or movable in images of the patient captured at runtime) are removed from the 3D model to generate skin markers. For example, portions of the 3D model corresponding to one or more joints (e.g., joints in the shoulder / axillary region) connected to the target body part (e.g., the patient's torso) may be removed. As another example, portions of the 3D model corresponding to portions of the patient's body adjacent to such joints (e.g., skin areas near joints in the shoulder / axillary region) may be removed. As yet another example, portions of the 3D model corresponding to portions of the patient's body connected to the target body part via such joints (e.g., arms connected to the patient's torso via joints in the shoulder / axillary region) may be removed.

[0115] Additional examples of identified noise components

[0116] For brain surgery, portions of the 3D model corresponding to the patient's neck and below (position may vary) and hair (the patient may be wearing a headgear) may be removed. For a lung biopsy, portions of the 3D model corresponding to the patient's neck and above (position may vary), shoulder joints and arms (position may vary), abdomen (the patient may have been fasting), and hip joints and below (position may vary) may be removed.

[0117] Determination of medical procedures

[0118] In some implementations, the skin marking generation process is specific to a medical procedure. The medical procedure may be determined by a human operator and / or input into a computing device that performs the skin marking generation process. As another example, the computing device that performs the skin marking generation process may automatically determine the medical procedure based on the medical images without human input. In other implementations, the skin marking generation process is not specific to a medical procedure.

[0119] Remove invisible or obscured parts

[0120] In some implementations, when generating skin markers from the 3D model, one or more portions of the 3D model are removed, the one or more portions corresponding to portions of the patient's body that may be displayed in a medical image, but cannot be displayed in an image of the patient's body at runtime (e.g., after the patient has been prepared for surgery and the patient is lying on a surgical bed). Such portions may include the patient's back (in the case of chest surgery), chest (in the case of back surgery), body parts facing the surgical bed, body parts that are not related to the medical procedure (because such body parts may be covered by a surgical drape), body parts that will be covered during the medical procedure (e.g., hair, eyebrows, etc.), body parts facing away from the surgeon, etc. Such portions may also be referred to herein as "noise portions."

[0121] Not all invisible or occluded parts need to be removed

[0122] In some implementations, not all potentially invisible or occluded portions are removed from the 3D model when the skin marker is generated. For example, if a 3D depth scan of a patient during operation occurs before the patient is covered for a medical procedure (described in more detail below), portions of the 3D model corresponding to parts of the patient's body that will be covered during the medical procedure may remain in the skin marker. Such portions may be used to align the skin marker to the 3D image of the patient. In other implementations, all potentially invisible or occluded portions are removed from the 3D model when the skin marker is generated.

[0123] Example of invisible or obscured parts (patient's back)

[0124] Figure 4A According to an example implementation, Figure 3B The result of removing the patient’s back from the 3D model. Figure 4A As shown, the portion of the 3D model corresponding to the user's back has been removed because the patient's back may be visible in the medical image taken before runtime, but is not visible in the 3D image captured at runtime.

[0125] Remove movable parts

[0126] In some implementations, when generating skin markers from the 3D model, one or more portions of the 3D model are removed that correspond to portions of the patient's body that may be moved or repositioned from the time the medical image is taken to the time the patient is positioned for the medical procedure (e.g., the patient is already on a surgical table for the surgical procedure). For example, if a patient takes an MRI image with her arms down, but will undergo the surgical procedure with her arms up, the portions of the 3D model corresponding to her arms will not help (and may even interfere) with aligning the 3D model with the patient's image at runtime because the orientation of her arms will be different in the 3D model and the image (e.g., due to the presence of joints). As another example, if a patient takes an MRI image with a full stomach, but will undergo the surgical procedure with an empty stomach, the portions of the 3D model corresponding to her abdomen will not help (and may even interfere) with aligning the 3D model with the patient's image at runtime because the size and shape of her abdomen will be different in the 3D model and the image. As yet another example, if the medical procedure involves intubation, the opening (e.g., nose or mouth) through which the tube will be inserted may be removed. As yet another example, changes in gravity and / or posture may affect the shape and position of a patient's body parts, and such body parts may be removed (e.g., breasts, male genitalia, etc.). Such parts may also be referred to herein as "noisy parts."

[0127] Not all movable parts need to be removed

[0128] In some implementations, not all movable portions are removed from the 3D model when the skin marker is generated. For example, if a portion of the 3D model corresponds to a movable body part of the patient, but the movable body part is not expected to be positioned differently during the medical procedure relative to the time when the medical image is captured, such portion may not be removed and remain part of the skin marker. Such portion may therefore be used to align the skin marker to the 3D image of the patient. In other implementations, all movable portions are removed from the 3D model when the skin marker is generated.

[0129] joint

[0130] As used herein, the term "joint" refers to an area where two bones of an animal skeleton are attached for the purpose of allowing the two bones to move about the area. For example, joints in the human body may include axle joints (such as those found between cervical vertebrae), hinge joints (such as those found in the elbows, ankles, fingers, toes, and knees), saddle joints (such as those found in the thumb), plane joints (such as those found in the wrist and ankle), condylar joints (such as those found in the wrist and arm), and ball and socket joints (such as those found in the shoulder and hip).

[0131] Movable joints

[0132] A typical human body includes more than 200 joints, some of which may be non-movable. In some implementations, only movable joints embodied in the 3D model are removed from the 3D model when skin tags are generated. In other implementations, all joints embodied in the 3D model are removed from the 3D model when skin tags are generated. In other implementations, only a subset of movable joints embodied in the 3D model are removed from the 3D model when skin tags are generated.

[0133] Examples of movable parts (joints of the patient)

[0134] In some implementations, a body part (e.g., head, chest, abdomen, lower torso, etc.) shown in a medical image is identified, and a skin marker is generated by removing all portions of the 3D model that correspond to joints specific to the identified body part. For example, if it is determined that the medical image shows a patient's head, a skin marker may be generated by removing portions of the 3D model that correspond to joints in the patient's head, such as the mandible and maxilla. As another example, if it is determined that the medical image shows a patient's chest, a skin marker may be generated by removing portions of the 3D model that correspond to joints near the patient's chest, such as joints in the shoulder, also known as the glenohumeral joint.

[0135] Remove the direction of the joint

[0136] In some implementations, as part of generating the skin markers, portions of the 3D model corresponding to the patient's joints and the patient's body parts on the other side of the joints are removed from the 3D model. For example, if the area of ​​interest for the medical procedure is in the patient's chest, the shoulder joint and arm may be removed. On the other hand, if the area of ​​interest for the medical procedure is in the patient's upper arm, the shoulder joint and chest may be removed. As another example, if the area of ​​interest for the medical procedure is in the patient's thigh (above the knee), the knee joint and calf may be removed. On the other hand, if the area of ​​interest for the medical procedure is in the patient's calf (below the knee), the knee joint and thigh may be removed.

[0137] Example of a movable part (the patient's armpit)

[0138] Figure 4B According to an example implementation, Figure 4A The result of removing the patient's armpit from the 3D model. Figure 4B As shown, the portion of the 3D model corresponding to the user's armpit has been removed because the position of the patient's arm may vary between the medical image taken before runtime and the 3D image captured at runtime.

[0139] Another example of a movable part (the patient's abdomen)

[0140] Figure 5A According to an example implementation, Figure 4B The result of removing the patient’s abdomen from the 3D model. Figure 5A As shown, the portion of the 3D model corresponding to the user's abdomen has been removed because the patient's abdomen may look different in the medical image taken before runtime and the 3D image captured at runtime.

[0141] Another example of a non-visible or occluded portion (the patient's contralateral chest)

[0142] Figure 5B According to an example implementation, Figure 5A The result of removing the left chest part of the patient from the 3D model. Figure 5B As shown, a portion of the 3D model corresponding to the user's left side chest (e.g., the opposite side of the right side chest where surgery is performed) has been removed because the patient's left side chest may be visible in medical images taken before runtime, but may be covered by a surgical drape (e.g., as shown in FIG. 1 ). Figure 5C shown) and is invisible.

[0143] Identify parts of a 3D model for removal

[0144] In some implementations, the portions of the 3D model to be removed (e.g., regions of the 3D model corresponding to body parts of the patient that are expected to be invisible or moving in the 3D image captured at runtime) are predetermined for the medical image (e.g., for a chest CT image or a 3D model near the chest region, the patient's armpits are removed; for a head CT image or a 3D model near the head region, the patient's jaw is removed, etc.). In other implementations, such portions are determined based on user input (e.g., an operator of the computing device 102 selects one or more portions of the 3D model for removal). In still other implementations, some of such portions are predetermined, while the rest of such portions are determined based on user input.

[0145] The target portion of the 3D model included in the skin marking

[0146] In some implementations, as part of determining which portions of the 3D model should be retained in the skin marker, the location of a target of interest (e.g., a lesion) in a medical procedure is determined in the medical image. Using the location of the target of interest, one or more bones closest to the target of interest are identified. Portions of the 3D model corresponding to the skin next to the identified one or more bones can then be included in the skin marker. Such portions are unlikely to change position or shape between the time the medical image is taken and the time the patient has been prepared for the medical procedure. Therefore, such portions are used as part of the skin marker to facilitate alignment and registration described herein. Such portions may include the patient's clavicle, sternum, thorax, ribs, cheekbones, ears, etc. Such portions are also referred to herein as "target portions."

[0147] The target portion may not include the skin area closest to the area of ​​interest

[0148] In some implementations, the skin next to one or more bones closest to the target of interest (e.g., a lesion) that is included in the skin marker as described above is located farther away than the skin closest to the target of interest. In such implementations, the skin closest to the target of interest may not be included in the skin marker (e.g., because the skin is considered to be a noisy component that may interfere with the alignment process, which will be described in more detail below). For example, if the target of interest is in the patient's abdomen, the closest bone may be the sternum or the lowest rib, and the skin next to such a bone will be closer to the patient's chest and will be farther away from the area of ​​interest than, for example, the skin on the patient's abdomen. In other implementations, the skin marker includes the patient's skin closest to the target of interest.

[0149] Skin marker generation based on target part and noise part

[0150] In some implementations, the skin marker includes only the target portion described herein. In such implementations, the skin marker is generated by identifying the target portion as described herein and removing all other portions from the 3D model. In other implementations, the skin marker includes only those portions of the 3D model that are not removed as noise portions. In such implementations, the skin marker is generated by identifying the noise portion as described herein and removing all identified noise portions from the 3D model. In still other implementations, the skin marker includes the intersection of the identified target portion and those portions that are not removed as noise portions (e.g., two sets). In still other implementations, the skin marker includes the joint portion of the identified target portion and those portions that are not removed as noise portions (e.g., either set).

[0151] Not all parts of a skin tag may be connected

[0152] In some implementations, the skin marker includes two portions that are not in contact or adjacent to each other. For example, the skin marker may include a patch near the patient's lowest rib and another patch near the patient's clavicle, where the two patches are not connected and are completely separate from each other. In other implementations, the entire skin marker is a single patch of the 3D model (e.g., as shown in FIG. Figure 5B shown).

[0153] Procedure-specific skin tag generation process

[0154] In some implementations, the portion of the 3D model that is removed to generate the skin mark is different depending on the medical procedure. For example, for chest surgery, the back portion of the 3D model may be removed (because the back will contact the surgical table during surgery and is therefore not in the patient's camera view), while for back surgery, the chest portion of the 3D model may be removed (because the chest will contact the surgical table during surgery and is therefore not in the patient's camera view). The medical procedure and / or the portion to be removed may be specified by an operator of the computing device 102, or another computing device different from the computing device used to provide the real-time view at runtime (e.g., a physician performing the medical procedure). In other implementations, the portion of the 3D model that is removed to generate the skin mark is determined without regard to the specific medical procedure to be performed (e.g., by removing only generally distracting features such as joints, abdomen, etc.).

[0155] Example skin tag generation for neurosurgery

[0156] In the instance of neurosurgery involving the patient's skull, portions of the 3D model corresponding to non-visible or movable features of the patient may be removed, such as the patient's jaw (because its shape may change due to the presence of joints), mouth (because its shape may change due to cannulas passing through the mouth), nose (because its shape may change due to cannulas passing through the nose), hair (because it will be covered by a headgear worn by the patient), eyebrows (because it will be covered during surgery), forehead (because it will be covered during surgery), etc., thereby leaving only those portions of the 3D model corresponding to the patient's visible but non-movable features in the skin markings, such as cheekbones, cheeks, ears, etc., and the connecting portions therebetween.

[0157] Processing performed at runtime

[0158] As described above, registering the 3D model coordinate system to the computing device coordinate system and overlaying the medical image on the real-time camera view may include certain steps that are performed when preparing the patient for the medical procedure or during the medical procedure (runtime processing), such as aligning pre-generated skin markers with an image of the patient (e.g., an image of the patient lying on a surgical table) and displaying the medical image overlaid on the real-time camera view of computing device 102. These processes are described in more detail below.

[0159] Access medical images, 3D models, and skin markers

[0160] At runtime, the computing device 102 accesses a medical image of the patient, a 3D model generated based on the medical image, and / or a skin marker generated based on the 3D model. In some implementations, one or more of the medical image, the 3D model, and the skin marker is accessed from an internal storage device of the computing device 102. In other implementations, one or more of the medical image, the 3D model, and the skin marker is accessed from an external storage device (e.g., via wired or wireless communication).

[0161] 3D depth scan of the patient

[0162] In operation, computing device 102 uses the 3D depth camera to scan a patient, and uses data generated by the 3D depth camera to register the coordinate system of the skin markers with the coordinate system of computing device 102. For example, a physician performing a medical procedure on a patient may point a 3D depth camera connected to computing device 102 at a patient lying on a surgical bed and initiate the scanning process (e.g., by providing a user command via a digital button provided on a display screen of computing device 102).

[0163] Data captured during 3D depth scanning

[0164] During the scan, computing device 102 causes a 3D depth camera of computing device 102 (e.g., integrated into computing device 102 or externally connected to computing device 102) to capture one or more 3D images of the patient's body. Each 3D image may be associated with: (i) one or more objects shown in the image and their corresponding depths, (ii) an orientation of computing device 102 when the 3D image was captured (e.g., the direction the 3D depth camera was pointing), and (iii) a position of computing device 102 when the 3D image was captured (e.g., the position of computing device 102 in three-dimensional space).

[0165] Using tags at runtime

[0166] In some implementations, one or more markers are affixed to or placed on the patient's body to aid in the registration process. For example, such markers will appear in 3D images of the patient captured at runtime, and indications of the markers will be mapped to specific portions of skin markings and / or will be used as reference points during the registration process. In other implementations, such markers are not used at runtime, and the 3D scanning and / or registration described herein are performed without visible markers affixed to or placed on the patient's body.

[0167] When to use 3D depth scanning

[0168] In some implementations, the scan is performed after the patient has been positioned for the medical procedure (e.g., placed on a surgical table for surgery) but before the patient's body is draped for the medical procedure (e.g., after the patient has been placed on a surgical table for surgery). Figure 5C In other implementations, the scan occurs after the patient has been positioned for the medical procedure (e.g., after the patient has been placed on a surgical table for the surgical procedure) and after the patient's body has been covered for the medical procedure (e.g., after the patient has been placed on a surgical table for the surgical procedure). Figure 5C After being covered as shown) occurs.

[0169] The amount of 3D images captured

[0170] In some implementations, computing device 102 continuously captures 3D images of the patient and analyzes the captured 3D images in real time until the registration process is complete. In some implementations, computing device 102 captures a fixed number of 3D images of the patient and performs the registration process based on the captured 3D images.

[0171] Align the 3D model coordinate system to the user's computing device coordinate system

[0172] Implementations of the present disclosure relate to systems and techniques for registering a coordinate system associated with a 3D model (and therefore with medical images and skin markers) with another coordinate system used by a computing device (e.g., a tablet computer operated by a physician in an operating room). Thus, the AR system of the present disclosure can use the registration to determine the 3D position associated with each medical image of the patient within a real-time camera view provided by the computing device.

[0173] Matching Features

[0174] Registration can be performed by identifying pre-generated skin markers and matching points / regions in a 3D image (or a 3D model generated based on the 3D image) of the patient captured at runtime (e.g., while the patient is lying on a surgical table). In some implementations, a point pair feature (PPF) algorithm is used. In other implementations, one or more other 3D matching algorithms are used instead. In some implementations, an iterative closest point (ICP) algorithm is used to improve the accuracy of the registration process.

[0175] Overlay medical images on live camera view

[0176] After registration, Figure 1A As shown, computing device 102 may display the medical images as superimposed on a real-time camera view 104 displayed on a display of computing device 102. For example, computing device 102 displays the medical images at corresponding locations on the patient's body (e.g., based on the registration) in response to user input, one medical image at a time. The size and orientation of the superimposed medical images may change as the position and orientation of the camera of computing device 102 change. Computing device 102 may also indicate the location of a region of interest in the superimposed medical image and the distance between the region of interest and the patient's proximal skin on the real-time camera view displayed on the display.

[0177] Medical images using overlays

[0178] By scrolling through the medical images, physician 101 may identify a medical image showing an area of ​​interest (e.g., a lesion) and make certain decisions about how to perform a medical procedure. For example, physician 101 may (e.g., using user interface controls 108) cause a medical image of a medical image showing a tumor inside a patient's breast to be displayed on a display and determine where an incision should be made on the patient's breast to remove the tumor (e.g., a location that will provide the easiest and / or closest access to the tumor).

[0179] Data superimposed on the camera view of the patient

[0180] Although some implementations described herein describe superimposing one or more medical images (e.g., medical images showing lesions or areas of interest) on a live camera view or one or more images / photos of a patient captured at runtime, other data may additionally or alternatively be superimposed on the live camera view or images / photos of the patient, such as (i) skin markings or portions thereof, (ii) a 3D model or portions thereof, (iii) a tissue 3D model or portions thereof, (iv) a 3D mesh or portions thereof, (v) markers indicating specific locations of interest within one or more of the foregoing medical images, skin markings, skin 3D models, tissue 3D models, or 3D meshes, and / or the like.

[0181] Pre-processing and runtime processing on different devices

[0182] In some implementations, one or more of the pre-processing operations described herein (e.g., processing of medical images, generation of 3D models, and / or generation of skin markers) are performed on a different device than the device that performs the runtime processing operations (e.g., 3D depth scanning, registration, and / or overlaying the medical image on a live camera view). In other implementations, all pre-processing operations and runtime processing operations are performed on the same device (e.g., computing device 102).

[0183] Instructions stored on a non-transitory computer readable medium (CRM)

[0184] The techniques described herein may be implemented as a set of instructions stored on a non-transitory computer-readable medium that, when executed by a computer (e.g., a server computer, a tablet computer, etc.), causes the computer to perform one or more operations described herein. The set of instructions may be packaged as an application or a set of software modules and downloaded to a computer. The set of instructions may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of non-transitory computer-readable storage medium.

[0185] Same set of instructions on different CRMs

[0186] In some implementations, the same set of instructions (e.g., as the same application) is stored on one CRM that communicates with a first computer and on another CRM that communicates with a second computer, wherein the set of instructions causes the first computer to perform the pre-processing steps described herein and causes the second computer to perform the runtime processing steps described herein.

[0187] Different instruction sets on different CRMs

[0188] In some implementations, different sets of instructions (e.g., as two different applications) are stored on different CRMs on different computers, where one set of instructions stored on one CRM causes a first computer to perform the pre-processing steps described herein, and another set of instructions stored on another CRM causes a second computer to perform the runtime processing steps described herein.

[0189] Same set of instructions on a single CRM

[0190] In some implementations, one set of instructions (eg, as a single application) is stored on a CRM of a computer, wherein the set of instructions causes the computer to perform the pre-processing steps described herein, and also causes the computer to perform the runtime processing steps described herein.

[0191] Different sets of instructions on a single CRM

[0192] In some implementations, different sets of instructions are stored (e.g., as two different applications) on a single CRM of a computer, where one set of instructions causes the computer to perform the pre-processing steps described herein and another set of instructions causes the computer to perform the runtime processing steps described herein.

[0193] Alternative timing of skin tag generation

[0194] In some implementations, the skin markings are generated prior to runtime (e.g., prior to preparing the patient for the medical procedure and prior to a 3D depth scan of the patient). In other implementations, the skin markings are generated at runtime (e.g., while or after preparing the patient for the medical procedure) but prior to a 3D depth scan of the patient. In other implementations, the skin markings are generated at runtime (e.g., while or after preparing the patient for the medical procedure) and after a 3D depth scan of the patient.

[0195] Flowchart for preprocessing medical images and generating skin markers

[0196] Figure 6 is a flow chart illustrating a method for preprocessing medical images and generating skin markings according to an example implementation. Figure 6 The steps shown may be performed, for example, by a computing device 102 operated by a medical professional performing a medical procedure on a patient; another computing device different from the computing device used to provide the real-time view at runtime; a server computer in communication with the computing device 102, to name a few examples. For convenience, the method 600 is described as being performed by the computing device 102.

[0197] Obtaining medical images

[0198] At block 602, computing device 102 obtains medical images of a patient. For example, one or more medical images of the patient, such as CT or MRI images, may have been taken to visualize the interior of the patient's body and provide an appropriate diagnosis. Such images may be transferred to computing device 102 for pre-processing prior to run time.

[0199] Preprocessing medical images

[0200] At block 604, computing device 102 processes each medical image to extract contour lines (eg, sets of vertices or pixels / voxels) corresponding to the contours of the patient's skin. For example, the segmentation process described herein may be used to extract the contour lines.

[0201] Generate 3D Model

[0202] At block 606, computing device 102 generates a 3D model by combining contours extracted from the medical image. For example, the meshing process described herein may be used to generate the 3D model.

[0203] Generate skin tags

[0204] At block 608, computing device 102 generates a skin marker by removing one or more portions of the 3D model. For example, as described herein, a skin marker may be generated based on a target portion and / or a noise portion.

[0205] Flowchart for processing patient images and overlaying medical images

[0206] Figure 7 is a flow chart illustrating a method for processing an image of a patient and overlaying a medical image on a live camera view according to an example implementation. Figure 7 The steps shown may be performed, for example, by a computing device 102 operated by a medical professional performing a medical procedure on a patient. For convenience, method 700 is described as being performed by computing device 102 .

[0207] Get medical images and pre-generated skin markers

[0208] At block 702 , computing device 102 obtains a medical image of a patient and pre-generated skin markings corresponding to the medical image.

[0209] Capturing patient images

[0210] At block 704, computing device 102 captures a 3D image of a patient lying on a surgical table. For example, computing device 102 may capture a 3D image using a 3D depth camera to which it is connected, as described herein.

[0211] Register pre-generated skin landmarks to captured patient images

[0212] At block 706, computing device 102 registers the pre-generated skin marker to the captured 3D image. For example, features in the skin marker may be aligned to features in the 3D image (or a 3D model generated based on the 3D image), and a coordinate system of the skin marker may be registered to a coordinate system of computing device 102 based on the alignment.

[0213] Display medical images on live camera view

[0214] At block 708 , computing device 102 displays one of the medical images on a display device over a real-time camera view of the patient lying on the surgical bed.

[0215] Scroll through other medical images

[0216] At block 710 , the computing device 102 displays other ones of the medical images on the real-time camera view of the patient, one at a time, in response to the user input.

[0217] Example architecture of a user computing system

[0218] Figure 8 1 to 2 show a method for performing the method described herein or Figure 7 An example architecture of a computing device 102 illustrating one or more technologies. Figure 8 The general architecture of the computing device 102 shown includes an arrangement of computer hardware and software modules that can be used to implement one or more aspects of the present disclosure. The computing device 102 may include Figure 8 Those more (or fewer) elements shown. However, it is not necessary to show all of these elements in order to provide an implementable disclosure. As shown, the computing device 102 includes a processor 190, a network interface 192, a computer-readable medium 194, and an input / output device interface 196, all of which can communicate with each other via a communication bus. The network interface 192 can provide a connection to one or more networks or computing systems. The processing 190 can also communicate with the memory 180, and further provide output information to one or more output devices (such as a display (e.g., display 199), a speaker, etc.) via the input / output device interface 196. The input / output device interface 196 can also accept input from one or more input devices, such as a camera (e.g., a 3D depth camera 198), a keyboard, a mouse, a digital pen, a microphone, a touch screen, a gesture recognition system, a speech recognition system, an accelerometer, a gyroscope, etc.

[0219] The memory 180 may contain computer program instructions (grouped into modules in some implementations) that the processor 190 executes to implement one or more aspects of the present disclosure. The memory 180 may include RAM, ROM, and / or other persistent, auxiliary, or non-transitory computer-readable media. The memory 180 may store an operating system 184 that provides computer program instructions for use by the processor 190 to generally manage and operate the computing device 102. The memory 180 may also include computer program instructions and other information for implementing one or more aspects of the present disclosure. For example, in one implementation, the memory 180 includes a user interface module 182 that generates a user interface (and / or instructions for the user interface) for display, for example, via a browser or application installed on the computing device 102.

[0220] In addition to and / or in combination with the user interface module 182, the memory 180 may include a medical image processing module 185, a skin marker generation module 186, a 3D depth scanning module 187, a marker and device registration module 188, and a medical image overlay module 189 that may be executed by the processor 190. In one implementation, the medical image processing module 185 processes the medical image of the patient to generate a 3D model of the patient, the skin marker generation module 186 removes portions of the generated 3D model to generate skin markers, the 3D depth scanning module 187 causes the 3D depth camera 198 to capture a 3D image of the patient, the marker and device registration module 188 uses the skin markers to register the coordinate systems of the 3D model and the medical image with the coordinate system of the computing device 102, and the medical image overlay module 189 causes the medical image to be superimposed on a real-time camera view of the 3D depth camera 198 (or another camera on the computing device 102) displayed via the display 199 of the computing device 102. The above reference to FIGS. 1 to 13 illustrates a medical image processing module 185 that processes the medical image of the patient to generate a 3D model of the patient, the skin marker generation module 186 removes portions of the generated 3D model to generate skin markers, the 3D depth scanning module 187 causes the 3D depth camera 198 to capture a 3D image of the patient, the marker and device registration module 188 uses the skin markers to register the coordinate systems of the 3D model and the medical image with the coordinate system of the computing device 102, and the medical image overlay module 189 causes the medical image to be superimposed on a real-time camera view of the 3D depth camera 198 (or another camera on the computing device 102) displayed via the display 199 of the computing device 102. Figure 7 The operations and algorithms of modules 185-189 are described in more detail.

[0221] Despite Figure 8 , a single processor, a single network interface, a single computer-readable medium, a single input / output device interface, a single memory, a single 3D depth camera, and a single display are shown in the example, but in other implementations, computing device 102 may have multiples of one or more of these components (e.g., two or more processors and / or two or more memories).

[0222] Other considerations

[0223] The logic blocks, modules, or units described in conjunction with the implementations disclosed herein may be implemented or executed by a computing device having at least one processor, at least one memory, and at least one communication interface. The elements of the methods, processes, or algorithms described in conjunction with the implementations disclosed herein may be directly embodied in hardware, in a software module executed by at least one processor, or in a combination of the two. Computer executable instructions for implementing the methods, processes, or algorithms described in conjunction with the implementations disclosed herein may be stored in a non-transitory computer-readable storage medium.

[0224] Although the implementation of the present invention has been disclosed in the context of certain implementations and examples, it will be understood by those skilled in the art that the present invention will extend to other alternative implementations and / or uses of the present invention and obvious modifications and equivalent forms thereof beyond the specifically disclosed implementations. In addition, although many variations of the present invention have been shown and described in detail, based on this disclosure, it will be easy for those skilled in the art to clearly understand other modifications within the scope of the present invention. It is also conceivable that various combinations or sub-combinations of the specific features and aspects of these implementations may be produced and they still belong to one or more inventive forms. Therefore, it should be understood that the various features and aspects of the disclosed implementations may be combined or substituted with each other to form different modes of the disclosed invention. Therefore, the scope of the present invention disclosed herein should not be intended to be limited by the specific disclosed implementations described above, and various changes in form and detail may be made without departing from the spirit and scope of the present disclosure as set forth in the following claims.

Claims

1. A method for a medical procedure, the method comprising: Prior to treating or examining a target tissue of a patient, processing medical image data to generate a 3D skin model defining a 3D contour of the patient's skin, the medical image data comprising a plurality of 2D cross-sectional images of the patient's body; Still before performing the treatment or examination on the target tissue, the skin 3D model is processed to generate a skin marking, wherein the processing of the skin 3D model includes removing one or both of the following items from the skin 3D model: (1) a first portion of the 3D skin model corresponding to a skin area of ​​the patient that will contact or directly face a support surface on which the patient will lie during the treatment or examination, and (2) a second portion of the 3D skin model corresponding to a skin area of ​​the patient adjacent to a shoulder joint of the patient's body, the skin area will significantly change its position relative to the target tissue and / or significantly change its contour when an arm connected to the shoulder joint moves around the shoulder joint, causing the skin marking to include at least a portion of the skin 3D model excluding one or both of the first portion and the second portion; capturing an image of the patient lying on a bed using a handheld computing device while or during the treatment or examination of the target tissue; Still while or during said treatment or examination of said target tissue, processing the captured images of said patient and said skin markings to register them to achieve augmented reality without using artificial physical markers affixed to or placed on said patient's body; as well as Still while or during the treatment or examination of the target tissue, a real-time augmented reality view is presented on a display screen of the handheld computing device, the real-time augmented reality view comprising at least one of the multiple 2D cross-sectional images of the patient's body superimposed on one or more additional real-time images of the patient captured after the alignment.

2. The method of claim 1, wherein the processing of the skin 3D model comprises removing both the first portion and the second portion such that the skin marker comprises at least a portion of the skin 3D model from which both the first portion and the second portion are excluded.

3. A method as claimed in claim 1, wherein the patient's body includes a target body part covering the target tissue and a non-target body part connected to the target body part via a joint, wherein the skin 3D model defines a 3D contour of the patient's skin at the target body part and a 3D contour of the patient's skin at the non-target body part, wherein the processing of the skin 3D model includes further removing the 3D contour of the patient's skin at the non-target body part.

4. A method as claimed in claim 1, wherein in the real-time augmented reality view, at least one of the multiple 2D cross-sectional images of the patient's body showing the target tissue is displayed on the display screen over the one or more additional real-time images of the patient for the surgeon to determine the incision location on the patient's body.

5. The method of claim 1, further comprising: Prior to performing the treatment or examination, the medical image data is processed to generate a tissue 3D model defining a 3D contour surrounding the target tissue, wherein the tissue 3D model is located within the skin 3D model, and wherein the real-time augmented reality view also includes the tissue 3D model superimposed on one or more additional captured images of the patient.

6. The method of claim 1, wherein the processing of the skin 3D model comprises further removing a portion of the skin 3D model corresponding to a skin area of ​​the patient that will be covered by a medical drape and therefore not exposed during a medical procedure.

7. The method of claim 1, wherein the processing of the skin 3D model comprises further removing a portion of the skin 3D model corresponding to a skin area of ​​the patient that will significantly change its contour before and after the patient eats.

8. The method of claim 1, wherein the skin mark comprises at least two portions that are not connected to each other corresponding to skin areas of the patient.

9. The method of claim 1, further comprising: A medical drape is placed on the patient lying on the bed in a manner that leaves a skin portion of the patient's body exposed, wherein the placing of the medical drape is performed before or after capturing the image of the patient.

10. The method of claim 1, further comprising: A medical drape is placed on the patient lying on the bed while exposing a skin area, wherein the placing of the medical drape is performed after the image of the patient is captured, and wherein the skin marking does not include a portion of the skin 3D model corresponding to the exposed skin area.

11. The method of claim 1, wherein the skin marker does not include a portion of the skin 3D model that corresponds to a skin area of ​​the patient that is closest to the target tissue.

12. The method of claim 1, wherein the processing of the skin 3D model comprises further removing at least one portion of the skin 3D model corresponding to at least one skin area of ​​a body part of the patient, the body part of the patient being selected from the group consisting of an abdomen, an axilla, and a limb.

13. The method of claim 1, wherein the processing of the skin 3D model comprises further removing at least one portion of the skin 3D model corresponding to at least one skin area of ​​a body part of the patient, the body part of the patient being selected from the group consisting of a pubic area, an area of ​​densely gathered hair, and a mouth.

14. The method of claim 1, wherein the processing of the skin 3D model includes retaining a portion of the skin 3D model corresponding to a cheekbone of the patient.

15. The method of claim 1, wherein the processing of the skin 3D model includes retaining a portion of the skin 3D model corresponding to an ear of the patient.

16. A non-transitory storage medium storing a plurality of instructions executable by a computer, wherein the plurality of instructions, when executed, cause the computer to: receiving medical image data comprising a plurality of 2D cross-sectional images of a patient's body covering a target tissue; processing the medical image data to generate a 3D skin model defining the 3D contours of the patient's skin; receiving information related to the target tissue and information related to a proposed medical procedure for treating or examining the target tissue; Identifying one or both of: (1) a first skin area of ​​the patient that will contact or directly face a lying surface on which the patient will lie during the proposed medical procedure, and (2) a second skin area of ​​the patient that is adjacent to a shoulder joint of the patient's body and that will significantly change its position relative to the target tissue and / or significantly change its contour when an arm connected to the shoulder joint moves about the shoulder joint; as well as Removing one or both of the following from the skin 3D model: (1) a first portion of the skin 3D model corresponding to the first skin area; and (2) a second portion of the skin 3D model corresponding to the second skin area, thereby providing a skin marker comprising at least a portion of the skin 3D model from which one or both of the first portion and the second portion are excluded.

17. The non-transitory storage medium of claim 16, wherein the computer is a handheld computing device, wherein the plurality of instructions, when executed, further cause the handheld computing device to: capturing an image of the patient; processing the captured images of the patient and the skin markers to register them to achieve augmented reality without using artificial physical markers affixed to the patient's body or worn by the patient; and A real-time augmented reality view is presented on the handheld computing device, the real-time augmented reality view comprising at least one of the plurality of 2D cross-sectional images of the patient's body superimposed on one or more additional images of the patient captured after the registration.

18. A non-transitory storage medium as described in claim 17, wherein the plurality of instructions, when executed, further cause the handheld computing device to present one or both of the following items on the handheld computing device: (a) a mesh of the skin 3D model; and (b) a mesh of the skin markings.

19. The non-transitory storage medium of claim 16, wherein the skin mark includes at least two portions that are not connected to each other corresponding to a skin area of ​​the patient.

20. The non-transitory storage medium of claim 16, wherein both the first portion and the second portion are removed from the skin 3D model to provide the skin marking, the skin marking comprising at least a portion of the skin 3D model from which both the first portion and the second portion are excluded.

Images