Method and system for enhancing medical scan image information on extended reality images

By identifying and matching feature points in three-dimensional images, and generating enhanced extended reality images, the problem of inconsistency between virtual and entity information in the prior art is solved, and the accuracy and safety of the surgery are improved.

CN115428443BActive Publication Date: 2024-11-01BRAIN NAVI BIOTECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202180030417.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-04-22
Filing Date
2021-04-21
Publication Date
2024-11-01
Estimated Expiration
2041-04-21

AI Technical Summary

Technical Problem

现有的延展实境技术无法充分协调手术途径中的虚拟图像与医学扫描信息之间的差异,导致在手术过程中信息不一致,影响手术的准确性和安全性。

Method used

By identifying feature points in three-dimensional images, matching and superimposing virtual and entity information, we generate enhanced extended reality images to ensure the accurate alignment of surgical tools and patient tissues, and using wearable devices to display the combination of virtual and entity reality. image.

Benefits of technology

It improves the consistency and accuracy of information during the operation, enhances the decision-making ability of surgeons, reduces the risk of surgery, and increases the success rate of surgery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115428443B_ABST
    Figure CN115428443B_ABST
Patent Text Reader

Abstract

A method for enhancing medical scan information about a target object on a first extended reality image. The method includes obtaining a three-dimensional image about the target object (210); identifying a first set of three-dimensional feature points from the three-dimensional image (220); identifying anatomical points based on the first set of three-dimensional feature points (230); obtaining the first extended reality image about the isoanatomical points (240); selecting a second set of three-dimensional feature points from the first extended reality image (250); performing a first image matching between the first set of three-dimensional feature points and the second set of three-dimensional feature points (260); and superimposing the three-dimensional image on the first extended reality image based on the first image matching (270).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-references

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 013,687, filed on April 22, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments of the present invention generally relate to methods and systems for enhancing medical scan image information on extended reality images. Background Art

[0004] Unless otherwise indicated in the specification, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section. Summary of the invention

[0005] In surgery, planning of the surgical approach is critical. The surgical approach may be defined by a number of points, such as a safe point and a preoperative point away from the patient, an entry point on the patient's tissue, and a target point at the target of the surgery.

[0006] Before surgery, the patient undergoes a medical scan (e.g., CT or MRI). The medical scan can provide images of the patient's tissues, organs, and organ systems. The surgical approach is planned based on the medical scan images. For example, artificial intelligence can be used to suggest to the surgeon the best approach with the least amount of damage.

[0007] Extended reality technology generally refers to technology that includes one or more real and virtual combined environments and one or more human-machine interfaces (generated by computer technology and one or more wearable devices). In the medical field, extended reality including virtual reality, augmented reality, and mixed reality are increasingly used. For example, extended reality can display virtual images of tissues, organs, and organ systems in close proximity to the surgical approach, and enhance medical scan information (e.g., medical scan images) on the virtual images to facilitate the surgery.

[0008] However, there is a significant time difference between when the medical scan is taken on the patient and when the surgery is performed. Conventional extended reality techniques cannot adequately reconcile the difference between the virtual image immediately adjacent to the surgical path and the information obtained in the medical scan. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example diagram showing multiple spatial relationships among a number of points that may be encountered when performing surgery on a patient;

[0010] Figure 2A flowchart illustrating an example procedure for associating a three-dimensional image of a target object with an extended reality image of the target object;

[0011] Figure 3 is a flow chart illustrating an example procedure for associating a three-dimensional image of a target object with a medical scan image (including the target object);

[0012] Figure 4A and Figure 4B Illustrate example processing images based on medical scan images;

[0013] Figure 5 A flowchart illustrating an example process for enhancing information obtained from a medical scan of a target object on an extended reality image of the target object; and

[0014] Figure 6 A flow chart illustrating an example process for obtaining an extended reality image of a target object as the target object is displaced when a surgical tool reaches a subsurface region of the target object, all of which are configured according to some specific embodiments of the present invention. DETAILED DESCRIPTION

[0015] In the following embodiments, reference is made to the drawings forming a part thereof. Unless otherwise indicated, in the multiple drawings, the same number generally indicates similar parts. In the embodiments, drawings, and claims, the multiple illustrative embodiments described are not intended to be limiting. Other embodiments may be utilized and other changes may be made without departing from the spirit or scope of the objects described herein. It will be readily appreciated that the multiple aspects of the invention (e.g., as generally described in this specification and illustrated in the drawings) may be configured, replaced, combined, and designed in a wide variety of different configurations, all of which are expressly contemplated by this specification.

[0016] Throughout the following paragraphs, "extended reality (XR)" generally refers to an environment that combines virtual and physical reality, where the "X" represents a variable for any current or future spatial computing technology. In some embodiments, extended reality is an umbrella term for all environments that combine virtual and physical reality. For example, extended reality includes augmented, virtual, and mixed reality. "Extended reality image" broadly refers to an image that contains information about both virtual and physical reality. "Wearable", "Wearable technology", and "Wearable device" are used interchangeably to generally refer to a hands-free device that can be worn on a person as an accessory, embedded in clothing, implanted in the human body, etc. Such a device can generally detect, collect, analyze, and / or communicate information about the wearer, such as vital signs, movement data, and / or ambient data. Examples of wearable devices may include, but are not limited to, head-mounted devices, smart glasses, etc. A three-dimensional (3D) model generally refers to a set of points in a 3D space connected by various geometric entities (eg, triangles, straight lines, curved surfaces, etc.).

[0017] Figure 1 An example diagram showing various spatial relationships among several points that may be encountered when performing surgery on a patient is configured according to some specific embodiments of the present invention. Figure 1 , the surgical path 110 may include a safety point 120, a pre-operative point 130, an entry point 140, a transient point 150, and a target point 160. The safety point 120 may be a point at which the robotic arm may be operated to move or rotate without causing any harm to the patient. The pre-operative point 130 may correspond to a point at which the robotic arm (configured to perform the surgery but not yet in physical contact with the patient) is positioned and angled. The entry point 140 may be a point at which the robotic arm first contacts the patient along the surgical path 110. The target point 160 may be a point related to the target tissue or target organ of the surgery. The transient point 150 may be a point between the entry point 140 and the target point 160.

[0018] Figure 2A flowchart of an example program 200 for associating a three-dimensional image of a target object with an extended reality image of a target object is provided, which is configured according to some specific embodiments of the present invention. Program 200 may include one or more operations, functions, or actions, as illustrated by steps 210, 220, 230, 240, 250, 260, and / or 270, which may be performed by hardware, software, and / or firmware. The multiple steps are not intended to be limited to the specific embodiments. The steps and operations described are provided only as examples, and some of the steps and operations may be combined into fewer steps and operations, or expanded into additional steps and operations as needed, without detracting from the essence of the disclosed specific embodiments. Although the blocks are illustrated in a sequential order, the steps may also be performed in parallel and / or in an order different from that described in this specification.

[0019] The process 200 may start from step 210 “obtaining a three-dimensional (3D) image of a target object”. In some embodiments, for example only, the target object may be a portion of a tissue, organ, or organ system of a patient. One of the three-dimensional images may correspond to an image captured by a three-dimensional camera (e.g., a camera with a depth sensor) or a set of images captured by a two-dimensional camera. Another of the three-dimensional images may also correspond to an image obtained by another source (such as a medical scanning device (e.g., an ultrasound scanner, a computerized tomography (CT) scanner, a magnetic resonance imaging (MRI) device, etc.)). In some embodiments, any of the three-dimensional images may correspond to surface anatomical information of the tissue, organ, or organ system of the patient. In some embodiments, the camera(s) may be configured to capture an image of the patient's head to capture the appearance and contours of the patient's head (e.g., eyes, ears, nose tip, nostrils, earlobes, etc.). The 3D camera or the 2D camera may be coupled to a wearable device of a surgeon performing the operation on the patient. Alternatively, the 3D camera or the 2D camera may be coupled to an endoscope or a surgical tool controlled by a robotic arm.

[0020] It should be noted that these 3D images will be considered to include physical reality information captured by various devices in the physical reality (e.g., 3D camera, 2D camera, medical scanning device, etc.).

[0021] Step 210 may be followed by step 220 of “identifying a first set of 3D feature points from the 3D image”. In some embodiments, an artificial intelligence engine may be used to identify the first set of 3D feature points from the 3D image obtained in step 210. The artificial intelligence engine may identify the first set of 3D feature points based on edges, contrasts, and shapes. In some alternative embodiments, the first set of 3D feature points may be identified by a surgeon through a wearable device.

[0022] Step 220 may be followed by step 230 of “identifying anatomical points based on the first set of 3D feature points”. In some specific embodiments, the first set of 3D feature points are displayed or marked on the 3D image obtained in step 210 to identify the anatomical points of the patient corresponding to the first set of 3D feature points. For example, by displaying or marking the first set of 3D feature points on the 3D facial image of the patient, the anatomical points of the patient corresponding to the first set of 3D feature points (e.g., eyes, ears, nose tip, nostrils, and earlobes) may be identified. Alternatively, by displaying or marking the first set of 3D feature points on the 3D endoscopic image of the patient, the anatomical points of the patient corresponding to the first set of 3D feature points (e.g., blood vessels of an organ) may be identified. In step 230, one or more tissues, one or more organs, and one or more organ systems of the patient (including the anatomical points) may also be identified.

[0023] Step 230 may be followed by step 240 of “obtaining extended reality images of relevant anatomical points”. In some embodiments, based on the identified one or more tissues, one or more organs, or one or more organ systems of the patient (including the anatomical points), it is possible to obtain extended reality images of the one or more tissues, one or more organs, or one or more organ systems of the patient. For example, this extended reality image may be an XR image of the surface of the patient's head to be displayed in a wearable device (e.g., a head-mounted device, smart glasses, etc.). In alternative embodiments, this extended reality image may be an XR image of the surface of the patient's colored organ (e.g., liver or brain) to be displayed in the wearable device. These XR images include information captured in the physical reality (e.g., one or more images of the patient's head, one or more images of the patient's organs, etc.), and the displayed image in the virtual reality.

[0024] Step 240 may be followed by step 250 of “selecting a second set of 3D feature points from the extended reality image”. In some embodiments, based on the identified anatomical points, the second set of 3D feature points are selected from the extended reality image obtained in step 240. The second set of 3D feature points may correspond to the identified anatomical points.

[0025] Step 250 may be followed by step 260 of “performing image matching between the first set of 3D feature points and the second set of 3D feature points”. In some embodiments, the first set of 3D feature points is matched with the second set of 3D feature points to determine the relationship between the first set of 3D feature points and the second set of 3D feature points, sometimes iteratively to minimize the difference between the two sets of 3D feature points. The image matching may be based on some image comparison methods, such as iterative closest point (ICP). Based on the determined relationship between the first set of 3D feature points and the second set of 3D feature points, the three-dimensional image of the target object is the extended reality image of the target object. In some embodiments, such as in a rectangular coordinate system, the determined relationship may include but is not limited to a first shift along the X axis; a second shift along the Y axis; a third shift along the Z axis; a first rotation angle along the X axis; a second rotation angle along the Y axis; and a third rotation angle along the Z axis. In various coordinate systems, the determined relationship may be different.

[0026] Step 260 may be followed by step 270 of “superimposing the 3D image on the extended reality image”. In some embodiments, based on the relationship between the first set of 3D feature points and the second set of 3D feature points determined in step 260, the three-dimensional image of the target object is an extended reality image of the target object. Therefore, the three-dimensional image of the target object obtained in step 210 may be superimposed on the extended reality image of the anatomical point obtained in step 240 to enhance the additional information obtained in step 210 on the extended reality image obtained in step 240.

[0027] Figure 3 Flowchart of an example process 300 for associating a three-dimensional image of a target object with a medical scan image (including the target object) is configured according to some embodiments of the present invention. Process 300 may include one or more operations, functions, or actions, as illustrated by steps 310, 320, 330, 340, 350, 360, and / or 370, which may be performed by hardware, software, and / or firmware. In some embodiments, in combination with Figure 1 , procedure 300 may be performed as the surgical tool reaches entry point 140. The various steps are not intended to be limited to the described embodiments. The outlined steps and operations are provided only as examples, and some of the steps and operations may be combined into fewer steps and operations, or expanded into additional steps and operations as desired without detracting from the essence of the disclosed embodiments. Although the steps are illustrated in sequence, the steps may also be performed in parallel and / or in an order different from that described in this specification.

[0028] The process 300 may start from step 310 "obtaining a three-dimensional (3D) image of a target object". In some embodiments, for example only, the target object may be a portion of a patient's tissue, organ, or organ system. The 3D image may include an image captured by a 3D camera (e.g., a camera with a depth sensor); or, a set of images captured by a 2D camera. In some embodiments, the 3D image may correspond to surface anatomical information of the patient's tissue, organ, or organ system. In some embodiments, the camera(s) may be configured to capture an image of the patient's head to capture the appearance and contours of the patient's head (e.g., eyes, ears, nose tip, nostrils, earlobes, etc.). The 3D camera or the 2D camera may be fixed to a wearable device of a surgeon performing surgery on the patient. Alternatively, the 3D camera or the 2D camera may be fixed to an endoscope or surgical tool controlled by a robotic arm.

[0029] Step 310 may be followed by step 320, "Construct 3D model based on medical scan". Prior to surgery, some medical scans may be used to capture a snapshot of the patient's condition so that a surgical plan can be made. The surgical plan may include a planned surgical approach, as previously described. For example, the surgeon may order a medical scan (e.g., CT or MRI) to obtain a medical scan image that includes a target object (e.g., one or more tissues or organs of the patient). This medical scan may be performed a few days (e.g., 3 to 5 days) before surgery. A three-dimensional model of the target object may be constructed based on information from the image obtained from the medical scan data using some known methods.

[0030] Figure 4A and Figure 4B The processed images based on medical scan images configured according to some specific embodiments of the present invention are illustrated. Figure 4A and Figure 4B , step 320 may be followed by step 330 "procedural information obtained from a medical scan", which is according to some embodiments of the present invention. The 3D image obtained in step 310 is only about the surface information of the target object, but not about the subsurface information of the target object. However, the image obtained from a medical scan is usually about both the surface and the subsurface information. Image processing is performed to remove the subsurface information from the medical scan.

[0031] exist Figure 4AIn some specific embodiments, assuming that the surgical target is inside the patient's head, the binary image 410 can be derived from the original MRI image (along the axis from the patient's head to the toes). Region 411 is the patient's skull and is usually represented by white in the original MRI image. The periphery 412 of region 411 can refer to the patient's skin, which is surface information about the target object (i.e., the patient's head). Using the set threshold method, an image 410 including region 413 of foreign objects outside the skull (all black) and region 415 inside the skull can be established. Image 410 can be further processed to form image 420. In image 420, region 413 is assigned a grayscale to distinguish between black and white to form region 423.

[0032] exist Figure 4B In the present invention, image 420 can be further processed to form image 430. In image 430, each area other than the grayscale of region 413 is designated as black to form region 431. Then, region 423 in image 430 can be designated as white to form image 440. Therefore, each point along perimeter 441 can correspond to the skin of the patient with respect to the surface information of the target object. Therefore, each point along perimeter 441 does not include the subsurface information obtained by the medical scan.

[0033] Step 330 may be followed by step 340 of "selecting a first set of 3D feature points from the 3D image". In some embodiments, an artificial intelligence engine may be used to select the first set of 3D feature points from the 3D image obtained in step 310. The artificial intelligence engine may select the first set of 3D feature points based on edges, contrasts, and shapes. In some embodiments, the first set of 3D feature points may correspond to anatomical feature points, such as vascularity of an organ or tissue texture.

[0034] Step 340 may be followed by step 350, "Select a second set of 3D feature points from the processed information". In some embodiments, an artificial intelligence engine may be employed to select the second set of 3D feature points from the information obtained from the medical scan processed in step 330. The artificial intelligence engine may select the second set of 3D feature points based on edges, contrasts, and shapes. In some embodiments, the second set of 3D feature points may correspond to the same anatomical feature points (corresponding to the first set of 3D feature points selected in step 340).

[0035] Step 350 may be followed by step 360 of “performing image matching between the first set of 3D feature points and the second set of 3D feature points”. In some embodiments, the first set of 3D feature points are matched with the second set of 3D feature points to determine a relationship between the first set of 3D feature points and the second set of 3D feature points, sometimes iterating to minimize the difference between the two sets of 3D feature points. The image matching may be based on some image comparison methods, such as iterative closest point (ICP). Based on the determined relationship between the first set of 3D feature points and the second set of 3D feature points, the three-dimensional image of the target object is the processed image (e.g., image 440) of the target object. In some embodiments, such as in a Cartesian coordinate system, the determined relationship may include but is not limited to a first shift along the X axis; a second shift along the Y axis; a third shift along the Z axis; a first rotation angle along the X axis; a second rotation angle along the Y axis; and a third rotation angle along the Z axis. In various coordinate systems, the determined relationship may be different.

[0036] Step 360 may be followed by step 370 of “Matching the 3D model to the matched surface”. In some embodiments, based on the relationship between the aligned first set of 3D feature points and the second set of 3D feature points determined in step 360, the first surface (related to the target object obtained in step 310) is matched with the second surface (related to the 3D model constructed based on the medical scan in step 320). Several points of the 3D model may define the second surface. Therefore, based on the determined relationship as described above, the 3D model constructed in step 320 may be rotated and / or shifted to match the surface defined by several points of the 3D model with the second surface.

[0037] Figure 5 A flowchart of an example procedure for enhancing information obtained from a medical scan of a target object on an extended reality image of the target object is provided, which is configured according to some embodiments of the present invention. Procedure 500 may include one or more operations, functions, or actions, as illustrated by steps 510, 520, and / or 530, which may be performed by hardware, software, and / or firmware. The various steps are not intended to be limited to the described embodiments. The steps and operations outlined are provided as examples only, and some of the steps and operations may be combined into fewer steps and operations, or expanded into additional steps and operations as needed without detracting from the essence of the disclosed embodiments. Although the steps are illustrated in a sequential order, the steps may be performed in parallel and / or in an order different from that described in this specification.

[0038] Process 500 may start from step 510 of “superimposing a three-dimensional (3D) image on a first extended reality image”. In some embodiments, in combination with Figure 2 , step 510 may correspond to step 270. For example, in step 510, the three-dimensional image of the target object obtained in step 210 may be superimposed on the first extended reality image of the anatomical point obtained in step 240 to enhance the additional information obtained in step 240 on the first extended reality image obtained in step 210.

[0039] Step 510 may be followed by step 520 of “obtaining a second extended reality image based on the matched 3D model”. Figure 3 , step 370 may be followed by step 520. For example, in step 520, the second extended reality image may be obtained based on the 3D model matched in step 370. The second extended reality image may include one or more surface and subsurface (e.g., tissue or organ) XR images of the patient's head, which are to be displayed in a wearable device (e.g., a head-mounted device, smart glasses, etc.). In various alternative embodiments, the second extended reality image may include one or more surface and subsurface XR images of the patient's tissue (e.g., blood vessels) or organ (e.g., liver or brain) to be displayed in the wearable device. These XR images include information captured in physical reality (e.g., medical scan images of the patient), as well as images displayed in virtual reality.

[0040] Step 520 may be followed by step 530 of “superimposing the second extended reality image on the first extended reality image”. In some specific embodiments, because the second extended reality image is obtained based on the 3D model matched in step 370, the surface image of the second extended reality image is to be matched with the first surface (related to the target object obtained in step 310) (which is also a part of the three-dimensional image of the target object obtained in step 210). After identifying the first surface from the three-dimensional image of the target object obtained in step 210, the second extended reality image may be superimposed on the first extended reality image based on the first surface. Because the second extended reality image is obtained from the 3D model in step 370 (which is also constructed based on the medical scan in step 320), as discussed above, the information obtained from the medical scan (including subsurface information) may be enhanced on the first extended reality image of the target object.

[0041] Figure 6Flowchart of an example process for obtaining an extended reality image of a target object as a surgical tool moves toward a subsurface region of the target object, configured in accordance with some embodiments of the present invention. Process 600 may include one or more operations, functions, or actions, as illustrated by steps 610, 620, 630, and / or 640, which may be performed by hardware, software, and / or firmware. In some embodiments, in combination with Figure 1 , the procedure 600 may be performed as the surgical tool passes through the entry point 140 and reaches the instantaneous point 150. In some embodiments, in combination with Figure 5 , process 600 may be performed after step 530. The steps described are not intended to be limited to the specific embodiments described. The steps and operations outlined are provided as examples only, and some of the steps and operations may be combined into fewer steps and operations, or expanded into additional steps and operations as needed without detracting from the essence of the disclosed specific embodiments. Although the steps are illustrated in a sequential order, the steps may also be performed in parallel and / or in an order different from that described in this specification.

[0042] Process 600 may begin at step 610 of “obtaining a third extended reality image”. In some embodiments, the third extended reality image may correspond to a subsurface region of the target object (eg, Figure 1 150) to simulate the field of view when the surgical tool reaches the subsurface area of ​​the target object. The surgical tool can be attached to a robotic arm in an operating room (which has a known spatial relationship with the patient). In some embodiments, combined with Figure 5 After step 530, the second extended reality image is superimposed on the first extended reality image. Therefore, in step 610, the third extended reality image can be obtained based on the second extended reality image and the known spatial relationship.

[0043] Step 610 may be followed by step 620 of “obtaining a three-dimensional (3D) image of the subsurface region”. In some embodiments, the 3D image of the subsurface region may be obtained by a camera or an ultrasonic sensor attached to the surgical tool when the surgical tool physically reaches the subsurface region.

[0044] Step 620 may be followed by step 630 of “calculating the deviation between the third extended reality image and the 3D image”. In some embodiments, the deviation between the third extended reality image obtained in step 610 and the 3D image obtained in step 620 is calculated by any technically feasible method. The deviation may be caused by the invasion of the surgical tool. For example, the brain includes very soft tissues. These tissues are easily displaced from their original positions with the invasion of foreign objects (e.g., the surgical tool).

[0045] Step 630 may be followed by step 640 of “obtaining a fourth extended reality image”. In some embodiments, the third extended reality image obtained in step 610 may be updated by compensating for the deviation calculated in step 630 to obtain a fourth extended reality image. Therefore, the fourth extended reality image may correspond to an image of the relevant subsurface area to simulate the field of view when the surgical tool physically reaches the subsurface area of ​​the target object. Therefore, the fourth extended reality image may include information obtained from the medical scan and may facilitate the surgeon to perform surgery in response to the displacement of one or more tissues or one or more organs.

[0046] In some embodiments, methods 200, 300, 500, and 600 may be connected to a wearable device (e.g., HoloLens) computer. The computer can provide an extended reality platform that provides a real-world experience (e.g., images) on the wearable device. The wearable device is constructed as described above. Figure 2 , Figure 3 , Figure 5 ,and Figure 6 The extended reality image is described in the display.

[0047] The above examples may be implemented by hardware (including hardware logic circuits), software, or firmware, or a combination thereof. The above examples may be implemented by any suitable computing device, computer system, wearable device, etc. The computing device may include a processor, a memory, and a physical network adapter (NIC), which may communicate with each other via a communication bus, etc. The computing device may include a non-transitory computer-readable medium having instructions or program codes stored thereon that are executed by the processor, so that the processor executes the instructions of this specification. Figure 2 , Figure 3 , Figure 5 ,and Figure 6 For example, the computing device may communicate with a wearable device and / or one or more sensors.

[0048] The techniques described above may be implemented in dedicated hard-wired circuits, in software and / or firmware in conjunction with programmable circuits, or in a combination thereof. The dedicated hard-wired circuits may be in the form of, for example, one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc. The term "processor" is broadly interpreted to include a processing unit, an ASIC, a logic unit, or a programmable gate array, etc.

[0049] Some aspects of the specific embodiments disclosed in this specification may be (in whole or in part) equivalently implemented in an integrated circuit as one or more computer programs executed on one or more computers (e.g., as one or more programs executed on one or more computing systems), one or more programs executed on one or more processors (e.g., as one or more programs executed on one or more microprocessors), firmware, or substantially any combination thereof, and the circuit may be designed and / or the code may be written for the software and / or firmware based on the contents of this disclosure.

[0050] Software and / or other instructions for implementing the techniques described herein may be stored on a non-transitory computer-readable storage medium and executed by one or more general or special purpose programmable microprocessors. As used herein, "computer-readable storage medium" includes any mechanism that provides (i.e., stores and / or transmits) information in a form that can be accessed by a machine (e.g., a computer, a network device, a personal digital assistant (PDA), a mobile device, a manufacturing tool, any device with one or more processors, etc.). Computer-readable storage media may include recordable / non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic or optical storage media, flash memory devices, etc.).

[0051] From the foregoing, it will be understood that the various embodiments of the present invention are described in this specification for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the present invention. Therefore, the various embodiments disclosed in this specification are not intended to be limiting.

Claims

1. A method for enhancing medical scan information about a target object on an extended reality image, comprising: Obtaining a three-dimensional image of the target object; Using an artificial intelligence engine to identify a first set of three-dimensional feature points from the three-dimensional image based on edges, contrast, and shapes; identifying anatomical points based on the first set of three-dimensional feature points; obtaining a first extended reality image related to the anatomical point; The artificial intelligence engine selects a second set of three-dimensional feature points from the first extended reality image based on edges, contrast, and shapes; performing first image matching between the first set of three-dimensional feature points and the second set of three-dimensional feature points; and superimposing the three-dimensional image on the first extended reality image based on the first image matching; Wherein the method further comprises: constructing a three-dimensional model based on the medical scan information; selecting a third set of feature points from the three-dimensional model; Performing a second image matching between the first set of three-dimensional feature points and the third set of three-dimensional feature points to identify matching surfaces between the three-dimensional image and the three-dimensional model; matching the three-dimensional model with the matching surface; Obtaining a second extended reality image based on the matching three-dimensional model; as well as The second extended reality image is superimposed on the first extended reality image based on the matching surface to enhance medical scan information about the target object. 2 . The method of claim 1 , wherein the identifying anatomical points further comprises marking the first set of three-dimensional feature points on the three-dimensional image. 3 . The method of claim 1 , wherein the first extended reality image comprises one or more tissues, one or more organs, or one or more organ systems of the patient including the anatomical point. 4 . The method of claim 1 , wherein the matching the three-dimensional model further comprises rotating or shifting the three-dimensional model to match a surface defined by a plurality of points of the three-dimensional model with the matching surface.

5. The method of claim 1, wherein the matching surface is a surface of one or more tissues, one or more organs, or one or more organ systems of the patient.

6. The method as claimed in claim 1 further comprises obtaining a third extended reality image corresponding to a subsurface area of ​​one or more tissues, one or more organs, or one or more organ systems of the patient, wherein the third extended reality image is an image related to the subsurface area to simulate the field of view when the surgical tool reaches the subsurface area.

7. The method of claim 6, further comprising: As the surgical tool physically reaches the subsurface region, obtaining additional three-dimensional images of the subsurface region; and A deviation between the third extended reality image and the further three-dimensional image is calculated, wherein the deviation corresponds to a displacement of the one or more tissues, one or more organs, or one or more organ systems of the patient. 8 . The method of claim 7 , further comprising obtaining a fourth extended reality image corresponding to the subsurface area based on the third extended reality image and the deviation.

9. A system for enhancing medical scan information about a target object on an extended reality image, comprising: one or more processors; a wearable device coupled to the processor; as well as A non-transitory computer readable medium having instructions stored thereon for causing the one or more processors to, when executed by the one or more processors, perform a method for enhancing medical scan information related to a target object on a first extended reality image, the method comprising: Obtaining a three-dimensional image of the target object; Using an artificial intelligence engine to identify a first set of three-dimensional feature points from the three-dimensional image based on edges, contrast, and shapes; identifying anatomical points based on the first set of three-dimensional feature points; obtaining a first extended reality image related to the anatomical point; The artificial intelligence engine selects a second set of three-dimensional feature points from the first extended reality image based on edges, contrast, and shapes; performing first image matching between the first set of three-dimensional feature points and the second set of three-dimensional feature points; and Superimposing the three-dimensional image on the first extended reality image based on the first image matching A non-transitory computer-readable medium having stored thereon additional instructions for execution by the one or more processors, causing the one or more processors to: constructing a three-dimensional model based on the medical scan information; selecting a third set of feature points from the three-dimensional model; Performing a second image matching between the first set of three-dimensional feature points and the third set of three-dimensional feature points to identify matching surfaces between the three-dimensional image and the three-dimensional model; matching the three-dimensional model with the matching surface; Obtaining a second extended reality image based on the matching three-dimensional model; as well as The second extended reality image is superimposed on the first extended reality image based on the matching surface to enhance the medical scan information about the target object.

10. The system of claim 9, wherein the wearable device is configured to display one or more of the first extended reality image and the second extended reality image.

11. The system of claim 9, wherein a non-transitory computer readable medium has stored thereon additional instructions for execution by the one or more processors, causing the one or more processors to mark the first set of three-dimensional feature points on the three-dimensional image.

12. The system of claim 9, wherein a non-transitory computer-readable medium has stored thereon additional instructions that, upon execution by the one or more processors, cause the one or more processors to rotate or translate the three-dimensional model to match a surface defined by a plurality of points of the three-dimensional model with the matching surface.

13. A system as described in claim 9, having a non-temporary computer-readable medium having additional instructions stored thereon so as to be executed by the one or more processors, so that the one or more processors obtain a third extended reality image corresponding to a subsurface area of ​​one or more tissues, one or more organs, or one or more organ systems of a patient, wherein the third extended reality image is an image of the subsurface area to simulate the field of view when the surgical tool reaches the subsurface area.

14. The system of claim 13, wherein the non-transitory computer readable medium has stored thereon additional instructions for execution by the one or more processors, causing the one or more processors to: As the surgical tool physically reaches the subsurface region, obtaining additional three-dimensional images of the subsurface region; and A deviation between the third extended reality image and the further three-dimensional image is calculated, wherein the deviation corresponds to a displacement of the one or more tissues, one or more organs, or one or more organ systems of the patient.

15. The system of claim 14, wherein a non-transitory computer-readable medium has stored thereon additional instructions that, when executed by the one or more processors, cause the one or more processors to obtain a fourth extended reality image corresponding to the subsurface area based on the third extended reality image and the deviation.

16. The system of claim 15, wherein the wearable device is configured to display one or more of the first extended reality image, the second extended reality image, the third extended reality image, and the fourth extended reality image.