Augmented reality system for real-space navigation and surgical system using it

Abstract

The present disclosure relates to an augmented reality system for real space navigation, the augmented reality system comprising: a navigation module for determining a set of spatial coordinates for each of a plurality of navigational landmarks corresponding to a location in a three-dimensional real space; and a virtual image display module for displaying a virtual image associated with one of the plurality of navigational landmarks such that the virtual image is perceived by a user as being at a location in the three-dimensional real space, the virtual image being composed of at least one binocular pixel, each of the binocular pixels being formed by a first optical signal projected onto a first retina of the user and a second optical signal projected onto a second retina of the user.

Description

Related Applications

[0001] This application claims priority to Provisional Application No. 63 / 228,171, filed August 2, 2021, entitled "Apparatus and Method with Mixed Reality for Guiding Needle Insertion." [Technical Field]

[0002] The present invention relates to an augmented reality based navigation system, and more particularly to an augmented reality based navigation system capable of displaying virtual images that guide a user to perform and complete tasks. The present invention further relates to an augmented reality based navigation system for guiding medical personnel to perform and complete medical procedures, such as surgery. [Background technology]

[0003] In traditional practice, medical professionals must rely on medical records displayed on a computer screen as a reference for performing medical procedures. Medical professionals often need to look back and forth between the patient and the computer screen, which can be tedious. Furthermore, determining the correct location and path for manipulating surgical instruments during a medical procedure often relies heavily on the medical professional's experience. As an example, the insertion of an epidural needle into the spinal region for epidural anesthesia requires precise insertion location and orientation of the epidural needle to prevent damage to collateral tissue. Anesthesiologists often must rely on ultrasound imaging devices to determine the location and orientation of the epidural needle insertion. However, ultrasound imaging devices cannot be placed directly at the insertion site but rather to the side of the insertion site, which undesirably limits the accuracy of the images and increases the likelihood of procedural failure.

[0004] In recent years, many augmented reality-assisted medical procedures have been proposed. However, the technology for accurately mapping the positions of three-dimensional virtual objects perceived by a user to real physical space has not yet been fully developed. Therefore, an augmented reality-based navigation system for assisting medical procedures has not yet been realized.

[0005] Furthermore, many current waveguide-based augmented reality / virtual reality displays capable of displaying virtual images with varying depths suffer from focal rivalry, which arises from the fact that the distance from the user's eyes to the display screen at which the user's eyes are focused does not match the depth perception of the virtual image perceived by the observer. This can cause discomfort to the user, and the user may not be able to focus on the real object and the virtual image simultaneously. Summary of the Invention [Problem to be solved by the invention]

[0006] Based on the above reasons, a novel augmented reality system for real space navigation that can solve the above problems is desired. [Means for solving the problem]

[0007] The present invention is advantageous over the prior art in overcoming focal rivalry and vergence accommodation conflict (VAC) in virtual / mixed reality displays. In the field of augmented reality or mixed reality, the depth perception and three-dimensional effect of virtual images are often rendered via parallax imaging technology. Parallax images of virtual objects for the left and right eyes are displayed on a screen at a certain distance from the observer's eyes, respectively, but this distance is often temporally different from the depth perception of the apparent point in the rendered virtual image. Furthermore, when the goal is to create augmented reality or mixed reality by superimposing a virtual image on a real object, the virtual image displaced by the screen and the real object cannot be simultaneously focused by the observer's eyes because the depth of the real object and the screen are at different distances from the observer's eyes.

[0008] The present invention eliminates the use of a display screen and implements direct retinal scanning technology, which projects an image onto the retina of the observer's eye. As a result, the observer does not need to stare at a fixed screen. Furthermore, the virtual image is projected to the observer's eye at a convergence angle that matches the binocular natural vision. In other words, the depth perception of the virtual image matches the convergence angle in natural vision. This eliminates both focus conflict and VAC.

[0009] An augmented reality assistance system for performing a medical procedure on a patient includes: a navigation module for determining a set of spatial coordinates corresponding to a position in three-dimensional real space for each of a plurality of navigational landmarks based on diagnostic information of the patient, the plurality of navigational landmarks corresponding to a target position or a target orientation of a surgical instrument in the three-dimensional real space for performing the medical procedure; and a virtual image display module for displaying a virtual image associated with one of the plurality of navigational landmarks such that the virtual image is perceived by a user as being at a position in the three-dimensional real space, the virtual image being composed of at least one binocular pixel, each of the binocular pixels being formed by a first optical signal projected onto a first retina of the user and a second optical signal projected onto a second retina of the user. The virtual image display module includes a left optical signal projector and a right optical signal projector. The left optical signal projector and the right optical signal projector may use lasers as light sources. In one embodiment, the left and right optical signal projectors are laser beam scanning projectors (LBS projectors), which may include red, green, and blue lasers, optical color adjusters such as a dichroic combiner and a polarization combiner, and a 2D adjustable reflector such as a two-dimensional (2D) electromechanical system ("MEMS") mirror. The 2D adjustable reflector can be replaced with two 1D reflectors such as two 1D MEMS mirrors. As an example, the LBS projector sequentially generates and scans optical signals one by one to form a 2D image with a predetermined resolution, e.g., 1280x720 pixels per frame. Thus, one optical signal per pixel is generated and projected toward the combiner at a time.

[0010] The depth coordinate perceived by the user in real space for each of at least one binocular pixel having a specific horizontal and vertical coordinate is rendered by projecting a first light signal and a second light signal, respectively, onto a pair of specified positions on the surfaces of the first and second retinas that are specific to the perception of the depth coordinate, regardless of the projection angle of the first light signal onto the first retina and the projection angle of the second light signal onto the second retina.

[0011] According to one embodiment of the present invention, the virtual image display module is a head-wearable device, and the series of spatial, depth, horizontal, and vertical coordinates are measured relative to the position of the head-wearable device. In another embodiment of the present invention, the series of spatial, depth, horizontal, and vertical coordinates are measured relative to the navigation module. The vertical or horizontal coordinate perceived by the user in real space of at least one binocular pixel of the virtual image is rendered by projecting the first and second optical signals to a pair of specified positions on the surfaces of the first and second retinas having vertical or horizontal positions corresponding to the vertical or horizontal coordinate, regardless of the projection angle of the first optical signal onto the first retina and the projection angle of the second optical signal onto the second retina.

[0012] According to an embodiment of the present invention, the pair of designated positions comprises a first designated position and a second designated position, and a change in the depth coordinate of each of the at least one binocular pixel perceived by the user is rendered by changing the relative distance between the first designated position and the second designated position.

[0013] According to an embodiment of the present invention, a surgical instrument includes an insertion portion for insertion into a patient, and a medical imaging device is provided coaxially adjacent to the insertion portion. The surgical instrument may further include an orientation detection module for determining an orientation of the surgical instrument with respect to real space. The surgical instrument may further include a penetration depth detection module for determining a penetration depth of the surgical instrument into the patient.

[0014] According to an embodiment of the present invention, a medical imaging device may provide diagnostic information regarding the spatial position of each of a plurality of physiological or anatomical features of a patient. The medical imaging device may provide real-time information regarding the patient. The medical imaging device may further provide real-time information regarding the spatial deviation of one of a plurality of alignment reference points relative to one of a plurality of navigation landmarks.

[0015] According to an embodiment of the present invention, a plurality of alignment reference points are assigned to a surgical instrument, and the position of each of the plurality of alignment reference points is determined by a navigation module. The navigation module determines a spatial deviation of each of the plurality of alignment reference points relative to one of the plurality of navigation landmarks. The virtual image module may output a visual cue to a user when the spatial deviation is greater than a first predetermined value or less than a second predetermined value. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 illustrates a navigation landmark according to an embodiment of the present invention.

[0017] [Figure 2] FIG. 2 shows a virtual image of a navigation landmark and an augmented reality system for real-space navigation according to an embodiment of the present invention.

[0018] [Figure 3A] FIG. 3A illustrates a navigational landmark and an augmented reality system for real space navigation according to an embodiment of the present invention.

[0019] [Figure 3B] FIG. 3B illustrates an exemplary coordinate system according to an embodiment of the present invention.

[0020] [Figure 4] FIG. 4 illustrates the principle of natural binocular vision according to the present invention.

[0021] [Figure 5] FIG. 5 illustrates the principle of rendering different depth perceptions at specific horizontal and vertical coordinates according to an embodiment of the present invention.

[0022] [Figure 6] FIG. 6 illustrates the principle of rendering multiple binocular pixels in a 3D real space according to an embodiment of the present invention.

[0023] [Figure 7A] FIG. 7A shows a lookup table containing pairs of designated locations with corresponding spatial coordinates.

[0024] [Figure 7B] FIG. 7B shows how to precisely project an optical signal to a desired, specified location.

[0025] [Figure 8] FIG. 8 illustrates the principle of rendering multiple binocular pixels in a 3D real space according to another embodiment of the present invention.

[0026] [Figure 9A] FIG. 9A illustrates an exemplary implementation of the present invention.

[0027] [Figure 9B] FIG. 9B illustrates an exemplary implementation of the present invention.

[0028] [Figure 10] FIG. 10 illustrates an exemplary embodiment of a surgical procedure implementing an augmented reality system for real-space navigation according to the present invention.

[0029] [Figure 11] FIG. 11 illustrates an exemplary surgical instrument according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0030] The terms used in the following description are intended to be interpreted in the broadest reasonable manner, even when used in conjunction with a detailed description of specific embodiments of the technology. Although certain terms may be emphasized below, terms intended to be interpreted in a restrictive manner shall be specifically defined as such in this detailed description section.

[0031] In the present invention, navigational landmarks are used to guide a user to complete a task requiring precise manipulation of an instrument, tool, or human hand in three-dimensional space; an example of such manipulation may be a medical or dental procedure. In some cases, navigational landmarks are a series of spatial coordinates indicating the correct location or orientation of an instrument manipulated by a human to complete the task. For example, (see FIG. 1 ), a navigational landmark may include coordinates indicating a location on a patient for an anesthesiologist to administer an epidural anesthesia injection. A navigational landmark may also include multiple coordinates indicating the exact location of the tip of an epidural needle or the tail of a syringe to indicate the correct orientation / angle of the injection. Referring to FIG. 1 , the navigational landmarks are denoted as NL1, NL2, etc. The coordinates of the navigational landmarks in this example may be predetermined based on, for example, medical image data of a patient, which may indicate the location of the space between the patient's vertebrae. The coordinates of the navigational landmarks indicate the proper orientation and location of epidural needle insertion to successfully insert the epidural needle into the space between the patient's vertebrae while avoiding damage to the nervous system. In some other examples, the navigational landmarks may indicate multiple location coordinates for a dentist to perform a dental implant. In the present invention, an augmented reality-based display may display a virtual image corresponding to the location of the navigational landmarks, so that the user can see the correct location and orientation in three-dimensional real space to perform a task. Those skilled in the art will understand that the present invention can be applied to other applications without departing from the spirit of the present invention.

[0032] An augmented reality system for real-space navigation that can be applied to various situations will now be described. Referring to FIG. 2 , the augmented reality system for real-space navigation includes a navigation module 100 and a virtual image display module 200. The navigation module 100 determines a set of spatial coordinates for each of a plurality of navigation landmarks, each corresponding to a position in a three-dimensional real space. More specifically, the navigation module 100 may receive data related to a reference for determining the position of the navigation landmark for a specific task. The navigation module 100 converts the data into three-dimensional coordinates in the real space relative to a reference point. The reference point is designated as the origin of the coordinate system (e.g., having coordinates of (0,0,0)). As an example, the navigation module 100 may set the reference point at a point on the virtual image display module 200, a specified position in the real space, or a point on the navigation module 100, depending on the embodiment. The plurality of navigation landmarks correspond to a target position or a target orientation of a task subject in the three-dimensional real space for performing a task. For example, if the task is to inject an intravenous anesthetic, the task subject may be an epidural needle. For other types of medical procedures, the task subject may be a corresponding tool for performing the medical procedure. In some embodiments, the navigation module 100 is provided on the virtual image display module 200, or the navigation module 100 is an indoor positioning system provided separately from the virtual image display module 200.

[0033] By way of example, a navigation module 100 according to the present invention may include a positioning unit for determining the user's position (if the origin is not set on the user's head-wearable device), the position of a surgical tool or surgical site, etc. The positioning unit may be implemented with GPS (indoor or outdoor), a cellular network, or Wi-Fi for outdoor positioning methods. The positioning unit may be implemented with UWB, Bluetooth, a wireless network, or a beacon for indoor positioning. In embodiments in which the navigation module 100 is provided on a head-wearable device, the navigation module 100 may include a depth-sensing unit. The depth-sensing unit may be used to measure the distance between any point on the target object and the user (more specifically, the midpoint between the user's eyes). The position of the user's hand or surgical tool may be measured by the depth-sensing unit. A depth map may be used to track the movement of the target object, surgical tool, and / or hand. The depth map may be created by the depth-sensing unit and / or a camera. The depth map is further used to cluster the target object and the hand. The depth sensing unit may be implemented in the form of a depth sensing camera. The depth sensing camera captures two-dimensional or three-dimensional real-time images of the target object, and the distance between the camera and the target object can also be determined by the depth sensing unit. The navigation module 100 may further include an inertial measurement unit (IMU) to assist in determining the user's position and orientation.

[0034] In some embodiments of the present invention, the virtual image display module 200 may include an eye-tracking unit for determining the user's selection of a target object according to the user's visual axis. The user's gaze is determined by the eye-tracking unit. Eye-tracking is achieved by an eye-tracking camera or by measuring electrical signals of eye movements. The virtual image display module 200 may also include a hand gesture recognition unit. Hand gestures and hand positions may be captured by a depth-sensing unit or a camera. The depth-sensing unit or the camera provides information about the hand gesture to the hand gesture recognition unit, which then recognizes the hand gesture. The virtual image display module 200 may further include an object recognition unit. An image and position of the target object are captured by a depth-sensing unit or a depth-sensing camera. The object recognition unit performs object recognition of the target object based on the image of the target object. In some cases, the object recognition unit may recognize surgical instruments, the user's hands, and the patient.

[0035] The virtual image display module 200 is configured to display a virtual image associated with any one of multiple navigational landmarks to enable a user to visualize the navigational landmarks for navigational purposes. In particular, the virtual image is perceived by the user as being at a location in three-dimensional real space. As an example, if a user of the present invention is performing a medical procedure, the virtual image may be a circular spot indicating the insertion position of a surgical instrument on a patient, or the virtual image may resemble a virtual image of a surgical instrument indicating the desired position or orientation of the actual surgical instrument, as shown in FIG. 2. While the navigational landmarks may be represented as coordinates of points in three-dimensional real space, the visual representation of the navigational landmarks may be centered on the coordinates of the navigational landmarks and occupy a small area in three-dimensional space, or the virtual image may be rendered based on the location of the navigational landmarks. Thus, the virtual image may be composed of at least one binocular pixel (as shown in FIG. 2). For a retinal scanning-based augmented display system, each binocular pixel is formed by a first optical signal projected onto the user's first retina and a second optical signal projected onto the user's second retina. The present invention uses retinal scanning technology to render virtual images of navigational landmarks. The method for rendering the virtual image and binocular pixels as perceived by the user at a particular real space location is described in detail below.

[0036] 3A, in the present invention, the virtual image display module 200 includes a first optical signal generator 10 (e.g., a right optical signal generator), a first combiner (e.g., a right combiner), a second optical signal generator 20 (e.g., a left optical signal generator), and a second combiner (e.g., a left combiner). The first optical signal generator generates a first optical signal, and the first combiner 11 redirects the first optical signal toward a first retina of a user to display a first pixel p1. Similarly, the second optical signal generator 20 generates a second optical signal, and the second combiner 21 redirects the second optical signal toward a second retina of a user to display a second pixel p2.

[0037] The first and second optical signal projectors 10, 20 included in the virtual image display module may use lasers as light sources. In one embodiment, the first and second optical signal projectors 10, 20 are laser beam scanning projectors (LBS projectors), which may include red, green, and blue lasers, optical color adjusters such as a dichroic combiner and a polarization combiner, and a 2D adjustable reflector such as a two-dimensional (2D) electromechanical system ("MEMS") mirror. The 2D adjustable reflector can be replaced with two 1D reflectors such as two one-dimensional (1D) MEMS mirrors. As an example, the LBS projector sequentially generates and scans optical signals one by one to form a 2D image with a predetermined resolution, e.g., 1280x720 pixels per frame. Thus, one optical signal per pixel is generated and projected toward the combiner at a time. For a viewer to see such a 2D image from one eye, the LBS projector must generate a light signal for each pixel, e.g., 1280 x 720, consecutively within the time period of the image persistence, e.g., 1 / 18 of a second. Thus, each light signal has a duration of approximately 60.28 nanoseconds.

[0038] After the user's first and second eyes perceive the first and second optical signals, the human brain creates an image of a binocular pixel (e.g., BP1) through the fusion of the images of the first and second optical pixels. The binocular pixel is perceived by the user as having specific three-dimensional coordinates. For convenience of describing the present invention, the origin of the coordinate system may be set at the center of the virtual image display module 200 (which may be a head-wearable device), and the three-dimensional coordinates correspond to specific horizontal, vertical, and depth coordinates relative to the head-wearable device (as shown in FIG. 3B ).

[0039] It is easy to see that the horizontal and vertical positions of binocular pixels in three-dimensional space perceived by a user are directly related to the horizontal and vertical positions on the first and second retinas where the first and second optical signals are projected and received, respectively. However, the present invention considers that the depth positions of binocular pixels perceived by a user are also related to the horizontal and vertical positions on the first and second retinas where the first and second optical signals are projected and received, respectively. Referring to FIG. 4, this figure illustrates the perception of the horizontal, vertical, and depth positions of objects in 3D space by natural human binocular vision. For the convenience of explaining the principles of human vision and retinal scanning, the retinas of the user's first and second eyes are depicted as matrices, with each matrix element corresponding to a specific horizontal and vertical position on the retina. According to natural vision, a first right light instance R1 from an object arrives at matrix element R22 on the first retina. A corresponding second light instance L1 from the object arrives at matrix element L22 on the second retina. In addition to the disparity information of an object contained in R1 and L1, a user's depth perception also depends on the convergence angle CA1 between the first light instance R1 and the second light instance L1. As the observer's perceived depth of an object increases, the convergence angle decreases; conversely, as the observer's perceived depth of an object decreases, the convergence angle increases. Specifically, as shown in Figure 4, when an object is moved from position p1 to p2, the convergence angle changes from CA1 to CA2 (CA2 > CA1), while the location on the first retina receiving the first light instance changes from R22 to R23, and the location on the second retina receiving the second light instance changes from L22 to L12. Clearly, the depth perception of an object is at least partially related to the convergence angle (in addition to the disparity image) between the first and second light instances entering the observer's eyes. In natural vision, due to scattering of light, there may be an infinite number of first and second light instances from a point on an object, but due to the effect of the eye's lens, the first and second instances all converge to a single location, so only a single instance is shown in Figure 4.Furthermore, according to FIG. 4, it can be seen that each convergence angle formed between the first light instance R1 and the second light instance L1 has a corresponding relative horizontal distance (denoted as d1 and d2) between the first light instance R1 and the second light instance L1 on the two retinas. Thus, the depth of an object perceived by a user can also be considered to be associated with the relative horizontal distance between the position where the first light instance R1 is projected on the first retina and the corresponding position where the second light instance L1 is projected on the second retina. In other words, the deeper an object is perceived by a user, the smaller the relative horizontal distance between the position on the retina receiving the first light signal and the position on the retina receiving the second light signal. However, from another perspective, the relative distance between the first light instance and the second light instance can be measured at a position closer to the anterior region of the pupil. In this regard, the relative horizontal distance between two light instances that form a larger convergence angle (the object is closer to the observer) will be smaller than the relative horizontal distance between two light instances that form a smaller convergence angle (the object is farther from the observer). That is, the further back the actual object is perceived by the user, the greater the relative horizontal distance between the light instances that form the image of the actual object before entering the pupil. Based on the above principles, the depth perception of an object can be manipulated by changing the relative distance between the light instances that form the image before entering the eye, or by changing the relative distance between the locations on the retina that receive the light instances.

[0040] Referring to FIG. 5, this figure illustrates a method for rendering depth perception based on the above-described principles according to the present invention. FIG. 5 illustrates a first binocular pixel BP1 formed by fusing a first optical signal S1 and a second optical signal S2 having a first convergence angle CA1, and a second binocular pixel BP2 formed by fusing a third optical signal S3 and a fourth optical signal S4 having a second convergence angle CA2. The first binocular pixel BP1 is rendered by projecting optical signals onto a pair of designated positions R22 (first designated position) and L22 (second designated position). The first binocular pixel BP1 is perceived by the user as having a greater depth (i.e., being farther away from the user) than the second binocular pixel BP2. The second binocular pixel BP2 is rendered by projecting optical signals onto a pair of designated positions R32 (first designated position) and L12 (second designated position). The horizontal distance between the third light signal S3 and the fourth light signal S4 on the retina (the distance between R32 and L12) is greater than the horizontal distance between the first light signal S1 and the second light signal S2 on the retina (the distance between R22 and L22). As shown in FIG. 5 , to render a binocular pixel having a depth coordinate d1 at a horizontal coordinate h1 and a vertical coordinate v1, it is necessary to provide light signals to a pair of designated positions R22 and L22. To render a binocular pixel having a depth coordinate d2 at the same horizontal coordinate h1 and vertical coordinate v1, it is necessary to provide light signals to a pair of designated positions R32 and L12. Therefore, by projecting light signals to different first and second designated positions, it is possible to render a change in the depth coordinate of each of at least one binocular pixel perceived by a user.

[0041] Based on the above principle, in one embodiment in which the origin of the three-dimensional coordinate system is set at the center of the head-wearable device, the depth coordinate perceived by the user in real space for each binocular pixel having a specific horizontal and vertical coordinate is rendered by projecting a first optical signal and a second optical signal onto a pair of designated positions (e.g., R22 and L22 or R32 and L12) on the surfaces of the first and second retinas, respectively. Each pair of designated positions renders the perception of a specific depth coordinate to the user. In the above description, a 3×3 matrix is ​​used to explain this principle of human binocular vision, but it is clear that the retina can be divided into a matrix larger than 3×3 (e.g., a 100×100 matrix or a 1000×1000 matrix). Furthermore, this example is used to illustrate the idea that for every depth coordinate, there exists a designated position on the first retina and another corresponding designated position (pair of designated positions) on the second retina, onto which optical signals can be projected so that the user can perceive the binocular pixel at that specific depth coordinate. Furthermore, the vertical or horizontal coordinate of at least one binocular pixel of the virtual image perceived by the user in real space is rendered by projecting the first light signal and the second light signal to a pair of specified positions on the surfaces of the first retina and the second retina, the vertical or horizontal positions corresponding to the vertical or horizontal coordinate, regardless of the projection angle of the first light signal onto the first retina and the projection angle of the second light signal onto the second retina. As long as the light signals are projected to specific positions on the retina, the human eye can recognize the binocular pixel at the corresponding position in real space, regardless of the angle of light incident on the retina.

[0042] Furthermore, FIG. 6 illustrates the relationship between binocular pixels and pairs of designated locations that form the binocular pixels. In this example, the first retina and the second retina are each divided into 36 (6×6) designated locations. It is known that for visual fusion to occur, the image projected to the right eye and the corresponding image projected to the left eye must have similar vertical positions (relative to the human eye). Therefore, pairs of designated locations must have substantially the same vertical positions (relative to the human eye). In FIG. 6, the optical path extension of one optical signal intersects with the optical path extension of the corresponding optical signal on the same row (i.e., the same vertical position). Based on this premise and taking into account the vertical and horizontal coordinates, a total of 216 (6×6×6) virtual binocular pixels with different three-dimensional coordinates (indicated by dots) can be created.

[0043] 7A, a lookup table may be constructed for the virtual image display module 200 to quickly identify the correct pair of designated positions for rendering depth coordinates at specific vertical and horizontal coordinates for binocular pixels. For example, 216 virtual binocular pixels numbered 1 to 216 are formed by projecting light signals to 36 (6x6) designated positions on a first retina and 36 (6x6) designated positions on a second retina. A first binocular pixel BP(1) having horizontal coordinate h1, vertical coordinate v1, and depth coordinate d1 is rendered by a pair of designated positions R(11) and L(11), and a second binocular pixel BP(2) having horizontal coordinate h2, vertical coordinate v2, and depth coordinate d2 is rendered by a pair of designated positions R(12) and left pixel L(11). Thus, to display binocular pixels at specific three-dimensional coordinates in real space, the optical signals projected to the user's left and right eyes must be received at corresponding designated locations on the surface of the user's retina based on information in the lookup table.

[0044] In practice, to accurately project a light signal onto a desired, designated location on the observer's retina, the location where the projected light signal enters the pupil is an important factor to consider. In other words, the locations where the first and second light signals enter the pupil must be controlled so that the light signals are incident on the correct location on the retina, rendering binocular pixels at specific spatial locations. The area just before the pupil receives the incident light signal can also be viewed as a matrix, and this area can be divided into several subunit areas, similar to the aforementioned areas of designated locations on the retina (see FIG. 7B). Each subunit area SA corresponds to a designated location DL on the retina. Therefore, when a light signal enters the pupil at a specific angle through a specific subunit area, the corresponding designated location on the retina that will receive the light signal can be predicted. In one embodiment, a lookup table may be constructed to determine the location where the light signal enters the pupil, ensuring that the light signal is received at the desired designated location on the retina. In one embodiment, the relationship between the subunit areas and the corresponding designated areas on the retina is shown in FIG. 7B. In this example, for the light signal to be received at a pair of designated locations R32 and L12 on the retina, the light signal must pass through subunits R12 and L32 before entering the area before entering the pupil. Thus, to change the depth of the binocular pixels perceived by the observer from one location to another, the light signal can be projected through a different pair of subunit areas SA associated with the target designated locations on the retina, thereby allowing the target pair at the designated locations to receive the light signal.

[0045] Further, referring to FIG. 8 , based on the above principle for rendering depth coordinates of binocular pixels, the depth perception of binocular pixels is independent of the projection angles of the first and second optical signals onto the user's first and second eyes. As long as a pair of designated positions corresponding to a particular depth coordinate receives an optical signal, binocular pixels having the depth coordinate can be rendered regardless of the projection angle of the first optical signal onto the first retina and the projection angle of the second optical signal onto the second retina. In the above-described method for rendering depth perception, each designated position has a fixed relative position on the retina. This means that the designated positions cannot be assigned randomly or artificially; they are the result of human physiology and retinal anatomy. When stimulated by an optical signal, each pair of designated positions can render unique three-dimensional coordinates for binocular pixels, and each pair of designated positions has a unique position on the user's retina. Furthermore, in order for the first optical signal and the second optical signal to be fused by the human brain to generate a single binocular pixel, the information contained in the first optical signal and the second optical signal must be substantially the same.

[0046] By using the aforementioned method of rendering binocular pixels that can be perceived by a user as being at specific locations in three-dimensional real space, the virtual image display module 200 can display an image at a specific location in three-dimensional real space corresponding to a navigational landmark (as shown in FIGS. 9A and 9B ). An image corresponding to a navigational landmark may be composed of at least one binocular pixel. In one example, an image of a navigational landmark may be displayed to a user so that the user knows exactly where (where in real three-dimensional space) to perform a task (e.g., surgery or epidural needle insertion). The navigational landmark may also indicate the correct position and orientation of equipment for performing the task (e.g., the orientation of a scalpel or epidural needle for performing surgery or epidural needle insertion).

[0047] 10, the following is an exemplary embodiment of a surgical procedure implementing an augmented reality system for real space navigation according to the present invention. The surgical procedure may comprise the following steps: (s1) The navigation module assigns a unified global coordinate system in real space to the patient, the virtual image display module, the surgical instruments, and the medical personnel. The virtual image display module and / or the medical imaging device perform coordinate calibration with respect to the unified global coordinate system (e.g., vertical coordinate, horizontal coordinate, and depth coordinate). (s2) A medical professional performs imaging of the patient over a general area near the target surgical site using a medical imaging device. The medical imaging device may be a magnetic resonance imaging device, a CT scan, or an ultrasound imaging device. The image data is converted into a set of spatial coordinate data relating to anatomical features near the target surgical site on the patient. (s3) The set of spatial coordinate data for the anatomical features is sent to a navigation module to construct a three-dimensional model of the patient's anatomy. (s4) The navigation module determines appropriate positions and angles (or orientations) for manipulating the surgical instrument in the 3D real space based on a three-dimensional model of the patient's anatomy. The navigation module generates a plurality of navigation landmarks associated with the positions and angles (or orientations) of the surgical instrument for performing the surgical procedure. (s5) The virtual image associated with the navigation landmark is displayed to the medical professional via a virtual image display module (e.g., a head-wearable device). (s6) The medical professional can align the surgical instrument with the virtual image for navigation. The navigation module determines the deviation between the surgical instrument and the navigation landmarks and provides feedback to the medical professional.

[0048] In the following description, epidural anesthesia will continue to be used as an example to illustrate the application of the augmented reality system for real space navigation according to the present invention.

[0049] The augmented reality assistance system may include a navigation module 100 for determining a series of spatial coordinates corresponding to a position in three-dimensional real space for each of a plurality of navigational landmarks. The navigational landmarks are defined based on diagnostic information of a patient. For example, the diagnostic information may be a real-time ultrasound scan image of the patient's spine, which indicates the locations of vertebrae and spaces between the vertebrae. A plurality of navigational landmarks may be defined by the navigation module 100, and the navigational landmarks correspond to a target position or a target orientation of a surgical instrument (e.g., an epidural needle) in three-dimensional real space for performing a medical procedure (e.g., epidural anesthesia). For example, the navigational landmarks may indicate the best position and orientation for an epidural needle to be inserted into the space between vertebrae for epidural anesthesia.

[0050] The virtual image display module 200 displays a virtual image associated with a plurality of navigation landmarks so that the virtual image is perceived by the user as being at a specific position in three-dimensional real space. The virtual image display device in this embodiment may be a head-wearable device. The origin of a coordinate system representing the coordinates (depth coordinate, horizontal coordinate, and vertical coordinate) of the navigation landmarks and the virtual image is set to the position of the head-wearable device. In this embodiment, the navigation module 100 may be provided on the head-wearable device.

[0051] The virtual image may be an arrow or an epidural needle to guide the healthcare professional in holding the epidural needle in the correct orientation when inserting it into the patient. The virtual image is composed of at least one binocular pixel, and as described above, each binocular pixel is formed by a first optical signal projected onto the user's first retina and a second optical signal projected onto the user's second retina. In some embodiments, the position of the virtual image perceived by the user in three-dimensional space corresponds to the correct position for inserting the epidural needle into the patient. Two additional virtual images of two navigational landmarks indicating the correct positions of the two ends of the epidural needle may be displayed so that the healthcare professional can match the positions of the actual ends of the epidural needle to the two virtual images of the navigational landmarks visible to the healthcare professional to obtain the correct orientation of the epidural needle.

[0052] In an alternative embodiment of the present invention, the virtual image may resemble an epidural needle to allow the healthcare professional to match the position and orientation of the actual epidural needle with the virtual image of the epidural needle. The relative distance between the healthcare professional and the virtual image perceived by the healthcare professional (wearing a head-wearable device) may be dynamically adjusted based on the healthcare professional's movement or changes in position. The relative orientation of the virtual image perceived by the healthcare professional can also be dynamically adjusted in response to changes in the healthcare professional's position. This may be achieved by a position module that dynamically calculates the three-dimensional coordinates in real space relative to the origin of the navigation landmark (or virtual image) (which may be set to the position of the head-wearable device and may move with the healthcare professional), and the virtual image display module 200 (i.e., the head-wearable device) then dynamically adjusts and renders the virtual image based on changes in the healthcare professional's position. In this embodiment, the origin of the coordinate system may be set to the position of the virtual image display module 200 (i.e., the head-wearable device). However, as mentioned above, the origin of the coordinate system may be set to a position other than the position of the virtual image display module 200, especially when the navigation module 100 is not provided on a head-wearable device. For example, in some cases, the origin may be set to the navigation module 100, and the position of the navigation module 100 may be fixed relative to the room where the surgery is performed. Nevertheless, the coordinates of the navigation landmarks and the head-wearable device can be measured and calculated relative to the navigation module 100, and the position of the virtual image perceived by the medical personnel can be adjusted based on the relative position between the origin of the coordinate system and the medical personnel (the person wearing the head-wearable device).

[0053] To enhance the alignment between the actual epidural needle and the virtual image and allow the medical professional to administer epidural anesthesia more accurately, in some embodiments of the present invention, the navigation module 100 may assign multiple alignment reference points on the actual epidural needle. In this embodiment, the navigation module 100 may further include an object recognition module for recognizing features of the actual epidural needle and assigning specific features as alignment reference points, and a position sensing module for sensing the positions of these alignment reference points. Navigational landmarks may be associated with the alignment reference points. That is, in some cases, the navigational landmarks relate to the correct coordinates where the alignment reference points should be located in real space during epidural needle insertion. The navigation module 100 may be able to compare the positions of the alignment reference points with corresponding navigational landmarks to determine the spatial deviation of the alignment reference points relative to their corresponding navigational landmarks. Additionally, the virtual image display module 200 may output a visual cue to alert the medical personnel if the spatial deviation is greater than a predetermined upper limit of the allowable spatial deviation, or alternatively, the virtual image display module 200 may output another visual cue to confirm that the epidural needle is on the correct path for insertion if the deviation is less than a predetermined lower limit.

[0054] In one embodiment of the present invention, diagnostic information is received from a medical imaging device attached to the surgical instrument. The diagnostic information may include the spatial location of each of multiple physiological or anatomical features of the patient. In an example where the surgical instrument is an epidural needle, the navigation module 100 may determine the best route for performing epidural needle insertion based on the diagnostic information, and the navigation module 100 defines navigational landmarks based on the best route for performing epidural needle insertion. To obtain accurate and distortion-free diagnostic information, it is preferable for the medical imaging device to perform real-time imaging as close as possible to the epidural needle insertion site. Referring to FIG. 11 , to solve this problem, the medical imaging device 50 may be coaxially attached near the insertion portion 40 (i.e., the epidural needle) of the surgical instrument. Furthermore, because the patient is not always stationary (e.g., the patient may change body position), the medical imaging device 50 needs to update the diagnostic information about the patient in real time, and therefore, the spatial coordinates corresponding to the positions of the navigational landmarks are configured accordingly in real time. Meanwhile, the medical imaging device 50 and / or the navigation module 100 may provide real-time information regarding the spatial deviation of the alignment reference points relative to the navigation landmarks.

[0055] In some embodiments of the present invention, the surgical instrument may further include an orientation detection module 60 (see FIG. 10 ) for determining the orientation of the surgical instrument relative to real space. The orientation detection module 60 then provides orientation data of the surgical instrument to the navigation module 100 for determining the spatial deviation of the alignment reference point relative to the navigation landmark. By way of example, the orientation detection module 60 may be a gyroscope. The surgical instrument may also include a penetration depth detection module for determining the penetration depth of the surgical instrument into the patient. By way of example, the depth detection module may be a pressure sensor or an optical sensor. The penetration depth detection module may assist in detecting the position of the surgical instrument in a three-dimensional coordinate system of real space to determine whether the surgical instrument has reached an appropriate penetration depth relative to the surface of the surgical target (e.g., the patient's skin).

[0056] The following are exemplary embodiments showing a practical implementation of epidural anesthesia using an augmented reality-assisted system according to the present invention, in which an epidural needle has an insertion portion for insertion into a patient, and a medical imaging device is provided coaxially adjacent to the insertion portion.

[0057] In a first embodiment, epidural anesthesia may comprise the following steps: (1) The epidural needle is in a retraction mode. The epidural needle equipped with a medical imaging device can be moved to the patient's skin area to obtain 3D image data (diagnostic information) of the anatomical structure from the medical imaging device (e.g., a circular ultrasound transducer or 3D ultrasound) and construct a 3D model of the patient's anatomical structure. (2) Build a 3D model of the patient's anatomy. (3) Determine the appropriate position and angle (or orientation) for inserting the epidural needle in 3D real space, which is displayed to the medical professional via a head-wearable device. (4) A head-worn navigation device allows the healthcare professional to align the epidural needle in space with a projected virtual image of the epidural needle. (5) The central epidural needle is inserted manually or by an automated pushing device. During insertion, the depth of the epidural needle is monitored by a medical imaging device (e.g., an anterior ultrasound transducer near the tip of the epidural needle). Once the specified depth is reached, the central epidural needle is removed. (6) Attach the syringe to the epidural needle. Push the entire epidural needle deeper until a loss of resistance is detected by the pressure or optical sensor.

[0058] In a second embodiment, epidural anesthesia may comprise the following steps: (1) A medical imaging device (e.g., a 3D ultrasound scan) is used to scan a patient's skin area to obtain 3D image data of the anatomical structures and construct a 3D model of the patient. (2) Determine the appropriate position and angle (or orientation) for the insertion of the epidural needle in 3D real space. The position and angle are displayed to the medical staff via a head-wearable device. The medical staff can then use the head-wearable device to confirm the path of the epidural needle from the simulation results. (3) The head-wearable device projects a virtual image of the epidural needle at a preferred insertion location on the skin with a preferred insertion angle according to the results of step (2). (4) The healthcare professional can align the epidural needle with a virtual image of the epidural needle projected in 3D space by a head-wearable device. (5) The central epidural needle is inserted manually or by an automated pushing device. During insertion, the depth of the epidural needle is monitored by a medical imaging device (e.g., an anterior ultrasound transducer near the tip of the epidural needle). Once the designated area is reached, the central epidural needle is removed. (6) Attach the syringe to the epidural needle. Push the entire epidural needle deeper until a loss of resistance is detected by the pressure or optical sensor.

[0059] In a third embodiment, epidural anesthesia may comprise the following steps: (1) A medical imaging device (e.g., a 3D ultrasound scan) is used to scan a patient's skin area to obtain 3D image data of the anatomical structures and construct a 3D model of the patient. (2) Determine the appropriate position and angle (or orientation) for the insertion of the epidural needle in 3D real space. The position and angle are displayed to the medical staff via a head-wearable device. The medical staff can then use the head-wearable device to confirm the path of the epidural needle from the simulation results. (3) Remove the medical imaging device. The head-wearable device projects a virtual image of the epidural needle at the preferred insertion position on the skin with the preferred insertion angle according to the results of step (2). (4) The healthcare professional can align the epidural needle with a virtual image of the epidural needle projected in 3D space by a head-wearable device. (5) The central epidural needle is inserted manually or by an automated pushing device. During insertion, the depth of the epidural needle is monitored by a medical imaging device (e.g., an anterior ultrasound transducer near the tip of the epidural needle). Once the designated area is reached, the central epidural needle is removed. (6) Attach the syringe to the epidural needle. Push the entire epidural needle deeper until a loss of resistance is detected by the pressure or optical sensor.

[0060] The foregoing description of the embodiments is provided to enable any person skilled in the art to make and use the subject matter. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the novel principles and subject matter disclosed herein may be applied to other embodiments without the exercise of innovative faculty. The claimed subject matter is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. Additional embodiments are contemplated within the spirit and true scope of the disclosed subject matter. Thus, it is intended that the present invention cover modifications and variations that come within the scope of the appended claims and their equivalents.

Claims

1. 1. An augmented reality assistance system for performing a medical procedure on a patient, comprising: a navigation module for determining a set of spatial coordinates corresponding to a location in three-dimensional real space for each of a plurality of navigational landmarks based on patient diagnostic information, the plurality of navigational landmarks corresponding to a target position or a target orientation of a surgical instrument in the three-dimensional real space for performing the medical procedure; a virtual image display module for displaying a virtual image associated with one of the plurality of navigation landmarks such that the virtual image is perceived by a user as being at the location in the three-dimensional real space, the virtual image being composed of at least one binocular pixel, each of the binocular pixels being formed by a first light signal projected onto a first retina of the user and a second light signal projected onto a second retina of the user; a depth coordinate perceived by the user in real space for each of the at least one binocular pixel having a specific horizontal coordinate and vertical coordinate is rendered by projecting the first light signal and the second light signal, respectively, onto a pair of designated positions on the surfaces of the first retina and the second retina that are specific to the perception of the depth coordinate; the first optical signal and the second optical signal are projected onto the pair of specified positions, respectively, such that the user perceives the depth coordinate regardless of a projection angle of the first optical signal onto the first retina and a projection angle of the second optical signal onto the second retina; the pair of designated positions comprises a first designated position and a second designated position, the first designated position having a fixed relative position on the first retina, and the second designated position having a fixed relative position on the second retina; An augmented reality assistance system, wherein changes in the depth coordinates of each of the at least one binocular pixel perceived by the user are rendered by changing the relative distance between the first specified position and the second specified position.

2. The augmented reality support system of claim 1 , wherein the first light signal and the second light signal have the same image information.

3. The augmented reality support system of claim 1 , wherein the virtual image display module is a head-wearable device, and the series of spatial coordinates, the depth coordinate, the horizontal coordinate, and the vertical coordinate are measured relative to the position of the head-wearable device.

4. The augmented reality assistance system of claim 1 , wherein the set of spatial coordinates, the depth coordinate, the horizontal coordinate, and the vertical coordinate, are measured relative to the navigation module.

5. 2. The augmented reality assistance system of claim 1, wherein the vertical or horizontal coordinate perceived by the user in the real space of the at least one binocular pixel of the virtual image is rendered by projecting the first light signal and the second light signal to a pair of specified positions on the surfaces of the first retina and the second retina having a vertical or horizontal position corresponding to the vertical or horizontal coordinate, regardless of the projection angle of the first light signal onto the first retina and the projection angle of the second light signal onto the second retina.

6. The augmented reality assistance system of claim 1 , wherein the diagnostic information is received from a medical imaging device mounted on the surgical instrument.

7. The augmented reality support system according to claim 6 , wherein the surgical instrument comprises an insertion portion for insertion into the patient, and the medical imaging device is provided coaxially near the insertion portion.

8. The augmented reality-assisted system of claim 1 , wherein a plurality of alignment reference points are assigned to the surgical instrument, and the position of each of the plurality of alignment reference points is determined by the navigation module.

9. 7. The augmented reality assistance system of claim 6, wherein the medical imaging device provides real-time information about the patient, and the series of spatial coordinates corresponding to positions in three-dimensional real space for each of a plurality of navigation landmarks are set according to the real-time information.

10. The augmented reality assistance system of claim 1 , wherein the virtual image display module further comprises a left light signal projector and a right light signal projector that project a plurality of left light signals and a plurality of right light signals, respectively.

11. The augmented reality assistance system of claim 1 , wherein one of the plurality of navigational landmarks is associated with a location for performing a medical procedure on an object of interest.

12. A surgical instrument according to claim 1, wherein a plurality of alignment reference points are assigned to the surgical instrument, and the position of each of the plurality of alignment reference points is determined by the navigation module; The augmented reality assistance system of claim 6 , wherein the medical imaging device provides real-time information regarding a spatial deviation of one of the plurality of alignment reference points relative to one of the plurality of navigation landmarks.

Images