Reference to anatomical structures

Abstract

This paper describes a technique for referencing anatomical structures using an AR headset. This method may include registering markers located on distal anatomical structures. The markers may have a posture, including their position and orientation in the 3D coordinate system of the AR headset. Another action may be to identify the proximal anatomical structures to which the distal anatomical structures are connected. The markers may be registered in a second posture having a second position and a second orientation. Joint rotation axes and angles may be determined for the proximal anatomical structures by comparing the first and second postures. The joint rotation axes may then be displayed using the AR headset. This technique may also automatically find anatomical pathways and anatomical lengths using a pointer device.

Description

TECHNICAL FIELD

[0001] Mixed reality or augmented reality is an area of computing technology where views from the physical world and images from the virtual computing world can be combined into a mixed real world. In mixed reality, people, places, and objects from the physical world and the virtual world form a blended visual and auditory environment. Along with the use of VR (Virtual Reality) or AR (Augmented Reality) headsets, mixed reality experiences may be provided via existing commercial or custom software.

[0002] Augmented reality (AR) is an example of mixed reality in which a live direct view (or indirect view) of a physical real-world environment is augmented or supplemented by computer-generated sensory inputs such as audio, video, graphics or other data. The augmentation is performed in relation to environmental elements, with the real-world position being observable. With the support of advanced AR technologies (e.g., the addition of computer vision and object recognition), information about the real world around the user may be interactive and digitally modified. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] [Figure 1] This is a diagram illustrating an example of identifying and referencing a joint pivot axis at the knee. [Figure 2] This illustrates an example of a variable circle (or slice of a cylinder) that defines an isometric radius for ligament insertion from the hinge point of the femur to the patella. [Figure 3] This is a diagram illustrating an example of a first marker on the distal anatomical structure of a joint and a second marker on the proximal anatomical structure of the joint. [Figure 4A] This shows an example of identifying the joint pivot axis or hinge point of the knee joint between the tibia and the femur. [Figure 4B] This illustrates another example of identifying the joint pivot axis of the knee joint between the tibia and the femur. [Figure 5] This illustrates an example of tracking the tibia in a leg during extension to identify the joint pivot axis of the knee joint between the tibia and the femur. [Figure 6] This is an example of an optical marker near the junction of the femur and tibia (distal femur). [Figure 7] This flowchart illustrates an exemplary method for referencing anatomical structures using an AR headset. [Figure 8] This paper illustrates an example of referencing anatomical distances using pointing devices, virtual lines, image datasets, and augmented reality (AR) headsets. [Figure 9] Figure 7 illustrates an example of a sagittal view of the vertebrae and pointing device. [Figure 10] This example illustrates how to measure anatomical distances for use in fracture repair. [Figure 11] An example of a 3D scan or 3D image dataset of a knee in an extended position with markers is illustrated. [Figure 12] This is an example of a sagittal view of the knee in an extended or straight position. [Figure 13] An example of a bone segment identifiable near a human joint will be illustrated. [Figure 14] This example illustrates how anatomical segments can rotate around joint rotation axes in image datasets. [Figure 15] This example illustrates how anatomical segments can rotate around joint rotation axes in image datasets. [Modes for carrying out the invention]

[0004] Herein, we will refer to the embodiments illustrated in the drawings and describe them using the specific language used herein. Nevertheless, it will be understood that no limitation of the scope of this art is intended therein. Modifications and further alterations to the features described herein, as well as additional uses of the embodiments described herein that a person skilled in the art who owns this disclosure might conceive, should be considered within the scope of this specification.

[0005] This technology may include the use of augmented reality (AR) systems to improve the ability to identify anatomical features (e.g., joint rotation axes), provide guidance in medical procedures related to human joints (e.g., knee, elbow, or hip surgery), and reposition dislocated and / or broken portions of limbs or human anatomical structures. The guidance may enable medical professionals to determine the distance from a joint rotation axis or point of rotation to another site on the bone, thereby allowing the bone to be cut at the appropriate site. Furthermore, the length of anatomical structures may be tracked and measured using an augmented reality (AR) headset and a medical image dataset aligned with visible human anatomical structures. While conventional medical imaging can show a patient's anatomical structures aligned with the patient, this technology can go further and inform medical professionals about the location of joint rotation axes and isometric sites for ligament attachment around hinges.

[0006] In one example, during knee replacement surgery, a medical professional may need to know the length of the leg or part of the leg. The medical professional may also want to take measurements to perform a saw cut so that the leg being operated on is the same length as the other leg. This technique further allows the medical professional to locate the joint rotation axis, the pivot point of the ball joint (e.g., the hip joint), the upper part of the leg, or other important anatomical sites that may be difficult to identify during the medical procedure. In another example, this technique may be used to measure the length of the femur by determining its origin.

[0007] A system and method for referencing anatomical structures using an AR headset may be provided. Figure 1 illustrates an example of identifying and referencing the joint rotation axis at the knee. To identify the joint rotation axis point, a marker 110 may be registered on a distal anatomical structure 112. The marker 110 may have a posture including its position and orientation in the 3D coordinate system of the AR headset. The distal anatomical structure 112 may be a readily movable part of the leg. Multiple points may be registered in the optical code. This may allow determining the position and orientation of the marker in relation to its posture, and / or identifying the center, corner, or edge of the marker.

[0008] In addition, a proximal anatomical structure 114 to which a distal anatomical structure 112 is connected can be identified. The proximal anatomical portion 114 of the structure may be part of a leg attached to the patient's body. This proximal anatomical portion 114 may be considered fixed because the patient may be attached to an operating table or a wheeled stretcher. The term fixed may also define a proximal anatomical structure or portion that does not move during a medical procedure due to some restraint or the weight of the proximal anatomical structure. The patient's proximal anatomical portion may also be fixed when the patient is fixed in a position or similar fixed position when the patient is lying down or sitting. Alternatively, the distal anatomical structure of a person may be fixed, while the proximal portion of a person may be moved (e.g., using a lift, a wheeled stretcher, etc.).

[0009] Marker 110 may also be detected in a second posture 116 having a position and orientation relative to a proximal anatomical structure 112'. The AR headset may receive a command from the user and detect marker 110' in the second posture. In one example, the AR headset may be configured to continuously track marker 110 from the first posture until the marker stops in the second posture.

[0010] The joint rotation axis 118 can be determined by comparing a first posture with a second posture. The joint axis point 118 can then be displayed using an AR headset. In one example, the joint rotation axis 118 is located along the surface of the joint bone substantially parallel to the marker in the first posture. It can be identified by using the intersection of a first line extending (or through the axis of symmetry of the bone) and a second line extending along the surface of the bone (or through the axis of symmetry of the bone) substantially parallel to the second posture of the marker.

[0011] In one example, a medical professional may place a marker 110, which is an optical tracker, on the patient's arm, leg, or other anatomical structure while the patient is lying down. As the marker 110 moves, sensors communicating with an AR headset can locate the new location of the marker 110'. Thus, the location of the joint rotation axis and the distance from the tracker 110 to the joint rotation axis can be calculated.

[0012] The term "attitude" can refer to the position and orientation of a single marker in a 3D coordinate system or relative to another marker. A marker may have a 2D region and an identification pattern on that region, so multiple points on the marker can be registered and used to determine the marker's position and orientation. When the attitudes of a single marker are compared for two different positions (e.g., two different times), the second attitude can be analyzed in relation to the first attitude. The comparison involves calculating the changes in position and rotation from attitude 1 to attitude 2 (e.g., from time 1 to time 2). For example, the movement of a marker from point 1 to point 2 can be measured, and then the rotation of the marker in degrees around the center of the marker can be measured and displayed. The rotational component for comparing these two attitudes is around the axis of the marker. The rotation of the marker may be around an axis passing through the center of the marker, but the rotation may also be around any other parallel axis (which also causes a change in position).

[0013] A rotation axis can also be determined by considering the positional change of a marker in a plane perpendicular to the rotation axis. This rotation axis corresponds to the joint rotation axis, insofar as the marker is attached to the rigid part (e.g., bone) on which the rotation occurs. There may also be some unconsidered movement parallel to the rotation axis, which can be represented as bone displacement resulting from the rotation. Calculating the joint rotation axis using a single marker assumes that at least one part of the joint (e.g., the proximal part) is fixed in 3D space. In an alternative configuration without this assumption, a second marker may be used on the other side of the joint, and the calculation may be related to a moving coordinate system that includes both the proximal and distal anatomical structures of the joint (instead of a fixed world system).

[0014] Distal and proximal anatomical structures may include bones that form joints. For example, the proximal anatomical structure may be the femur, and the distal anatomical structure may be the patella or tibia. Other pairs of bones can also be identified as distal and proximal anatomical structures. For example, the humerus may be the proximal anatomical structure, and the radius or ulna may be used as the distal anatomical structure.

[0015] Figure 2 shows an example of a variable circle or variable cylinder defining an isometric radius for ligament insertion from the articular rotation axis 208 (or hinge point) of the femur 216 to the patella 214. Figure 2 may display a plurality of points 202 surrounding the articular rotation axis 208 and being isometric with respect to the articular rotation axis 208. In addition, a calculated rotation angle 204 may be displayed. The plurality of points 202 may form a circle as illustrated, which is part or a slice of a cylinder extending perpendicular to the plane illustrated by Figure 2. The circle, variable circle, or variable cylinder is a virtual circle or virtual cylinder provided as an overlay in an AR headset for viewing by a medical professional or user. The plurality of points may form a virtual display object which is at least one of an arc, circle, cylinder, curved surface, or irregular shape. Furthermore, the arrangement of the points does not have to be continuous as illustrated, and may be one or more disjoint points. Multiple isometric points 202 surrounding the joint rotation axis 208 can be partially displayed based on a defined ligament length to identify, using an AR headset, the locations on bones (e.g., patella, femur, tibia, humerus, radius, etc.) where ligaments can be attached at an isometric distance relative to the joint rotation axis 208 during medical procedures. Multiple points 202 within the circle (or cylinder) may be visible as an overlay on a CT (computed tomography) scan, and the location of the ring (or cylinder) is known in space. In one configuration, a single point at an isometric radius from the joint rotation axis 208 may be shown.

[0016] Markers may be positioned on the femur and patella to find isometric insertion points for the patella and femur. Alternatively, the marker 210 may be placed on the skin of the leg, while the isometric insertion point may be defined based on the expected location of the bone. Figure 2 illustrates the marker 210 on the leg in a flexed position, while the medical image shows the leg in an extended position.

[0017] The joint pivot axis is the center of the pivot axis of the joint and associated ligaments. The isometric point in the form of a circle or cylinder can be dragged out to be larger or smaller from the joint pivot axis based on the measured size of the ligament. For example, the desired ligament length may be equal to the radius of a circle, cylinder, cylinder segment, or arc. The circle or cylinder provides an isometric trajectory on which the end of the ligament can be placed. In the case of Figure 2, the size of the circle (or cylinder) can be adjusted to be larger or smaller until the circle overlies the patella. This circle identifies where the isometric point for the patella is located. The other site where the second point of the ligament is to be placed can be found at the joint pivot axis of the femur or on other related anatomical structures such as the surface of the bone. The region of bone surrounding the joint pivot axis is a location where a drill hole can be positioned to reach the joint pivot axis. In that case, the attachment point may be within the bone at the joint pivot axis.

[0018] In a more specific repair example, a procedure for repairing the patella at the knee may be used. When the knee flexes, the patella rotates anteriorly past the spherical portion of the distal femur. If this ligament is torn, a medical procedure may reattach the patella to the femur. However, if the ligament is too short, the ligament will rotate around the bone, stretch and tear. If the ligament is too long, the ligament will be too loose, and the patient may not be able to properly control the anatomical structure. Therefore, it is important that the second end of the ligament can be accurately attached to an isometric hinge point that is equidistant from the initial ligament attachment as the ligament moves.

[0019] More generally, there is a pivot point where the joint rotates, and the first end of the ligament can be attached to that site. This site varies based on the anatomical structure of the structure, the size of the person, any anatomical abnormalities, etc. A hole may be drilled in the bone to access the joint rotation axis, and a new ligament or ligament replacement can be passed through the hole. Unfortunately, in the prior art, the placement of both ends of the ligament is speculative. As mentioned above, if the ligament is not isometric, when the person moves, the ligament may be too tight and rupture, or too loose, preventing the patient from properly controlling the joint. Preferably, the ligament may be isometric across the hinge point of the elbow or knee. Otherwise, the ligament may stretch and be damaged.

[0020] Accurately identifying hinge points, such as joint rotation axes or joint rotation points, before or during medical procedures can be difficult. This technology enables medical professionals to quickly and accurately identify joint rotation axes or joint rotation points during medical procedures. Joint rotation axes or joint rotation points can be identified without even aligning and overlaying a 3D image dataset onto a human anatomical structure in the 3D space of an AR headset, and without performing medical imaging (e.g., CT (computed tomography) scans or MRI (magnetic resonance imaging) scans) immediately before or during surgery.

[0021] Medical professionals can determine where the ligament should be inserted on each side of the ligament, so that the distance of the ligament endpoints remains isometric with respect to the hinge point as the joint moves through its range of motion. As mentioned above, the marker may be placed on the distal anatomical structure or on the structures of both joints (for example, fixed to the patella with a bone pin). Then the joint hinge The joint axis may be calculated, the joint hinge axis may be used as the first connection point of the ligament, and isometric points or isometric geometric shapes may be used to find the second connection point. The motion discussed here takes place in one plane, but the motion of the joint under consideration can be in any direction.

[0022] The system can calculate where the two ends of a ligament can be more precisely positioned. These positions are measured and calculated. Markers allow for the identification of the joint rotation axis. The joint rotation axis 208 may be an axis passing perpendicularly through the image in Figure 2, and the joint rotation axis 208 is the central axis of the knee. The ligament may need to be positioned on the surface of the bone or on the patella, as illustrated. A surgeon can virtually see where the bones are located when a medical image dataset (e.g., CT scan or MRI) is aligned with the patient. As described, an arc, circle, or cylinder can be centered on the joint rotation axis and, based on measurements, guide where the ligament should be positioned. According to the illustrated example in Figure 2, one end of the ligament may be positioned on the patella at one of the isometric points of the arc, circle, or cylinder, and the second end of the ligament may be positioned on the joint rotation axis or hinge axis. The site where the patella attaches is an isometric point as defined by the arc, circle, or cylinder.

[0023] This process can be used for other types of ligament reconstruction, such as ACL reconstruction. For example, the system can define where the normal attachment of the anterior cruciate ligament on the tibia occurs when a medical professional fabricates a graft through the bone. The medical professional may be provided with a graphical guide for drilling the hole. After the initial attachment point is created, the isometric guide then helps identify where to attach the second end of the ligament. This technique can be useful for isometric ligament reconstruction and similar procedures.

[0024] As described, the AR headset can calculate the change between a first posture and a second posture. Figure 2 shows that the angle between the first and second postures can also be calculated and displayed as an arc 204, or calculated and displayed as an overlay using the AR headset. A numerical output 206 of the angle that defines the change in angle as the marker moves around the joint rotation axis can also be displayed. In addition, the numerical angle 206 of the change between the first and second postures can be displayed as an overlay as the angle changes in real time. More specifically, the angle of displacement of distal anatomical structures can be calculated using the first and second postures, along with a graphical reference for angular displacement displayed using the AR headset. This technique can track the angle of movement, joint axis, distance, etc., using only markers on anatomical structures without requiring additional medical imaging.

[0025] In one example, the process can establish a starting point for measuring angles. First, a marker 210 on a distal anatomical structure 212 may be detected. This site becomes the starting point for angle measurement, and then any movement from a first posture may be tracked, and a report is provided on how far the marker 210 has moved from its starting point. The starting angle, displacement angle, and / or ending angle may be displayed within the AR headset. Optionally, relative angle measurements can be obtained using markers on a person's body or proximal anatomical structures.

[0026] When a medical professional has a CT image open, they can activate a slice view. In this example, the point where the axis intersects with the slice of the CT scan can be seen as point 208 on the angle of the rotation and arc segments. The circle or cylinder encloses the joint rotation axis and indicates where the ligaments are fixed. As mentioned above, the circle, ring 202, or cylinder may be called an isometric ring, isometric cylinder, or isometric orbit.

[0027] The user in an anatomical slice of a CT or MRI scan perpendicular to the joint rotation axis. As the scan progresses (for example, moving deeper or shallower relative to the viewpoint), the joint rotation axis appears throughout the slice. Typically, medical professionals are interested in visualizing the bone during ligament repair procedures. In contrast, when ACL ligament repair is being performed, medical professionals may want to use CT or MRI scan slices to visualize deeper parts of the inside of the knee. Once a medical professional finds the point where the ACL will attach, they may mark that point.

[0028] Once the joint rotation axis 208 is identified in a slice of the image dataset, the ring 202 is at a distance from the joint rotation axis 208 that is the length of the ligament to which it will be reattached (unless the ring moves), so any point on the ring 202 is isometric from the joint rotation axis 208. The ligament can always be attached to the central axis. However, if the ligament is attached to a point on the isometric ring, the ligament must be in the same cross-section of the axis, as is actually illustrated in the slice being viewed. Otherwise, a different distance that is not in the slice (or in another slice of the CT scan) may be used, and the ligament may not be isometrically located.

[0029] The identification and presentation of isometric points can be extended to ball hinges or joints such as those in the hip and shoulder. When the joint rotation point is identified for a ball hinge, isometric points can be drawn using curved surfaces.

[0030] While some joints cannot be simply adapted to illustrate isometric points as in Figure 2, adjustments and representations may be provided in the graphical user interface to accommodate anatomical joint patterns or individual human diversity. For example, in the knee, the joint rotation axis may move to different points as the user moves through knee rotation. This technique can track where that joint rotation axis changes. Representing isometric points for joints where the joint rotation axis does not move (such as the patella and tibia) may not need to be more complex.

[0031] An example of using rotational displacement might be a person who has fractured their femur or hip in an accident. Not only is the bone broken, but it is also rotated around its longitudinal axis from its normal position. A medical professional (e.g., a surgeon) would want to know how much the bone is rotated so that the bone and leg can be returned to their correct or normal position. A CT scan of the leg may be taken before surgery, and the medical professional can determine that the bone is displaced and / or incorrectly rotated and is stationary at a specific angle. This angle is measurable, and the distance is also measurable. The angle of displacement from the correct angle of the leg can be determined by comparing it to the other leg. Assume the bone of the leg has a rotational offset of 30 degrees. Using the known rotation angle, a marker can be placed on the end of the leg (i.e., the moving broken portion). An AR headset can then track this marker in 3D coordinate space. A starting point can be set based on the current position of the bone, and then the bone and leg can be rotated over the measured 30 degrees. Once the correct point is reached, the medical professional knows the procedure is complete. After the initial scan (e.g., CT scan or MRI scan) and measurements, no further imaging is required for such a procedure. Furthermore, this process does not require a navigation system or registration with a sensor array or the like.

[0032] In another example of rotational displacement, an anatomical structure may have rotated 60 degrees, while the bone may have moved or translated to a position away from the other parts of the fractured bone. The system can track displacement and the rotational angle of the displacement in space. A bone may be translated 8 cm (centimeters) and rotated 60 degrees. Multiple points on the marker are tracked, and by tracking many points on the marker, six degrees of freedom orientation are possible. This detailed tracking means that there is only one rotation and translation around a single axis, which can produce rotation using multiple points on the marker (e.g., optical code). Rotation and displacement of anatomical structures (legs, feet, arms, etc.) can be displayed. The user can move an anatomical structure 8 cm while using 60-degree rotation. Movement may be instructed. As described, the axis of rotation may be along the axis of symmetry of the anatomical structure (e.g., along the length of a leg, arm, or bone). When rotation and translation are occurring, displacement or translation in space may be reported to a medical professional or user.

[0033] Image datasets can also be aligned to the human body using markers. Medical images can be acquired and aligned to the human body. For example, CT (computed tomography) scans, MRI (magnetic resonance imaging) images, or other imaging can be displayed as an overlay on the patient and used to acquire measurements of the anatomical aspects of the joint or limb being operated on. U.S. Patent Nos. 9,892,564, 10,475,244, 11,004,271, 10,010,379, 10,945,807, 11,266,480, 10,825,563, 11,237,627, 11,287,874, U.S. Patent Application No. 17 / 706462, “Using Optical Codes with Augmented Reality Displays,” “Image U.S. Patent Application No. 17 / 536009, entitled “Data Set Alignment for an AR Headset Using Anatomic Structures and Data Fitting,” and U.S. Patent Application No. 17 / 978,962, entitled “3D Spatial Mapping in a 3D Coordinate System of an AR Headset Using 2D Images,” describe methods and systems for aligning image datasets from medical imaging devices with the human body, and these descriptions are incorporated herein by reference in their entirety.

[0034] In another configuration, markers can be used to move a portion of an anatomical structure, such as a bone, that is detected and located in a dislocated or fractured position, to the correct position. To perform this function, a target position and target rotation can be set in 3D space to which the marker is to be moved. The positional distance and rotation angle of the displacement of the distal anatomical structure can then be tracked from the starting point using the marker. A rotation axis, which is the longitudinal axis of the bone within the distal or proximal anatomical structure, can be defined. A graphical reference may then be displayed for the positional distance and rotation angle of the displacement as the distal anatomical structure is moved and rotated from the starting point. A graphical message may also be displayed when the target position and target rotation are reached. For example, the distal and proximal anatomical structures could be a fractured bone, a dislocated joint, or a portion of a dislocated anatomical structure.

[0035] Figure 3 shows the use of a first marker 310 on the distal anatomical structure 322 of the joint and a second marker 312 on the proximal anatomical structure 320 of the joint. The distal anatomical structure 322 and the proximal anatomical structure 320 can be referenced to each other, but not in the 3D space of the AR headset.

[0036] A first pose position of a second marker 312 in 3D space can be registered. Then, a target position and target rotation of the second marker 312 can be set in 3D space. The positional distance and rotational angle of the displacement of the distal anatomical structure 322 can be calculated using the first pose position 324 and the second pose position 324'. The positional distance and rotational angle when the distal anatomical structure is moved can also be displayed between the first and second pose positions using an AR headset. When the target position and target area are reached, visual, auditory, tactile (e.g., vibration), or graphical feedback can be provided. For example, a graphical icon may be displayed when the target position or target rotation is reached.

[0037] The first marker 310 and distal anatomical structure 322 may relate to anatomical structures that are not joints. In one configuration, the distal and proximal anatomical structures relate to the fractured part of the bone. It is possible.

[0038] The structure of anatomical structures related to joints can also be segmented using machine learning, machine vision, edge detection, and similar processes. The segments of the anatomical structure can be moved so that they are visible through an AR headset, using changes in the position and orientation of one or more markers. For example, the system could segment different bones related to a joint or human anatomical structure. The segments can be moved within the AR headset and virtually moved using visible markers.

[0039] The joint rotation point of a ball joint for a patient can also be identified and indicated using this technique. Types of ball joints from which the joint rotation point can be determined may include hip or shoulder joints. The joint axis point in a ball joint can be determined by taking multiple pairs of postures, each pair of postures being used to find the joint rotation axis and angle for that posture pair. Multiple rotation axes may intersect each other, thereby identifying a single point that is the rotation point of the ball joint.

[0040] A more specific detail of this joint rotation point discovery process is that a marker may be registered in a first position on the anatomical structure that forms a joint with the proximal anatomical structure. A second position of the marker after the anatomical structure has moved in a first plane relative to the proximal anatomical structure may then be registered. The first joint rotation axis may be registered by identifying the intersection of a first line parallel to the plane created by the point of the first marker and a second line parallel to the plane created by the point of the second marker. The distal anatomical structure (e.g., leg or arm) may then be moved in a second plane of movement, and third and fourth positions with markers may be registered to identify the second joint rotation axis. The intersection of the first and second joint rotation axes of the joint rotation point of the human ball joint may then be found or calculated.

[0041] The image dataset can be aligned to a person using a first marker. The joint rotation point, multiple points within a curved surface surrounding the joint rotation point that are isometric with respect to the joint rotation point, and the image dataset can be displayed using an AR headset. The multiple points within the curved surface can be used to determine isometric points for ligament attachment.

[0042] This technology is convenient because it can be used with markers (e.g., visible or optical markers), thereby avoiding the use of medical images for rotation and alignment. Orthopedic medical professionals frequently move bones, and this system can assist in orthopedic procedures. The ability to track the movement of anatomical structures moved by the medical professional's hands in real time while the bone is moving is beneficial because it avoids the use of external machines, protractors, compasses, etc. Medical professionals can also receive display information about the angle of movement, the axis of rotation of the joint, or an isometric circle or cylinder for the axis of rotation of the joint, which may be useful in such procedures.

[0043] Medical professionals also use AR, but they can perform these procedures without prior planning. For example, rotating a fractured bone may not need to be planned in advance. A medical professional may decide to rotate a femur by 30 degrees for therapeutic purposes, but they do not need to perform imaging during the medical procedure because the system can identify changes in such angles and positions during the medical procedure.

[0044] Figure 4A shows an example of tracking the tibia in an extended leg and identifying the articular rotation axis 402 or hinge point of the knee joint of the leg 400 between the tibia and the femur. A circle 404 (or slice of cylinder) or the entire cylinder may be used with an isometric radius that defines where one end of the ligament may attach to the bone (e.g., the patella), and the center of the circle is where the other end of the ligament may attach (e.g., the femur). It can be defined whether it can attach (on bone). The circle or cylinder may start with a default size of 405, and the user may then drag out the circle 416 or cylinder to increase the size of the circle until it covers or draws the area where the ligament can attach isometrically to the patella. The numerical size of the isometric radius of the circle 416 or cylinder may also be displayed (for example, the default radius may be 10 mm, and the expanded radius may be larger). The joint rotation axis 402 drawn in Figure 4A is illustrated from the right side, as viewed from the viewpoint of the joint rotation axis 402, so that the joint rotation axis 402 extends away from the AR headset viewer in 3D space.

[0045] A marker containing an optical code 406 (more specifically, containing multiple 2D optical codes) can be attached to the leg 400 using, for example, bone pins, wraps, adhesive layers 408 or adhesive pads, and connecting marker structures. The marker 406 can be used to identify the movement of the tibia relative to the femur. The movement of the tibia can be tracked between a first position marked by a first line 410 and a second position marked by a second line 412. The intersection of the first line 410 and the second line 412 can be used to identify the joint rotation axis 402. The first line 410 and the second line 412 may be located in the skin of the limb, along the surface of the bone, or at the center of the bone. The displacement or angle of movement between the first and second positions (e.g., 47.8 degrees), along with the distance of the lateral translational movement between the first and second positions (e.g., 15.4 mm) and the length of the structure (i.e., leg) from the joint rotation axis to the marker (e.g., 138.55 mm), can be displayed as an overlay within the AR headset (414). Figure 4B shows an example of tracking the tibia of a flexed leg (also shown in Figure 4A) and identifying the joint rotation axis of the knee joint between the tibia and femur.

[0046] Figure 5 shows an example of tracking the tibia 506 in an extended leg and identifying the joint rotation axis of the knee joint between the tibia and femur, as shown in Figures 4A and 4B. In addition, Figure 5 shows that an image dataset from a medical imaging device may be displayed as an overlay on the leg. For example, the image dataset may be a CT scan, MRI, X-ray-based image, or another type of medical image. The image dataset may provide a view of a specific slice of the image dataset. In the example in Figure 5, cross-sectional slices of the patella 502, femur 504, and tibia 506 may be seen.

[0047] Figure 6 shows a marker 610 attached to a leg (e.g., at or near the base of the femur). The leg is then moved to various different positions (e.g., two or more positions) using the marker 610, and one joint rotation axis 612 of the joint (one axis at the top of the femoral head) can then be identified. The length of the femur can also be identified by providing the ability to measure from the joint rotation axis to the end of the femur using an image dataset displayed as an overlay using an AR headset. Alternatively, the distance from the joint rotation axis 612 to the marker may be measured. A 2D (two-dimensional) image dataset representing the pelvis (not the hip joint discussed) is drawn to illustrate the ability to include aligned and displayed images as an overlay, although a hip joint image dataset can also be used as an overlay within the AR headset.

[0048] The articular rotation point 612 of the ball joint may be determined by using two posture groups in at least two posture planes of the leg. For example, two postures may be captured in plane A, which can make it possible to identify the articular rotation axis 612. Then, two or more postures may be identified in plane B (not shown), and a second articular rotation axis (not shown) can be identified. The intersection of the articular rotation axes of planes A and B can then be used to identify the articular rotation point of the ball joint of the leg.

[0049] Figure 7 illustrates an exemplary method for referencing anatomical structures using an AR headset. This is a flowchart. This method may include registering markers located on movable anatomical structures, as in block 710. The markers may have a first orientation, including their position and orientation in the 3D coordinate system of the AR headset.

[0050] As shown in block 720, the proximal anatomical structure to which the movable anatomical structure is connected can be identified. In some situations, the movable anatomical structure may also be considered a distal anatomical structure. The proximal anatomical structure may be a structure that is substantially immobile during use of the AR headset, or is substantially fixed in place due to constraint or weight. As shown in block 730, the marker may be detected in a second posture of the movable anatomical structure having a second position and a second orientation.

[0051] As in block 740, the joint rotation axis and angle can be determined by comparing the first posture with the second posture. Then, as in block 750, the joint rotation axis can be displayed using an AR headset. In one example, the proximal anatomical structure may include bone, and the movable anatomical structure (or distal anatomical structure) may be the patella.

[0052] In an alternative configuration, a second marker having a third position and third orientation on the proximal anatomical structure may be registered. The third marker can then be used to calculate the first and second postures relative to the proximal anatomical structure, thereby enabling free movement of the movable anatomical structure (e.g., distal anatomical structure) and the proximal anatomical structure. The marker may be at least one of the following: a visible marker, an optical code, a linear barcode, a 2D barcode, an infrared marker, or a radiopaque marker.

[0053] Multiple points can be displayed around the joint rotation axis, each isometric with respect to the joint rotation axis. These points may form curves, circles, or other geometric shapes or surfaces. For example, the points may form at least one of arcs, circles, cylinders, curved surfaces, spheres, smooth surfaces, or irregular shapes. The multiple isometric points, displayed around the joint rotation axis based on a defined ligament length, can identify areas on the bone where a ligament can be attached at an isometric distance relative to the joint rotation axis during a medical procedure.

[0054] The length of a bone or part of a bone can also be measured from the joint rotation axis to a defined point on the bone. Image datasets aligned to humans using markers can be used to more accurately measure bone length or to measure the distance between the joint rotation axis and a point on the bone. In one exemplary use case, an anatomical landmark may be marked on the femur, and a medical professional can measure the length of the leg from the landmark to the end of the bone.

[0055] The change in angle between the first and second postures can also be calculated. Then, a numerical output may be displayed for the angle that defines the change in angle as the marker moves around the joint rotation axis.

[0056] A graphical guide may be provided to guide surgical access to a joint rotation axis or multiple isometric points on the bone associated with the joint. This may include graphical images or guiding icons to guide medical professionals to the joint rotation axis or isometric points on the bone (e.g., drilling routes, scalpel routes, etc.). For example, the graphical guide may illustrate a graphical shape on the bone indicating where to drill or a channel to be used to drill the bone. The graphical guide may be displayed to the user through an AR headset.

[0057] This technology can be used with any type of mixed reality device. Examples of mixed reality devices in which this technology can be used include VR (virtual reality) devices, flat panel displays using augmented cameras, head-up displays, or interfaces of this technology. Other systems may include those in which the patient's face can be displayed along with a real-time view or real-time video image.

[0058] Figure 8 illustrates a system and method for referencing anatomical distances or automated measurements of medical hardware (e.g., screws or implants) using an augmented reality (AR) headset. More specifically, the technology may use a pointer device to automatically determine anatomical pathways, anatomical lengths, or the length, width, or height of medical implants, thereby determining anatomical pathways and structures during medical procedures. An initial operation may be to determine at least one boundary of an anatomical structure for the human body. An exemplary boundary of an anatomical structure may be the boundary of bones 810 or the shape of bones such as vertebrae. The boundary may include an entire 3D volume, a 2D shape of an anatomical structure displayed in an image dataset, or only one or more parts of the boundary of an anatomical structure.

[0059] Image datasets from medical imaging devices (e.g., CT scans or MRIs) may be aligned to a human body using markers on the human body or anatomical structures. The markers may be optical codes or other types of markers as referred herein. The image datasets may be viewable through an AR headset so that they appear as a graphic overlay on top of a person's actual field of view through the AR headset. The image datasets may be used by an anatomical structure detection service to detect the boundaries of anatomical structures within the image dataset. The boundaries of specific anatomical structures may be identified using edge detection, feature detection, shape detection, morphometric detection, machine learning, or other shape or volume detection processes that may be applied to the image dataset.

[0060] The pointer device 812 may be registered with the AR headset, and the pointer device 812 may have a position and orientation relative to the 3D coordinate system of the AR headset or the anatomical structure 810. A marker 818, such as an optical cord, connected to the pointer device may enable the AR headset to register the position and orientation of the pointer device.

[0061] A virtual line 814 may be generated having a major axis aligned with the major axis of the pointer device 812, and the virtual line 814 may pass through a portion of an anatomical structure. For example, the virtual line may extend from the tip 820 of the pointer device 812 in one direction (or two directions) away from the pointer device. A medical professional may point the tip of the pointer device 812 towards an anatomical structure while the virtual line 814 is being generated. Although the virtual line 814 is shown as a dotted line, the virtual line may be a solid line (not dotted), semi-one-dimensional (e.g., one physical display element in width or height), two-dimensional (e.g., having a limited width), or three-dimensional (e.g., having a limited height and width).

[0062] The measurement can be determined from the entry point 816 where the virtual line 814 enters the boundary of the anatomical structure 810 to an additional point 822 within the anatomical structure. The AR headset may display a numerical output for the length of the virtual line between the entry point 816 where the virtual line enters the anatomical structure and the additional point 822 defined for the anatomical structure. The length of the virtual line 814 can be used to determine the length of a medical device used with the anatomical structure, based on the length of the virtual line 814 between the entry point 816 where the virtual line enters the anatomical structure and the additional point within the anatomical structure. For example, the medical device may be at least one of an implant, screw, needle, stent, or trocar.

[0063] The additional points may be user-defined points within an anatomical structure at any location along the virtual line. In another example, the additional points may be where the virtual line exits an anatomical structure. Alternatively, the additional points on the line may be set distances calculated from the entry point 816. For example, the line may extend 5 cm from the entry point 816.

[0064] In an alternative configuration, the additional points may be locations along the virtual line determined using an inward distance offset 830 from an exit point 824 from which the virtual line can exit an anatomical structure. For example, the exit point 824 of the virtual line may be identified, and then an offset point on the virtual line 814 located 1 cm (centimeter) inward from the exit point (or, if the virtual line 814 is shorter, the exit point) may be calculated. The length of the virtual line 814 may then be measured from the entry point 816 to the offset point forming the additional point 822.

[0065] The measured distance can be used to determine the length of a medical tool (e.g., a screw or rod) that may be used in a portion of an anatomical structure where the line is displayed as an overlay. In an exemplary use of this measurement technique shown in Figure 8, a needle or instrument may be used as a pointer device 812, and a marker 818 may be located on or attached to the device. As a medical professional guides the pointer device 812 toward the anatomical structure, the image dataset can be reconfigured to display different planes relative to the tip of the instrument. The plane or slice of the viewed image dataset may be controlled by the location of the needle relative to the anatomical structure (e.g., showing the layer closest to the coordinates of the tip of the pointer device). Alternatively, the plane or slice of the viewed image dataset may be controlled by a medical professional wearing an AR headset.

[0066] Figure 8 illustrates an axial view of a vertebra with a needle and a virtual line or projected ray projected from the top of the needle. Figure 9 illustrates a sagittal view of the vertebra and needle in Figure 8. AR headsets can provide smaller inserted orthogonal views of image datasets and pointer devices within the AR headset. Image datasets can be viewed as MIP (Maximum Projection) views, direct volume rendering images, or other views of 3D image datasets.

[0067] When a virtual line is projected from a pointing device or the tip of a needle, the virtual line intersects with a first part of an anatomical structure (e.g., bone), and this point can then be automatically recorded or marked by a medical professional as an entry point. The AR headset can track the needle in three dimensions. As previously mentioned, the image dataset can be aligned to the patient via the AR headset and displayed as an overlay on the patient, allowing measurements to be taken. The aligned image dataset can also be used to help determine when the virtual line intersected with an anatomical structure in the 3D coordinate system of the AR headset. The intersection of the virtual line with any boundary of an identified anatomical structure identified from the image dataset can be calculated.

[0068] In a further example, a virtual screw of a defined length may be created to determine whether the virtual screw can be fitted into an identified anatomical structure (e.g., bone). If the screw is too long, it could extend from the other end of the vertebral body and potentially kill the patient (e.g., the screw could strike the aorta). Consequently, by using virtual lines and a pointer device, the virtual screw can be sized to have a safe or desired length. Furthermore, the system may suggest implant or screw sizes based on detected boundaries of anatomical structures and / or lines within anatomical structures. The system may also identify the exit boundary of a virtual line crossing a bone boundary and calculate the safety margin of the screw. This makes it possible to calculate the desired screw length based on the entry point of the virtual line into the anatomical structure and the endpoint in the bone, which has a safety distance or safety factor to consider for the screw length. Furthermore, virtual sizing can be performed in any direction or dimension, such as length, width, or height. For example, the diameter of a screw may be determined to determine whether the screw could damage narrow bone (e.g., the narrow pedicle of the spine).

[0069] This technology can also be used for automated depth measurement. When a medical professional is about to perform a puncture for fluid collection, it is important to know the depth of the area filled with fluid. The medical professional can point a pointer device or instrument towards the area of ​​an anatomical structure where fluid is expected to be present, and a virtual line may point to the area containing the fluid. This allows the medical professional to determine the depth to which the needle or trocar needs to move to strike the fluid area. In addition, this technology may allow medical professionals to measure the anatomical structure around a stent and determine the length of the stent before inserting it into the patient.

[0070] Figure 10 illustrates the measurement of anatomical distances for use in repairing a femoral fracture. A virtual line from a pointing device can be used to identify the length of the bone portion into which a nail or screw will be embedded to bridge the fracture in the broken femur. Knowing the required length of the screw or nail in advance is useful to ensure that the implant is neither too long nor too short when the screw or nail is placed. The virtual line used with the pointer device can help locate the appropriate site and then measure the desired path within the bone. Safety limits can also be determined during the measurement operation, as previously mentioned.

[0071] When augmented reality (AR) headsets are used in medical procedures, planning using the headset during or after a procedure can be cumbersome because medical professionals do not have a mouse or workstation nearby. In addition, the images may change or move based on the medical professional's viewpoint. This technology could give physicians the ability to equip or measure something without interrupting or leaving a procedure unattended.

[0072] Another example of using this technology is the procedure of placing a ventricular catheter in the brain. The generation of virtual lines can be used to determine the distance to the ventricles. This technology can automatically locate the ventricles in a CT scan, and then measurements can be taken. Measurements via an AR headset are useful because medical professionals may not have time to perform measurements due to receiving emergency patients, or because their time is limited based on insurance periods and billing limits.

[0073] In medical procedures, there may be situations where medical professionals wish to align image datasets (e.g., CT scans or MRI scans) with a person or patient's body in order to perform the procedure. However, the image dataset may represent different scan locations of the human anatomical structure than the desired location of the anatomical structure in which the medical procedure is to be performed. For example, in the case of ligament insertion into a joint, the scan for the image dataset may be performed on the medical imaging device when the joint (e.g., leg, elbow, knee) is extended or straight. However, if an endoscopic medical tool is used on a joint (e.g., knee) during the procedure, the medical professional may need to significantly bend the knee (e.g., to about 90 degrees) to insert the endoscopic tool. Furthermore, due to insufficient space within the medical imaging device (e.g., CT scanner or MRI scanner), it is not possible to significantly bend the human anatomical structure within the medical imaging device, nor is it possible to bend the scanner itself. Essentially, medical scanners do not have the practical leeway to bend parts of the body by any significant amount during a medical image scan.

[0074] This technology provides the ability to virtually reposition anatomical structures in an image dataset. The modified image dataset can then be displayed on an AR headset and aligned with a person or patient. The image dataset in Figure 11 shows a 3D scan of a knee in an extended position with markers 1116, 1118, and 1120 on the femur 1110, tibia 1112, and patella 1114 (e.g., using 2D optical codes). The 3D scan may be a 3D CT scan or an MRI scan. Figure 12 shows the knee in an extended position. This is a sagittal view of the knee in an upright or unbent position.

[0075] Image datasets can be aligned to a person using markers associated with or on the person (e.g., 2D optical codes or other markers). Markers located on the person's movable anatomical structures can be registered in an AR headset. The markers can then be detected in a first pose, including their position and orientation in the AR headset's 3D coordinate system.

[0076] The joint rotation axis and angle can be determined by using markers to compare a first and second posture of the movable anatomical structure. For example, the position and orientation of the markers can be detected when the distal portion of the joint is moved to the first and second postures by a person or medical professional.

[0077] The structure of anatomical structures within an image dataset can be segmented to form anatomical segments. For example, the structure of an anatomical structure may relate to a human joint. Figure 13 shows bone segments 1310 and 1312 identified near a human joint. Further examples include the boundaries of bones, organs, blood vessels, or anatomical structures being identified within an image dataset. Segmenting the structure of an anatomical structure may involve detecting bones and anatomical structures using edge detection, feature detection, shape detection, or machine learning.

[0078] Figures 14 and 15 illustrate that anatomical segments can be rotated around joint rotation axes within an image dataset. Figure 14 may be an example of rotating an anatomical segment from the image dataset in Figure 11. Identified segments can be rotated around joint rotation axes to represent flexion in the image dataset and to represent the mobile anatomical structure of a person. For example, an anatomical segment can be rotated around a joint rotation axis to an angle that matches the flexion angle of an actual joint that is visible through and detectable by an AR headset. If the angle of a patient's visible joint is 45 degrees, as visible through and detectable by an AR headset, the anatomical segment can be rotated by 45 degrees. Anatomical segments can be virtually moved and / or rotated within the 3D image space of the image dataset. When the image dataset is aligned with a person, the moved anatomical segments in the modified image dataset may appear to match the visible anatomical posture of the person. More specifically, the modified image dataset can be viewed using an AR headset. In another example, a bone may be segmented, and then an application, service, or process running using an AR tracker can be used to virtually rotate the bone and perform the aforementioned process.

[0079] Traditionally, bones and anatomical structures can be found and segmented within image datasets using AI and other segmentation techniques. However, there has been no method to modify image datasets after they have been captured by medical imaging devices to model changes in human anatomical structures (e.g., joint flexion). Furthermore, there has been no ability to move segmented bones and anatomical structures in a way that is useful in medical procedures. This technique can be used to predict the movement and trajectory of bones and anatomical structures and then apply them to image datasets. More specifically, this technique can identify joint rotation axes and use them to calculate the precise rotational movement of bones and anatomical structures around those joint rotation axes. These modifications can be applied to image datasets without capturing additional image datasets using medical imaging machines. For example, these modifications can be applied during medical procedures, as mentioned above.

[0080] In the case of the leg, with the knee joint in an extended position, the femur and tibia can be segmented from the image dataset. Then, the tibia, as described above, has markers on the joints of a human body. Using the joint rotation axis found with the AR headset, the femur can be virtually and accurately rotated within the image dataset. Insertion points for ligaments, which can be used to plan medical procedures, remain valid in this scenario as well. By segmenting using AI, bones and other identified tissues (e.g., skin, blood vessels, organs, etc.) can rotate around a joint rotation axis or virtual hinge point. This allows medical professionals to maintain the relative alignment of two bones even if the bone and anatomical structure were not scanned at that location. Using the joint rotation axis or other information (e.g., skin location) obtained from using markers while moving the leg (e.g., while finding the joint rotation axis), two independent structures (e.g., bones) can be virtually moved and / or correctly related at the joint.

[0081] Modeling soft tissue can be difficult because it stretches or bends. Tissues proximal to a joint may deform or stretch more when the joint is bent, while soft tissues more distal to the joint can be modeled more predictably because their deformation is reduced. Over time, accurate models of soft tissue and how it deforms in the event of bending or other movements can be developed. These soft tissue deformation models can then be applied to the movement of segmented anatomical structures.

[0082] Some of the functional units described herein are labeled as modules to more specifically emphasize the independence of their implementations. For example, modules may be implemented as hardware circuits including custom VLSI circuits or off-the-shelf semiconductors, transistors, or other discrete components such as gate arrays, logic chips, etc. Modules may also be implemented in programmable hardware devices such as field-programmable gate arrays, programmable array logic, or programmable logic devices.

[0083] Modules may also be implemented in software for execution on various types of processors. An identified module of executable code may contain, for example, one or more blocks of computer instructions that can be organized as objects, procedures, or functions. Nevertheless, the executable files of an identified module do not need to be physically located together and may contain heterogeneous instructions stored in different locations that constitute the module and achieve the purpose stated for the module when logically combined.

[0084] In fact, a module of executable code may be a single instruction or many instructions, and may even be distributed across several different code segments, across different programs and across several memory devices. Similarly, operational data may be identified and exemplified herein within a module, embodied in any preferred form, and organized within any preferred type of data structure. Operational data may be collected as a single dataset or distributed across different locations, including across different storage devices. A module may be passive or active, including an agent capable of performing a desired function.

[0085] The technologies described herein may be stored in computer-readable storage media, including volatile and non-volatile, removable and non-removable media, implemented with any technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, optical disks or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other computer storage media that can be used to store the desired information and the technologies described herein.

[0086] The apparatus described herein may also include communication connections or networking devices and networking connections that enable the apparatus to communicate with other apparatus. A communication connection is an example of a communication medium. Communication medium typically includes any information distribution medium that embodies computer-readable instructions, data structures, program modules, and other data in modulated data signals, such as carrier waves or other transport mechanisms. "Modulated data signal" means a signal having one or more of its characteristics set or modified in such a manner that it encodes the information in the signal. Examples, but not limited to, communication mediums include wired mediums such as wired networks or direct wiring connections, and wireless mediums such as acoustic, radio frequency, infrared, and other wireless mediums. As used herein, the term computer-readable medium includes communication mediums.

[0087] Furthermore, the described features, structures, or properties may be combined in any preferred manner in one or more embodiments. In the preceding description, numerous specific details, such as embodiments of various configurations, were provided to provide a complete understanding of embodiments of the described art. However, those skilled in the art will recognize that the art may be practiced without one or more of the specific details, or with other methods, components, apparatus, etc. In other cases, well-known structures or operations are not illustrated or described in detail to avoid obscuring embodiments of the art.

[0088] While this subject matter has been described in language specific to its structural features and / or operation, it should be understood that the subject matter as defined in the attached claims is not necessarily limited to the specific features and operation described above. Rather, the specific features and operation described above are disclosed as exemplary forms that implement the claims. Numerous modifications and alternative configurations can be devised without departing from the spirit and scope of the described technology.

Claims

1. A method for referencing anatomical structures using an AR headset, The registration of a marker located on a movable anatomical structure, wherein the marker has a first posture including its position and orientation in the 3D coordinate system of the AR headset. Identifying the marker in a second posture of the movable anatomical structure having a second position and a second orientation, A method comprising determining the joint rotation axis and angle by comparing the first posture and the second posture.

2. The method according to claim 1, further comprising using the AR headset to display the joint rotation axis.

3. To register a second marker having a third position and a third orientation on the proximal anatomical structure in the second posture, The method according to claim 1, further comprising using the second marker to calculate the first and second postures relative to the proximal anatomical structure to enable free movement of the movable anatomical structure and the proximal anatomical structure.

4. The method according to claim 1, further comprising displaying a plurality of points surrounding the joint rotation axis, which are isometric with respect to the joint rotation axis.

5. The method according to claim 4, wherein the plurality of points form at least one of an arc, a circle, a cylinder, a curved surface, or an irregular shape.

6. The method according to claim 1, further comprising using the AR headset to display a plurality of isometric points surrounding the joint rotation axis based on a defined ligament length in order to identify a site on the bone where the ligament can be attached at an isometric distance with respect to the joint rotation axis.

7. The method according to claim 1, wherein the length of the bone is measured from the joint rotation axis to a point defined on the bone.

8. Calculating the change in the angle between the first posture and the second posture, The method according to claim 1, further comprising displaying a numerical output of the angle that defines the angle change based in part on the movement of the marker around the joint pivot axis.

9. The method according to claim 1, further comprising aligning an image dataset to a person using the marker attached to the person.

10. The method according to claim 1, further comprising identifying the joint rotation axis as the intersection of a first line extending along the surface of the joint bone substantially parallel to the first posture of the marker and a second line extending along the surface of the bone of the anatomical structure in the second posture.

11. The method according to claim 1, wherein the movable anatomical structure and the proximal anatomical structure include bones that form a joint.

12. The method according to claim 1, wherein the proximal anatomical structure includes bone.

13. To guide surgical access to multiple isometric points on the bone associated with the joint rotation axis or joint. The method according to claim 1, further comprising providing a graphical guide for

14. The method according to claim 1, wherein the marker is at least one of an optical code, a 2D barcode, an infrared marker, or a radiopaque marker.

15. The method according to claim 1, further comprising identifying a fixed proximal anatomical structure in the 3D coordinate system, wherein the movable anatomical structure is connected to the proximal anatomical structure.

16. The method according to claim 15, further comprising determining the joint rotation axis and angle relative to the proximal anatomical structure by comparing the first posture with the second posture.

17. A method for referencing human anatomical structures using an AR headset, To register markers on distal anatomical structures connected to immobile proximal anatomical structures during medical procedures, Identifying a first orientation of the marker and a second orientation of the marker in a 3D (three-dimensional) coordinate system, Using the first and second postures, the rotational angle of the displacement of the distal anatomical structure is calculated, A method comprising using the AR headset to display a graphical reference for the rotation angle of the displacement.

18. Setting the target position and target rotation for the marker in the aforementioned 3D coordinate system, Using the aforementioned marker, the positional distance of the displacement of the distal anatomical structure from the starting point and the rotation angle are tracked. The method according to claim 17, further comprising displaying the graphical reference for the displacement position distance and rotation angle when the distal anatomical structure is moved and rotated from the starting point.

19. The method according to claim 18, further comprising providing at least one of a visual indicator, an auditory indicator, or a tactile indicator when the marker reaches the target position and target rotation.

20. The method according to claim 17, wherein the distal anatomical structure and the proximal anatomical structure are part of at least one of a fractured bone, a rotational anomaly, a dislocation, or a dislocated anatomical structure.

21. A method for referencing the anatomical structure of human joints using an AR headset, To register a first marker on the proximal anatomical structure of the joint and a second marker on the distal anatomical structure of the joint, Identifying the first orientation position of the second marker in 3D (three-dimensional) space, Setting the target position and target rotation of the second marker in the aforementioned 3D space, Identifying a second orientation position of the second marker in 3D space, Using the first and second posture positions, the positional distance and rotational angle of the displacement of the distal anatomical structure are calculated. A method comprising using the AR headset to display the positional distance and rotation angle when the distal anatomical structure is moved between the first and second posture positions.

22. The method according to claim 21, further comprising aligning an image dataset to the person using the first marker or the second marker attached to the person.

23. The method according to claim 21, wherein visual feedback, auditory feedback, tactile feedback, or graphical feedback is provided when the target position and target part are reached.

24. The method according to claim 23, further comprising displaying a graphical icon when the target position or target rotation is reached.

25. Segmenting the structure of the anatomical structures related to the aforementioned joint, The method according to claim 21, further comprising moving the segment so that it can be viewed through the AR headset, using a change in the position and orientation of the first marker or the second marker.

26. The method according to claim 21, wherein the proximal anatomical structure and the distal anatomical structure include a fractured portion of the bone.

27. The method according to claim 21, further comprising defining a rotational axis which is the longitudinal axis of the bone in the proximal anatomical structure or the distal anatomical structure.

28. The method according to claim 21, wherein the first marker and the second marker are at least one of an optical code, a 2D optical code, an infrared marker, or a radiopaque marker.

29. The method according to claim 21, further comprising registering a plurality of points on an optical code.

30. The method according to claim 21, wherein the proximal anatomical structure is fixed in a suitable location within the 3D space.

31. A method for identifying the anatomical structure of a human ball joint using an AR headset, To register markers in the first posture on distal anatomical structures connected to proximal anatomical structures, After the distal anatomical structure is moved relative to the proximal anatomical structure, the second posture of the marker is registered. By comparing the first posture and the second posture, the first joint rotation axis and angle relative to the proximal anatomical structure are determined, The third and fourth postures are registered in the marker to identify the second joint rotation axis, A method comprising determining the intersection point of a first joint rotation axis representing the joint rotation point of the ball joint and the second joint rotation axis.

32. The method according to claim 31, further comprising using the AR headset to display the joint rotation point.

33. The method according to claim 31, further comprising using the AR headset to display a plurality of points surrounding the joint rotation point based on a defined ligament length in order to identify a location on the bone to which the ligament can be attached at an isometric distance from the joint rotation point.

34. The method according to claim 33, wherein the plurality of points are at least one of an arc, a circle, a cylinder, a curved surface, or an irregular shape.

35. The method according to claim 31, wherein the joint pivot point is a ball joint in the person.

36. A method for referencing anatomical distances using an augmented reality (AR) headset, Using a medical image dataset aligned to the human body and an anatomical structure detection service, determine at least one boundary of the anatomical structure of the human body. Using the aforementioned AR headset, register a pointer device having a position and orientation relative to the anatomical structure, The method involves generating a virtual line having an axis aligned with the long axis of the pointer device and passing through a portion of the anatomical structure. A method comprising determining a measurement from an entry point where the imaginary line enters the boundary of the anatomical structure to an additional point within the anatomical structure.

37. The method according to claim 36, further comprising displaying a numerical output for the length of the virtual line between the entry point where the virtual line enters the anatomical structure and the additional point within the anatomical structure.

38. The method of claim 36, further comprising determining the length of a medical device used in conjunction with the anatomical structure based on the length of the virtual line between the entry point where the virtual line enters the anatomical structure and the additional point within the anatomical structure.

39. The method according to claim 38, wherein the medical device is at least one of an implant, a screw, a needle, a stent, or a trocar.

40. The method according to claim 36, wherein the aforementioned additional points are points defined by the user.

41. The method according to claim 36, wherein the additional point is the location where the imaginary line emerges from the anatomical structure.

42. The method according to claim 36, wherein the additional point is a portion along the imaginary line having a distance offset inward from the point where the imaginary line exits the anatomical structure.

43. The method according to claim 36, further comprising determining the boundaries of the anatomical structure using edge detection, feature detection, shape detection, morphometric detection, or machine learning.

44. The method according to claim 36, further comprising using an optical cord connected to the pointer device to enable the AR headset to register the position and orientation of the pointer device.

45. The method according to claim 36, wherein the virtual line extends from the tip of the pointer device.

46. The method according to claim 36, wherein the proximal anatomical structure is fixed in a suitable location in 3D space.

47. A method for the virtual repositioning of anatomical structures within an image dataset, Aligning the image dataset to the person using a marker associated with that person, The joint rotation axis and angle are determined by comparing the first and second postures of the human's movable anatomical structure using the aforementioned markers. To segment the structure of anatomical structures within the image dataset of the person concerned to form anatomical segments within the image dataset, A method comprising rotating the anatomical segment in the image dataset around the joint rotation axis so that it can be viewed using an AR headset.

48. The method according to claim 47, further comprising rotating the anatomical segment around the joint rotation axis to represent the flexion of the movable anatomical structure in the image dataset.

49. The method according to claim 47, further comprising rotating the anatomical segment about the joint rotation axis to an angle that matches the flexion angle of a joint detectable by the AR headset.

50. The method according to claim 47, further comprising segmenting the structure of an anatomical structure to identify bones using edge detection, feature detection, shape detection, or machine learning.

51. The method according to claim 47, further comprising registering a marker located on a human mobile anatomical structure, wherein the marker has a first posture including a position and orientation in the 3D coordinate system of the AR headset.

52. The method according to claim 47, wherein the anatomical segment is moved within the 3D image space of the image dataset.

Images