Augmented Reality Patient Assessment Module

Abstract

Patient mobility assessment can be improved using an augmented reality patient assessment module. To reduce subjectivity of mobility assessment, a depth camera can be used to determine precise patient motion, generate a skeletal model of the patient, and determine range of motion of various patient joints. The augmented reality head mounted display can include a transparent display screen and can be used to display the skeletal model superimposed on the patient while viewing the patient through the transparent display screen. A physician can guide the patient through a series of musculoskeletal assessment activities that can be used to generate a skeletal model of the patient and determine range of motion of various patient joints. The skeletal model and range of motion information can be used to generate a predicted post-operative skeletal model, which can display an improved range of motion based on the surgical procedure.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 180456, filed April 27, 2021, and concurrently claims the benefit of U.S. Provisional Patent Application No. 63 / 303683, filed January 27, 2022, the priority benefit of each of which is hereby claimed, and each of which is incorporated herein by reference in its entirety.

[0002] This application relates to patient mobility assessment using augmented reality. [Background technology]

[0003] A patient's mobility can be affected by various changes in the patient's musculoskeletal system. For example, the range of motion of a patient's joints may be reduced by arthritis or due to injury. When conducting an assessment of a patient's mobility, a physician typically leads the patient through a series of exercises to determine range of motion. However, a patient's range of motion is often a subjective determination by the physician.

[0004] Diagnostics are used to evaluate patients to determine whether they require surgical procedures for the upper extremity (e.g., shoulder or elbow), lower extremity (e.g., knee, hip, etc.), or the like. Hundreds of thousands of these procedures are performed annually in the United States. Surgical advances have enabled surgeons to use preoperative planning, display devices, and imaging to improve diagnosis and surgical outcomes.

[0005] Augmented reality (AR) or mixed reality (MR) devices (AR and MR are used interchangeably) allow a user to see a representation of virtual objects that appear to be projected onto a real (visible) environment. AR devices typically include two display lenses or screens, one for each eye of the user. Light can pass through the two display lenses so that aspects of the real environment are visible, while virtual elements project light visible to the user of the AR device. [Brief explanation of the drawings]

[0006] [Figure 1A] FIG. 1A shows a diagram of a skeletal modeling system according to some embodiments. [Figure 1B] FIG. 1B shows a diagram of a skeletal modeling system according to some embodiments. [Figure 2A] FIG. 2A shows a diagram of a skeletal kinematic modeling system according to some embodiments. [Figure 2B] FIG. 2B shows a diagram of a skeletal kinematic modeling system according to some embodiments. [Figure 3] FIG. 3 shows a diagram of a remote skeletal modeling system according to some embodiments. [Figure 4] FIG. 4 shows a diagram of an augmented reality articulated viewing system according to some embodiments. [Figure 5A] FIG. 5A shows a diagram of a spherical skeleton modeling system according to some embodiments. [Figure 5B] FIG. 5B shows a diagram of a spherical skeleton modeling system according to some embodiments. [Figure 6] FIG. 6 shows a flow chart illustrating an augmented reality patient assessment method according to some embodiments. [Figure 7] FIG. 7 illustrates a user interface for selecting an application according to some embodiments. [Figure 8] FIG. 8 illustrates a first user interface for patient selection according to some embodiments. [Figure 9] FIG. 9 illustrates a second user interface for patient selection according to some embodiments. [Figure 10] FIG. 10 illustrates a user interface for selecting a patient according to some embodiments. [Figure 11] FIG. 11 illustrates a user interface for displaying patient information according to some embodiments. [Figure 12]FIG. 12 illustrates a rating user interface for selecting a rating according to some embodiments. [Figure 13] FIG. 13 illustrates a user interface for displaying aspects of an AR evaluation according to some embodiments. [Figure 14] FIG. 14 illustrates user interfaces and components for displaying aspects of AR assessment and skeletal overlay according to some embodiments. [Figure 15] FIG. 15 illustrates user interfaces and components for displaying aspects of AR assessment and skeletal overlay according to some embodiments. [Figure 16] FIG. 16 illustrates a user interface for displaying the results of an AR evaluation according to some embodiments. [Figure 17] FIG. 17 illustrates a user interface for selecting a surgical demonstration (hereinafter referred to as a "surgical demo") in augmented reality according to some embodiments. [Figure 18] FIG. 18 illustrates a user interface for selecting a surgical demo in augmented reality, according to some embodiments. [Figure 19] FIG. 19 illustrates user interfaces and components for displaying aspects of the AR demonstration and 3D bone model according to some embodiments. [Figure 20] FIG. 20 shows an example of a 3D bone model AR view according to some embodiments. [Figure 21] FIG. 21 shows an example of a 3D bone model in an AR view according to some embodiments. [Figure 22] FIG. 22 shows a flowchart illustrating a technique for performing a patient assessment using augmented reality in accordance with at least one embodiment of the present disclosure. [Figure 23] FIG. 23 shows a flowchart illustrating a technique for displaying a surgical demonstration using augmented reality in accordance with at least one embodiment of the present disclosure. [Figure 24]FIG. 24 illustrates an example block diagram of a machine capable of implementing any one or more of the techniques discussed herein, according to some embodiments. Summary of the Invention [Problem to be solved by the invention]

[0007] This disclosure describes technical solutions to the technical problems faced in patient mobility assessment. To reduce the subjectivity of mobility assessment, a depth camera (e.g., a depth sensor) can be used to determine precise patient motion, generate a skeletal model of the patient, and determine the range of motion of various patient joints. An AR or MR head-mounted display (HMD) can include a transparent display screen and can be used to display the skeletal model superimposed on the patient while viewing the patient through the transparent display screen. [Means for solving the problem]

[0008] A physician can guide a patient through a series of musculoskeletal assessment activities. Depth sensor information captured during the assessment activities can be used to generate a skeletal model of the patient and determine the range of motion of various patient joints. The assessment activities can be displayed on an HMD while viewing the patient through a transparent display screen. The display of the assessment activities can include a display of the patient's current range of motion for one or more joints.

[0009] The skeletal model and range of motion information can be used to generate a predicted postoperative skeletal model. The predicted postoperative skeletal model can indicate an improved range of motion based on a surgical procedure. For example, femoroacetabular impingement may limit hip joint mobility, and the predicted postoperative skeletal model can indicate an improved hip joint range of motion based on an acetabular resurfacing procedure. The assessment activity can be used to identify one or more surgical procedures that can improve the joint's range of motion. For example, a hip flexion and extension assessment activity can indicate a decrease in hip joint range of motion and suggest an acetabular resurfacing procedure or other hip procedure to the physician to improve the hip joint range of motion. The predicted postoperative skeletal model can be output for display and superimposed on the patient while the patient is viewed through a transparent display screen. The predicted postoperative skeletal model can be displayed on a patient viewing device, such as a patient HMD, tablet, or other display device.

[0010] An additional musculoskeletal assessment activity can be used to reassess patient mobility, such as after a surgical procedure. A post-operative assessment activity can be used to collect post-operative depth sensor data and generate a post-operative skeletal model. This post-operative skeletal model can be compared to the pre-operative skeletal model, such as by displaying the post-operative model superimposed on the pre-operative model. One or both of the pre-operative and post-operative models can be superimposed on the user, such as by viewing the patient through a transparent HMD screen and superimposing both models on the patient.

[0011] An optical camera (e.g., an image capture device) can capture images (e.g., still images or video), such as during a pre- or post-operative evaluation. The captured images can be saved along with an associated pre- or post-operative skeletal model and used by a physician or patient to view the skeletal model superimposed on the patient. The captured images allow a physician or patient to view the position of a particular joint (e.g., full flexion or full extension) or view a video of the patient's current range of motion for a joint.

[0012] The systems and methods described herein can be used to evaluate a patient before, during, or after completion of an orthopedic surgical procedure on a portion of the patient's body. Orthopedic surgical procedures can include joint repair, replacement, revision, etc. Patient evaluation is an important pre-operative, intra-operative, and post-operative aspect of the treatment journey. Range of motion and quality of motion information can be particularly useful in determining a patient's limitations before an intervention, the degree of repair during surgery, and the progress of recovery after an intervention.

[0013] The systems and methods described herein can be used to display features, user interfaces, components (e.g., three-dimensional (3D) models, overlays, etc.), or the like, in augmented or virtual reality. The 3D models can include bone models, such as generic bone models or patient-specific bone models (e.g., generated from patient imaging). The overlays can include skeletal overlays, such as a set of joints and segments connecting the joints, representing the patient's joints and bones or other internal anatomy. The overlays can be displayed superimposed on the patient (e.g., the overlays are displayed virtually in an augmented or mixed reality system with the patient as they would appear in the real world).

[0014] Augmented reality (AR) devices allow users to see displayed virtual objects that appear to be projected onto a real environment (which also appears similar). AR devices typically include two display lenses or screens, one for each eye of the user. Light can pass through the two display lenses so that aspects of the real environment are visible while projecting light to allow the user of the AR device to see virtual elements.

[0015] Augmented reality is a technology for displaying virtual or "augmented" objects or visual effects overlaid on a real environment. The real environment can include a room or a specific area, or it can be broader, including the world at large. The virtual forms overlaid on the real environment can be represented as fixed or in a set position relative to one or more aspects of the real environment. For example, a virtual object such as a menu or model can be configured to appear to be resting on a table. An AR system can display virtual forms that are fixed to real objects without considering the field of view or viewer of the AR system. For example, a virtual object may be located in a room and visible to a viewer of the AR system within that room, but invisible to a viewer of the AR system outside that room. A virtual object within the room can be displayed to a viewer outside the room when the viewer enters the room. In this example, the room can function as the real object to which the virtual object is fixed in the AR system.

[0016] An AR device can include one or more screens, such as one screen or two screens (e.g., one for each eye of the user). The screens allow light to pass through them so that aspects of the real environment are visible while the virtual objects are displayed. The virtual objects can be made visible to the wearer of the AR device by projecting light. The virtual objects can appear to have some transparency or can be opaque (i.e., blocking aspects of the real environment).

[0017] An AR system can be viewable by one or more viewers and can include differences between the views available to one or more viewers while maintaining some aspects common between the views. For example, a heads-up display can change between two views while anchoring a virtual object to a real object or area in both views. Aspects such as object color, lighting, or other changes can be made between views without changing the anchoring position of at least one virtual object.

[0018] A user can view virtual objects in an AR system, displayed as opaque or with several levels of transparency. In one example, a user can interact with a virtual object by moving it from a first position to a second position or selecting a representation (e.g., in a menu). For example, a user can move or select an object with a gesture or hand placement. This can be done virtually in the AR system by determining when the hand has moved to a position coincident with or adjacent to the object (e.g., using one or more cameras that can be attached to the AR device and be stationary or controllable to move) and causing the object to move or respond accordingly. The virtual form can include a virtual representation of a real-world object or visual effects such as lighting effects. The AR system can include rules governing the behavior of the virtual object, such as whether the virtual object is subject to gravity or friction, or other predefined rules that ignore real-world physical constraints (e.g., floating objects, perpetual motion, etc.). The AR device can include a camera on the AR device. The AR device camera can include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, etc. AR devices can project virtual items over a representation of the real environment that can be seen by the user. DETAILED DESCRIPTION OF THE INVENTION

[0019] In the figures (not necessarily to scale), like reference numbers may refer to like components in different figures. Like numbers with different letter suffixes may represent different instances of like components. The drawings illustrate generally, by way of example, and not by way of limitation, various embodiments discussed in the present application.

[0020] 1A-1B show diagrams of a skeletal modeling system 100 according to some embodiments. The system 100 can include an HMD 110 with an AR display 115. The HMD 110 can be worn by a physician 120 and can be used to display patient information 125 while viewing the patient 130 through the display 115. The HMD 110 can identify the patient 130 using depth sensors, image sensors, and other sensors. As shown in FIG. 1A, the HMD 110 can overlay a circle 135 to indicate that the patient has been identified. The HMD 110 can use one or more of the depth sensors, image sensors, and other sensors to generate a pre-operative skeletal model. The pre-operative skeletal model can be generated based on a stationary or moving patient. As shown in FIG. 1B, the HMD 110 can overlay a pre-operative skeletal model 140 on the patient 130. In one embodiment, a generalized initial pre-operative skeletal model is generated based on a stationary view of the patient, and the pre-operative skeletal model is continuously updated based on additional depth and image sensor data. The model can be updated based on a series of musculoskeletal assessment activities, such as those shown in Figures 2A-2B.

[0021] 2A-2B show diagrams of a skeletal motion modeling system 200 according to some embodiments. The system 200 can include an HMD 210 with an AR display 215. The HMD 210 can be worn by a physician 220 and can be used to display range of motion and other patient information 225 while viewing a patient 230 through the display 215. The physician 220 can guide the patient 230 through a series of musculoskeletal assessment activities. The assessment activities can be used to update a skeletal model 250 of the patient 230. As shown in FIG. 2A, the assessment activities can be used to determine a shoulder joint range of motion 255. The HMD 210 can display information related to range of motion, such as an angular range of motion display 260 superimposed on the skeletal model 250 and the patient 230. The HMD 210 can also display current and historical range of motion information within the patient information 225.

[0022] The skeletal model 250 and range of motion information can be used to generate a predicted postoperative skeletal model including an associated improved range of motion based on the surgical procedure. The system 200 can identify reduced shoulder joint range of motion and generate a predicted postoperative skeletal model including an improved shoulder joint range of motion. As shown in FIG. 2B , the HMD 210 can display the reduced shoulder joint range of motion 265 and the predicted postoperative range of motion 255. The reduced shoulder joint range of motion 265 can be displayed in a dashed line or semi-transparent format to distinguish it from the predicted postoperative range of motion 255. In one embodiment, the system 200 suggests an acromio-clavicular resurfacing procedure or other shoulder joint procedure to improve the shoulder joint range of motion. The system 200 can display one or more surgical procedures to the physician 220 for selection, and the predicted postoperative range of motion 255 can be generated based on the selected procedure or procedures.

[0023] After surgery, a post-operative musculoskeletal assessment activity can be used to reassess the skeletal model 250 and range of motion information. The post-operative assessment activity can be used to collect post-operative depth sensor data and generate a post-operative skeletal model. Similar to FIG. 2B, the HMD 210 can display a pair of patient skeletal models, such as displaying the reduced pre-operative shoulder joint range of motion next to the post-operative range of motion.

[0024] FIG. 3 shows a diagram of a remote skeletal modeling system 300 according to some embodiments. The system 300 can include a skeletal modeling device 310, which can include depth sensors, image sensors, and other sensors to generate a skeletal model 340 of a patient 320. A display device 330 can be used to display the skeletal model 340 and joint range of motion 345 overlaid on a captured image of the patient 320. The image and depth sensor data captured by the skeletal modeling device 310 can be stored along with the associated pre- or post-operative skeletal model and used by a physician or patient to view images or videos of the skeletal model 340 overlaid on the patient 320. In one embodiment, the skeletal modeling device 310 can be used in the patient's home or other location remote from the physician. This can improve remote assessments, such as by allowing the patient 320 to participate in pre- or post-operative musculoskeletal assessment activities without the need to visit a physician.

[0025] FIG. 4 illustrates an AR joint viewing system 400 according to some embodiments. The system 400 can include one or more AR displays, such as a patient AR display 410 worn by a patient 415 and a doctor AR display 420 worn by a doctor 425. The doctor 425 can use the system 400 to demonstrate pre- or post-operative joint mobility. For example, the doctor 425 can display an AR image of a pre-operative knee joint condition on both the patient AR display 410 and the doctor AR display 420. The AR image can be fixed at a location on a table or at another real-world location. The doctor 425 can interact with the AR image, such as by highlighting one or more bones or soft tissues to help explain the pre-operative knee joint condition. The system 400 can be used to display and discuss one or more skeletal models, such as a pre-operative skeletal model, a predicted post-operative skeletal model, or a post-operative skeletal model.

[0026] 5A-5B show diagrams of a spherical skeletal modeling system 500 according to some embodiments. The system 500 can include an augmented reality head-mounted display (ARHMD) 510 worn by a physician 520 while viewing a patient 530. The system 500 can display an evaluation sphere 540 around the patient 530 while viewing the patient 530 through the HMD 510. The evaluation sphere 540 can include one or more angle guidelines (e.g., lines of latitude or longitude) that can be used to assess patient mobility. For example, shoulder joint rotations from 0° to 180° shown in FIGS. 2A-3 can be displayed by corresponding guidelines on the evaluation sphere 540. The evaluation sphere 540 can be used to view a stationary patient, as shown in FIG. 5A, or a moving patient, as shown in FIG. 5B. The evaluation sphere 540 can be fixed to a patient skeletal model 550 and can rotate or move with the patient 530. In one embodiment, the position of evaluation sphere 540 can be fixed to the top joint of skeletal model 550 while keeping its rotation aligned with a predetermined coordinate system. For example, the position of evaluation sphere 540 can move with walking patient 530, but the rotation of evaluation sphere 540 can be locked within a Cartesian coordinate system defined by the walls and floor of a room.

[0027] 6 is a flowchart illustrating an AR patient assessment method 600 according to some embodiments. Method 600 includes generating 605 depth sensor data about a patient during a musculoskeletal assessment activity and generating 610 a pre-operative skeletal model based on the depth sensor data. Method 600 includes generating 615 a predicted post-operative skeletal model based on the pre-operative skeletal model. The predicted post-operative skeletal model indicates an improved range of motion based on the surgical procedure. Method 600 includes outputting 620 the predicted post-operative skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0028] In one embodiment, method 600 includes generating 625 post-operative depth sensor data for the patient during a post-operative musculoskeletal assessment activity. Method 600 can include generating 630 an improved post-operative skeletal model based on the post-operative depth sensor data. Method 600 can include outputting 635 the improved post-operative skeletal model overlaid on the pre-operative skeletal model for display on an ARHMD, thereby allowing a viewer to compare the improved post-operative skeletal model to the pre-operative skeletal model. Method 600 can include outputting 640 the improved post-operative skeletal model overlaid on the predicted post-operative skeletal model for display on an ARHMD, thereby allowing a viewer to compare the improved post-operative skeletal model to the predicted post-operative skeletal model.

[0029] In one embodiment, the method 600 includes receiving 645 a selection of a surgical procedure, and the predictive post-operative skeletal model is further based on the selection of the surgical procedure. The method 600 can include identifying 650 a list of surgical procedures associated with the musculoskeletal assessment activity and outputting 655 a selection prompt for the list of surgical procedures for display on the ARHMD.

[0030] In one embodiment, method 600 includes capturing 660 an image of the patient, wherein the pre-operative skeletal model is further based on the captured image of the patient. Method 600 may include receiving 665 a selection of range of motion exercises. Method 600 may include generating 670 a plurality of range of motion images associated with the selected range of motion exercises, wherein the plurality of range of motion images includes the post-operative skeletal model overlaid on the captured image of the patient.

[0031] In one embodiment, the method 600 includes outputting 675 the guided musculoskeletal activity for display on the ARHMD. The guided musculoskeletal activity can provide patient movement commands to guide the musculoskeletal assessment activity.

[0032] In one embodiment, the method 600 includes receiving 680 motion sensor data or medical imaging data. The motion sensor data can be received from a motion sensor attached to the patient, and the motion sensor data can characterize the patient's musculoskeletal motion. The generation of the pre-operative skeletal model can be further based on the received sensor data or the received medical imaging data.

[0033] FIG. 7 illustrates a user interface 700 for selecting an application (e.g., using augmented reality) according to some embodiments. Menus can be displayed in virtual or augmented reality or on a traditional screen. The user interface 700 illustrated in FIG. 7 is AR, where a view of the surrounding area is visible. The AR view can be displayed using glasses, a visor, goggles, or other immersion / AR setup. Menus can be selectable with a user's finger (e.g., as detected in real space using an AR device). Selections can be made by a clinician, a patient, or other interested party. For example, a selection can include a patient assessment in AR with display 702 or a surgical demonstration in AR with display 704.

[0034] FIG. 8 illustrates a first user interface 800 for patient selection according to some embodiments. FIG. 8 includes a menu 802 (e.g., an AR menu) that displays selectable indicators through QR code representations. Upon selecting a selectable indicator, a camera view can be displayed (e.g., a reticle framing a capture area, as shown in AR in FIG. 9). Other selectable options in the menu 802 include selecting a patient, starting an assessment, or viewing a dashboard. In examples where the menu 802 is shown in AR, the menu 802 can be automatically aligned to a wall, next to a recognized patient, etc.

[0035] FIG. 9 illustrates a second user interface 900 for patient selection according to some embodiments. A QR code 902 can be scanned using a camera (e.g., attached to the AR device displaying the menu 802). The QR code 902 can be patient-specific, clinician-specific, assessment-specific, instrument-specific, or a combination thereof. The QR code 902 can be scanned as shown in FIG. 9 to display a new menu. After scanning the QR code 902, the AR display (e.g., menu) can automatically populate with information such as patient information, a patient list, instrument information, assessment information, a facility or group patient list, etc. An example patient list is shown in FIG. 10, discussed below.

[0036] In one embodiment, the QR code helps locate the location of a camera (which may include a lidar camera). The camera allows the spatial location of skeletal joints to be located in camera coordinates. The AR device can convert these joint coordinates into real-world coordinates.

[0037] FIG. 10 illustrates an interface 1000 for selecting a patient according to some embodiments. The user interface 1000 can be displayed in response to a user selection, scanning a QR code (e.g., as described above in connection with FIGS. 8-9), a user login, or the like. When displayed in AR, the user interface 1000 can be moved within the field of view. The user interface 1000 can be fixed to various locations in the real world, such as a wall, or moved by maintaining a fixed distance to a user wearing AR equipment. In one example, a patient can be selected from the user interface 1000. When selected, further details about the patient can be displayed (discussed below in connection with FIG. 11).

[0038] FIG. 11 illustrates a user interface 1100 for displaying patient information according to some embodiments. The user interface 1100 can be displayable in response to patient selection (e.g., on the user interface 1000, via voice command, via gesture, etc.). Once a patient is selected, patient details such as name, age, last evaluation, treatment (e.g., treatment completed or upcoming), and attending physician's name can be shown on the user interface 1100. A user of the AR device displaying the user interface 1100 can confirm or cancel the patient selection. In some instances, a QR code may be unavailable or missing, such as if the patient is a new patient and new input is required. In such instances, a virtual keyboard can be displayed for manual entry of the patient's name or other details. After the patient is confirmed and selected, an evaluation user interface can be displayed.

[0039] FIG. 12 illustrates an assessment user interface 1202 for selecting an assessment according to some embodiments. The assessment user interface 1202 identifies broad, selectable assessment types, such as upper limb or lower limb. After selecting a broad category, an assessment user interface 1204 can be displayed containing various categories or subcategories of assessments for selection. The displayed options can be limited to those applicable to the previously selected patient, or a broader set of assessment types can be displayed. The assessment types displayed in the assessment user interface 1204 can include multiple parts (e.g., a first assessment corresponding to a first leg or first movement and a second assessment corresponding to a second leg or second movement). Example assessments can include those shown in Tables 1 and 2 below, such as single-joint, single-plane movement for the upper limb or multi-plane, multi-joint movement for the lower limb. These two example assessments can be categorically referred to as a single-joint, single-plane, traditional range of motion assessment or a multi-joint, multi-plane functional assessment. [Table 1] [Table 2] After a particular rating is selected, the AR rating can be displayed, for example, as discussed below in connection with FIG.

[0040] FIG. 13 illustrates a user interface 1300 for displaying aspects of an AR assessment, according to some embodiments. FIG. 13 illustrates a first user interface 1302, a second user interface 1304, and a skeleton overlay 1306 superimposed on a patient 1308 for displaying aspects of the AR assessment. The interface or component of FIG. 13 can be displayed in response to receiving a selection of an assessment (e.g., in the assessment user interface 1204 of FIG. 12 ). A menu 1304 including the name of the assessment can be displayed along with selectable options related to the assessment for starting, recording (e.g., from the AR display), canceling, restarting steps, etc. A second menu 1302 can be displayed including patient demographic information, instructions for the patient to complete the exercises (which can be written in a variety of languages, including, for example, patient instructions or instructions a clinician would use to instruct a patient, e.g., general or clinical terms), etc. The second menu 1302 can include a selectable indicator to start the assessment. The second menu 1302 can include a goal of the assessment (e.g., general or patient-specific, such as post-operative, target range of motion based on completed surgical procedures, physical therapy, time since procedure, etc.). The second menu 1302 can include a component for displaying a virtual demo of the assessment. The component can be activated automatically or by the user of the AR device.

[0041] Menus 1302 and 1304 can be independently movable, fixed to a portion of a room, fixed relative to patient 1308 or other movable object, fixed to each other, or set at a fixed distance from the wearer of the AR display showing menu 1302 or 1304, etc.

[0042] In one example, the skeletal frame 1306 can be displayed overlaid on the patient 1308. The skeletal frame 1306 can include joints, segments (e.g., corresponding to bones or other body parts), or the like. The skeletal overlay can move with the patient 1308, and the augmented reality image of the skeletal overlay tracks the real-world movements of the patient 1308. While the skeletal frame 1306 is described as tracking the real-world movements of the patient 1308, the skeletal frame 1306 can also be displayed to appear to move relative to the wearer of the AR device displaying the skeletal frame 1306. For example, as the wearer moves, the projection of the skeletal frame 1306 can change to remain between the wearer and the patient 1308. In another example, the skeletal frame 1306 does not move relative to the wearer, such that the skeletal frame 1306 is partially or completely obscured when the wearer's field of view changes.

[0043] The skeletal frame 1306 can be generated from the patient 1308 using a camera, such as a lidar camera, a depth camera, etc. The camera can be part of the AR device or can be separate. Data from the camera can be sent to a processor controlling the display of the AR device, for example via an API, and the skeletal frame 1306 can be output for display using the AR device based on the received data.

[0044] In one embodiment, a lidar camera can be used to capture and identify the patient 1308 via projected light. A skeletal frame 1308 can be derived from the lidar camera data using, for example, image recognition and skeletal assignment, and can optionally be personalized to the patient 1306. A visualization of the skeletal frame 1308 can be provided and displayed via an AR device. Range of motion data for the patient 1306 can be determined based on the patient's 1306 movements (e.g., captured via a lidar camera, a camera attached to an AR device, etc.) and compared to predicted movement in space based on the known dynamics of the skeletal frame and the captured or known anatomy of the patient 1306 (e.g., height). Information about the patient 1306 (e.g., height, arm span, etc.) can be stored in a connected health cloud.

[0045] After receiving a selection to begin (either in one of menus 1302 or 1304, via gesture, voice command, etc.), a real-time indicator of range of motion can be displayed, as described below in connection with FIG.

[0046] FIG. 14 illustrates a user interface 1400 for displaying aspects of AR assessment and skeletal overlay, according to some embodiments. FIG. 14 illustrates user interfaces (e.g., 1402 and 1404) and components (e.g., including portions 1406 and 1408) for displaying aspects of AR assessment and skeletal overlay. User interface 1402 includes a real-time indicator of the patient's range of motion. As the patient moves during the assessment, the real-time indicator can be updated with the current or total range of motion obtained (e.g., displayed as 2 degrees in user interface 1402, such as immediately after the assessment begins). The "Start" selectable indicator in user interface 1402 changes to a "Done" or "End" selectable indicator (e.g., as the assessment progresses and the range of motion changes), which, when selected, can stop the assessment.

[0047] The display of the range of motion in the user interface 1404 progresses (e.g., toward a target range of motion), such as using a completion bar, circle, etc. In some examples, effects can be added or modified to indicate progress, such as a color change, a pop-up, an audible sound, or other indication in the user interface 1404, for example, to indicate the degree or amount of progress. Progress can correspond to passing a previous personal record, achieving a target range of motion, etc.

[0048] The user interface 1404 includes a skeletal overlay superimposed on the user, the skeletal overlay including portions 1406 and 1408. Display enhancements can be shown on the skeletal overlay to show the movement path (e.g., target and current portion 1406 at limb portion 1408), final destination, starting point, etc. The display enhancements are shown in real time and can be modified as the patient moves during the assessment.

[0049] User interfaces 1402 and 1404 show both user interface menus and the patient's skeletal frame. The simultaneous display of the user interface menus and the skeletal frame allows a user (e.g., a clinician, such as a surgeon) to see data about the patient, the skeletal frame, and movement in one view. This improves visibility of information by eliminating the need for the clinician to stare at the screen (and lose sight of the patient). The skeletal frame can provide depth information in some embodiments. As the clinician or other user of the AR device moves, the user interface components can move with the clinician or remain stationary (e.g., near the patient). In some embodiments, the user interface components appear to rotate spatially with the user of the AR device, allowing the user to view the user interface components at any angle while obtaining various perspectives of the patient and skeletal frame. This allows the user to see the exact major plane movement, which can be seen visually on the skeletal frame and, optionally, as values ​​in the user interface components.

[0050] In one embodiment, after the assessment is complete, after range of motion (full range of motion or quality of motion) is achieved, or after a completion indicator is selected, the AR assessment can be displayed as described below in connection with FIG. 15.

[0051] FIG. 15 illustrates a user interface 1500 for displaying aspects of the AR assessment and skeletal overlay, according to some embodiments. FIG. 15 illustrates a user interface 1502 and a component 1504 for displaying aspects of the AR assessment and skeletal overlay. The user interface 1502 includes details of the assessment (e.g., a completed assessment as described above). The skeletal overlay or display enhancement can be maintained for review. The user interface 1502 can include a display of the best achieved range of motion, for example, displayed against a goal or personal best. The component 1504 can show the final position of the limb during the range of motion assessment, for example, to indicate progress or achievement to the user.

[0052] If the assessment has multiple parts (e.g., two limbs, two exercises, etc.), the system can move to the next part following selection of the "Done" indicator (as described above). In some instances, the next part begins immediately, and in other instances, the user can select the "Start" indicator to begin the next part. Once one or all parts of the assessment are complete, the overall results of the AR assessment can be displayed, as described below in connection with FIG. 16.

[0053] 16 illustrates a user interface 1600 for displaying the results of an AR assessment according to some embodiments. The user interface 1600 can include a menu showing the results of the assessment. In the example shown in FIG. 16, a range of movement angles (e.g., 119 degrees and 108 degrees) is displayed for the horizontal adduction assessment of both the left and right upper limbs.

[0054] Once the user selects the "Done" indicator on the user interface 1600, indicating that they have verified the AR assessment results, the system can return to the previous menu for further assessment if necessary, or to complete, save, or submit the assessment results. The system can provide further instructions (e.g., exercises to continue to improve range of motion, education on the benefits of improving range of motion, instructions to contact a clinician).

[0055] An augmented or virtual reality view of the patient's anatomy can be displayed after the AR assessment is completed in some examples. The patient's anatomy can be displayed depending on the role of the viewer of the anatomy, such as a patient view or a clinician view. The patient view can be simpler than the clinician view in terms of anatomy or clinical information. In some examples, the patient view can include additional information, such as descriptions, educational materials, colors or other display effects or the like, including the patient's anatomy.

[0056] Completed or to-be-completed procedures may be shown (e.g., a roadmap including an indication of where the patient is on the roadmap). The augmented reality patient anatomy may be displayed in various states, such as before the procedure, during or after the procedure, pre-operative including a predicted view of the outcome, and post-operative for comparison with the pre-operative predicted outcome. In one example, the patient anatomy may be shown in dynamic, static, or exploded view, or the like. The patient anatomy displayed in the augmented reality view may be rotatable, movable, zoomable, or the like.

[0057] 17 illustrates a user interface 1700 for selecting a surgical demo in augmented reality according to some embodiments. User interface 1702 allows a user to select a role, such as clinician or patient. User interface 1704 can be displayed after a selection is made on user interface 1702. User interface 1704 can include selections for starting the demo, selecting a patient (e.g., if the user is a clinician), sharing, etc. In one example, when a user selects "patient" in user interface 1702, user interface 1704 can be skipped and proceed to a patient selection screen, for example, to confirm the patient.

[0058] FIG. 18 illustrates a user interface 1800 for selecting a surgical demo in augmented reality. When a user selects a start demo indicator (e.g., on the user interface 1104 of FIG. 11B), the user interface 1800 can display available demos. The user can select the demo they want to view (corresponding to the patient, clinician, type of procedure, etc.). The user interface 1800 includes a demo for total knee surgery or hip surgery. The patient can select to personalize the demo in some embodiments. The demo can include an AR demonstration, as described below, and the demo can correspond to an upcoming or previously completed surgical procedure.

[0059] FIG. 19 illustrates a user interface 1900 for displaying aspects of an AR demonstration and a 3D bone model, according to some embodiments. FIG. 19 illustrates a user interface 1904 and a component 1902 for displaying aspects of the AR demonstration and a 3D bone model. The component 1902 can include a virtually manipulated 3D representation of the 3D bone model displayed in AR. The 3D bone model, in some examples, can include a custom 3D model based on patient imaging. The control user interface 1904 can display components such as rotate, explode, play, cancel, and share for controlling the display of the component 1902. In some examples, the component 1902 can be controlled by “moving” the component 1902 (e.g., using hands or gestures).

[0060] In one embodiment, selecting the "Play" button in the user interface 1904 shows the full rotation or range of motion of the 3D bone model. When viewing the 3D bone model in AR, the incisions, trial placement, exploded views, implants, rotations, etc. can be displayed (e.g., animated). In this way, the user can see an end-to-end view of the procedure in 3D AR. The patient can be given a visual walk-through of the procedure while the clinician-user uses the 3D AR view to visualize spots. When finished, the user can select the "Confirm" or "Check" button.

[0061] Component 1902 shows a three-dimensional rendering of a patient anatomy, implants, trials, etc. in an augmented reality display according to some embodiments. The AR display includes component 1902, which can include a patient anatomy generated using, for example, an X-ray, MRI, CT scan, or the like. The AR display can show animations or allow control or movement of a three-dimensional virtual representation of the patient anatomy or implant in component 1902 (e.g., bone). The three-dimensional virtual representation can be interacted with by a clinician viewing the AR display using, for example, buttons, a remote, gestures, input on a menu in user interface 1904, etc. Interaction can involve manipulating component 1902 (e.g., rotating, moving, zooming, etc.). Manipulating component 1902 can allow a clinician to visualize a surgical procedure, such as pre-operatively or post-operatively.

[0062] FIG. 20 illustrates an example of a 3D bone model AR view 2000 according to some embodiments. The AR view 2000 is an exploded view of the 3D bone model and can include a visual representation of the implant or trial, the incision, etc. The AR view 2000 can be controlled, for example, to expand or compress (e.g., reassemble), rotate, move, etc. In some examples, a component such as a trial or implant can be selected and then removed, moved independently of the model, replaced with another, resized, etc. This allows a clinician to quickly visualize different sized trials or implants by looking at the expanded model. In some examples, the incision can be modified, for example, the cutting depth, cutting angle, etc., to allow the clinician to visualize changes to the incision.

[0063] In one embodiment, the AR view 2000 includes a demonstration system for a surgical procedure. In some embodiments, the AR view 2000 can use a non-patient-specific bone structure, while in other embodiments, the AR view 2000 can use a patient-specific bone structure (e.g., based on patient imaging). The AR view 2000 can be used to show medical device components such as a knee system (e.g., a total or partial knee), a hip system, etc.

[0064] FIG. 21 shows an example of a 3D bone model in an AR view 2100 according to some embodiments. The AR view 2100 includes a visual indicator 2102 of where movement of the model occurs. In this example AR view 2100, the 3D bone model can be stable (e.g., immovable) except for the points of movement indicated by the visual indicator 2102 (in this case, a blue sphere). A second visual indicator 2104 can be used to indicate a rotational joint that fixes the movement caused by moving the visual indicator 2102. For example, in the AR view 2100, the visual indicator 2102 is at the knee joint, and when it is moved, it rotates the femur at the hip joint as indicated by the second visual indicator 2104. The visual indicator can be displayed intrinsically (e.g., as a sphere, in a different color, by changing when a cursor or finger hovers over it, etc.). In the AR view 2100, the knee joint can be moved to show, for example, the hip joint rotation or range of motion of the patient-specific 3D bone model.

[0065] The AR view 2000 or 2100 can be used, for example, in a multi-user system, to display user interface components, models, techniques, skeletal frames, or the like, as described herein. In some embodiments, the multi-user system can be used by a clinician and a patient using synchronized or connected AR devices. The clinician's AR device can be used to control or display aspects of patient recovery, a surgical procedure, or the like, in the patient AR. In some embodiments, the body structure shown in the AR view 2000 or 2100 can move in a pre-set manner (e.g., animated). In other embodiments, the clinician can control the body structure by rotating, spinning, disassembling, playing, pausing, speeding up, slowing down, etc.

[0066] 22 is a flowchart illustrating a technique 2200 for conducting a patient assessment using augmented reality in accordance with at least one embodiment of the present disclosure. The technique 2200 includes an operation 2202 for using an augmented reality (AR) device to initiate a first user interface including a selectable indicator corresponding to the assessment. In one embodiment, the patient can make a selection using the AR user interface (or multiple AR user interfaces).

[0067] Technique 2200 includes an operation 2204 for receiving a selection of a selectable indicator on a first user interface, the selectable indicator corresponding to an assignment on the user interface. Technique 2200 includes an operation 2206 for displaying a second user interface including a current range of motion indication corresponding to the current position of the patient performing the assessment, i.e., the patient as viewed through the AR device. Technique 2200 includes an operation 2208 for outputting range of motion results of the assessment for display using the AR device. In one embodiment, technique 2200 may include displaying a skeletal overlay on the patient instead of or in addition to operations 2206 or 2208.

[0068] The skeletal overlay can be displayed along with the AR display via the AR device. The skeletal overlay can move when the patient moves. The user interface can be controlled by the user or automatically to follow the user's gaze of the AR display, or can remain stationary or fixed at a specific distance from the object (e.g., the patient). When the user of the AR device moves, the user interface can follow, or rotate, etc. (e.g., when set to a fixed distance).

[0069] 23 is a flowchart illustrating a technique 2300 for displaying a surgical demonstration using augmented reality in accordance with at least one embodiment of the present disclosure. Technique 2300 includes an operation 2302 for displaying a user interface including a selectable indicator that, when selected, displays a surgical demonstration in virtual or augmented reality using a virtual or augmented reality display device. Technique 2300 includes an operation 2304 for receiving user input corresponding to the selectable indicator. Technique 2300 includes an operation 2306 for displaying a surgical demonstration in virtual or augmented reality using a 3D model generated from images of the patient's anatomy, the surgical demonstration including a 3D view of at least one of a full rotation of the 3D model, a range of motion of the 3D model, an incision of the 3D model, placement of a trial or implant in the 3D model, or an exploded view of the 3D model.

[0070] FIG. 24 is an example block diagram of a machine 2400 capable of implementing any one or more of the techniques (e.g., methodologies) discussed herein, according to some embodiments. In another embodiment, machine 2400 operates as a standalone device or can be connected (or networked) to other machines. In a network deployment, machine 2400 can operate as a server machine, a client machine, or both in a server-client network environment. Machine 2400 can be a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, a switch, or a bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions the machine should take. Furthermore, while a single machine is illustrated, “machine” can also be interpreted to include any collection of machines individually or collectively executing a set (or sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations, etc.

[0071] An embodiment as described herein may include or operate within logic or multiple components, modules, or similar mechanisms. Such mechanisms are tangible entities (e.g., hardware) that, when activated, can perform specified operations. In one embodiment, the hardware may be tangibly configured to perform specific operations (e.g., wiring). In one embodiment, the hardware may include configurable execution units (e.g., transistors, circuits, etc.) and instructions contained on a computer-readable medium, which, when activated, configure the execution units to perform specific operations. Configuration may occur under the direction of the execution units or a loading mechanism. Thus, the execution units are communicatively coupled to the computer-readable medium when the device is operating. For example, in operation, the execution units may be configured by a first set of instructions to implement a first set of features at one time and reconfigured by a second set of instructions to implement a second set of features.

[0072] The machine (e.g., computer system) 2400 may include a hardware processor 2402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 2404, and a static memory 2406, some or all of which may communicate with each other via an interlink (e.g., a bus) 2408. The machine 2400 may further include a display unit 2410, an alphanumeric input device 2412 (e.g., a keyboard), and a user interface (UI) navigation device 2414 (e.g., a mouse). In one embodiment, the display unit 2410, the alphanumeric input device 2412, and the UI navigation device 2414 may be touchscreen displays. The display unit 2410 may include goggles, glasses, an augmented reality (AR) display, a virtual reality (VR) display, or other display component. For example, the display unit may be worn on a user's head and provide a head-up display to the user. The alphanumeric input device 2412 may include a virtual keyboard (e.g., a virtually displayed keyboard in a VR or AR setting).

[0073] The machine 2400 may further include a storage device (e.g., a drive unit) 2416, a signal generation device 2418 (e.g., a speaker), a network interface device 2420, and one or more sensors 2421, such as a Global Positioning System (GPS) sensor, a compass, an accelerometer, or other sensor. The machine 2400 may include an output controller 2428, such as a serial (e.g., Universal Serial Bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection, for communicating with or controlling one or more peripheral devices.

[0074] Storage device 2416 may include non-transitory machine-readable medium 2422 on which is stored one or more data structures or instructions 2424 (e.g., software) that implement or are utilized by any one or more of the techniques or functions described herein. The instructions 2424 may reside, completely or at least partially, within main memory 2404, static memory 2406, or hardware processor 2402 during execution thereof by machine 2400. In one embodiment, one or any combination of hardware processor 2402, main memory 2404, static memory 2406, or storage device 2416 may constitute a machine-readable medium.

[0075] Although the machine-readable medium 2422 is described as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) configured to store one or more instructions 2424.

[0076] A "machine-readable medium" may include any medium that can store, encode, or retain instructions for execution by machine 2400 and cause machine 2400 to perform any one or more of the techniques of this disclosure or that can store, encode, or retain data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory and optical and magnetic media. Specific examples of machine-readable media include non-volatile memory such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, CD-ROM and DVD-ROM disks, and the like.

[0077] The instructions 2424 may further be transmitted or received over a communications network 2426 using a transmission medium via network interface device 2420 utilizing any one of a number of transfer protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Examples of communications networks include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), plain old telephone (POTS) networks, and wireless data networks (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard known as Wi-Fi®, the personal area network standard known as Bluetooth® promulgated by the Bluetooth® Special Interest Group, and peer-to-peer (P2P) networks), among others. In one embodiment, the network interface device 2420 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connecting to the communications network 2426. In one embodiment, network interface device 2420 may include multiple antennas for wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO). "Transmission medium" is intended to include any non-tangible medium capable of storing, encoding, or carrying instructions for execution by machine 2400, including digital or analog communication signals or other non-tangible medium for facilitating the communication of such software.

[0078] Each of these non-limiting examples can be taken alone or combined with one or more of the other examples in various permutations or combinations.

[0079] Example 1 is a system for augmented reality patient assessment. The system includes an augmented reality (AR) head-mounted display (HMD), a depth sensor for generating depth sensor data about a patient during a musculoskeletal assessment activity, a processing circuit, and a memory containing instructions that, when executed by the processing circuit, cause the processing circuit to generate a skeletal model based on the depth sensor data, track the patient's movements during the musculoskeletal assessment activity, determine a current ROM based on the patient's movements, and output the current ROM superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0080] In example 2, the subject matter of example 1 further includes instructions for causing the processing circuit to receive a selection of the musculoskeletal assessment activity and output a description of the musculoskeletal assessment activity for display on the ARHMD while viewing the patient through the ARHMD.

[0081] In example 3, the subject matter of example 1 or example 2 further includes instructions for causing the processing circuitry to determine a target ROM based on the musculoskeletal assessment activity and output a graphical representation of the target ROM for display on the ARHMD while viewing the patient through the ARHMD.

[0082] In Example 4, the content of Examples 1 to 3 further includes instructions for causing the processing circuitry to output guided musculoskeletal activities for display on the ARHMD, the guided musculoskeletal activities providing patient movement commands for performing musculoskeletal assessment activities.

[0083] In Example 5, the subject matter of Examples 1 to 4 further includes instructions for causing the processing circuitry to receive motion sensor data from a motion sensor attached to the patient, the motion sensor data characterizing musculoskeletal motion of the patient, and generation of the skeletal model further based on the sensor data.

[0084] In Example 6, the subject matter of Examples 1 to 5 further includes instructions for causing the processing circuitry to receive medical imaging data of the patient's musculoskeletal joints, and wherein generating the skeletal model is further based on the medical imaging data.

[0085] In Example 7, the content of Examples 1 to 6 further includes instructions for causing the processing circuitry to receive a selection of a model surgical procedure, generate a patient procedure model based on the model surgical procedure and the skeletal model, and output the patient procedure model for display on the ARHMD while viewing the patient through the ARHMD.

[0086] In Example 8, the content of Examples 1 to 7 further includes instructions to cause the processing circuit to generate a predicted post-operative skeletal model based on the skeletal model, wherein the skeletal model includes the pre-operative skeletal model, and the predicted post-operative skeletal model includes an improved range of motion (ROM) based on the surgical procedure, and instructions to output the predicted post-operative skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0087] In Example 9, the subject matter of Example 8 further includes a depth sensor for generating post-operative depth sensor data about the patient during a post-operative musculoskeletal assessment activity, and the instructions further cause the processing circuit to generate an improved post-operative skeletal model based on the post-operative depth sensor data and output the improved post-operative skeletal model overlaid on the pre-operative skeletal model for display on the ARHMD.

[0088] In example 10, the subject matter of example 9 further includes instructions for causing the processing circuitry to output the refined post-operative skeletal model overlaid on the predicted post-operative skeletal model for display on the ARHMD.

[0089] In Example 11, the subject matter of Examples 8 to 10 further includes instructions for causing the processing circuitry to receive a selection of a surgical procedure, wherein the predicted post-operative skeletal model is further based on the selection of the surgical procedure.

[0090] In example 12, the subject matter of example 11 further includes instructions for causing the processing circuit to identify a list of surgical procedures associated with the musculoskeletal assessment activity and output a selection prompt for the list of surgical procedures for display on the ARHMD.

[0091] In Example 13, the subject matter of Examples 8 to 12 includes an image sensor for capturing a plurality of images of the patient, and the pre-operative skeletal model is further based on the plurality of images of the patient.

[0092] In Example 14, the subject matter of Example 13 further includes instructions for causing the processing circuit to receive a selection of a ROM exercise and generate a plurality of ROM images associated with the ROM exercise, the plurality of ROM images including a predicted postoperative skeletal model overlaid on the plurality of images of the patient.

[0093] In Example 15, the subject matter of Examples 1 to 14 includes a patient-use ARHMD, and the instructions further cause the processing circuit to output a predicted skeletal model for display on the ARHMD while the physician views the patient through the ARHMD, capture an image of the patient as seen by the physician through the ARHMD, and output the predicted skeletal model superimposed on the image of the patient for display on the patient-use ARHMD.

[0094] In Example 16, the content of Examples 1 to 15 further includes instructions to cause the processing circuit to output a multi-pose skeletal model, wherein the multi-pose skeletal model is configured to display multiple positions of a patient's body part based on the improved ROM when the multi-pose skeletal model is superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0095] In Example 17, the content of Example 16 further includes instructions to cause the processing circuit to output a kinematic skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD, the kinematic skeletal model indicating the movement of the patient's body parts based on the improved ROM.

[0096] In Example 18, the subject matter of Example 17 further includes instructions for causing the processing circuit to receive a skeletal model movement pause input and freeze the movement of the patient body part in the display on the patient's ARHMD.

[0097] In Example 19, the subject matter of Examples 1 to 18 further includes instructions for causing the processing circuit to receive a selection of a surgical procedure.

[0098] In example 20, the subject matter of example 19 further includes instructions for causing the processing circuitry to prompt the user for an improved ROM surgical procedure, the improved ROM surgical procedure providing a larger ROM than the surgical procedure.

[0099] In Example 21, the subject matter of Examples 1 to 20 further includes instructions to cause the processing circuit to receive a skeletal model modification input, generate a modified skeletal model based on the skeletal model modification input, and output the modified skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0100] In Example 22, the content of Example 21 includes, wherein the skeletal model correction input includes at least skeletal model joint repositioning, limb length adjustment, limb pose adjustment, and skeletal model reset input.

[0101] In Example 23, the subject matter of Examples 12 to 22 further includes instructions for causing the processing circuit to identify a surgical procedure suggestion associated with at least one element of a list of surgical procedures associated with the musculoskeletal assessment activity, and wherein outputting the selection prompt for the list of surgical procedures includes displaying the surgical procedure suggestion for display on the ARHMD.

[0102] In Example 24, the subject matter of Example 23 includes, wherein the surgical procedure implications include at least one of recovery time, recovery physical therapy requirements, and predicted ROM.

[0103] In Example 25, the subject matter of Examples 1 to 24 further includes instructions for causing the processing circuit to receive a surgical avoidance selection and generate a predicted surgical avoidance skeletal model based on the skeletal model, the predicted skeletal model including a reduced ROM based on the avoidance from the surgical procedure, and output the predicted surgical avoidance skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0104] In Example 26, the content of Examples 1 to 25 further includes instructions for causing the processing circuit to receive an aging progression input and generate a plurality of aging skeletal models based on the skeletal model, the plurality of aging skeletal models including a plurality of reduced ROM values ​​based on the aging progression input, and outputting the progression of the plurality of aging skeletal models superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0105] In Example 27, the subject matter of Examples 14 to 26 further includes instructions for causing the processing circuit to capture patient movement, and the selection of ROM exercises is based on the patient movement.

[0106] Example 28 is a method for augmented reality patient assessment. The method includes generating depth sensor data regarding a patient during a musculoskeletal assessment activity, generating a skeletal model based on the depth sensor data, tracking patient movement during the musculoskeletal assessment activity, determining a current ROM based on the patient movement, and outputting the current ROM superimposed on the patient for display on an augmented reality (AR) head-mounted display (HMD) while viewing the patient through the ARHMD.

[0107] In Example 29, the content of Example 28 includes receiving a selection of a musculoskeletal assessment activity and outputting a description of the musculoskeletal assessment activity for display on the ARHMD while viewing the patient through the ARHMD.

[0108] In Example 30, the content of Example 28 or Example 29 includes determining a target ROM based on the musculoskeletal assessment activity and outputting a graphical representation of the target ROM for display on the ARHMD while viewing the patient through the ARHMD.

[0109] In Example 31, the contents of Examples 28 to 30 include outputting guided musculoskeletal activities for display on an ARHMD, the guided musculoskeletal activities providing patient movement commands for performing musculoskeletal assessment activities.

[0110] In Example 32, the subject matter of Examples 28 to 31 includes receiving motion sensor data from a motion sensor attached to the patient, the motion sensor data characterizing musculoskeletal motion of the patient. The generation of the skeletal model is further based on the sensor data.

[0111] In Example 33, the subject matter of Examples 28 to 32 includes receiving medical imaging data of the patient's musculoskeletal joints, and generating the skeletal model is further based on the medical imaging data.

[0112] In Example 34, the contents of Examples 28 to 33 include receiving a selection of a model surgical procedure, generating a patient procedure model based on the model surgical procedure and the skeletal model, and outputting the patient procedure model for display on the ARHMD while viewing the patient through the ARHMD.

[0113] In Example 35, the contents of Examples 28 to 34 include generating a predicted postoperative skeletal model based on a skeletal model, where the skeletal model includes a preoperative skeletal model, and the predicted postoperative skeletal model includes an improved range of motion (ROM) based on the surgical procedure, and also outputting the predicted postoperative skeletal model superimposed on the patient for display on an augmented reality (AR) head-mounted display (HMD) while viewing the patient through the ARHMD.

[0114] In Example 36, the content of Example 35 includes generating postoperative depth sensor data of a patient during a postoperative musculoskeletal assessment activity, generating a modified postoperative skeletal model based on the postoperative depth sensor data, and outputting the modified postoperative skeletal model overlaid on the preoperative skeletal model for display on an ARHMD.

[0115] In Example 37, the contents of Example 36 include outputting an improved post-operative skeletal model overlaid on the predicted post-operative skeletal model for display on an ARHMD.

[0116] In Example 38, the content of Examples 35 to 37 includes receiving a selection of a surgical procedure, and the predictive post-operative skeletal model is further based on the selection of the surgical procedure.

[0117] In Example 39, the content of Example 38 includes identifying a list of surgical procedures associated with the musculoskeletal assessment activity and outputting a selection prompt for the list of surgical procedures for display on the ARHMD.

[0118] In Example 40, the subject matter of Examples 35 to 39 includes capturing a plurality of images of the patient, and the pre-operative skeletal model is further based on the plurality of images of the patient.

[0119] In Example 41, the content of Example 40 includes receiving a selection of a ROM exercise and generating a plurality of ROM images associated with the selected ROM exercise, the plurality of ROM images including a postoperative skeletal model overlaid on a captured image of the patient.

[0120] In Example 42, the contents of Examples 28 to 41 include an ARHMD for a patient, and further include outputting a predicted skeletal model for display on the ARHMD while the physician views the patient through the ARHMD, capturing an image of the patient as the physician views the patient through the ARHMD, and outputting the predicted skeletal model superimposed on the image of the patient for display on the patient ARHMD.

[0121] In Example 43, the contents of Examples 28 to 42 include outputting a multi-pose skeletal model, wherein the multi-pose skeletal model is configured to display multiple portions of a patient's body part based on the improved ROM when the multi-pose skeletal model is overlaid on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0122] In Example 44, the content of Example 43 includes outputting a kinematic skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD, the kinematic skeletal model showing the movement of the patient's body parts based on the improved ROM.

[0123] In Example 45, the content of Example 44 includes receiving a skeletal model movement interruption input and freezing the movement of the patient's body part on the display on the patient's ARHMD.

[0124] In Example 46, the content of Examples 28 to 45 includes receiving a selection of a surgical procedure.

[0125] In Example 47, the subject matter of Example 46 includes providing a prompt to the user for an improved ROM surgical procedure, where the improved ROM surgical procedure provides a greater ROM than the surgical procedure.

[0126] In Example 48, the contents of Examples 28 to 47 include receiving a skeletal model modification input, generating a modified skeletal model based on the skeletal model modification input, and outputting the modified skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0127] In Example 49, the content of Example 48 is such that the skeletal model correction input includes at least skeletal model joint repositioning, limb length adjustment, limb pose adjustment, and skeletal model reset input.

[0128] In Example 50, the content of Examples 39 to 49 includes identifying a surgical procedure suggestion associated with at least one element of a list of surgical procedures associated with the musculoskeletal assessment activity, and outputting a selection prompt for the list of surgical procedures includes displaying the surgical procedure suggestion for display on the ARHMD.

[0129] In Example 51, the subject matter of Example 50 includes, wherein the surgical procedure implications include at least one of recovery time, recovery physical therapy requirements, and predicted ROM.

[0130] In Example 52, the contents of Examples 28 to 51 include receiving a surgical avoidance selection; generating a predicted surgical avoidance skeletal model based on the skeletal model, where the predicted skeletal model includes a reduced ROM based on the avoidance from the surgical procedure; and outputting the predicted surgical avoidance skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0131] In Example 53, the contents of Examples 28 to 52 include receiving an aging progression input; generating a plurality of aging skeletal models based on the skeletal model, wherein the plurality of aging skeletal models include a plurality of ROM value reductions based on the aging progression input; and outputting the progression of the plurality of aging skeletal models superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0132] In Example 54, the contents of Examples 41 to 53 include capturing patient movement, and the selection of ROM exercises is based on the patient movement.

[0133] Example 55 is a non-transitory machine-readable storage medium that includes instructions that, when executed by a processing circuit of a computer-controlled device, cause the processing circuit to generate depth sensor data for a patient during a musculoskeletal assessment activity, generate a skeletal model based on the depth sensor data, track movement of the patient during the musculoskeletal assessment activity, determine a current ROM based on the patient's movement, and output the current ROM superimposed on the patient for display on an augmented reality (AR) head-mounted display (HMD) while viewing the patient through the ARHMD.

[0134] In example 56, the subject matter of example 55 further includes instructions for causing the processing circuit to receive a selection of the musculoskeletal assessment activity and output a description of the musculoskeletal assessment activity for display on the ARHMD while viewing the patient through the ARHMD.

[0135] In Example 57, the content of Example 55 or Example 56 further includes instructions for causing the processing circuit to determine a target ROM based on the musculoskeletal assessment activity and output a graphical representation of the target ROM for display on the ARHMD while viewing the patient through the ARHMD.

[0136] In Example 58, the content of Examples 55 to 57 further includes instructions to cause the processing circuit to output guided musculoskeletal activities for display on the ARHMD, the guided musculoskeletal activities providing patient movement commands for performing musculoskeletal assessment activities.

[0137] In Example 59, the content of Examples 55 to 58 further includes instructions for causing the processing circuit to receive motion sensor data from a motion sensor attached to the patient, the motion sensor data characterizing the musculoskeletal motion of the patient, and the generation of the skeletal model is further based on the sensor data.

[0138] In Example 60, the subject matter of Examples 55 to 59 further includes instructions for causing the processing circuitry to receive medical imaging data of the patient's musculoskeletal joints, and wherein generating the skeletal model is further based on the medical imaging data.

[0139] In Example 61, the content of Examples 55 to 60 further includes instructions to cause the processing circuit to receive a selection of a model surgical procedure, generate a patient procedure model based on the model surgical procedure and the skeletal model, and output the patient procedure model for display on the ARHMD while viewing the patient through the ARHMD.

[0140] In Example 62, the contents of Examples 55 to 61 further include instructions to cause the processing circuit to generate a predicted postoperative skeletal model based on the skeletal model, the skeletal model including the preoperative skeletal model, and the predicted postoperative skeletal model including an improved range of motion (ROM) based on the surgical procedure, and also include instructions to output the predicted postoperative skeletal model superimposed on the patient for display on an AR (augmented reality) HMD (head-mounted display) while viewing the patient through the ARHMD.

[0141] In Example 63, the content of Example 62 further includes instructions to cause the processing circuit to generate post-operative depth sensor data of the patient during a post-operative musculoskeletal assessment activity, generate an improved post-operative skeletal model based on the post-operative depth sensor data, and output the improved post-operative skeletal model overlaid on the pre-operative skeletal model for display on the ARHMD.

[0142] In example 64, the subject matter of example 63 further includes instructions for causing the processing circuit to output the refined post-operative skeletal model overlaid on the predicted post-operative skeletal model for display on the ARHMD.

[0143] In Example 65, the subject matter of Examples 62 to 64 further includes instructions for causing the processing circuitry to receive a selection of a surgical procedure, wherein the predicted post-operative skeletal model is further based on the selection of the surgical procedure.

[0144] In example 66, the subject matter of example 65 further includes instructions for causing the processing circuit to identify a list of surgical procedures associated with the musculoskeletal assessment activity and output a selection prompt for the list of surgical procedures for display on the ARHMD.

[0145] In Example 67, the content of Examples 55 to 66 further includes instructions to the processing circuit to operate the patient ARHMD, and the instructions further cause the processing circuit to output a predicted skeletal model for display on the ARHMD while viewing the patient through the ARHMD, capture an image of the patient as seen by the physician through the ARHMD, and output the predicted skeletal model superimposed on the image of the patient for display on the patient ARHMD.

[0146] In Example 68, the content of Examples 55 to 67 further includes instructions to cause the processing circuit to output a multi-pose skeletal model, the multi-pose skeletal model configured to display multiple positions of a patient's body part based on the improved ROM when the multi-pose skeletal model is overlaid on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0147] In Example 69, the content of Example 68 further includes instructions to cause the processing circuit to output a kinematic skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD, the kinematic skeletal model showing the movement of the patient's body parts based on the improved ROM.

[0148] In example 70, the subject matter of example 69 further includes instructions for causing the processing circuit to receive a skeletal model movement pause input and freeze the movement of the patient body part in the display on the patient's ARHMD.

[0149] In Example 71, the subject matter of Examples 55 to 70 further includes instructions for causing the processing circuit to receive a selection of a surgical procedure.

[0150] In example 72, the subject matter of example 71 further includes instructions for causing the processing circuitry to prompt the user for an improved ROM surgical procedure, the improved ROM surgical procedure providing a larger ROM than the surgical procedure.

[0151] In Example 73, the content of Examples 55 to 72 further includes instructions for causing the processing circuit to receive a skeletal model modification input, generate a modified skeletal model based on the skeletal model modification input, and output the modified skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0152] In Example 74, the content of Example 73 includes that the skeletal model correction input includes at least skeletal model joint repositioning, limb length adjustment, limb pose adjustment, and skeletal model reset input.

[0153] In Example 75, the content of Examples 66 to 74 further includes instructions for causing the processing circuit to identify a surgical procedure suggestion associated with at least one element of a list of surgical procedures associated with the musculoskeletal assessment activity, and outputting the selection prompt for the surgical procedure list includes displaying the surgical procedure suggestion for display on the ARHMD.

[0154] In Example 76, the subject matter of Example 75 includes, wherein the surgical procedure implications include at least one of recovery time, recovery physical therapy requirements, and predicted ROM.

[0155] In Example 77, the content of Examples 55 to 76 further includes instructions to cause the processing circuit to receive a surgical avoidance selection, generate a predicted surgical avoidance skeletal model based on the skeletal model, the predicted skeletal model including a reduced ROM based on the avoidance from the surgical procedure, and output the predicted surgical avoidance skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0156] In Example 78, the contents of Examples 55 to 77 further include instructions for causing the processing circuit to receive aging progression input and generate a plurality of aging skeletal models based on the skeletal model, the plurality of aging skeletal models including a plurality of reduced ROM values ​​based on the aging progression input, and also include instructions for outputting the progression of the plurality of aging skeletal models superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD.

[0157] In Example 79, the content of the undefined embodiment further includes instructions for causing the processing circuit to capture patient movement, and the selection of ROM exercises is based on the patient movement.

[0158] Example 80 is a system for patient assessment comprising: a depth sensor for generating depth sensor data regarding a patient; an image sensor for capturing multiple images of the patient; a processing circuit; and a memory containing instructions that, when executed by the processing circuit, cause the processing circuit to generate a pre-operative skeletal model based on the depth sensor data, generate a post-operative skeletal model based on the pre-operative skeletal model, the post-operative skeletal model including an improved range of motion based on a preset surgical procedure, and output multiple superimposed post-operative skeletal models of the patient for display.

[0159] Example 81 is a method of evaluation, including generating depth sensor data regarding a patient, capturing a plurality of images of the patient, generating a pre-operative skeletal model based on the depth sensor, generating a post-operative skeletal model based on the pre-operative skeletal model, the post-operative skeletal model including an improved range of motion based on a preset surgical procedure, and outputting for display the post-operative skeletal model superimposed on the plurality of images of the patient.

[0160] Example 82 is at least one machine-readable medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 1 to 81.

[0161] Example 83 is an apparatus having means for realizing any of Examples 1 to 81.

[0162] Example 84 is a system for realizing any of Examples 1 to 81.

[0163] Example 85 is a method for realizing any of Examples 1 to 81.

[0164] The example methods described herein can be implemented at least in part by a machine or computer. Some embodiments can include an encoded computer-readable medium or machine-readable medium containing instructions operable to configure electronic equipment to perform the methods as described in the above embodiments. Implementations of these methods can include code such as microcode, assembly language code, high-level language code, or the like. Such code can include computer-readable instructions for performing various methods. The code can form part of a computer program product. Furthermore, in one embodiment, the code can be tangibly stored, such as during execution or at other times, on one or more volatile, non-transitory, or non-volatile tangible computer-readable media. Examples of these tangible computer-readable media can include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memory (RAM), read-only memory (ROM), and the like. According to aspect (1), there is provided a system for augmented reality patient assessment, the system comprising: Augmented reality (AR) head-mounted display (HMD) and a depth sensor for generating depth sensor data about the patient during a musculoskeletal assessment activity; a processing circuit; a memory containing instructions that, when executed by the processing circuitry, cause the processing circuitry to: generating a skeletal model based on the depth sensor data; tracking the patient's movements during said musculoskeletal assessment activity; determining a current ROM based on the patient's motion; outputting the current ROM superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD; Memory and The system comprises: According to aspect (2), the instructions further cause the processing circuit to: receiving a selection of the musculoskeletal assessment activity; A description of the musculoskeletal assessment activity is output for display on the ARHMD while viewing the patient through the ARHMD. According to aspect (3), the instructions further cause the processing circuit to: determining a target ROM based on the musculoskeletal assessment activity; A graphical representation of the target ROM is output for display on the ARHMD while viewing the patient through the ARHMD. According to aspect (4), the instructions further cause the processing circuit to output guided musculoskeletal activities for display on the ARHMD, the guided musculoskeletal activities providing patient motor commands for performing the musculoskeletal assessment activities. According to aspect (5), the instructions further cause the processing circuit to receive motion sensor data from a motion sensor attached to the patient, the motion sensor data characterizing musculoskeletal motion of the patient; The generation of the skeletal model is further based on the sensor data. According to aspect (6), the instructions further cause the processing circuit to receive medical imaging data of the patient's musculoskeletal joint; The generation of the skeletal model is further based on the medical imaging data. According to aspect (7), the instructions further cause the processing circuit to: receiving a selection of a model surgical procedure; generating a patient procedure model based on the model surgical procedure and the skeletal model; The patient treatment model is output for display on the ARHMD while viewing the patient through the ARHMD. According to aspect (8), the instructions further cause the processing circuit to: generating a predicted post-operative skeletal model based on the skeletal model, the skeletal model including a pre-operative skeletal model, the predicted post-operative skeletal model including an improved range of motion (ROM) based on the surgical procedure; The predicted post-operative skeletal model is outputted superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD. According to aspect (9), the depth sensor further generates post-operative depth sensor data about the patient during a post-operative musculoskeletal assessment activity; The instructions further cause the processing circuitry to: generating an improved post-operative skeletal model based on the post-operative depth sensor data; The refined post-operative skeletal model is output overlaid on the pre-operative skeletal model for display on the ARHMD. According to aspect (10), the instructions further cause the processing circuit to receive a selection of a surgical procedure, and the predicted post-operative skeletal model is further based on the selection of the surgical procedure. According to aspect (11), the instructions further cause the processing circuit to: identifying a list of surgical procedures associated with the musculoskeletal assessment activity; A selection prompt is output for a list of the surgical procedures to display on the ARHMD. According to aspect (12), there is provided a method for augmented reality patient assessment, the method comprising: generating depth sensor data about the patient during a musculoskeletal assessment activity; generating a skeletal model based on the depth sensor data; tracking the patient's movements during the musculoskeletal assessment activity; and determining a current ROM based on the patient's motion; and outputting the current ROM superimposed on the patient for display on an augmented reality (AR) head-mounted display (HMD) while viewing the patient through the AHMD; The method includes: According to aspect (13), further, receiving a selection of the musculoskeletal assessment activity; outputting a description of the musculoskeletal assessment activity for display on the ARHMD while viewing the patient through the ARHMD; Includes. According to aspect (14), further, determining a target ROM based on the musculoskeletal assessment activity; and outputting a graphical representation of the target ROM for display on the ARHMD while viewing the patient through the ARHMD; Includes: According to aspect (15), the method further includes outputting guided musculoskeletal activities for display on the ARHMD, the guided musculoskeletal activities providing motor commands for the patient to perform the musculoskeletal evaluation activities. According to aspect (16), the method further includes receiving motion sensor data from a motion sensor attached to the patient, the motion sensor data characterizing musculoskeletal motion of the patient; The generation of the skeletal model is further based on the sensor data. According to aspect (17), the method further includes receiving medical imaging data of the patient's musculoskeletal joint; The generation of the skeletal model is further based on the medical imaging data. According to aspect (18), further, receiving a selection of a model surgical procedure; generating a patient procedure model based on the model surgical procedure and the skeletal model; outputting the patient treatment model for display on the ARHMD while viewing the patient through the ARHMD; Includes: According to aspect (19), further, generating a predicted post-operative skeletal model based on the skeletal model, the skeletal model including a pre-operative skeletal model, the predicted post-operative skeletal model including an improved range of motion (ROM) based on the surgical procedure; outputting the predicted post-operative skeletal model superimposed on the patient for display on an augmented reality (AR) head-mounted display (HMD) while viewing the patient through the AHMD; Includes: According to aspect (20), further, generating post-operative depth sensor data for the patient during a post-operative musculoskeletal assessment activity; generating an improved post-operative skeletal model based on the post-operative depth sensor data; outputting the refined post-operative skeletal model overlaid on the pre-operative skeletal model for display on the ARHMD; Includes: According to aspect (21), the method further includes receiving a selection of a surgical procedure, and the predicted post-operative skeletal model is further based on the selection of the surgical procedure. According to aspect (22), further, identifying a list of surgical procedures associated with the musculoskeletal assessment activity; outputting a selection prompt for the list of surgical procedures for display on the ARHMD; Includes: According to aspect (23), a non-transitory machine-readable storage medium containing instructions, the instructions, when executed by a processing circuit of a computer-controlled device, causing the processing circuit to: generating depth sensor data about the patient during a musculoskeletal assessment activity; generating a skeletal model based on the depth sensor data; tracking the patient's movements during said musculoskeletal assessment activity; determining a current ROM based on the patient's motion; outputting the current ROM superimposed on the patient for display on an augmented reality (AR) head-mounted display (HMD) while viewing the patient through the ARHMD; It is a non-transitory machine-readable storage medium. According to aspect (24), the instructions further cause the processing circuit to: receiving a selection of the musculoskeletal assessment activity; A description of the musculoskeletal assessment activity is output for display on the ARHMD while viewing the patient through the ARHMD. According to aspect (25), the instructions further cause the processing circuit to: determining a target ROM based on the musculoskeletal assessment activity; A graphical representation of the target ROM is output for display on the ARHMD while viewing the patient through the ARHMD.

Claims

1. 1. A system for augmented reality patient assessment, the system comprising: an augmented reality (AR) head-mounted display (HMD); a depth sensor for generating depth sensor data about the patient during a musculoskeletal assessment activity; a processing circuit; a memory containing instructions that, when executed by the processing circuitry, cause the processing circuitry to: generating a skeletal model in which feature points including joints are represented by coordinate data based on a skeletal estimation technique using the depth sensor data; tracking images of the patient's movements during said musculoskeletal assessment activity; determining a current ROM based on a comparison of the patient motion image and the patient rest image; outputting the current ROM superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD; Memory and Equipped with The instructions cause the processing circuitry to: Determine improved range of motion (ROM) based on musculoskeletal surgery; generating a predicted post-operative skeletal model based on the improved range of motion (ROM) and a pre-operative skeletal model of the patient, the predicted post-operative skeletal model including the improved range of motion (ROM); outputting the predicted post-operative skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD; system.

2. The instructions further cause the processing circuitry to: receiving a selection of the musculoskeletal assessment activity; outputting a description of the musculoskeletal assessment activity for display on the ARHMD while viewing the patient through the ARHMD; The system of claim 1 .

3. The instructions further cause the processing circuitry to: determining a target ROM based on the depth sensor data of the patient during the musculoskeletal assessment activity; outputting a graphical representation of the target ROM for display on the ARHMD while viewing the patient through the ARHMD; The system of claim 1 .

4. 2. The system of claim 1, wherein the instructions further cause the processing circuit to output musculoskeletal activities for realizing a target ROM for display on the ARHMD, the musculoskeletal activities for realizing the target ROM being displayed on the ARHMD in the form of motor commands for the patient to perform the musculoskeletal evaluation activities.

5. The instructions further cause the processing circuit to receive motion sensor data from a motion sensor attached to the patient, the motion sensor data characterizing musculoskeletal motion of the patient; the generation of the skeletal model is further based on the motion sensor data. The system of claim 1 .

6. The instructions further cause the processing circuit to receive medical imaging data of the patient's musculoskeletal joint; generating the skeletal model further based on the medical imaging data, including X-ray, MRI, and CT scan; The system of claim 1 .

7. The instructions further cause the processing circuitry to: receiving a selection of model surgical procedures to consider for possible need for said patient; generating a necessary procedure for the patient based on the model surgical procedure and the skeletal model to determine whether it is necessary for the patient; outputting a treatment required for the patient for display on the ARHMD while viewing the patient through the ARHMD; The system of claim 1 .

8. the depth sensor further generates post-operative depth sensor data regarding the patient during a post-operative musculoskeletal assessment activity; The instructions further cause the processing circuitry to: generating a postoperative skeletal model based on the postoperative depth sensor data; outputting the post-operative skeletal model superimposed on the pre-operative skeletal model for display on the ARHMD; The system of claim 1 .

9. The system of claim 1 , wherein the instructions further cause the processing circuitry to receive a selection of a surgical procedure, and wherein the predicted post-operative skeletal model is further based on the selection of the surgical procedure.

10. The instructions further cause the processing circuitry to: identifying a list of the surgical procedures associated with the musculoskeletal assessment activity; outputting a selection prompt for a list of said surgical procedures for display on said ARHMD; The system of claim 9.

11. A method for augmented reality patient assessment performed by the system of claim 1, the method comprising: generating the depth sensor data about the patient during the musculoskeletal assessment activity; generating a skeletal model based on the depth sensor data; tracking the patient's movements during the musculoskeletal assessment activity; and determining the current ROM based on the patient's motion; outputting the current ROM superimposed on the patient for display on an augmented reality (AR) head-mounted display (HMD) while viewing the patient through the AR HMD; A method comprising:

12. Furthermore, receiving a selection of the musculoskeletal assessment activity; outputting a description of the musculoskeletal assessment activity for display on the ARHMD while viewing the patient through the ARHMD; The method of claim 11 , comprising:

13. Furthermore, determining a target ROM based on the depth sensor data of the patient during the musculoskeletal assessment activity; outputting a graphical representation of the target ROM for display on the ARHMD while viewing the patient through the ARHMD; The method of claim 11 , comprising:

14. 12. The method of claim 11, further comprising outputting musculoskeletal activities for realizing a target ROM for display on the ARHMD, wherein the musculoskeletal activities for realizing the target ROM are displayed on the ARHMD in the form of motor commands for the patient to perform the musculoskeletal evaluation activities.

15. further comprising receiving motion sensor data from a motion sensor attached to the patient, the motion sensor data characterizing musculoskeletal motion of the patient; The generation of the skeletal model is further based on the motion sensor data. The method of claim 11.

16. further comprising receiving medical imaging data of the patient's musculoskeletal joint; generating the skeletal model further based on the medical imaging data, including X-ray, MRI, and CT scan; The method of claim 11.

17. Furthermore, receiving a selection of model surgical procedures to consider for possible need for said patient; generating a necessary procedure for the patient based on the model surgical procedure and the skeletal model to determine whether it is necessary for the patient; outputting a treatment required for said patient for display on said ARHMD while viewing said patient through said ARHMD; The method of claim 11 , comprising:

18. Furthermore, generating post-operative depth sensor data for the patient during a post-operative musculoskeletal assessment activity; generating a post-operative skeletal model based on the post-operative depth sensor data; outputting the post-operative skeletal model overlaid on the pre-operative skeletal model for display on the ARHMD; The method of claim 11 , comprising:

19. The method of claim 11 , further comprising receiving a selection of a surgical procedure, wherein the predicted post-operative skeletal model is further based on the selection of the surgical procedure.

20. Furthermore, identifying a list of the surgical procedures associated with the musculoskeletal assessment activity; outputting a selection prompt for the list of surgical procedures for display on the ARHMD; 20. The method of claim 19, comprising:

21. A non-transitory machine-readable storage medium containing instructions that, in response to being executed by a processing circuit of a computer-controlled device, cause the processing circuit to: generating depth sensor data about the patient during a musculoskeletal assessment activity; generating a skeletal model in which feature points including joints are represented by coordinate data based on a skeletal estimation technique using the depth sensor data; tracking images of the patient's movements during said musculoskeletal assessment activity; determining a current ROM based on a comparison of the patient motion image and the patient rest image; outputting the current ROM superimposed on the patient for display on an augmented reality (AR) head-mounted display (HMD) while viewing the patient through the AR HMD; Determine improved range of motion (ROM) based on musculoskeletal surgery; generating a predicted post-operative skeletal model including the improved range of motion (ROM) based on the improved range of motion (ROM) and the patient's pre-operative skeletal model; outputting the predicted post-operative skeletal model superimposed on the patient for display on the ARHMD while viewing the patient through the ARHMD; Non-transitory machine-readable storage medium.

22. The instructions further cause the processing circuitry to: receiving a selection of the musculoskeletal assessment activity; outputting a description of the musculoskeletal assessment activity for display on the ARHMD while viewing the patient through the ARHMD; 22. The non-transitory machine-readable storage medium of claim 21.

23. The instructions further cause the processing circuitry to: determining a target ROM based on the depth sensor data of the patient during the musculoskeletal assessment activity; outputting a graphical representation of the target ROM for display on the ARHMD while viewing the patient through the ARHMD; 22. The non-transitory machine-readable storage medium of claim 21.

Images