Augmented reality therapy mobile display and gesture analyzer

Abstract

Systems and methods for displaying augmented reality clinical movements can use an augmented reality device to display various aspects of clinical movements. The systems and methods can use a motion capture device to capture clinical movements. A method can include analyzing information about a clinical movement to determine a motion path representing at least a portion of the clinical movement. The method can automatically define a path area or a virtual target in an augmented reality environment overlaid on a real environment. The method can display the path area or the virtual target on the augmented reality display.

Description

[0001] This application is a divisional application of application number 201780081935.9, filed on November 3, 2017, entitled 'Augmented Reality Therapy Mobile Display and Gesture Analyzer'.

[0002] Cross-references to related applications

[0003] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 416869, filed November 3, 2016; the benefit of U.S. Provisional Patent Application No. 62 / 421001, filed November 11, 2016; and the benefit of U.S. Provisional Patent Application No. 62 / 440588, filed December 30, 2016. Priority to each of the foregoing documents is hereby claimed, and each of the foregoing documents is incorporated herein by reference in its entirety. Background Technology

[0004] Remote rehabilitation systems are typically used to remotely assess or monitor patients undergoing rehabilitation activities. Due to the remote nature of remote rehabilitation, current systems are often limited or not used for occupational therapy or physical therapy. Occupational therapy or physical therapy includes exercises or activities used to recover from injury or surgery, or to otherwise improve mobility. Patients often forget how to perform rehabilitation-related activities or cannot understand the instructions provided describing the activities. Attached Figure Description

[0005] The accompanying drawings are not necessarily drawn to scale. In the drawings, the same numbers may describe similar parts in different views. The same numbers with different letter suffixes may represent different instances of similar parts. The accompanying drawings illustrate various embodiments discussed herein by way of example and not limitation.

[0006] Figure 1 The illustrations depict the real and virtual aspects of an augmented reality and exercise creation system according to some embodiments.

[0007] Figure 2 The illustration shows an enhanced reality display according to some embodiments.

[0008] Figure 3 An automatic gesture display system according to some embodiments is illustrated.

[0009] Figure 4 The illustration shows a patient evaluation display system according to some embodiments.

[0010] Figure 5 The illustration shows a remote rehabilitation system according to some embodiments.

[0011] Figure 6The illustration shows a flowchart illustrating a technique for remote rehabilitation according to some embodiments.

[0012] Figure 7 The illustration shows a flowchart, according to some embodiments, of a technique for displaying directions related to a patient's treatment movement in an augmented reality environment.

[0013] Figure 8 The illustration shows a flowchart illustrating techniques for displaying enhanced realistic clinical movement according to some embodiments.

[0014] Figure 9 The diagram illustrates, in general, an example of a block diagram of a machine on which any one or more of the technologies discussed herein according to some embodiments may be executed. Detailed Implementation

[0015] This paper describes systems and methods for remote rehabilitation feedback. Specifically, it describes the use of motion capture devices to capture remote rehabilitation information from therapists or patients. Remote recovery systems and methods can present feedback using user interfaces or video displays, augmented reality (AR) displays, virtual reality (VR) displays, auditory alarms, haptic feedback, non-contact alarms, and the like.

[0016] In this example, a remote rehabilitation system could be used to provide a mechanism for capturing video of rehabilitation exercises and automatically identifying key points of the therapist performing the exercises within the video. These key points can be used to create visual goals for the patient's exercise as they attempt it. The system can virtually display the rehabilitation exercises, including these visual goals. These visual goals can include "bubbles," such as spherical or circular visual representations that can "pop" (e.g., visual representations of bubbles being removed from the display by showing animations or visual effects as the patient or object interacts with them). The system can capture video of the patient performing the exercises and can analyze the patient's attempts. The visual goals can be used to demonstrate the effectiveness of the patient's attempts; for example, a popping bubble can represent the successful completion of an exercise or a portion of an exercise. The system can provide feedback to the patient (e.g., popping or unpopped bubbles).

[0017] The systems and methods described herein can, for example, automatically determine the placement of gesture bubbles (e.g., bubbles for gesture placement, such as the start or end of a gesture for an exercise) based on the therapist's movement. For instance, the therapist may stand in front of a motion capture device (e.g., a sensor, sensor array, camera, infrared camera, two or more cameras, depth camera, etc.) and move, and the system can automatically place gesture bubbles. The system can place bubbles in determined locations and continue to a second bubble. For example, a first bubble may correspond to a first position (e.g., a start position), and a second bubble may correspond to a second position (e.g., an end position). A path region may include a path between the first and second positions, or may include either a start or end position. In the example, a gesture video can be created using a path region, one or more bubbles, video captured from the therapist, video captured from the patient, animation clips, etc. In the example, a gesture may include movement in an exercise, or may be an exercise (e.g., a gesture may include movement having multiple movements constituting an exercise, or a gesture may include movement in an exercise, which may be repetitive or include other gestures or exercises to form a routine).

[0018] Augmented Reality (AR) is a technology used to display virtual or “augmented” objects or visual effects overlaid on a real environment. The real environment can include a room or a specific area, or more generally, the entire world. Virtual aspects overlaid on the real environment can be represented as being anchored or situated in a predetermined position relative to one or more aspects of the real environment. For example, a virtual object can be configured to appear as if it is placed on a table. An AR system can present virtual aspects anchored to a real object regardless of the perspective of one or more observers within the AR system. For example, a virtual object can exist in a room, visible to an observer within the room but invisible to an observer outside the room. When an observer enters the room, the virtual object within the room can be displayed to the observer outside. In this example, the room can act as the real object anchoring the virtual object in the AR system.

[0019] An AR system can be visible to one or more observers and can include differences between views that are available to one or more observers, while maintaining some aspects of the views as common aspects. For example, the heads-up display can change between two views when a virtual object can be pinned to a real object or area in two views. Aspects such as changing the color of an object, changing lighting, or other changes can be made between views without altering the fixed position of at least one virtual object.

[0020] Users can perceive virtual objects presented in an AR system as opaque or as having some degree of transparency. In the example, a user can interact with the virtual object, for instance, by moving it from a first position to a second. For example, a user can move the object with their hand. This can be done virtually in the AR system by determining (e.g., using one or more cameras) that the hand has been moved to a position overlapping with or adjacent to the virtual object and causing the virtual object to move responsively. The virtual aspect can include virtual representations of real-world objects or visual effects such as lighting effects. The AR system can include rules for managing the behavior of virtual objects (e.g., subjecting virtual objects to gravity or friction) or other predefined rules that violate real-world physical constraints (e.g., floating objects, permanent motion, etc.).

[0021] This paper describes systems and methods for augmented reality (AR) remote rehabilitation. The systems and methods described herein use AR to display virtual targets for the patient, allowing the patient to perform clinical movements. In one example, a virtual therapist may be displayed. In another example, a real therapist with AR enhancements (e.g., highlighting of body parts, virtual orientation, etc.) may be displayed. Either the virtual or real therapist can demonstrate clinical movements. Clinical movements may include the use of objects. In one example, AR may be used to display objects the patient intends to use. In another example, AR may be used to display the orientation the patient takes to perform clinical movements. For example, the real therapist could be an on-site therapist present with the patient, or it could be a video recording of the therapist. A real therapist can be defined as any non-virtual therapist.

[0022] The systems and methods described herein can provide overlaid virtual graphics to help users understand how to move. Paths or targets can be highlighted, and virtual representations of movement can be shown to patients or therapists (e.g., virtual representations overlaid on the user as a viewpoint technique for illustrating movement). Virtual targets can be used to help patients visually observe the range of movement (e.g., the final endpoint of the movement).

[0023] Figure 1The illustration depicts the real and virtual aspects of an augmented reality and exercise creation system 100 according to some embodiments. In the example, system 100 includes a video capture device 105, a display device 107, and an input device 109, which a therapist 102 can use to generate model exercises for a rehabilitation patient. The therapist 102 can perform the exercises, which are captured by the video capture device 105 and displayed on the display device 107. The input device 109 can be used to edit or enhance the displayed exercises, or to select one or more exercises for a routine. System 100 can automatically edit the captured video to remove irrelevant portions that appear before or after the therapist 102 performs the exercises. In the example, a series of exercises can be performed by the therapist 102 and captured by the video capture device 105, and system 100 can separate the captured video of the series of exercises into individual exercise videos. In the example, the video capture device 105 can be a motion sensor from Microsoft Corporation in Redmond, Washington.

[0024] Input device 109 can be used to select various aspects of the exercise. The selected aspects may include a start position, an end position, or a transition movement. When a start position is selected, display device 107 can display the selection of the appropriate time in the captured exercise video. For example, a circle can be drawn around a displayed body part (e.g., a foot, hand, etc.), and this circle can be displayed in the video captured for the exercise. Similarly, the end position can be highlighted. When a transition movement is selected, a path that tracks the selected movement can be displayed during the captured video. The start position, end position, or transition movement can include an area in the captured video larger than the area occupied by the body part (e.g., an area with a radius larger than the radius around the center point of the body part).

[0025] In this example, a video capture device 105 can be used to calibrate the system 100. The video capture device 105 can use infrared light to detect the therapist 102 in the field of view. The system 100 can evaluate the detection results to identify the therapist 102's joints, limbs, appendages, head, etc. These identified body parts can be used with later-captured workout videos to label specific body parts.

[0026] After the video capture device 105 captures the exercise video, the therapist 102 (or another user) can edit the captured video. In the example, the therapist 102 can select a portion of the captured video and add tags (e.g., "Introduction", "Exercise", "First Repetition", "Second Repetition", "Ending Section", etc.). In the example, a single captured repetition can be repeated in the edited video to show multiple repetitions observed by the patient.

[0027] A final edited video of the exercise can be created. This video can be named and categorized, such as by body part, muscle group, post-operative type, patient-specific tags, etc. The final edited video can be saved for later use by, for example, therapist 102 in building a routine. In another example, the final edited video can be saved to a database for sharing with other users (e.g., other users co-caring for patients with therapist 102, other publicly available therapists in a company, group, or hospital, etc.). In this example, system 100 can be used to repeat the exercise a specified number of times, allowing the patient to observe the exercise for a specified number of repetitions to complete a routine or part of a routine.

[0028] System 100 can be used to capture three-dimensional motion. For example, video capture device 105 may include a motion capture device. The motion capture device may include two or more infrared sensors or cameras to detect or capture three-dimensional motion. Video capture device 105 may include a camera for capturing video in conjunction with the motion captured by infrared sensors. The captured motion may include video.

[0029] System 100 can interact with a real therapist (e.g., therapist 102) or may include a virtual therapist 103 displayed within system 100. System 100 can be used by patient 104. System 100 includes an AR device 108. In an example, system 100 may include a reference wall 106, a physical object 112, a virtual object 116, or a camera 110. In an example, physical object 112 may include a sensor 114. In another example, the sensor may be embedded in an implant of patient 104. In an example, a camera may be coupled to AR device 108. AR device 108 may include headphones, glasses, goggles, a touch point, a projector, etc. In an example, reference wall 106 may be identified during configuration or calibration of AR device 108.

[0030] AR device 108 may include camera 110. Camera 110 may include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, etc. AR device 108 can project virtual items onto a representation of a real environment, which can be observed by patient 104. In an example, the real environment may include a display of floors, rooms, and physical props, etc. Therapist 102 may exist in the real environment, and virtual aspects may be superimposed on therapist 102 in the AR environment generated by AR device 108. In another example, virtual therapist 103 may include virtual aspects, such as highlighting body parts, movement arrows, blurring to indicate movement, etc. Virtual therapist 103 may be placed at a predetermined location in the real environment, such as a location within an area of ​​the real environment visible to patient 104 or AR device 108. For example, virtual therapist 103 may be located in front of AR device 108 within system 100, which is visible to patient 104, so that patient 104 can observe virtual therapist 103. When the patient 104 rotates the AR device 108, the virtual therapist 103 can be designed to disappear from the view, similar to how the real therapist 102 would disappear when the patient's head is turned.

[0031] In the example, the virtual therapist 103 can be shown as being overlaid on the real environment to demonstrate the exercise. In the example, a virtual display of props or objects used for the exercise can be overlaid on the real environment. For example, virtual object 116 can represent physical object 112. The AR device 108 can be used to instruct the patient 104 to locate physical object 112 in the room by displaying, for example, virtual object 116. In the example, virtual object 116 can be displayed when used by the virtual therapist 103, or it can be displayed near, above, or shown as floating nearby the real therapist 102.

[0032] Physical object 112 may include sensor 114. Sensor 114 can be used to track patient progress, such as the duration or number of repetitions completed by patient 104. In this example, sensor 114 can be used to identify physical object 112 for AR device 108. Once identified, AR device 108 can select virtual object 116 corresponding to physical object 112. For example, if patient 104 has a 10-pound weight and a 5-pound weight available, these weights can be identified via a weight-related sensor, and AR device 108 can determine that the 5-pound weight will be used for the exercise, and virtual object 116 can resemble the 5-pound weight. Virtual object 116 can change to resemble a 10-pound weight in another exercise or when the weight is increased to increase the difficulty of the exercise. In this example, AR device 108 can display virtual movement of the limb or virtual exercise, or it can display virtual bubbles to indicate the start, end, or path of the movement or exercise.

[0033] In the example, an implant sensor may be embedded in an implant in patient 104. The implant sensor can be used to track the movement (e.g., number of repetitions), non-movement, etc., of patient 104. This tracked movement can be used to enhance movement captured by camera 110 or a patient-facing motion capture device (e.g., a sensor, sensor array, camera, infrared camera, two or more cameras, depth camera, etc.). Patient-facing motion capture devices, camera 110, implant sensors, or any motion capture device can be used to track the movement of patient 104. In the example, the prescribed treatment output or treatment can be modified or updated based on sensor data from the implant sensor. For example, in total knee replacement surgery, the knee prosthesis (implant) may include sensors for monitoring pressure generated during movement, and when too much pressure exists on one side of the implant, an AR device 108 can be used to display indications for easy movement, changing movement, or stopping movement. After patient 104 has moved, completed a scheduled procedure, or performed at least part of a scheduled treatment, therapist 102 or other caregiver (e.g., surgeon) can use data from implanted sensors or AR device 108 (e.g., whether patient 104 successfully moved, percentage of successful execution, metrics related to repetition count, weight used, etc.). This data can be used to adjust prescription treatments, movements, exercises, medications, surgical schedules, etc. Patient 104 and real therapist 102 can share the augmented experience using AR device 108 or multiple AR devices. For example, real therapist 102 can have an AR device, and therapist 102's AR device can display augmented and virtual aspects superimposed on the same real environment in the environment shared with AR device 108. Real therapist 102 can manipulate virtual or real aspects of the shared environment so that patient 104 can see such manipulation. For example, real therapist 102 can cause virtual bubbles to burst or virtual objects to rise, wherein the virtual bubbles or virtual objects can be visible to both patient 104 and real therapist 102, as referenced below. Figure 2 Further discussion and descriptions in the following paragraphs and elsewhere.

[0034] Figure 2An augmented reality (AR) display 200 is illustrated according to some embodiments. The AR display 200 can be used by a patient 201 to display virtual aspects in a real-world environment. The AR display 200 may include virtual identifiers of the patient 201's joints 202, a reference wall 218 displayed in the real-world environment, or multiple virtual reference indicators. Reference indicators may include a virtual start bubble 206, a virtual end bubble 210, a virtual movement path 208, virtual edges 214 and 216 to a path region, a virtual start limb position 204, or a virtual end limb position 212. In an example, reference indicators may include a virtual example start bubble 220, a virtual example end bubble 222, or a virtual example path region 224. The virtual example start bubble 220, virtual example end bubble 222, or virtual example path region 224 may be displayed as overlaid on a representation of a therapist present in the real-world environment within the AR environment. In another example, the virtual example start bubble 220, virtual example end bubble 222, or virtual example path region 224 may be displayed as overlaid on a virtual therapist. In yet another example, the virtual example start bubble 220, the virtual example end bubble 222, or the virtual example path area 224 may be displayed together with a virtual limb (e.g., a body part of a virtual therapist, but not the entire virtual therapist). In yet another example, the virtual example start bubble 220, the virtual example end bubble 222, or the virtual example path area 224 may be displayed without a virtual element or in the absence of a therapist in the real environment.

[0035] In the example, a virtual start bubble 206, a virtual end bubble 210, a virtual motion path 208, or virtual edges 214 or 216 to the path area can be virtually displayed. The virtual start bubble 206, virtual end bubble 210, virtual motion path 208, or virtual edges 214 or 216 to the path area can be displayed using color or a variety of color variations, and can be displayed sequentially, etc. In the example, if the patient makes a movement corresponding to the virtual representation, the virtual start bubble 206, virtual end bubble 210, virtual motion path 208, or virtual edges 214 or 216 to the path area can be removed from the view by "bursting" or otherwise. For example, if the patient physically places a body part in an occupied position in AR using the virtual start bubble 206, the virtual start bubble 206 can burst. In the example, when a body part moves from the virtual start bubble 206 to the virtual end bubble 210, the virtual motion path 208 can include multiple bubbles to burst.

[0036] In this example, an AR device can be used to generate an AR display 200. The AR device may include a projection screen, goggles, glasses, etc. In this example, the AR device can project animations around the patient 201, allowing the patient 201 to see a virtual start bubble 206, a virtual end bubble 210, a virtual movement path 208, virtual edges 214 and 216 to the path area, a virtual start limb position 204, or a virtual end limb position 212. As the patient 201 moves, the AR device can present an avatar (e.g., a virtual therapist) to deliver a virtual treatment experience. The virtual therapist can be pre-programmed or controlled by the therapist in real time. In this example, the patient 201 may have the opportunity to ask questions to the virtual therapist using the AR display 200 or an input device within the AR device 108. The treatment for the patient 201 can be modified based on the questions, answers, or interactions with the patient 201. For example, the exercise can be reduced or its difficulty or duration increased based on the patient 201's interactions. In this example, the patient 201 may ask the therapist to repeat or describe the exercise. The avatar can guide the patient 201 to a physical location, model the exercise, provide encouragement, correction, modification, and display success or failure, etc. In the example, the avatar can occupy a virtual physical presence (e.g., a static location in a room (e.g., relative to reference wall 218)).

[0037] AR display 200 can allow patient 201 to select joints or locations on the AR avatar. By selecting joints or locations on the AR avatar, patient 201 can indicate where the injury is or requires attention, select one side of the body, indicate pain level, etc. In another example, patient 201 can select options from a user interface within AR display 200, such as a pain scale. AR display 200 may include questionnaires for patient responses, allowing therapists to assess progress or determine patient exercise.

[0038] In the example, patient 201 may have a personalized movement profile. The AR device can apply the personalized movement profile to personalize the AR display 200. The personalized AR display 200 may include specific exercises, movements, restrictions, etc., tailored for patient 201. For example, if patient 201 has received implants (e.g., implants received during total knee arthroscopy, hip replacement, cardiac device implantation surgery, etc.), the AR display 200 may include exercises designed with these implants or surgeries in mind. The AR display 200 may include education for patient 201 to, for example, improve outcomes or mitigate movements that may harm the patient.

[0039] In the example, AR display 200 may include virtual mirrors. Virtual mirrors can be used for mirrorboxing techniques. Mirrorboxing can be used to create mirrors of limbs, for example, when a mirror limb is not present. For example, if patient 201 has lost their left leg below the knee, a mirror of the right leg can be created in AR display 200, allowing AR display 200 to display the left leg below the knee by mirroring the right leg. In another example, the left leg below the knee can be virtually displayed based on the right leg, or the left leg below the knee can be constructed based on virtual parts. Mirrorboxing can be used to alleviate pain in patient 201 from the phantom limbs, allowing the patient to attempt to regain use of the limbs, reduce limb pain, etc. Virtually displayed limbs can be used to map movement in patient 201's brain, for example, by overlapping limbs and issuing commands to patient 201 to virtually move the limbs. For example, patient 201 cannot control their right arm but can move their left arm. When the left arm moves in the real environment, this can cause the right arm to move virtually in AR display 200. Patient 201's brain can map pathways to actually control his right arm by moving his left arm and observing the movement of his right arm. This technique could be used to cure or restore the ability to move his right arm.

[0040] Figure 3 An automated gesture display system 300 according to some embodiments is illustrated. The automated gesture display system 300 includes a user interface 302. The user interface 302 includes a video / animated automated gesture display component 304. The video / animated automated gesture display component 304 can be used to manually or automatically add movement bubbles (e.g., 308 or 314) or path regions. In the example, movement bubbles (e.g., 308 or 314) can be automatically added to motion path 312. For example, parameters including success or failure parameters can be added to motion path 312 to create a path region (e.g., a region around motion path 312 between edge 318 and edge 320). The path region can be used to show a patient how to perform clinical movements. The path region can be used to determine whether a movement performed by the patient was successful. In the example, the path region or path movement can be displayed on a display (e.g., a screen, AR display, VR display, etc.).

[0041] In one example, the path area can be used to determine whether the patient has successfully completed the exercise. For example, if the patient completes the exercise within the path area, the exercise can be considered successfully completed. If the patient moves outside the path area while attempting to complete the exercise, the exercise can be considered unsuccessfully completed. In another example, moving bubbles can be used to analyze patient video to determine whether the patient has performed the exercise correctly. For example, if the patient is able to pop the bubbles (e.g., all the bubbles in the exercise) while performing the exercise, the exercise can be considered successfully completed. In this example, popping one or more bubbles indicates partial completion of the exercise. In yet another example, both the path area and the bubbles can be used to determine whether the exercise has been successfully completed.

[0042] The path region can be represented in two or three dimensions. For example, the path region can include a two-dimensional extension of motion path 312. In another example, the path region can include a three-dimensional extension of motion path 312. For example, the path region can include a region within a radial distance (e.g., two-dimensional or three-dimensional) away from motion path 312. In the example, a start position or an end position can be indicated for motion path 312. The start position or end position can include a radial distance (e.g., two-dimensional or three-dimensional) away from the start point or end point of motion path 312, respectively. For example, the start position can include a circle or a sphere surrounding the start point of motion path 312. The start point can be indicated by a therapist or can be detected automatically.

[0043] The video / animation automatic gesture display component 304 includes a start bubble 308 at the start position (e.g., corresponding to...). Figure 2 The start position 208) and the end position end bubble 314 (e.g., corresponding to the ... Figure 2 The video / animation auto-gesture display component 304 includes a captured video display or animation joint 306 and limb positions 310 and 316 corresponding to the start and end, respectively. The video / animation auto-gesture display component 304 can automatically generate a start bubble 308, an end bubble 314, an edge 318, or an edge 320. In the example, edges 318 and 320 can be automatically generated at a predetermined distance from the motion path 312 (e.g., multiple pixels, a distance determined using the zoom or animation distance of the captured video).

[0044] In one example, the clinical movements captured from the therapist can be animated later, and the animation can be used together with a path region to create a gesture video. In another example, the therapist can select a previously generated animation to create a gesture video. In yet another example, the therapist can select a previously captured video to create a gesture video. To create a gesture video, the therapist can select the joints to move (e.g., joint 306), such as the knee, hip, elbow, shoulder, neck, etc. Joint 306 can be automatically detected in the captured video, and a path region can be applied to movements extending from joint 306 (e.g., if the knee is selected, the path region can be used for the foot when the knee is extended, or in another example, the selected joint can be used as the path region), for example, movement along a motion path 312 between edges 318 and 320 from start bubble 308 to end bubble 314.

[0045] In the example, a motion capture device can be used to capture video, and start bubble 308 and end bubble 314 can be automatically added to the captured video. Start bubble 308 or end bubble 314 can be color-coordinated; for example, green start bubble 308 indicates the start point of movement, while red end bubble 314 indicates the end point of movement. In the example, the color can change as movement progresses. For example, start bubble 308 can be a first color, and then change to a second color as a specific body part or object is placed within start bubble 308 to indicate correct placement. Edges 318 or 320 can similarly change color based on whether the movement is within or outside edges 318 or 320. End bubble 314 can similarly change color when a body part or object is placed within end bubble 314. In the example, the start bubble 308 or the end bubble 314 can "pop" (e.g., animate to disappear), for example, when a user places a body part or object (e.g., a predetermined specific body part or object) within the start bubble 308 or the end bubble 314. In the example, a series of bubbles can be placed between the start bubble 308 and the end bubble 314, for example, along a motion path 312. A series of bubbles can change color or pop as described above with respect to the start bubble 308 or the end bubble 314. In another example, the start bubble 308 and the end bubble 314 can be in the same location or close to each other. For example, the motion path 312 can include a round trip or movement away from the start bubble 308 and then back towards or near the start bubble 308, which can transform into the end bubble 314 during the movement.

[0046] In the example, the therapist can select gesture tags to identify the joints the patient must move to meet exercise requirements. A motion capture device can be used to determine whether motion has been captured and whether the identified joint moved in the manner indicated by one or more gesture tags. For example, the gesture tag selected by the therapist could correspond to a joint tag identified by the motion capture device.

[0047] In the example, bubbles can be used to indicate restrictions on movement. For instance, a particular movement could include restricting head movement within a specific bubble to ensure the patient moves correctly.

[0048] According to some embodiments, Figure 3 The user interface 302 can be used as a video / animation creation and display system. The user interface 302 may include video / animation creation components. Movement can be completed along motion path 312. The video / animation creation components can be used to edit or display captured video, edit or display animation, or edit or display the location or path of movement (e.g., automatically generated location or motion path). In the example, the therapist can use a motion capture device (e.g., an infrared sensor or camera) to record clinical movement. Multiple cameras can be used for recording. The recorded clinical movement can be analyzed to determine the motion path 312.

[0049] The automated gesture display system 300 may include AR authoring tools. These tools can be used to enhance various aspects of detected gestures or clinical movements. For example, they can be used to alter the gesture or movement. In one example, the AR authoring tools can be used to create multiple different views of the gesture or movement. In another example, they can be used to enhance portions of a motion path or target object. For instance, the motion path can be enhanced with color, the distance from the center of the motion path can be selected (e.g., expanding or contracting the area surrounding the motion path), or waypoints can be set along the motion path to indicate or identify progress along it. Target objects can be enhanced with shapes, colors, styles (e.g., blinking, pulsating, glowing, etc.), transparency, etc.

[0050] In one example, the patient uses an augmented reality device to view a 3D animation presented in an augmented reality display. For example, the animation could include a virtual representation of the therapist performing clinical movements within an augmented reality environment superimposed on the real environment. In another example, the animation could be virtual reality animation. In yet another example, the animation could be augmented animation that enhances clinical movements performed by a therapist in front of the patient in a real environment. For example, the therapist's arms or legs or other moving limbs could be enhanced, for instance, with colors, outlines, arrows, etc., and the therapist's arms or legs or other moving limbs could be enhanced as the therapist performs the clinical movements.

[0051] In the example, a target range of motion (e.g., a diagnosis-based target range of motion) can be used to automatically create motion paths or target objects. For example, a therapist can perform a complete clinical movement, which may include one or more waypoints along the complete clinical movement, with waypoints representing points of progress. Waypoints may include an initial portion of the clinical movement corresponding to a first target object along the movement path, a second portion of the clinical movement corresponding to a second target object along the movement path, and so on. A final target can be placed at the end of the movement path. In this way, multiple motion path targets representing a complete range of motion can be established. The complete range of motion can be broken down into partial range of motion segments, which can be displayed to the patient (e.g., progressively) in an augmented reality environment. These segments can be coupled to progressively increase the range of motion targets (e.g., adding another target with a wider range of motion each day). The range of motion may include functional metrics that can be achieved by the patient. In the example, the range of motion can be varied based on the patient's height, weight, range of motion, proportions, etc.

[0052] In the example, creating motion paths or target objects can include automation, such as automatically creating motion paths or target objects based on anticipated experiences at home, in the clinic, or at work. For example, a pet, a smooth carpet, or other activities performed by a patient in daily life can be automatically added to the augmented reality environment to reflect daily activities in the treatment setting. These anticipated experiences can include fixed protocols that can be manipulated individually or can come from a database of shared activities.

[0053] In the example, the therapist can create complex paths for the patient. For instance, a multi-step movement can be created with specific waypoints where the patient stops during the movement. For example, a complex path could include a first path ending at a first waypoint and a second path ending at a second waypoint, where the patient raises their arm 90 degrees, and then at the second waypoint, moves their arm 90 degrees. Paths can be created individually and added together by the therapist to create a complex path, or a complex path can be created as a step-by-step process.

[0054] Figure 4A patient evaluation display system 400 according to some embodiments is illustrated. The patient evaluation display system 400 includes a user interface 402. In one example, user interfaces 202, 302, and 402 may be a single user interface with different views. In another example, credentials may be used to access user interfaces 202, 302, and 402, allowing access to one or more of user interfaces 202, 302, and 402, and optionally denying access to one or more of user interfaces 202, 302, or 402. User interface 402 includes a video / animation evaluation component 404. The video / animation evaluation component 404 includes the patient's joints 406 (e.g., using captured video of the patient or a live recording of the patient) and limb positions at start 410 and end 416. The video / animation evaluation component 404 includes the actual path performed by the patient, having an actual start position 408, an actual end position 414, and an actual movement path 412. The video / animation evaluation component 404 includes a desired path for the patient, having a desired start position 422 (e.g., ...). Figure 3 The starting bubble 308), the expected ending position 426 (e.g., Figure 3 The expected movement path 424 (e.g., movement path 312) includes the end bubble 314 and the intended movement path 424. The intended path may include intended edges 418 and 420 of the intended path region. In one example, if the actual movement path 412 falls between the intended edges 418 and 420, the patient's attempt at clinical movement can be determined to be successful. In another example, if the actual movement path 412 falls outside the intended edges 418 and 420, the patient's attempt at clinical movement can be determined to be unsuccessful. In yet another example, a certain amount of error, such as brief movements outside the intended edges 418 and 420, can be tolerated.

[0055] In the example, the actual start position 408 can be compared with the expected start position 422. If the actual start position 408 aligns with, falls within, or overlaps with the expected start position 422, then the actual start position 408 is considered successful. A similar alignment determination can be made for the actual end position 414 and the expected end position 426.

[0056] Using previously created and selected gesture videos or newly created gesture videos, a corresponding path region (e.g., a path region surrounded by edges 418 and 420) can be determined for the patient. For example, a video of a therapist moving a joint can include a start position, a path region, and an end position. The therapist's size can differ from the patient's, and the start position, path region, or end position can be automatically adjusted to fit the patient's size. The start position, path region, and end position can be converted into an expected start position 422, an expected movement path 424, and an expected end position 426, respectively, for example, using the therapist's size or the patient's size. For example, if the therapist is short and the patient is tall, and the video shows the therapist raising their arm above their head, the patient's arm might be raised to a higher height. The end position can be automatically moved to that higher height based on the detected patient size. The therapist's size can be automatically detected, and the therapist's size can be stored along with the video. In the example, the size of the start position, path region, or end position can be changed. These changes to position or size can be done automatically and can be scaled so that the movement the patient is to perform resembles the movement performed by the therapist.

[0057] In this example, the patient assessment display system 400 can be used to automatically detect or identify the patient's orientation relative to the motion capture device. This orientation can be compared to the orientation of the captured video or animation used to display the exercise. For example, the user interface 402 can be used to instruct the patient to rotate a specified degree, sit, stand, etc., so that the patient is in the correct starting position. In another example, one or more start bubbles (e.g., expected start position 422) can be used to guide the patient to the starting position. For example, the expected start position 422 can be used as the initial placement position for the body part to be moved during exercise. Additional start positions can be used, such as head position, torso position, leg position, arm position, etc., or visual indicators such as directional arrows can be displayed to provide the patient with the starting orientation for beginning the exercise. In another example, the patient's orientation can be identified, and the displayed video or animation can be rotated to correspond to the patient's orientation.

[0058] In the example, the expected start position 422, the expected movement path 424, the expected end position 426, or other bubbles can be automatically modified. For example, bubbles can be modified to create a next level to, for example, increase pressure, increase the challenge for the patient (e.g., by moving the bubble further away from the patient, changing the exercise, etc.). Automatic adjustments can be made based on progress (e.g., progress pre-selected by the therapist). In the example, the therapist can select the start and end points, and the video / animation evaluation component 404 can automatically insert points between the start and end points to adjust the bubbles to change the way the patient moves. For example, progress can be based on a unique start point of the patient's current success or current movement pattern (e.g., activity level), and then the expected movement path 424 can be automatically created to achieve the patient's unique completion goal or intermediate goal. In the example, the difficulty of the expected movement can be changed, for example, by changing the position of the bubble, changing the size of the bubble, changing the angle between the expected start position 408 and the expected end position 426 from joint 406, etc.

[0059] In the example, the video / animation evaluation component 404 can display a captured video or live stream of a patient making a movement. The captured video can show the patient's completed gestures (e.g., gestures automatically determined based on activation, arrival, popping bubbles, etc.), duration, heart rate, etc. The video capture can include the patient or the patient's skeleton and can darken any other background. In another example, the patient can self-report gestures or durations.

[0060] In the example, augmented reality device 501 can be used to display one or more prior attempts by a patient while performing a clinical movement. For example, camera 506 can capture the patient performing a clinical movement at a first time, including a first range of motion (e.g., the range of motion up to a first target object or a first patient-specific waypoint). The first clinical movement attempt can be stored in memory 504 or database 511. Then, for example, when the patient attempts the clinical movement at a second time or later, the first attempt can be shown to the patient in augmented reality display 510. One or more prior attempts made by the patient can be shown with specific effects (e.g., ghosting effects (e.g., fading, dimming, or ethereal)). When the patient attempts the clinical movement at a later time, the prior attempts can be shown in augmented reality display 510 (e.g., the prior attempts are displayed in real time along with the clinical movement attempted by the patient). In another example, prior attempts can be displayed to the therapist on display 510 to show the patient's progress. In this example, attempts further back in time can be shown more faintly. In another example, prior attempts can be color-coded or numbered. In the example, the therapist can use a before-and-after overlay to show the patient the progression of the range of motion in the patient's ongoing clinical movements. A first-person view can be used to show the patient previous attempts, thus displaying the range of motion that can be personalized for the patient with the progression.

[0061] Figure 5 The illustration depicts a system 500 for displaying augmented reality clinical motion according to some embodiments. System 500 includes a gesture analyzer device 503. The gesture analyzer device 503 may include a processor and memory, or may be connected to a device, such as an augmented reality device 501 including a processor 502 and memory 504. In an example, the gesture analyzer device 503 may include a motion capture device (e.g., a camera or motion sensor) 503. The augmented reality device 501 may include a feedback controller 508 or a display 510. The gesture analyzer device 503 may communicate with a database 511. The database 511 may include a video storage device 512 or an animation storage device 514. In an example, the augmented reality device 501 may be holographic glasses manufactured by Microsoft Corporation in Redmond, Washington.

[0062] Processor 502 can be used to receive information about the therapist's clinical movements (e.g., clinical movements captured using motion capture device 503). Processor 502 can analyze the clinical movements to determine the motion path of the clinical movements (e.g., the motion path on video captured by motion capture device 503). Processor 502 can automatically define a path area, for example, by using the motion path. Processor 502 can receive information about the patient's movements along the motion path (e.g., the patient's movements captured using motion capture device 503). Processor 502 can determine whether the movement is within the path area. In an example, processor 502 can send feedback to, for example, a feedback controller 508 or a display 510. The feedback can indicate whether the movement is within the path area. Display 510 can, for example, display feedback via visual indicators (e.g., on a user interface) about whether the movement is within or outside the path area, or where the movement might be outside the path area. Feedback controller 508 can be used to send feedback to display 510, issue an audible alarm, provide haptic feedback, etc. In an example, display 510 can be a screen, an augmented reality display, a virtual reality display, etc.

[0063] Processor 502 can automatically determine the start or end position of a clinical movement, and the start or end position can be included in the path area. For example, determining whether a movement is within the path area can include determining whether the movement begins at a start position and ends at an end position. The therapist can modify the path area, start position, or end position using display 510 (e.g., on a therapist user interface). Processor 502 can be used to create a video or animation using the path area and information about the clinical movement. For example, the video can include the path area superimposed on a captured video or animation of the clinical movement. The video can be played on display 510. While the video is playing on display 510, motion capture device 503 can be used to capture the patient's movement. The captured video can be stored in video storage device 512. The animation can be stored in animation storage device 514. In one example, the video can be retrieved from video storage device 512. The retrieved video can include the automatically added path area, start position, or end position. In another example, the animation can be retrieved from animation storage device 514. The retrieved animation can include the automatically added path area, start position, or end position.

[0064] The processor 502 of the augmented reality device 501 includes an augmented reality modeler 518. The augmented reality device 501 may include a camera 506. The system 500 may include a database 511, which can communicate with the augmented reality device 501.

[0065] Processor 502 can, for example, identify objects in the real-world environment by processing information received using camera 506. For instance, processor 502 can receive information such as images or image series from camera 506 and identify objects within the images or series. Processor 502 can create virtual targets (e.g., lines, bubbles, etc.) in the augmented reality (AR) environment. Virtual targets can have a fixed position, such as a fixed position relative to an object. For example, a virtual target can be positioned in the AR environment such that it remains fixed when the AR device 501 moves. In the example, the virtual target can be fixed, but not fixed relative to the view presented to the user of the AR device. In the example, the virtual target can be a sphere (e.g., a bubble) represented in the real-world environment in a fixed position relative to a patient. For example, a patient can be sitting and instructed to extend their legs (e.g., extend their knees from a flexed position). The sphere can be placed at the final destination of the patient's foot when extending their knee (e.g., approximately in front of the patient at knee height, approximately the length of the patient's leg). The sphere disappears when the patient's foot enters (or approaches) the sphere. The disappearance of the sphere can indicate a successfully performed movement. In another example, a series of spheres can be virtually displayed, for example, along the path of the foot from a bent knee position to an open knee position, and the series of spheres disappears as the foot enters each subsequent sphere (which can be superimposed to show the expected path of movement).

[0066] Display 510 can display an AR environment superimposed on the real environment. Display 510 can use AR device 501 to display virtual targets at fixed locations within the AR environment. In the example, display 501 can remove a virtual target from the display in the AR environment in response to detecting user interaction with the virtual target. For example, when the virtual target is a line or bubble, it can be removed (e.g., fade out, burst, explode, etc.) when the user interacts with it (e.g., kick out a bubble, move a body part through a line, etc.).

[0067] In the example, camera 506 can be used to identify objects in a real-world environment. Camera 506 can send information (e.g., an image) about the object to processor 502, and processor 502 can use the raw information (e.g., the raw image) to identify the object in the real-world environment. Augmented reality device 501 may include sensor 516, such as an infrared sensor. In another example, the sensor may be on the object. In this example, processor 502 can receive information from sensor 516 on the object to identify the object. Camera 506 or sensor 516 can be used to detect movement that can be interpreted by processor 502 as an attempted or anticipated interaction between a user and a virtual target.

[0068] Processor 502 can use augmented reality modeler 518 to create augmented reality environments. For example, augmented reality modeler 518 can receive the dimensions of a room, for example, from camera 506 or sensor 516 and create an augmented reality environment to fit the physical structure of the room. In another example, physical objects can exist in the room, and augmented reality modeler 518 can use physical objects to render virtual objects in the augmented reality environment. For example, augmented reality modeler 518 can use or detect a table present in the room and render a virtual object as if it were placed on the table. In the example, a user can interact with virtual targets using multiple physical items. The user can use virtual objects to perform physical therapy movements.

[0069] Figure 6 The illustration shows a flowchart of a technique 600 for remote rehabilitation according to some embodiments. Technique 600 includes operation 602 for analyzing clinical movement to determine the movement path of the clinical movement. Clinical movement can be received from a movement capturing device that captures clinical movement performed by a therapist. Analyzing the clinical movement may include determining a start position or an end position.

[0070] Technology 600 includes operation 604 for automatically defining a path region using a motion path. Automatically defining the path region may include defining a start or end region, for example, by using a start position or end position (e.g., the start region may be an area surrounding a start position, or the end region may be an area surrounding an end position). The start or end position may be determined based on the therapist's limb position in the video. In an example, the path region may include a predetermined area surrounding the motion path. Technology 600 includes operation 606 for receiving information about the patient's movement along the motion path. This information may include whether the movement remains within the motion path, moves outside the motion path, follows the motion path (e.g., within an error range around the motion path), etc. In an example, receiving information includes capturing and analyzing video. In an example, receiving information may include analyzing a live video of the patient, overlaying the live video onto an animation including the motion path and the path region to provide the patient with immediate visual feedback about the exercise / gesture being performed.

[0071] Technique 600 includes operation 608 for determining whether movement is within a path region. Determining whether movement is within a path region may include determining whether movement begins in a start region and whether movement ends in an end region. In the example, determining whether movement is within a path region includes determining whether movement is outside the path region. In the example, determining whether movement is within a path region includes determining whether movement is within the path region.

[0072] In the example, technique 600 may include monitoring patient movement. Patient movement can be compared to a path region to determine whether movement is within the path region, whether movement begins in a start region, and whether movement ends in an end region. Technique 600 may include sending feedback that includes a real-time depiction of the patient's movement. The real-time depiction may include displaying the movement on a display, for example, using visual indicators of the start region, end region, or path region. For example, bubbles may be used to represent the start region. The path region may be represented by a series of bubbles or visual indicators along its edges. In the example, the real-time depiction of patient movement includes an animation representing clinical movement including the path region. In another example, the real-time depiction includes a video of clinical movement including the path region. The animation or video representing the clinical movement may include color-coded visual effects to indicate compliance or non-compliance of patient movement with clinical movement. For example, the edge of the path region may turn red in response to detecting that the patient has moved outside the path region. The edge may be green when the patient is moving within the path region. The edge may turn yellow when the movement approaches within a specified distance of the edge of the path region. In the example, the start and end regions can be represented by bubbles that persist until the patient moves into either the start or end region. The bubbles burst when the patient moves into either the start or end region. In the example, the path region can be represented by a series of bubbles (e.g., starting after the start region bubble and ending before the end region bubble along the movement path). Patient movement through a series of bubbles causes this series of bubbles to burst sequentially upon completion of the movement.

[0073] Technology 600 includes operation 610 for sending feedback indicating whether the movement is within the pathway area. Sending feedback may include providing an alert to the patient or therapist when it is determined that the movement is outside the pathway area. Feedback may include an instruction to repeat the movement. Feedback may include an alert if the patient fails to complete the movement. Sending feedback may include providing an indication to the patient or therapist that the movement successfully simulates a clinical movement when the movement falls within the pathway area. Feedback may include an indication that the patient has successfully completed the movement. Feedback may include visual feedback, auditory feedback, tactile feedback, non-contact feedback, etc. Feedback may be presented on a user interface on a display. The user interface may include options for modifying the pathway area. The therapist can select modifications to the pathway area, start area, or end area.

[0074] In the example, technique 600 includes creating a video or animation using a path region and information about clinical movement. The video or animation may include a path region overlaid on a captured video or animation of clinical movement. The video or animation can be played on a monitor. In the example, the video or animation can be played on a monitor while a motion capture device is used to capture the patient's movement. The movement can be displayed on the monitor.

[0075] Figure 7 The illustration shows a flowchart of a technique 700, according to some embodiments, for displaying directions related to treatment movement for a patient in an augmented reality environment. Technique 700 includes operation 702 for displaying an augmented reality environment, such as an AR environment overlaid on a real-world environment. Operation 702 can be performed by an AR device. The AR environment can be created using an augmented reality modeler. Technique 700 includes operation 704 for recognizing objects in the real-world environment. Objects can be recognized using the camera of the AR device. In the example, sensors on the objects can be used to recognize them.

[0076] Technique 700 includes operation 706 for creating a virtual target in an augmented reality environment. The virtual target may have a fixed position relative to objects in the real environment. The virtual target may be fixed, but not fixed relative to the view of a user presented to the augmented reality device. For example, the virtual target may remain in a fixed position in the real environment as the AR device moves. For example, a user can use the virtual target to complete a physical therapy move. Displaying the virtual target in a fixed position may include displaying a bubble, for example, at the end of a physical therapy move to be completed by the user. In response to user interaction with the bubble (e.g., completing a therapy move), the bubble can be made to pop by displaying an animation. In the example, displaying the virtual target in a fixed position includes using a virtual avatar to display a virtual physical therapy move corresponding to a physical therapy move to be completed by the user. In the example, the virtual avatar is a virtual representation of the physical therapist.

[0077] Technique 700 includes operation 708 for displaying a virtual target in an augmented reality environment. The virtual target can be displayed at a fixed location for use by the augmented reality device during therapeutic movement within the augmented reality environment. The fixed location can be positioned at a predetermined start, end, or intermediate location of the therapeutic movement. Technique 700 includes operation 710 for removing the virtual target from the display in the augmented reality environment. Operation 710 can include removing the virtual target from the display in response to detecting user interaction with the virtual target, wherein the user instruction can indicate completion of the therapeutic movement. Detecting user interaction with the virtual target can include using a camera or sensor (e.g., a camera or sensor on an AR device).

[0078] Technology 700 may include displaying multiple physical items in an AR environment for a user to interact with virtual targets. In the example, multiple physical items in the AR environment may be identified, for example, by using visual indicators above or near the physical items or by displaying virtual representations of the physical items. Technology 700 may include displaying instructions in the augmented reality environment that include clinical information about the user's interaction.

[0079] Figure 8 The illustration shows a flowchart illustrating techniques for displaying enhanced realistic clinical movement according to some embodiments. Technique 800 includes operation 802 to analyze information about the clinical movement (e.g., clinical movement performed by a therapist) to determine a movement path representing at least a portion of the clinical movement. In examples, clinical movement may include movement for exercise, routine, stretching, occupational therapy movement, physical therapy movement, etc. A motion capture device can be used to capture the therapist's clinical movement. In an example, the motion capture device includes an infrared sensor, and the movement path is determined based on a series of snapshots, which include infrared sensor data from the infrared sensor.

[0080] Technique 800 includes operation 804 for automatically defining a path region and virtual targets in an augmented reality environment superimposed on a real environment using motion paths. The virtual targets may have a fixed position relative to objects in the real environment. For example, a virtual target may have a fixed position relative to aspects of a room (e.g., floor, wall, ceiling, etc.), fixed objects (e.g., table, chair, etc.), or moving objects (e.g., person, pet, skateboard, etc.). In the example, the path region is defined as an area encompassing a specified distance around the motion path. Operation 804 may include receiving modifications to the path region from a therapist on a user interface of a display.

[0081] Technique 800 includes operation 806 to display a path region and a virtual target at a fixed location in an augmented reality environment. Augmented reality devices can be used to display the path region and the virtual target. In the example, the fixed location can be positioned at the intended end point of the path region.

[0082] In the examples, displaying virtual targets includes displaying a movement task object representing a real-world object used in occupational therapy or physical therapy. For example, a real-world object could include tableware, and a movement task associated with the tableware could include displaying a virtual representation of the tableware to be virtually "moved" in the augmented reality environment to mimic the real-world task of placing the tableware. In another example, a real-world object could be a pet (e.g., a cat or dog), and a real-world task that could be simulated in the augmented reality environment could include stepping on the pet. In yet another example, a box could be added to represent stairs, and a task could include stepping on the box to simulate climbing stairs. Other examples could include other real-world objects used to avoid something, for everyday tasks, etc., such as stepping on the accelerator / brake pedal, picking up a cup of coffee, taking a photo, typing, brushing teeth, opening a door to get into a car, etc.

[0083] Augmented reality devices can be used to display multiple physical items to be used for interaction with virtual targets within an augmented reality environment. For example, a virtual representation of weight, corresponding to a weight to be lifted during clinical movement, can be displayed. In another example, resistance bands can be shown, for example, in terms of color, for use during clinical movement.

[0084] In one example, technology 800 may include displaying instructions in an augmented reality environment, which may include clinical information about the user interaction. For example, instructions may be displayed to the user to indicate whether a clinical movement has been successfully performed or not. The information displayed to the user may include encouragement or suggestions (e.g., “lift your leg slightly”). In another example, instructions may be displayed to the therapist to update the therapist’s understanding of the patient’s technique, successes, failures, progress, and level of effort.

[0085] While a virtual target is displayed, the user can interact with it within the augmented reality environment. The user's actions in the real environment can trigger effects within the augmented reality environment. For example, in response to detecting user interaction with the virtual target (e.g., completion of a clinical move), the virtual target can be removed from the display in the augmented reality environment. Removing the virtual target can indicate the completion of a clinical move. In the example, a camera can be used to detect user interaction with the virtual target that causes the virtual target to be removed from the augmented reality environment. For example, the camera and the augmented reality display device can communicate (or communicate with a third device) to determine whether the displayed virtual target is overlaid on the real environment and whether user interaction in the real environment occupies overlay space in the real environment.

[0086] In the example, technique 800 includes sending feedback that includes a real-time depiction of movement along a path region toward a virtual target, corresponding to patient movement attempting to reproduce clinical movement using an augmented reality device. In the example, the real-time depiction of the patient movement includes animation representing clinical movement encompassing the path region. For example, the animation representing clinical movement can be color-coded to indicate whether the patient movement conforms to or does not conform to clinical movement.

[0087] In the example, technique 800 includes receiving information about a user's movement along a motion path towards a virtual target. This information can be used to determine whether the movement is within the path area, for example, based on analysis of information about the user's movement. In response to determining that the movement is within the path area, technique 800 may include using an augmented reality device to display feedback indicating that the movement is within the path area.

[0088] In the example, a surgeon or patient can use an AR device post-surgery. For instance, the surgeon or patient can use the AR device to observe various aspects of the implant or surgery, such as using implant sensors, medical imaging (e.g., X-rays, MRI, CT scans, etc.), or images or videos taken during the surgery. The AR device can then display the stored video of various aspects of the implant or surgery as a 3D virtual object superimposed on the real environment. The surgeon or patient (or other user) can view the virtual object in 3D, making it appear as if the virtual object is in the same room as the surgeon or patient, and can be viewed from multiple different angles as the surgeon or patient moves the AR device around the virtual object. For example, the virtual object can be stationary or can move along with one or more points fixed to one aspect of the real environment (e.g., the virtual object might appear to be placed on a table). In the example, the surgeon and patient can view the virtual object together, and the surgeon can control the virtual object to show it moving or to point out certain aspects of it.

[0089] Figure 9The diagram illustrates an example block diagram of machine 900, which, according to some embodiments, can perform any or more of the techniques (e.g., methods) discussed herein. In alternative embodiments, machine 900 can operate as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, machine 900 can operate as a server machine, a client machine, or both in a server-client network environment. Machine 900 can be a personal computer (PC), tablet PC, personal digital assistant (PDA), mobile phone, network device, network router, switch, or bridge, or any machine capable of executing instructions (sequentially or otherwise) specifying the actions to be taken by that machine. Furthermore, although only a single machine is illustrated, the term "machine" should also be considered as including any collection of machines that individually or jointly execute a set (or more sets) of instructions to perform any or more of the methods discussed herein, such as cloud computing, Software as a Service (SaaS), or other computer cluster configurations.

[0090] As described herein, examples may include logic units or multiple components, modules, or similar mechanisms, or entities that can operate on the aforementioned items. Such a mechanism is a tangible entity (e.g., hardware) capable of performing a specified operation at operational time. In the examples, the hardware may be specifically configured to perform a particular operation (e.g., hardwiring). In the examples, the hardware may include configurable execution units (e.g., transistors, circuits, etc.) and a computer-readable medium containing instructions that configure the execution unit to perform a specific operation at operational time. Configuration may be performed under the guidance of the execution unit or loading mechanism. Thus, when the device is operational, the execution unit is communicatively coupled to the computer-readable medium. For example, under operation, the execution unit may be configured by a first set of instructions to implement a first set of features at a point in time and reconfigured by a second set of instructions to implement a second set of features.

[0091] Machine (e.g., computer system) 900 may include a hardware processor 902 (e.g., a central processing unit (CPU), graphics processing unit (GPU), hardware processor core, or any combination thereof), main memory 904, and static memory 906, some or all of which may communicate with each other via an interconnect link (e.g., bus) 908. Machine 900 may also include a display unit 910, an alphanumeric input device 912 (e.g., a keyboard), and a user interface (UI) navigation device 914 (e.g., a mouse). In the example, display unit 910, alphanumeric input device 912, and UI navigation device 914 may be a touchscreen display. Display unit 910 may include goggles, glasses, or other AR or VR display components. For example, the display unit may be worn on a user's head and may provide a heads-up display to the user. Alphanumeric input device 912 may include a virtual keyboard (e.g., a keyboard configured for VR or AR).

[0092] Machine 900 may additionally include a storage device (e.g., a drive unit) 916, a signal generation device 918 (e.g., a speaker), a network interface device 920, and one or more sensors 921 (e.g., a Global Positioning System (GPS) sensor, a compass, an accelerometer, or other sensors). Machine 900 may include an output controller 928 for communicating with or controlling one or more peripheral devices (e.g., a serial connection (e.g., Universal Serial Bus (USB), a parallel connection, or other wired or wireless connections (e.g., an infrared (IR) connection, a near field communication (NFC) connection, etc.)).

[0093] Storage device 916 may include a non-transient machine-readable medium 922 on which one or more sets of data structures or instructions 924 (e.g., software) are stored, implemented or used by any one or more of the techniques or functions described herein. Instructions 924 may also reside wholly or at least partially in main memory 904, in static memory 906, or in hardware processor 902 during execution by machine 900. In this example, one or any combination of hardware processor 902, main memory 904, static memory 906, or storage device 916 may constitute a machine-readable medium.

[0094] Although machine-readable medium 922 is illustrated as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database or associated cache and server) configured to store one or more instructions 924.

[0095] The term "machine-readable medium" can include any medium capable of storing, encoding, or carrying instructions for execution by machine 900 and causing machine 900 to perform any one or more of the technologies of this disclosure, or capable of storing, encoding, or carrying data structures used by or associated with those instructions. Examples of non-limiting machine-readable media can include solid-state memory, as well as optical and magnetic media. Specific examples of machine-readable media can include: non-volatile memory (e.g., semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices), magnetic disks (e.g., internal hard disks and removable hard disks), magneto-optical disks, and CD-ROM and DVD-ROM discs.

[0096] It is also possible to use any of a variety of transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.) to transmit or receive instructions 924 on the communication network 926 via the network interface device 920 using a transport medium. Example communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), conventional telephone (POTS) networks, and wireless data networks (e.g., referred to as…). The IEEE 802.11 standard series, published by the Bluetooth Special Interest Group, peer-to-peer (P2P) networks, etc., is known as... (A set of personal area network standards). In the example, network interface device 920 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connecting to communication network 926. In the example, network interface device 920 may include multiple antennas for wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technologies. The term "transmission medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions executed by machine 900, and includes digital or analog communication signals or other intangible media to facilitate communication of such software.

[0097] Various explanations and examples

[0098] Each of these non-restrictive examples can exist independently or can be combined with one or more examples from other examples in various permutations and combinations.

[0099] Example 1 is a method for displaying augmented reality clinical movement, the method comprising: analyzing information about clinical movement of a therapist captured using a motion capture device to determine a motion path representing at least a portion of the clinical movement; automatically defining a path region and a virtual target in an augmented reality environment superimposed on a real environment using the motion path, the virtual target having a fixed position relative to an object in the real environment; and displaying the path region and the virtual target at the fixed position in the augmented reality environment using an augmented reality device, the fixed position being positioned at the intended end position of the path region.

[0100] In Example 2, the subject of Example 1 optionally includes: receiving information about a user's movement along the motion path to the virtual target.

[0101] In Example 3, the subject of Example 2 optionally includes: determining whether the movement is within the path area based on the analysis of the information about the user's movement, and in response to determining that the movement is within the path area, using the augmented reality device to display feedback indicating that the movement is within the path area.

[0102] In Example 4, the subject of any one or more of Examples 1-3 optionally includes: removing the virtual target from the display in the augmented reality environment in response to detecting the completion of a user movement that reproduces the clinical movement by instructing the user to interact with the virtual target.

[0103] In Example 5, the subject matter of any one or more of Examples 1-4 optionally includes: wherein the motion capture device includes an infrared sensor, and the motion path is determined based on a series of snapshots, the snapshots including infrared sensor data from the infrared sensor.

[0104] In Example 6, the subject of Example 5 optionally includes: wherein the path region is defined as a region comprising a specified distance around the path of motion.

[0105] In Example 7, any one or more of the topics in Examples 1-6 optionally include: receiving modifications to the path area from the therapist at the user interface of the display.

[0106] In Example 8, any one or more of the topics in Examples 1-7 optionally include: sending feedback, the feedback including a real-time depiction of movement along the path area toward the virtual target, the movement corresponding to patient movement attempting to reproduce the clinical movement using the augmented reality device.

[0107] In Example 9, the subject of Example 8 optionally includes: wherein the real-time depiction of the patient's movement includes an animation representing the clinical movement including the area of ​​the path.

[0108] In Example 10, the subject of Example 9 optionally includes: wherein the animation representing the clinical movement is color-coded to indicate whether the patient movement is compliant or non-compliant with the clinical movement.

[0109] In Example 11, the subject of any one or more of Examples 1-10 optionally includes: displaying multiple physical items to be used in the user interaction with the virtual target in the augmented reality environment.

[0110] In Example 12, the subject of any one or more of Examples 1-11 optionally includes: detecting user interaction with the virtual target in data received from a camera, the user interaction causing the virtual target to be removed from the augmented reality environment.

[0111] In Example 13, the subject of any one or more of Examples 1-12 optionally includes: wherein displaying the virtual target includes displaying a mobile task object representing a real-world object used in occupational therapy.

[0112] In Example 14, any one or more of the topics in Examples 1-13 optionally include: displaying instructions in the augmented reality environment, the instructions including clinical information about user interaction.

[0113] Example 15 is an augmented reality device for displaying directions related to clinical movement for a patient in an augmented reality environment, comprising: a processor configured to: analyze information about clinical movement of a therapist captured using a motion capture device to determine a motion path representing at least a portion of the clinical movement; and automatically define a path region and a virtual target in an augmented reality environment superimposed on a real environment using the motion path, the virtual target having a fixed position relative to an object in the real environment; and a display configured to: display the path region and the virtual target at the fixed position in the augmented reality environment using the augmented reality device, the fixed position being positioned at a intended end position of the path region.

[0114] In Example 16, the subject matter of Example 15 optionally includes: wherein the processor is further configured to receive information about the user’s movement along the motion path to the virtual target.

[0115] In Example 17, the subject matter of Example 16 optionally includes: wherein the processor is further configured to determine whether the movement is within the path area based on analysis of the information about the user's movement, and in response to determining that the movement is within the path area, the display is further configured to use the augmented reality device to display feedback indicating that the movement is within the path area.

[0116] In Example 18, the subject matter of any one or more of Examples 15-17 optionally includes: wherein the display is further configured to remove the virtual target from the display in the augmented reality environment in response to detecting completion of user movement of the clinical movement by a user interaction instruction with the virtual target.

[0117] In Example 19, the subject matter of any one or more of Examples 15-18 optionally includes: wherein the motion capture device further includes an infrared sensor, and the motion path is determined based on a series of snapshots, the snapshots including infrared sensor data from the infrared sensor.

[0118] In Example 20, the subject of Example 19 optionally includes: wherein the path region is defined as a region comprising a specified distance around the path of motion.

[0119] In Example 21, any one or more of the topics in Examples 15-20 optionally include: wherein the processor is further configured to receive modifications to the path area from the therapist on the user interface of the display.

[0120] In Example 22, the subject matter of any one or more of Examples 15-21 optionally includes: wherein the processor is further configured to generate feedback comprising a real-time depiction of movement along the path region toward the virtual target, the movement corresponding to patient movement attempting to reproduce the clinical movement using the augmented reality device.

[0121] In Example 23, the subject matter of Example 22 optionally includes: wherein the real-time depiction of the patient movement includes animation representing the clinical movement including the path region.

[0122] In Example 24, the subject matter of Example 23 optionally includes: wherein the animation representing the clinical movement is color-coded to indicate whether the patient movement is compliant or non-compliant with the clinical movement.

[0123] In Example 25, the subject of any one or more of Examples 15-24 optionally includes: wherein the display is also used to display multiple physical items to be used in the user interaction with the virtual target in the augmented reality environment.

[0124] In Example 26, the subject of any one or more of Examples 15-25 optionally includes: wherein the processor is further configured to detect user interaction with the virtual target in data received from the camera, the user interaction causing the virtual target to be removed from the augmented reality environment.

[0125] In Example 27, the subject matter of any one or more of Examples 15-26 optionally includes: wherein, in order to display the virtual target, the display is also used to display a mobile task object representing a real-world object used in occupational therapy.

[0126] In Example 28, any one or more of the topics in Examples 15-27 optionally include: wherein the display is also used to display instructions in the augmented reality environment, the instructions including clinical information about user interaction.

[0127] Example 29 is a system comprising: a motion capture device including a camera for capturing information about a therapist’s clinical movements; and a processor configured to: analyze the information to determine a motion path representing at least a portion of the clinical movements; and automatically define a path region and a virtual target in an augmented reality environment superimposed on a real environment using the motion path, the virtual target having a fixed position relative to an object in the real environment; and an augmented reality display device configured to display the path region and the virtual target at the fixed position in the augmented reality environment, the fixed position being positioned at a intended end position of the path region.

[0128] In Example 30, the subject of Example 29 optionally includes a camera for capturing information about the user’s movement along the motion path to the virtual target.

[0129] In Example 31, the subject matter of Example 30 optionally includes: wherein the processor is further configured to determine whether the movement is within the path area based on analysis of the information about the user's movement, and in response to determining that the movement is within the path area, the enhanced reality display device is further configured to display feedback indicating that the movement is within the path area.

[0130] In Example 32, the subject matter of any one or more of Examples 29-31 optionally includes: wherein the augmented reality display device is further configured to: remove the virtual target from the display in the augmented reality environment in response to detecting completion of user movement that reproduces the clinical movement by instructing a user to interact with the virtual target.

[0131] In Example 33, the subject matter of any one or more of Examples 29-32 optionally includes: wherein the motion capture device further includes an infrared sensor, and the motion path is determined based on a series of snapshots, the snapshots including infrared sensor data from the infrared sensor.

[0132] In Example 34, the subject of Example 33 optionally includes: wherein the path region is defined as a region comprising a specified distance around the path of motion.

[0133] In Example 35, any one or more of the topics in Examples 29-34 optionally include: wherein the processor is further configured to receive modifications of the path region from the therapist on the user interface of the enhanced reality display device.

[0134] In Example 36, the subject matter of any one or more of Examples 29-35 optionally includes: wherein the processor is further configured to generate feedback comprising a real-time depiction of movement along the path region toward the virtual target, the movement corresponding to patient movement attempting to reproduce the clinical movement using the augmented reality device.

[0135] In Example 37, the subject matter of Example 36 optionally includes: wherein the real-time depiction of the patient's movement includes animation representing the clinical movement encompassing the path region.

[0136] In Example 38, the subject of Example 37 optionally includes: wherein the animation representing the clinical movement is color-coded to indicate whether the patient movement is compliant or non-compliant with the clinical movement.

[0137] In Example 39, the subject of any one or more of Examples 29-38 optionally includes: wherein the augmented reality display device is also used to display multiple physical items to be used in the user interaction with the virtual target in the augmented reality environment.

[0138] In Example 40, any one or more of the subjects in Examples 29-39 optionally include: wherein the processor is further configured to detect user interaction with the virtual target in data received from the camera, the user interaction causing the virtual target to be removed from the augmented reality environment.

[0139] In Example 41, the subject matter of any one or more of Examples 29-40 optionally includes: wherein, in order to display the virtual target, the enhanced reality display device is also used to display a mobile task object representing a real-world object used in occupational therapy.

[0140] In Example 42, the subject matter of any one or more of Examples 29-41 optionally includes: wherein the augmented reality display device is also used to display instructions in the augmented reality environment, the instructions including clinical information about user interaction.

[0141] Example 43 is at least one machine-readable medium comprising instructions for displaying augmented reality clinical movement, the instructions, when executed by a machine, causing the machine to: analyze information about clinical movement of a therapist captured using a motion capture device to determine a motion path representing at least a portion of the clinical movement; automatically define a path region and a virtual target in an augmented reality environment superimposed on a real environment using the motion path, the virtual target having a fixed position relative to an object in the real environment; and display the path region and the virtual target at the fixed position in the augmented reality environment using an augmented reality device, the fixed position being positioned at the intended end position of the path region.

[0142] In Example 44, the subject of Example 43 optionally includes instructions for receiving information about a user’s movement along the motion path to the virtual target.

[0143] In Example 45, the subject matter of Example 44 optionally includes instructions for: determining whether the movement is within the path area based on analysis of the information about the user's movement, and in response to determining that the movement is within the path area, using the augmented reality device to display feedback indicating that the movement is within the path area.

[0144] In Example 46, the subject matter of any one or more of Examples 43-45 optionally includes instructions for removing the virtual target from the display in the augmented reality environment in response to detecting completion of user movement that reproduces the clinical movement by instructing user interaction with the virtual target.

[0145] In Example 47, the subject matter of any one or more of Examples 43-46 optionally includes: wherein the motion capture device includes an infrared sensor, and the motion path is determined based on a series of snapshots, the snapshots including infrared sensor data from the infrared sensor.

[0146] In Example 48, the subject of Example 47 optionally includes: wherein the path region is defined as a region comprising a specified distance around the path of motion.

[0147] In Example 49, any one or more of the topics in Examples 43-48 optionally include instructions for receiving modifications to the path area from the therapist on a user interface of the display.

[0148] In Example 50, any one or more of the topics in Examples 43-49 optionally include instructions for generating feedback, which includes a real-time depiction of movement along the path region toward the virtual target, the movement corresponding to patient movement attempting to reproduce the clinical movement using the augmented reality device.

[0149] In Example 51, the subject of Example 50 optionally includes: wherein the real-time depiction of the patient movement includes animation representing the clinical movement including the path region.

[0150] In Example 52, the subject matter of Example 51 optionally includes: wherein the animation representing the clinical movement is color-coded to indicate whether the patient movement is compliant or non-compliant with the clinical movement.

[0151] In Example 53, any one or more of the themes in Examples 43-52 optionally include instructions for displaying multiple physical items to be used in the user interaction with the virtual target within the augmented reality environment.

[0152] In Example 54, any one or more of the subjects in Examples 43-53 optionally include instructions for detecting user interaction with the virtual target in data received from the camera, the user interaction causing the virtual target to be removed from the augmented reality environment.

[0153] In Example 55, the subject matter of any one or more of Examples 43-54 optionally includes: wherein the instructions for displaying the virtual target include instructions for displaying a mobile task object representing a real-world object used in occupational therapy.

[0154] In Example 56, any one or more of the subjects in Examples 43-55 optionally include instructions for displaying instructions in the augmented reality environment, the instructions including clinical information about user interaction.

[0155] Example 57 is a method for remote rehabilitation, the method comprising: receiving information about clinical movement of a therapist captured using a motion capture device; analyzing the clinical movement to determine a motion path representing at least a portion of the clinical movement; using the motion path to automatically define a path region; receiving information about movement of a patient along the motion path; determining, based on the analysis of the information about the patient's movement, whether the movement is within the path region; and sending feedback indicating whether the movement is within the path region.

[0156] In Example 58, the subject matter of Example 57 optionally includes: wherein analyzing the clinical movement includes determining the start and end positions of the clinical movement.

[0157] In Example 59, the subject of Example 58 optionally includes: wherein automatically defining the path region includes automatically defining the start region and the end region using the start position and the end position.

[0158] In Example 60, the subject matter of Example 59 optionally includes: wherein determining whether the movement is within the path region includes determining whether the movement begins in the start region and ends in the end region.

[0159] In Example 61, the subject matter of any one or more of Examples 57-60 optionally includes: wherein the motion capture device includes an infrared sensor, and the motion path is determined based on a series of snapshots, the snapshots including infrared sensor data from the infrared sensor.

[0160] In Example 62, the subject of Example 61 optionally includes: wherein the path region is defined as a region comprising a specified distance around the path of motion.

[0161] In Example 63, the subject of any one or more of Examples 57-62 optionally includes: wherein the path region includes a predetermined area surrounding the motion path.

[0162] In Example 64, the subject matter of any one or more of Examples 57-63 optionally includes: wherein determining whether the movement is within the path area includes determining that the movement is outside the path area.

[0163] In Example 65, the subject of Example 64 optionally includes: wherein sending the feedback includes providing the patient with an alert that the movement is outside the path area and that the movement will be repeated.

[0164] In Example 66, any one or more of the topics in Examples 64-65 optionally include: wherein sending the feedback includes providing the therapist with an alert that the patient has failed to complete the movement.

[0165] In Example 67, the subject matter of any one or more of Examples 57-66 optionally includes: wherein determining whether the movement is within the path area includes determining that the movement is within the path area.

[0166] In Example 68, the subject matter of Example 67 optionally includes: wherein sending the feedback includes providing the patient with an indication that the movement successfully mimics the clinical movement.

[0167] In Example 69, any one or more of the topics in Examples 67-68 optionally include: wherein sending the feedback includes providing the therapist with an indication that the patient has successfully completed the movement.

[0168] Example 70 is at least one machine-readable medium including instructions for receiving information, which, when executed by a machine, cause the machine to perform the following actions: receive information about clinical movement of a therapist captured using a motion capture device; analyze the clinical movement to determine a motion path of the clinical movement; use the motion path to automatically define a path region; receive information about movement of a patient along the motion path; determine whether the movement is within the path region; and send feedback indicating whether the movement is within the path region.

[0169] In Example 71, the subject of Example 70 optionally includes: wherein the feedback is visual, auditory, or tactile.

[0170] In Example 72, any one or more of the topics in Examples 70-71 optionally include instructions for receiving modifications to the path area from the therapist on a user interface of the display.

[0171] In Example 73, the subject matter of any one or more of Examples 70-72 optionally includes instructions for creating a video using the path region and the information about the clinical movement, the video including the path region superimposed on a captured video of the clinical movement.

[0172] In Example 74, the subject matter of Example 73 optionally includes instructions for playing the video on a display while capturing the patient's movement using the motion capture device.

[0173] Example 75 is a system comprising: a motion capture device; a memory; and a processor connected to the memory, the processor being configured to: receive information about a clinical movement performed by a therapist and captured using the motion capture device; analyze the clinical movement to determine a motion path of the clinical movement; use the motion path to automatically define a path region; receive information about a patient's movement along the motion path, the information being generated by the motion capture device; determine whether the movement is within the path region; and send feedback indicating whether the movement is within the path region.

[0174] In Example 76, the subject of Example 75 optionally includes: wherein, in order to automatically define the path region, the processor is further configured to automatically define the start region and the end region using the start position and the end position.

[0175] In Example 77, the subject matter of Example 76 optionally includes: wherein, in order to determine whether the movement is within the path region, the processor is further configured to determine whether the movement begins in the start region and ends in the end region.

[0176] In Example 78, any one or more of the topics in Examples 75-77 optionally include: wherein the processor is used to generate feedback including a real-time depiction of the patient's movement.

[0177] In Example 79, the subject of Example 78 optionally includes: wherein the real-time depiction of the patient's movement includes an animation representing the clinical movement including the area of ​​the path.

[0178] In Example 80, any one or more of the themes in Examples 78-79 optionally include: wherein the animation representing the clinical movement is color-coded to indicate the patient's compliance or non-compliance with the clinical movement.

[0179] Example 81 is a method for displaying directions related to therapeutic movements for a patient in an augmented reality environment, the method comprising: displaying the augmented reality environment superimposed on a real environment using an augmented reality device; identifying objects in the real environment; creating a virtual target in the augmented reality environment having a fixed position relative to the objects; displaying the virtual target at the fixed position using the augmented reality device for use in the therapeutic movements in the augmented reality environment, the fixed position being positioned at an expected end position of the therapeutic movements; and removing the virtual target from the display in the augmented reality environment in response to detecting a user interaction indication with the virtual target indicating completion of a user movement reproducing the therapeutic movements.

[0180] In Example 82, the subject of Example 81 optionally includes: displaying multiple physical items to be used in the user interaction with the virtual target within the augmented reality environment.

[0181] In Example 83, the subject of any one or more of Examples 81-82 optionally includes: wherein displaying the virtual target at the fixed position includes displaying a bubble at the expected end position.

[0182] In Example 84, the subject of Example 83 optionally includes: wherein removing the virtual target includes displaying an animation of the bubble bursting.

[0183] In Example 85, the subject of any one or more of Examples 81-84 optionally includes: wherein the virtual target is fixed relative to the object, but not relative to the view of the user presented to the augmented reality device.

[0184] In Example 86, the subject of any one or more of Examples 81-85 optionally includes: wherein identifying the object in the real environment includes using a camera of the augmented reality device.

[0185] In Example 87, the subject matter of any one or more of Examples 81-86 optionally includes: wherein identifying the object in the real environment includes using sensors on the object to identify the object.

[0186] In Example 88, any one or more of the topics in Examples 81-87 optionally include using the augmented reality modeler to create the augmented reality environment.

[0187] In Example 89, the subject of any one or more of Examples 81-88 optionally includes: wherein detecting the user interaction with the virtual target includes using a camera.

[0188] In Example 90, the subject of any one or more of Examples 81-89 optionally includes: wherein displaying the virtual target in the fixed location includes using a virtual avatar to display a virtual physical therapy movement corresponding to the physical therapy movement to be performed by the user.

[0189] In Example 91, the subject of Example 90 optionally includes: wherein the virtual avatar is a virtual representation of a physical therapist.

[0190] In Example 92, any one or more of the topics in Examples 81-91 optionally include: displaying instructions in the augmented reality environment, the instructions including clinical information about the user interaction.

[0191] Example 93 is an augmented reality device for displaying directions related to therapeutic movements for a patient in an augmented reality environment, the augmented reality device comprising: a processor configured to: identify objects in a real-world environment; and create a virtual target in the augmented reality environment having a fixed position relative to the objects; and a display configured to: display the augmented reality environment superimposed on the real-world environment; display the virtual target at the fixed position using the augmented reality device for use in the therapeutic movements within the augmented reality environment, the fixed position being positioned at an expected end position of the therapeutic movements; and remove the virtual target from the display in the augmented reality environment in response to detecting a user interaction indication with the virtual target indicating completion of a user movement reproducing the therapeutic movements.

[0192] In Example 94, the subject of Example 93 optionally includes: wherein the virtual target is fixed, but not fixed relative to the view of the user presented to the augmented reality device.

[0193] In Example 95, the subject of any one or more of Examples 93-94 optionally includes: a camera for identifying the object in the real environment.

[0194] In Example 96, the subject matter of any one or more of Examples 93-95 optionally includes: wherein, in order to identify the object in the real environment, the processor is further configured to receive information from sensors on the object to identify the object.

[0195] In Example 97, the subject of any one or more of Examples 93-96 optionally includes: wherein the processor is further configured to create the augmented reality environment using an augmented reality modeler.

[0196] In Example 98, any one or more of the subjects in Examples 93-97 optionally include: a camera for detecting user interaction with the virtual target.

[0197] In Example 99, the subject of any one or more of Examples 93-98 optionally includes: wherein the display is also used to display multiple physical items to be used by a user to interact with the virtual target in the augmented reality environment.

[0198] In Example 100, any one or more of the topics in Examples 93-99 optionally include: wherein a user uses the virtual target to perform physical therapy movement.

[0199] In Example 101, the subject matter of any one or more of Examples 1-99 may optionally include a technology or system that includes a display component to allow the therapist to select an object and place the object in an AR environment.

[0200] In Example 102, the subject of Example 101 may optionally include: generating feedback when user interaction with the placed object is detected.

[0201] In Example 103, the subject of any one or more of Examples 101-102 optionally includes: wherein detecting user interaction with a placed object includes receiving data from a camera, the user interaction causing the placed object to be removed from the AR environment.

[0202] The method examples described herein may be implemented, at least in part, by a machine or computer. Some examples may include a computer-readable or machine-readable medium encoded with instructions that operate to configure an electronic device to perform the methods described in the examples above. Implementations of this method may include code, such as microcode, assembly language code, higher-level language code, etc. This code may include computer-readable instructions for performing various methods. The code may form part of a computer program product. Additionally, in the examples, the code may be tangibly stored, for example, during execution or at other times, on one or more volatile, non-transient computer-readable media or non-volatile tangible computer-readable media. Examples of such tangible computer-readable media may include, but are not limited to, hard disks, removable disks, removable optical discs (e.g., optical discs and digital video discs), magnetic tapes, memory cards or sticks, random access memory (RAM), read-only memory (ROM), etc.

Claims

1. A system for evaluating clinical movement, comprising: a patient motion sensor for generating patient movement data; a display device; a memory; and a processor connected to the memory, the processor for: generating a plurality of movement path targets based on therapist clinical movements, the therapist clinical movements captured with a movement capture device; generating a three-dimensional movement path based on a movement target of the plurality of movement path targets, wherein the three-dimensional movement path indicates how to perform the therapist clinical movement by defining a path area having a virtual perimeter around the three-dimensional movement path; creating an animation of a portion of the therapist clinical movement including a visual representation of the path area; causing the display device to display the animation including the visual representation of the path area; displaying patient motion superimposed over the visual representation of the path area; determining the patient movement data with the patient motion sensor, the patient movement data indicating patient movement along the three-dimensional movement path; determining whether the patient movement is within the three-dimensional movement path based on analysis of the patient movement data regarding patient movement; and generating feedback indicating whether the patient movement is within the three-dimensional movement path.

2. The system of claim 1, further comprising a patient feedback device worn by the patient, wherein the processor is further configured to provide the feedback to the patient on the patient feedback device.

3. The system of claim 2, further comprising a patient treatment device worn by the patient, the patient treatment device including the patient feedback device and the patient motion sensor. the feedback including at least one of haptic feedback and non-contact feedback, wherein the non-contact feedback includes visual feedback and audible feedback.

5. The system of claim 1, wherein, 4. The system of claim 2, wherein, the determining whether the patient movement is within the three-dimensional movement path includes determining that the patient movement is within the three-dimensional movement path; and the generating feedback includes providing a success notification indicating that the patient correctly completed the patient movement.

6. The system of any one of claims 1-5, wherein, the determining whether the patient movement is within the three-dimensional movement path includes determining that the patient movement is not within the three-dimensional movement path; and the generating feedback includes providing a failure notification indicating that the patient did not correctly complete the patient movement.

7. The system of any one of claims 1-5, wherein, the creating an animation of a portion of the therapist clinical movement including a visual representation of the path area includes generating a start zone and an end zone based on the therapist clinical movement; and the determining whether the patient movement is within the three-dimensional movement path includes determining whether the patient movement begins at the start zone and ends at the end zone. ​ ​ 8. The system of one of claims 1-5, further comprising an infrared sensor within the movement capture device, the infrared sensor generating infrared sensor data indicative of the three-dimensional movement path of a therapist clinical movement, wherein the creating the animation of a portion of the therapist clinical movement including the visual representation of the path area is based on the generated infrared sensor data.

9. The system of one of claims 1-5, wherein, the creating the animation of a portion of the therapist clinical movement including the visual representation of the path area comprises generating a visual representation of a prescribed tolerance distance around the three-dimensional movement path; and the displaying the visual representation of the three-dimensional movement path comprises displaying the visual representation of the prescribed tolerance distance.

10. The system of one of claims 1-5, wherein, generating the three-dimensional movement path comprises adding a success parameter or a failure parameter to the movement path.

11. A method of evaluating a clinical movement, the method comprising: generating a plurality of movement path objectives based on movements of a therapist clinical movement, the therapist clinical movement being captured with a movement capture device; generating a three-dimensional movement path based on a movement objective of the plurality of movement path objectives, wherein the three-dimensional movement path indicates how to perform the therapist clinical movement by defining a path area having a virtual perimeter around the three-dimensional movement path; creating an animation of a portion of the clinical movement including a visual representation of the path area; displaying the animation including the visual representation of the path area; displaying a patient movement overlaid with the visual representation of the path area; receiving patient movement data from a patient movement sensor, wherein the patient movement data is indicative of patient movement along the three-dimensional movement path; determining whether the patient movement is within the three-dimensional movement path based on an analysis of the patient movement data regarding patient movement; and generating feedback indicating whether the patient movement is within the three-dimensional movement path.

12. The method of claim 11, further comprising providing the feedback to the patient on a patient feedback device worn by the patient.

13. The method of claim 12, wherein: the determining whether the patient movement is within the three-dimensional movement path comprises determining that the patient movement is within the three-dimensional movement path; and the generating feedback comprises providing a success notification indicating that the patient correctly completed the patient movement.

14. The method of one of claims 11-13, wherein, the determining whether the patient movement is within the three-dimensional movement path comprises determining that the patient movement is not within the three-dimensional movement path; and the generating feedback comprises providing a failure notification indicating that the patient did not correctly complete the patient movement.

15. The method of one of claims 11-13, wherein, the creating the animation of a portion of the clinical movement including the visual representation of the path area comprises generating a start zone and an end zone based on the therapist clinical movement; and ​ The determining whether the patient movement is within the three-dimensional movement path includes determining whether the patient movement originated at the start zone and terminated at the end zone.

16. The method of one of claims 11-13, wherein: The movement capture device further includes an infrared sensor to generate infrared sensor data indicative of the three-dimensional movement path of a therapist clinical movement; and The creating an animation of a portion of the clinical movement that includes a visual representation of the path zone is based on the generated infrared sensor data.

17. The method of one of claims 11-13, wherein, The creating an animation of a portion of the clinical movement that includes a visual representation of the path zone includes generating a visual representation of a prescribed tolerance distance around the three-dimensional movement path; and The displaying the animation that includes the visual representation of the path zone includes displaying the visual representation of the prescribed tolerance distance.

18. The method of one of claims 11-13, wherein, Generating a three-dimensional movement path includes adding a success parameter or a failure parameter to the movement path.

19. A machine-readable medium comprising instructions for receiving information, which when executed by a machine, cause the machine to perform operations of: generating a plurality of movement target paths based on therapist clinical movements, the therapist clinical movements being captured with a movement capture device; generating a three-dimensional movement path based on a movement target of the plurality of movement target paths, wherein the three-dimensional movement path indicates how to perform the therapist clinical movement by defining a path zone having a virtual perimeter around the three-dimensional movement path; creating an animation of a portion of the therapist clinical movement, the animation including a visual representation of the path zone; causing the animation including the visual representation of the path zone to be displayed; displaying a patient movement overlaid with the visual representation of the path zone; receiving patient movement data determined with a patient movement sensor, the patient movement data indicating patient movement along the three-dimensional movement path; determining whether the patient movement is within the three-dimensional movement path based on an analysis of the patient movement data regarding patient movement; and sending feedback indicating whether the patient movement is within the three-dimensional movement path.

20. The machine-readable medium of claim 19, the instructions further causing the machine to provide the feedback to the patient on a patient feedback device worn by the patient.

21. The machine-readable medium of claim 19, wherein: the determining whether the patient movement is within the three-dimensional movement path includes determining that the patient movement is within the three-dimensional movement path; and the sending feedback includes providing a success notification indicating that the patient correctly completed the patient movement.

22. The machine-readable medium of one of claims 19-21, wherein: the determining whether the patient movement is within the three-dimensional movement path includes determining that the patient movement is not within the three-dimensional movement path; and the sending feedback includes providing a failure notification indicating that the patient did not correctly complete the patient movement.

23. A system comprising: a movement capture device to capture a therapist clinical movement; a processor to: generate a plurality of movement target paths based on the therapist clinical movement; generate a three-dimensional movement path based on a movement target of the plurality of movement target paths, wherein the three-dimensional movement path indicates how to perform the therapist clinical movement by defining a path zone having a virtual perimeter around the three-dimensional movement path; create an animation of a portion of the therapist clinical movement, the animation including a visual representation of the path zone; cause the animation including the visual representation of the path zone to be displayed; display a patient movement overlaid with the visual representation of the path zone; receive patient movement data determined with a patient movement sensor, the patient movement data indicating patient movement along the three-dimensional movement path; determine whether the patient movement is within the three-dimensional movement path based on an analysis of the patient movement data regarding patient movement; and send feedback indicating whether the patient movement is within the three-dimensional movement path.

24. The system of claim 23, wherein: the determining whether the patient movement is within the three-dimensional movement path includes determining that the patient movement is within the three-dimensional movement path; and the sending feedback includes providing a success notification indicating that the patient correctly completed the patient movement.

25. The system of one of claims 23-24, wherein: the determining whether the patient movement is within the three-dimensional movement path includes determining that the patient movement is not within the three-dimensional movement path; and the sending feedback includes providing a failure notification indicating that the patient did not correctly complete the patient movement.

Images