Rehabilitation HMD device for providing object catching training based on vr / ar body image, and operating method thereof

Abstract

An HMD device according to one embodiment may comprise the operations of: acquiring information about a tracker worn on a first body, which is a cut or paralyzed body part of a patient; outputting a virtual body image onto the first body; outputting a virtual object; controlling the shape or motion of the virtual body image on the basis of the information about the tracker; and determining whether the virtual body image makes a preset motion at the location where the object is output.

Description

A rehabilitation HMD device providing object grasping training based on VR / AR body images and its operating method

[0001] The present invention relates to a rehabilitation HMD device for performing object grasping training based on a VR / AR virtual body image and an operating method thereof.

[0002] The number of patients undergoing limb amputation due to modern-day diseases like diabetes and Buerger's disease is increasing every year. Between 50% and 80% of amputees experience phantom pain. Phantom pain is a condition in which a person feels pain in a specific part of the body that is physically absent, perhaps due to an accident or illness.

[0003] A representative treatment for phantom pain is mirror therapy. Mirror therapy alleviates phantom pain by allowing amputees to view their other arm in a mirror and perform hand movements as if they had two arms. This method creates the illusion of pain-free movement of the amputated body part, reducing phantom pain in the amputated area. The effectiveness of this treatment has been proven by medical experts.

[0004] However, mirror therapy has the disadvantage of being inconvenient to apply because it must be performed in a position that precisely matches the angle of the patient's gaze to the mirror, and it has the disadvantage of being unrealistic because only a limited range of movement is possible right next to the mirror.

[0005] Additionally, the treatment method using mirrors was difficult to apply to patients with lower extremity amputations or paralysis because it was difficult to realistically align the angle between the patient's line of sight and the mirror.

[0006] The problem to be solved in the embodiment of the present invention is to provide a technology that contributes to increasing the effectiveness of treatment and rehabilitation training for alleviating phantom pain in amputees by implementing a VR / AR technology method that allows patients to perform various movements without a mirror while maintaining the treatment concept of mirror therapy that reduces phantom pain felt in the amputated area by creating an illusion with the visual effect of the amputated area moving without pain.

[0007] In particular, an embodiment of the present invention provides a technology to assist in the rehabilitation of a patient by providing a VR / AR-based object grasping simulation that allows the patient to indirectly recognize his or her own body by controlling the movement of a virtual body visible through an HMD device.

[0008] However, the technical problems to be solved by the embodiments of the present invention are not limited to the problems mentioned above, and various technical problems can be derived from the contents described below within a scope obvious to those skilled in the art.

[0009] An operation method performed by an HMD device according to one embodiment may include: an operation of acquiring information of a tracker worn on a first body, which is an amputated or paralyzed body part of a patient; an operation of outputting a virtual body image to the first body; an operation of outputting a virtual object; an operation of controlling a shape or movement of the virtual body image based on the information of the tracker; and an operation of determining whether the virtual body image takes a preset movement at a location where the object is output.

[0010] Additionally, the information of the tracker may include information about a resistance value that changes in response to a morphological deformation of the patient's ligaments or muscles, and the controlling operation may include an operation for controlling the shape or movement of the virtual body image based on the information about the resistance value.

[0011] In addition, the operation of outputting the virtual object may include an operation of setting a plurality of locations where the object is to be created; an operation of creating the virtual object at at least one of the plurality of locations; and an operation of changing the center point of the virtual object or the size of the object.

[0012] In addition, the controlling action may include an action of controlling the virtual body image to have a first shape having a grasping hand shape, and the determining action may include an action of determining that the object has been caught when the virtual body image is positioned at a position including the center point when the object has a preset size and the shape of the virtual body image has become the first shape.

[0013] In addition, the controlling action may include an action of sensing the shape or movement of a second body that is a body part symmetrical to the first body; and an action of controlling the virtual body image to have a shape or movement that is reversed left and right from the shape or movement of the second body based on the sensed information.

[0014] Additionally, the determining action may include an action of determining that the object has been caught when the object has reached a preset size and both the virtual body image and the second body are located within a preset distance from the center point.

[0015] In addition, the operation of outputting the virtual object includes an operation of setting a plurality of locations where the object is to be created; an operation of creating a first object and a second object at two or more locations among the plurality of locations; and an operation of changing the center point and size of each of the first object and the second object, and the operation of controlling includes an operation of controlling the virtual body image to have a first shape having a grasping hand shape, and the operation of determining includes an operation of sensing a shape or movement of a second body that is a body part symmetrical to the first body; And when the first object becomes a size corresponding to the first catch region information, and the center point of the first object is located within the first catch region information, and the shape of the virtual body image becomes the first shape, and when the second object becomes a size corresponding to the second catch region information, and the center point of the second object is located within the second catch region information, and the second body is sensed to assume a grasping shape while being positioned, it may include an operation of determining that the first object and the second object are caught.

[0016] In addition, the outputting operation may include an operation of outputting first catch region information of a preset size in an area where the virtual body image is located; and an operation of outputting second catch region information of a preset size in an area where the second body is located, and the determining operation may include an operation of determining that the first object and the second object have been caught when the first object has a size corresponding to the first catch region information, the center point of the first object is located within the first catch region information, and the shape of the virtual body image becomes the first shape, and when the second object has a size corresponding to the first catch region information, the center point of the second object is located within the second catch region information, and the second body is located and takes a grasping shape.

[0017] In addition, the above-described determining operation may further include an operation of calculating and outputting a catch accuracy based on the following mathematical expression 1 using the area (S1) of the first object and the area (S2) of the first catch area at the time when the virtual body image is controlled in the first form.

[0018] [Mathematical Formula 1]

[0019] Catch Accuracy = (1-|S1-S2| / S2)x100 (%)

[0020] (S1: Area of ​​the first object at the point in time when the virtual body image is controlled in the first form, S2: Area of ​​the first catch area)

[0021] In addition, the controlling operation may include an operation of sensing a shape or movement of a second body, which is a body part symmetrical to the first body; and an operation of controlling the virtual body image to have a shape or movement that is reversed left and right from the shape or movement of the second body based on the sensed information; and the determining operation may include an operation of determining that the object has been caught when the virtual body image and the second body are both located within a preset distance from the position of the virtual object when the object has a preset size.

[0022] An HMD device according to one embodiment includes one or more memories storing commands to perform a predetermined operation; and one or more processors operably connected to the one or more memories and configured to execute the commands; and a display outputting a virtual body image to a user, wherein the operation of the processor may include an operation of obtaining information of a tracker worn on a first body, which is an amputated or paralyzed body part of a patient; an operation of outputting a virtual body image to the first body; an operation of outputting a virtual object; an operation of controlling a shape or movement of the virtual body image based on the information of the tracker; and an operation of determining whether the virtual body image takes a preset movement at a location where the object is output.

[0023] According to an embodiment of the present invention, a virtual body can be implemented on a patient's amputated body part. The implemented virtual body can be implemented to have a virtual shape or movement based on information from a tracker worn on the amputated body part, or can be implemented to have movements symmetrical to the movements of the unamputated body. Through this, the patient can directly visually observe the movements of the virtual body implemented through virtual reality and control the movements, thereby perceiving the presence of a body on the amputated body part, thereby enabling treatment similar to mirror therapy.

[0024] Additionally, patients can perceive a virtual body image in front of their eyes at the same point in time as in real life through an HMD (Head Mounted Display) device, allowing them to perceive it vividly and from a natural point of view, just like their actual body.

[0025] Accordingly, patients can move a virtual body as if it were their actual arms and legs by thinking that they are moving the amputated body, and can visually perceive the movement of the body with a realistic sense, which can alleviate phantom pain and improve the effectiveness of rehabilitation training.

[0026] In addition, the patient's movements can be implemented so that the tracker senses the detailed movements of the nerves and muscles of the actual patient's arms or legs, so that the movements of the virtual body can be implemented to more accurately reflect the movements of the actual body.

[0027] Additionally, by providing simulations that include various missions that patients can perform with a virtual body, it is possible to provide treatment content that combines elements of interest to patients.

[0028] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0029] Figure 1 is a configuration diagram of a rehabilitation treatment system according to one embodiment of the present invention.

[0030] FIG. 2 is a functional block diagram of an HMD device according to one embodiment of the present invention.

[0031] FIG. 3 is an exemplary diagram showing a patient using a rehabilitation treatment system according to one embodiment of the present invention.

[0032] FIG. 4 is an exemplary diagram showing an operation of using a rehabilitation treatment system according to one embodiment of the present invention.

[0033] Figure 5 is a flowchart of an operation method performed by an HMD device according to one embodiment of the present invention.

[0034] FIGS. 6 to 8 are exemplary diagrams for explaining an operation in which an HMD device outputs a virtual object and a patient moves a virtual body image or a second body to catch the virtual object, according to one embodiment.

[0035] FIGS. 9 and 10 are exemplary diagrams illustrating an operation in which an HMD device outputs a virtual object and a patient moves a virtual body image and a second body together to catch the virtual object, according to one embodiment.

[0036] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the scope of the present invention is defined solely by the claims.

[0037] In describing embodiments of the present invention, specific descriptions of known functions or configurations will be omitted unless truly necessary. Furthermore, the terms described below are defined based on their functions in the embodiments of the present invention and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the overall content of this specification.

[0038] The functional blocks depicted in the drawings and described below are merely examples of possible implementations. Other implementations may utilize other functional blocks without departing from the spirit and scope of the detailed description. Furthermore, while one or more functional blocks of the present invention are depicted as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0039] Furthermore, the expression "including certain components" is an open expression, meaning that it simply indicates the presence of those components and should not be understood as excluding additional components.

[0040] Furthermore, when it is said that a component is connected or connected to another component, it should be understood that it may be directly connected or connected to that other component, but that there may also be other components in between.

[0041] Additionally, expressions such as 'first, second' are used only to distinguish between multiple components, and do not limit the order or other characteristics between the components.

[0042] The system (10) according to an embodiment of the present invention maintains the treatment concept of mirror therapy, which reduces phantom pain felt in the amputated area by creating an illusion with the visual effect of the amputated area moving without pain in patients, while implementing a treatment method that allows patients to perform various movements without a mirror, thereby providing a technology that contributes to increasing the effectiveness of treatment and rehabilitation training for alleviating phantom pain in patients with limb amputations.

[0043] A system (10) according to an embodiment of the present invention outputs a virtual body image (11) to a severed part of a patient's body through VR (Virtual Reality) or AR (Augmented Reality) technology, and can implement the shape or movement of the virtual body image (11) based on information obtained from the patient's body. To this end, the components included in the system (10) according to an embodiment of the present invention and the operation of each component will be described with drawings.

[0044] Figure 1 is a configuration diagram of a system (10) according to one embodiment of the present invention.

[0045] Referring to FIG. 1, a system (10) according to one embodiment of the present invention may include a tracker (100) and an HMD (Head Mounted Display) device (200).

[0046] A tracker (100) is a device attached to or worn on a patient's body to perform wired or wireless communication with an external device. The tracker (100) may include a physical mark (Mark) that allows the HMD device (200) to identify a portion of the patient's first body, such as a paralyzed or amputated body part, or a wearable device worn on the first body. For example, the wearable device may include a motion capture glove, a VR smart glove, or a motion recognition glove, and the wearable device may be worn on a portion of the patient's body, such as a hand, arm, foot, or ankle.

[0047] The tracker (100) is worn on one end of the patient's first body, and senses the movement of muscles or ligaments that change when the patient attempts to move the amputated body, generates information about the movement, and can determine the movement of the amputated part intended by the patient based on the information about the movement. For example, the tracker (100) may include a sensor capable of sensing changes in the shape of a surface through a resistance value that changes as the surface expands or contracts. The tracker (100) may generate information about the shape or movement of the first body sensed by the sensor and transmit it to the HMD device (200) via a wired or wireless network.

[0048] The HMD device (200) is a device that can be worn around the patient's eyes like glasses and outputs images containing virtual information to the patient. The HMD device (200) of the embodiment of the present invention may include a form such as a Face Mounted Display (FMD) or smart glasses.

[0049] Fig. 2 is a functional block diagram of an HMD device (200) according to an embodiment of the present invention. Referring to Fig. 2, the HMD device (200) according to an embodiment of the present invention may include a memory (210), a processor (220), a display (230), and a camera (240).

[0050] The memory (210) can store instructions for operating the processor (220) and data necessary for the operation of the processor (220).

[0051] The processor (220) can perform various data processing and calculations, including a central processing unit (CPU). The processor (220) can generate a virtual body image (11) of a patient based on information sensed by a tracker (100) or a camera (240).

[0052] The display (230) can output an image including a virtual body image (11) to the patient. The display (230) can output the virtual body image (11) generated by the processor (220) to one end of the patient's first body together with a VR (Virtual Reality) or AR (Augmented Reality) image.

[0053] The camera (240) can generate three-dimensional depth information of an actual object by calculating the depth values ​​of image pixels obtained from an actual object, including multiple camera modules. The camera (240) can photograph a patient's hand, arm, foot, ankle, etc. For example, the camera (240) can photograph a second body (12), which is a body part symmetrical to a first body corresponding to an amputated or paralyzed body part of the patient, and can sense the shape and movement of the second body (12) according to the patient's movement.

[0054] Figure 3 is an exemplary diagram showing the appearance of a patient using a system (10) according to one embodiment of the present invention.

[0055] Referring to FIG. 3, the patient in FIG. 3 has a first body (the left hand of the figure in FIG. 3) amputated. Accordingly, to utilize the system (10) according to one embodiment of the present invention, the patient may mark the area near the amputated surface of the first body with a tracker (100) or wear a wearable device. Additionally, the patient may wear an HMD device (200) around the eyeball.

[0056] FIG. 4 is an exemplary diagram showing an operation of using a system (10) according to one embodiment of the present invention.

[0057] Referring to FIG. 4, the HMD device (200) can generate a virtual body image (11) in the form of a VR image or AR image at the location of the first body, which is the amputated or paralyzed body part of the patient, and visually output it to the patient through the display (230).

[0058] According to one embodiment, the processor (220) of the HMD device (200) can output a virtual body image (11) to the first body, and can control the shape or movement of the virtual body image (11) by determining a motion that the patient intends to move the first body based on the information of the tracker (100). For example, the information of the tracker (100) can include information on a resistance value that changes in response to a morphological deformation or movement of a patient's ligament or muscle, and the HMD device (200) can control the shape or movement of the virtual body image (11) based on the information on the resistance value. As an example, the tracker (100) can include a plurality of strain gauges that output a corresponding resistance value according to a morphological deformation of a patient's muscle or ligament part, and can determine a motion intended by the user based on resistance value patterns of the plurality of strain gauges. The MD device (200) can reflect and output a movement corresponding to a virtual body image (11) based on the shape of the movement determined by the tracker (100). Accordingly, even if the body is amputated, the patient can control the movement of the virtual body image (11) through the movement of the ligaments, and can visually confirm the movement of the virtual body image (11).

[0059] According to one embodiment, the processor (220) of the HMD device (200) senses the shape or movement of a second body (12), which is a body part symmetrical to the first body, and controls the virtual body image (11) to have a shape or movement that is reversed left and right based on the sensed information.

[0060] FIG. 5 is a flowchart of an operation method performed by an HMD device (200) according to one embodiment of the present invention. Each step disclosed in FIG. 5 may be performed by the HMD device (200) according to one embodiment, and specifically, the operation of each step may be understood as an operation performed by a processor (220).

[0061] Each step disclosed in FIG. 5 is merely a preferred embodiment for achieving the purpose of the present invention, and some steps may be added or deleted as needed, and one step may be included and performed within another step. The order of each operation disclosed in FIG. 5 is merely arranged for convenience of understanding, and this order is not limited to a chronological order, and the order may be changed and performed differently depending on the designer's choice.

[0062] Referring to FIG. 5, at step S1010, the HMD device (200) can acquire information about a tracker (100) worn on a first body part, which is an amputated or paralyzed body part of the patient. For example, the information about the tracker (100) can include information about the tracker (100) of the example described with reference to FIGS. 3 and 4.

[0063] In step S1020, the HMD device (200) can output a virtual body image (11) to the first body. For example, the operation of step S1020 can be performed as an example operation described with reference to FIGS. 3 and 4.

[0064] At step S1030, the HMD device (200) can output a virtual object (1) (e.g., a baseball, a soccer ball, etc.) to the display (230) in the form of a VR image or AR image.

[0065] At step S1040, the HMD device (200) can control the shape or movement of the virtual body image (11) based on information from the tracker (100) (e.g., hand opening movement, hand clenching movement, etc.).

[0066] At step S1050, the HMD device (200) can determine whether the virtual body image (11) is making a preset movement at the location where the virtual object (1) is being output. In addition, the HMD device (200) can sense the movement of the second body (12) and perform additional judgment using the sensed information.

[0067] The operation of steps S1030 to S1050 will be examined in detail with the drawings below.

[0068] FIGS. 6 to 8 are exemplary diagrams for explaining an operation in which an HMD device (200) outputs a virtual object (1) according to one embodiment, and a patient moves a virtual body image (11) or a second body (12) to catch the virtual object (1).

[0069] First, as a first embodiment, an embodiment of catching a virtual object (1) through a virtual body image (11) is described with reference to FIGS. 6 to 8.

[0070] Referring to FIG. 6, the HMD device (200) can set multiple locations where a virtual object (1) is to be created. For example, the HMD device (200) can create a virtual object in at least one location among the six rectangular areas indicated by dotted lines in FIG. 6.

[0071] Referring to FIG. 7, the HMD device (200) can control the position and size of a virtual object (1). For example, the HMD device (200) can control the center point of the virtual object to move toward the patient and gradually increase the size of the virtual object, thereby outputting that the virtual object is approaching the patient.

[0072] Referring to FIG. 8, when the virtual object (1) reaches a preset size, the HMD device (200) can determine that the patient has caught the virtual object when the virtual body image (11) is positioned at the location of the virtual object (1) and the shape of the virtual body image (11) becomes the first shape (e.g., a shape of a hand being grasped).

[0073] Next, as a second embodiment, an embodiment of catching a virtual first object (1) through a virtual body image (11) and catching a virtual second object (2) with a second body (12) is described with reference to FIGS. 6 to 8.

[0074] Referring to FIG. 6, the HMD device (200) can set multiple locations where objects are to be created. For example, the HMD device (200) can create a virtual first object (1) and a virtual second object (2) at two or more locations among the six rectangular areas indicated by dotted lines in FIG. 6. At this time, the HMD device (200) can create the virtual first object (1) and the virtual second object (2) with a preset time difference.

[0075] Referring to FIG. 7, the HMD device (200) can sense the shape or movement of the second body (12), which is a body part symmetrical to the first body, through the camera (240). The HMD device (200) can determine the movement of the second body (12) and the movement of the virtual body image (11) together based on the sensed information. At this time, the HMD device (200) can output first catch area information (A) of a preset size to the area where the virtual body image (11) is located, and output second catch area information (B) of a preset size to the area where the sensed second body (12) is located.

[0076] At this time, the HMD device (200) can control the positions of each of the first virtual object (1) and the second virtual object (2) and the sizes of the virtual objects. For example, the HMD device (200) can output that the virtual first object (1) is approaching toward the left side of the patient by moving the center point of the first virtual object (1) toward the left side of the patient's body and controlling that the size of the first virtual object (1) gradually increases. The HMD device (200) can output that the virtual second object (2) is approaching toward the right side of the patient by moving the center point of the second virtual object (2) toward the right side of the patient's body and controlling that the size of the second virtual object (2) gradually increases. The HMD device (200) can control that the speed at which the sizes of the first virtual object (1) and the second virtual object (2) change differently so that the speed at which the first virtual object (1) and the second virtual object (2) approach the patient is perceived to be different.

[0077] Referring to FIG. 8, when the HMD device (200) controls the shape of the virtual body image (11) to the first shape (e.g., grasping shape) when the virtual first object (1) becomes a preset size and the virtual body image (11) is positioned at the position of the virtual first object (1), and when the virtual second object (2) becomes a preset size and the second body (12) is positioned at the position of the virtual second object (2) and assumes a grasping shape, it can be determined that the patient has caught the virtual first object (1) and the virtual second object (2).

[0078] For example, when the HMD device (200) senses that the shape of the virtual body image (11) is controlled to the first shape (e.g., grasping shape) when the center point of the virtual first object (1) is located within the first catch area information (A) when the size of the virtual first object (1) corresponds to the first catch area information (A), and the shape of the virtual body image (11) is controlled to the first shape (e.g., grasping shape), and when the size of the virtual second object (2) corresponds to the second catch area information (B), and the center point of the virtual second object (2) is located within the second catch area information (B), and the second body (12) is located and assumes the grasping shape, it can be determined that the patient has caught the virtual first object (1) and the virtual second object (2). At this time, when determining a catch by comparing the size of the catch area information and the size of the virtual object, a preset degree of error can be allowed when comparing each size.

[0079] In addition, when the user controls the virtual body image (11) to be in the first form, the HMD device (200) can guide the user to control his / her body more precisely by outputting the catch accuracy based on the difference between the size of the catch area information (A or B) and the size of the virtual object (the first object or the second object). For example, the HMD device (200) can calculate and output the catch accuracy based on the difference between the size of the first object at the time when the virtual body image is controlled in the first form and the size corresponding to the first catch area information. As an example, the HMD device (200) can calculate the catch accuracy based on the area (S1) of the first object (1) and the area (S2) of the first catch area (A) at the time when the virtual body image is controlled in the first form, as in Mathematical Expression 1 below.

[0080] [Mathematical Formula 1]

[0081] Catch Accuracy = (1-|S1-S2| / S2)x100 (%)

[0082] The embodiment of the above-described mathematical formula 1 is equally applied between the area (S1) of the second object (2) and the area (S2) of the second catch area (B) at the determined point in time when the second body (12) is determined to be in the shape of a hand being clenched, so that the HMD device (200) can calculate the catch accuracy for the second body.

[0083] FIG. 9 and FIG. 10 are exemplary diagrams for explaining an operation in which an HMD device (200) outputs a virtual object and a patient moves a virtual body image (11) and a second body (12) together to catch the virtual object, according to one embodiment.

[0084] Through FIGS. 9 and 10, an embodiment of catching a virtual object (1) using a virtual body image (11) and a second body (12) together is described.

[0085] Referring to FIG. 9, the HMD device (200) senses the shape or movement of a second body (12), which is a body part symmetrical to the first body, and based on the sensed information, controls a virtual body image (11) to have a shape or movement that is reversed left and right from the shape or movement of the second body (12).

[0086] Referring to FIG. 10, the HMD device (200) can determine that the virtual object (1) has been caught when the virtual object (1) has reached a preset size and both the virtual body image (11) and the sensed second body (12) are located within a preset distance from the position of the virtual object (1) (e.g., the position of the center point).

[0087] According to the above-described embodiment, a virtual body image (11) can be implemented on the amputated part of the patient's body, and at this time, the implemented virtual body image (11) can be implemented to have a virtual shape or movement based on information from a tracker (100) worn on the amputated part of the body, or can be implemented to have a movement symmetrical to the movement of the unamputated body. Through this, the patient can directly visually see the movement of the virtual body implemented through virtual reality and control the movement, thereby recognizing that a body exists in the amputated part of the body, thereby allowing treatment with a concept similar to mirror therapy to be performed.

[0088] In addition, the patient perceives a virtual body image (11) in front of his / her eyes at the same point in time as in real life through an HMD (Head Mounted Display) device, so he / she can perceive it vividly at a natural point in time as if it were a real body.

[0089] Accordingly, patients can move a virtual body as if it were their actual arms and legs by thinking that they are moving the amputated body, and can visually perceive the movement of the body with a realistic sense, which can alleviate phantom pain and improve the effectiveness of rehabilitation training.

[0090] In addition, the patient's movements can be implemented so that the tracker (100) senses the detailed movements of the nerves and muscles of the actual patient's arms or legs, so that the movements of the virtual body can be implemented to more accurately reflect the movements of the actual body.

[0091] Additionally, by providing simulations that include various missions that patients can perform with a virtual body, it is possible to provide treatment content that combines elements of interest to patients.

[0092] The various embodiments and terms used in this document are not intended to limit the technical features described in this document to specific embodiments, but should be understood to encompass various modifications, equivalents, or alternatives of the embodiments. In connection with the description of the drawings, similar reference numerals may be used to refer to similar or related components. The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise.

[0093] In this document, the phrases "A or B", "at least one of A and B", "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include all possible combinations of the items listed together in that phrase. Terms such as "1", "2", or "first" or "second" may be used merely to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a (e.g., a first) component is referred to as "coupled" or "connected" to another (e.g., a second) component, with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0094] The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0095] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., a memory) readable by a device (e.g., an electronic device). The storage medium may include random access memory (RAM), a memory buffer, a hard drive, a database, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), read-only memory (ROM), and / or the like.

[0096] Additionally, the processor of the embodiments of the present document can call at least one command among one or more commands stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to the at least one command called. These one or more commands may include code generated by a compiler or code executable by an interpreter. The processor may be a general-purpose processor, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), and / or the like.

[0097] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0098] The methods according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated on a machine-readable storage medium, such as a manufacturer's server, an application store's server, or a server's memory.

[0099] According to various embodiments, each component (e.g., a module or a program) of the described components may include a single or multiple entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In the method of operation performed by the HMD device, An action of acquiring information from a tracker worn on a first body part, which is an amputated or paralyzed body part of a patient; An action of outputting a virtual body image to the first body; The action of outputting a virtual object; An action of controlling the shape or movement of the virtual body image based on information from the tracker; and Including an action of determining whether the virtual body image takes a preset movement at the location where the object is being output. How the HMD device works.

2. In paragraph 1, The information of the above tracker is, Contains information about the resistance value that changes in response to the movement of the patient's ligaments or muscles, The above controlling action is, An operation for controlling the shape or movement of the virtual body image based on information about the resistance value, How the HMD device works.

3. In paragraph 1, The action of outputting the above virtual object is: An action that sets multiple locations; An operation of creating a virtual object at at least one of the plurality of locations; and Including an action of changing the position of the virtual object or the size of the virtual object, How the HMD device works.

4. In paragraph 3, The above controlling action is, An operation for controlling the virtual body image to have a first shape having a grasping hand shape based on information of the tracker, The above-mentioned judging action is, When the virtual object has a preset size, the virtual body image is positioned at the location of the virtual object, and the shape of the virtual body image becomes the first shape, including an action of determining that the object has been caught. How the HMD device works.

5. In paragraph 1, The action of outputting the above virtual object is: An action that sets multiple locations; An operation of generating a virtual first object and a virtual second object at two or more locations among the plurality of locations; and Including an operation of changing the center point and size of each of the first object and the second object, The above controlling action is, Including an action for controlling the virtual body image to have a first shape having a grasping hand shape, The above-mentioned judging action is, An action of sensing the shape or movement of a second body that is a body part symmetrical to the first body; and When the first object becomes a preset size, the virtual body image is positioned at a position including the center point of the first object and the shape of the virtual body image becomes the first shape, and when the second object becomes a preset size, the second body is positioned at a position including the center point of the second object and takes a grasping shape, including an action of determining that the first object and the second object have been caught. How the HMD device works.

6. In paragraph 5, The above output action is, An operation of outputting first catch area information of a preset size in an area where the above virtual body image is located; and Including an operation of outputting second catch area information of a preset size in an area where the second body is located, The above-mentioned judging action is, When the first object becomes a size corresponding to the first catch area information, and the center point of the first object is located within the first catch area information, and the shape of the virtual body image becomes the first shape, and when the second object becomes a size corresponding to the second catch area information, and the center point of the second object is located within the second catch area information, and the second body is sensed to assume a grasping shape while being positioned, an operation for determining that the first object and the second object are caught is included. How the HMD device works.

7. In paragraph 6, The above-mentioned judging action is, The above virtual body image further includes an operation of calculating and outputting the catch accuracy based on the following mathematical expression 1 using the area (S1) of the first object at the time when the above virtual body image is controlled in the first form and the area (S2) of the first catch area. [Mathematical Formula 1] Catch Accuracy = (1-|S1-S2| / S2)x100 (%) (S1: Area of ​​the first object at the point in time when the virtual body image is controlled in the first form, S2: Area of ​​the first catch area) How the HMD device works.

8. In paragraph 3, The above controlling action is, An action of sensing the shape or movement of a second body that is a body part symmetrical to the first body; and It includes an operation of controlling the virtual body image to have a shape or movement of the second body and a shape or movement that is reversed left and right based on the sensed information, The above-mentioned judging action is, When the object has reached a preset size, if both the virtual body image and the second body are located within a preset distance from the position of the virtual object, an action is included to determine that the object has been caught. How the HMD device works.

9. A computer-readable recording medium having recorded thereon a computer program including instructions for causing a processor to perform any one of the methods of clauses 1 to 8.

10. One or more memories storing commands to perform a predetermined operation; and one or more processors operatively connected to the one or more memories and configured to execute the commands; and a display outputting a virtual body image to a user, The operation of the above processor is An action of acquiring information from a tracker worn on a first body part, which is an amputated or paralyzed body part of a patient; An action of outputting a virtual body image to the first body; The action of outputting a virtual object; An action of controlling the shape or movement of the virtual body image based on information from the tracker; and Including an action of determining whether the virtual body image takes a preset movement at the location where the object is being output. HMD device.

Images