Gait training support program, gait training support method, and information processing device

Abstract

To perform effective gait training.SOLUTION: An information processing device 10 displays a training image 5 including a first object 7b arranged in a virtual space 7 in an HMD 2. The information processing device 10 acquires a captured image of a trainee 1 photographed by a camera 3. The information processing device 10 determines whether or not the trainee 1 has passed on the first object 7b safely on the basis of a posture change of the trainee 1 shown in the captured image and a first passage determination condition 4b indicating a condition on which to determine that the trainee 1 has passed on the first object 7b safely. On the basis of a determination result, the information processing device 10 decides a second object 7c arranged in the virtual space 7 in the successive gait training and a second passage determination condition 6b indicating a condition on which to determine that the trainee 1 has passed on the second object 7c safely.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a walking training support program, a walking training support method, and an information processing device. [Background technology]

[0002] Health promotion or rehabilitation is one aspect of health care for patients who have become unable to carry out their daily activities due to accidents or illness. In health promotion or rehabilitation, training is provided in an everyday environment to help patients acquire basic movements and mobility skills that are essential for daily life. For example, training is provided to cross the street when the pedestrian signal is green.

[0003] In addition, taking patients with reduced sensory or motor abilities onto public roads and training them to cross the street is practically impossible, considering the need to ensure the patient's safety and the inconvenience to others. Therefore, it is difficult to do this frequently. Therefore, efforts to promote the health of patients and improve the efficiency of rehabilitation by introducing VR devices are attracting attention. For example, it is possible to have patients wear a head mounted display (HMD) and perform walking training by displaying a virtual space on the HMD. By using an HMD, for example, walking training to avoid collisions with people, cars, and obstacles can be safely performed.

[0004] As a technology for supporting walking training, for example, a walking support system that can appropriately guide the walking motion of the user has been proposed. In addition, a walking training support device that can easily measure the walking state and visually grasp the movement of the feet has also been proposed. Furthermore, a sidewalk progress support system that can support the progress of people with walking disabilities on the sidewalk has also been proposed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 180144 [Patent Document 2] JP 2006-204730 A [Patent Document 3] JP 2019-46464 A Summary of the Invention [Problem to be solved by the invention]

[0006] While walking training systems using HMDs allow safe walking training, it is difficult to confirm the results of the training. For example, even if a trainee falls into a situation in the virtual space where he or she would normally fall, there is no mechanism in place to detect the trainee's fall. As a result, it is not possible to confirm the degree of achievement of the walking training, making it difficult to provide effective walking training that matches the trainee's current walking ability.

[0007] In one aspect, the present invention aims to enable effective gait training to be performed. [Means for solving the problem]

[0008] In one proposal, a walking training support program is provided that causes a computer to execute the following processes. The computer displays a training image including a first object placed in a virtual space on a head-mounted display worn by the trainee. The computer acquires a photographed image of the trainee taken by a camera. The computer judges whether or not the trainee has safely passed over the first object based on a change in the trainee's posture shown in the photographed image and a first passing judgment condition indicating a condition for judging that the trainee has safely passed over the first object. Then, based on the judgment result of whether or not the trainee has safely passed over, the computer determines a second object to be placed in the virtual space in the subsequent walking training and a second passing judgment condition indicating a condition for judging that the trainee has safely passed over the second object. Effect of the Invention

[0009] According to one embodiment, effective walking training can be performed. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram showing an example of a walking training support method according to the first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a walking training support system. [Diagram 3] FIG. 2 illustrates an example of computer hardware. [Figure 4] FIG. 1 is a block diagram showing functions for computer-based walking training assistance. [Diagram 5] FIG. 1 is a diagram showing an example of 3D data. [Figure 6] FIG. 4 is a diagram showing an example of target object information. [Figure 7] FIG. 13 is a diagram illustrating an example of a training log. [Figure 8] FIG. 13 is a diagram showing an example of an image displayed during walking training. [Figure 9] FIG. 11 is a diagram illustrating an example of a trainee's posture analysis process. [Figure 10] FIG. 11 is a diagram illustrating an example of a process for determining whether an obstacle is passed; [Figure 11] FIG. 13 is a diagram showing an example of training content after safely passing through an obstacle. [Figure 12] FIG. 13 is a diagram showing an example of training content after failing to pass an obstacle. [Figure 13] 13 is a flowchart illustrating an example of a procedure for a training determination process. [Figure 14] 13 is a flowchart illustrating an example of a procedure for a determination process in obstacle passing training. [Figure 15] 13 is a flowchart showing an example of a procedure for a pedestrian crossing training determination process. [Figure 16] 13 is a flowchart showing an example of a procedure for training content modification processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the present embodiment will be described with reference to the drawings. Note that each embodiment can be implemented in combination with a plurality of other embodiments as long as no contradiction occurs. First Embodiment The first embodiment is a walking training support method that enables effective walking training using an HMD.

[0012] Fig. 1 is a diagram showing an example of a walking training support method according to a first embodiment. Fig. 1 shows an information processing device 10 that implements the walking training support method. The information processing device 10 can implement the walking training support method according to the first embodiment by, for example, executing a walking training support program.

[0013] The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 is, for example, a memory or a storage device included in the information processing device 10. The processing unit 12 is, for example, a processor or an arithmetic circuit included in the information processing device 10.

[0014] The information processing device 10 is capable of wireless or wired communication with both an HMD 2 worn by a trainee 1 undergoing walking training, and a camera 3 that captures images of the trainee 1. The camera 3 is disposed in a position that captures images of the trainee 1, for example, from the side.

[0015] The storage unit 11 stores first object data 4a indicating a first object 7b arranged in the virtual space 7, and a first passage determination condition 4b indicating a condition for determining that the trainee 1 has safely passed over an area in which the first object 7b exists. The first object data 4a is, for example, three-dimensional model data indicating the shape of the first object 7b. The first object data 4a and the first passage determination condition 4b are stored in the storage unit 11 before the trainee 1 starts walking training.

[0016] After the trainee 1 starts walking training, second object data 6a and second passing judgment conditions 6b are stored in the storage unit 11. The second object data 6a is model data representing the shape of a second object 7c that is placed ahead of the first object 7b when viewed from the position of the trainee 1 in the virtual space 7. The second passing judgment conditions 6b are information indicating conditions for judging that the trainee 1 has safely passed over an area where the second object 7c exists.

[0017] During the walking training of the trainee 1, the processing unit 12 determines the second object 7c and the second passing determination condition 6b in the following procedure. The processing unit 12 displays a training image 5 including a first object 7b on the HMD 2 based on the position of the trainee 1 in the virtual space 7 and the first object data 4a. The training image 5 is, for example, an image in the virtual space 7 when viewed from the eye position of a trainee model 7a corresponding to the trainee 1 in the virtual space 7 in the direction in which the face of the trainee model 7a is facing.

[0018] The trainee 1 recognizes the presence of the first object 7b based on the training image 5 displayed on the HMD 2. The trainee 1 then performs walking training in which he passes over the first object 7b. The movement of the trainee 1 is captured by the camera 3. The processing unit 12 acquires the captured image of the trainee 1 from the camera 3. Then, the processing unit 12 determines whether or not the trainee 1 has safely passed over the first object 7b based on the change in posture of the trainee 1 shown in the captured image and the first passage determination condition 4b.

[0019] For example, the processing unit 12 generates a trainee model 7a indicating the posture of the trainee 1 based on a captured image of the trainee 1. The processing unit 12 determines the position of the trainee 1 in the virtual space 7 based on a change in the posture of the trainee model 7a. For example, when the trainee 1 steps on the spot, the processing unit 12 moves the position of the trainee 1 in the virtual space 7 based on the moving distance of the trainee 1 per step.

[0020] The processing unit 12 judges whether the posture of the trainee model 7a when the position of the trainee 1 reaches an area where the first object 7b is present is a posture that allows the trainee model 7a to safely pass over the first object 7b based on the first passing judgment condition 4b. For example, if the first object 7b is an obstacle, the first passing judgment condition 4b is set to a condition that the trainee model 7a's foot is raised to a height equal to or higher than the height of the obstacle. In this case, the processing unit 12 judges that the trainee model 7a can safely pass over the first object 7b if the heel of the raised foot (or the lowest part of the sole) is equal to or higher than the height of the obstacle.

[0021] The processing unit 12 determines a second object 7c to be displayed on the HMD 2 in the subsequent walking training and a second passing judgment condition 6b based on the result of the determination as to whether or not the first object 7b has been passed over safely. For example, when it is determined that the first object 7b has been passed over safely, the processing unit 12 determines the second object 7c and the second passing judgment condition 6b so that the difficulty of passing over safely is higher than that of the first object 7b. Also, when it is determined that the first object 7b has not been passed over safely, the processing unit 12 determines the second object 7c and the second passing judgment condition 6b so that the difficulty of passing over safely is lower than that of the first object 7b.

[0022] In this way, it is possible to judge whether the trainee 1 has safely passed over the first object 7b or not based on the captured image capturing the posture of the trainee 1, and it is possible to appropriately set the difficulty level for safely passing over the next second object 7c according to the judgment result. In this way, by sequentially setting objects with appropriate difficulty levels and performing walking training, effective walking training can be performed.

[0023] In the process of determining the second object 7c and the second passing determination condition 6b, the processing unit 12 determines the size of the second object 7c based on, for example, the determination result and the size of the first object 7b. Specifically, when it is determined that the trainee 1 has not safely passed over the first object 7b, the processing unit 12 determines the height of the second object 7c to be a value lower than that of the first object 7b. As a result, for example, when the trainee 1 cannot safely pass over an obstacle, the height of the next obstacle can be lowered to a height that the trainee 1 can pass over, and effective walking training can be performed to recover the ability to pass over obstacles.

[0024] Furthermore, when it is determined that the first object 7b has been safely passed over, the processing unit 12 determines the height of the second object 7c to be a value higher than that of the first object 7b. This allows for training to overcome an obstacle that is slightly higher than the obstacle that was safely overcome in the walking training to overcome an obstacle, and therefore allows for effective walking training to be performed to recover the ability to overcome obstacles.

[0025] The processing unit 12 may set an upper limit value for the height of the obstacle. When an upper limit value for the height is set, the processing unit 12 sets the height of the obstacle of the second object 7c within a range not exceeding the upper limit value. Furthermore, when it is determined that the trainee 1 has not passed over the first object 7b safely, the processing unit 12 may determine the length of the second object 7c in the moving direction of the trainee 1 in the virtual space 7 to be a value shorter than that of the first object 7b. For example, if the first object 7b is a crosswalk, a crosswalk shorter than that crosswalk is determined to be the second object 7c. As a result, for example, if the trainee 1 is unable to cross the crosswalk safely, the crosswalk can be shortened to a length that allows the trainee 1 to cross, and effective walking training can be performed to walk at a speed sufficient to cross the crosswalk safely.

[0026] Furthermore, when it is determined that the trainee 1 has safely passed over the first object 7b, the processing unit 12 may determine the length of the second object 7c in the moving direction of the trainee 1 in the virtual space 7 to be a value longer than the first object 7b. This allows, for example, when the trainee 1 has safely crossed a pedestrian crossing, to next undergo training to cross a pedestrian crossing that is slightly longer than the length that was safely crossed, thereby enabling effective walking training to be performed to improve walking speed.

[0027] The processing unit 12 may set an upper limit value for the length of the second object 7c. When an upper limit value for the length is set, the processing unit 12 sets the length of the second object 7c within a range not exceeding the upper limit value. The processing unit 12 can also adjust the difficulty of passing over the second object 7c by determining the second passing judgment condition 6b based on the judgment result and the first passing judgment condition 4b. For example, the first passing judgment condition 4b indicates a first time allowed for safe passing over the first object 7b. That is, if the trainee 1 can pass over the first object 7b (e.g., a crosswalk) within a first time (e.g., the time for a green light), it is determined that the trainee 1 has passed over the first object 7b safely. At this time, the processing unit 12 determines a second time allowed for safe passing over the second object 7c as the second passing judgment condition 6b based on the first time. This makes it possible to adjust the length of the green light for the next crosswalk, for example, based on whether the trainee 1 has crossed the crosswalk safely, and to perform effective crosswalk crossing training according to the walking speed of the trainee 1.

[0028] For example, if it is determined that the trainee 1 has not passed over the first object 7b safely, the processing unit 12 determines the second time to be a value longer than the first time. This makes it possible to perform crossing training at a crosswalk where the time for the trainee 1 to cross is set by lengthening the time for the green light at the next crosswalk, for example, when it is determined that the trainee 1 cannot cross safely.

[0029] Furthermore, when it is determined that the first object 7b has been passed over safely, the processing unit 12 determines the second time to be a value shorter than the first time. As a result, when it is determined that the crosswalk has been crossed safely, for example, the time for the green light for the next crosswalk is shortened to increase the difficulty of the training, thereby enabling effective walking training to be performed to cross the crosswalk in a short time.

[0030] Second Embodiment The second embodiment is a walking training support system that can efficiently carry out walking training based on the condition of each individual patient.

[0031] 2 is a diagram showing an example of a walking training support system. The walking training support system is composed of a computer 100 and an HMD 31 worn by a trainee 30. The HMD 31 can be connected to the computer 100 via wireless communication. The HMD 31 is compatible with VR (Virtual Reality) and can display an image of a virtual space generated by the computer 100. For example, an image of a virtual space in which a passing object for rehabilitation is arranged is displayed on the HMD 31. The VR-compatible HMD 31 is sometimes called a VR display.

[0032] The computer 100 has, for example, a built-in camera 107. The computer 100 captures an image of the trainee 30 with the camera 107, and analyzes a change in the posture of the trainee 30 based on the captured image. The computer 100 determines, based on the change in the posture of the trainee 30, whether or not the trainee 30 has safely passed over an object to be passed that is placed in the virtual space.

[0033] Furthermore, the computer 100 adjusts the degree of difficulty for passing over subsequent passing objects based on whether the trainee 30 has been able to safely pass over the passing object. 3 is a diagram showing an example of computer hardware. The entire computer 100 is controlled by a processor 101. A memory 102 and a plurality of peripheral devices are connected to the processor 101 via a bus 111. The processor 101 may be a multiprocessor. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP). At least a part of the functions realized by the processor 101 executing a program may be realized by an electronic circuit such as an application specific integrated circuit (ASIC) or a programmable logic device (PLD).

[0034] The memory 102 is used as a main storage device of the computer 100. The memory 102 temporarily stores at least a part of the OS (Operating System) program and application programs to be executed by the processor 101. The memory 102 also stores various data used in the processing by the processor 101. As the memory 102, for example, a volatile semiconductor storage device such as a RAM (Random Access Memory) is used.

[0035] The peripheral devices connected to the bus 111 include a storage device 103 , a GPU (Graphics Processing Unit) 104 , an input interface 105 , an optical drive device 106 , a camera 107 , a device connection interface 108 , a network interface 109 , and a wireless connection interface 110 .

[0036] The storage device 103 electrically or magnetically writes and reads data to and from a built-in recording medium. The storage device 103 is used as an auxiliary storage device for the computer 100. The storage device 103 stores an OS program, application programs, and various data. Note that, for example, a hard disk drive (HDD) or a solid state drive (SSD) can be used as the storage device 103.

[0037] The GPU 104 is an arithmetic device that performs image processing. The GPU 104 is an example of a graphic controller. The monitor 21 is connected to the GPU 104. The GPU 104 displays an image on the screen of the monitor 21 in accordance with an instruction from the processor 101. The monitor 21 may be a display device using an organic EL (Electro Luminescence) display device or a liquid crystal display device.

[0038] The input interface 105 is connected to the keyboard 22 and the mouse 23. The input interface 105 transmits signals sent from the keyboard 22 and the mouse 23 to the processor 101. Note that the mouse 23 is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.

[0039] The optical drive device 106 uses a laser beam or the like to read data recorded on the optical disc 24 or write data to the optical disc 24. The optical disc 24 is a portable recording medium on which data is recorded so that it can be read by reflection of light. The optical disc 24 includes a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (ReWritable), etc.

[0040] The camera 107 captures an image using, for example, a charge-coupled device (CCD) image sensor. The image captured by the camera 107 is analyzed by the processor 101. The device connection interface 108 is a communication interface for connecting peripheral devices to the computer 100. For example, a memory device 25 or a memory reader / writer 26 can be connected to the device connection interface 108. The memory device 25 is a recording medium equipped with a function for communicating with the device connection interface 108. The memory reader / writer 26 is a device for writing data to the memory card 27 or reading data from the memory card 27. The memory card 27 is a card-type recording medium.

[0041] The network interface 109 is connected to the network 20. The network interface 109 transmits and receives data to and from other computers or communication devices via the network 20. The network interface 109 is a wired communication interface that is connected by a cable to a wired communication device such as a switch or a router. The network interface 109 may also be a wireless communication interface that is communicatively connected by radio waves to a wireless communication device such as a base station or an access point.

[0042] The wireless connection interface 110 is an interface for wirelessly communicating with the HMD 31. For example, an image in a virtual space is transmitted to the HMD 31 via the wireless connection interface 110.

[0043] The computer 100 can realize the processing functions of the second embodiment by the above-mentioned hardware. The device shown in the first embodiment can also be realized by the same hardware as the computer 100 shown in FIG.

[0044] The computer 100 executes a program recorded in, for example, a computer-readable recording medium to realize the processing function of the second embodiment. The program describing the processing contents to be executed by the computer 100 can be recorded in various recording media. For example, the program to be executed by the computer 100 can be stored in the storage device 103. The processor 101 loads at least a part of the program in the storage device 103 into the memory 102 and executes the program. The program to be executed by the computer 100 can also be recorded in a portable recording medium such as the optical disk 24, the memory device 25, or the memory card 27. The program stored in the portable recording medium becomes executable after being installed in the storage device 103 under the control of, for example, the processor 101. The processor 101 can also directly read out and execute the program from the portable recording medium.

[0045] Such a computer 100 can be used to assist a trainee in walking training. 4 is a block diagram showing functions for assisting walking training of a computer. The computer 100 has a storage unit 120, a virtual space generation unit 130, a posture analysis unit 140, a trainee position determination unit 150, a training content setting unit 160, and a training result determination unit 170.

[0046] The storage unit 120 stores 3D data 121, object information 122, and a training log 123. The 3D data 121 is data indicating the shape of each 3D object placed in a virtual space. The object information 122 is information indicating the contents of an obstacle for the 3D object that is a passing object. The training log 123 is information indicating the results of the walking training of the trainee 30.

[0047] The virtual space generation unit 130 generates an image of the virtual space based on the 3D data 121. For example, the virtual space generation unit 130 generates an image of the virtual space seen from the viewpoint of the trainee 30 based on the position of the trainee 30 in the virtual space. If there is a passing object in front of the trainee 30 in the virtual space, an image including the passing object is generated. The virtual space generation unit 130 causes the HMD 31 to display the generated image.

[0048] The posture analysis unit 140 analyzes the posture of the trainee 30 based on the image of the trainee 30 captured by the camera 107. For example, the posture analysis unit 140 analyzes the posture of the trainee 30 shown in each captured frame, and transmits the obtained posture to the trainee position determination unit 150 and the training result determination unit 170.

[0049] The trainee position determining unit 150 determines the position of the trainee 30 in the virtual space based on the time-series change in the posture of the trainee 30. For example, the trainee position determining unit 150 moves the position of the trainee 30 by the distance of one step each time the trainee 30 takes one step. The moving distance of one step of the trainee 30 is calculated, for example, from the moving speed of the trainee 30 measured in advance and the number of steps per unit time.

[0050] The training content setting unit 160 determines the training content for the trainee 30. For example, the training content setting unit 160 receives an input of the level of athletic ability of the trainee 30, and determines the training content with a level of difficulty according to the athletic ability. Furthermore, if the trainee 30 has previously undergone walking training, the training content setting unit 160 refers to the training log 123 and determines the training content with a level of difficulty according to the training results of the past walking training. The determined training content includes a passing object to be placed in front of the trainee 30 in the virtual space.

[0051] The training result determination unit 170 determines whether or not the trainee 30 has safely passed over the passing objects based on the position and posture of the trainee 30. For example, the training result determination unit 170 refers to the object information 122 and recognizes the conditions (passage determination conditions) for safely passing over each passing object. If the movement of the position or posture of the trainee 30 satisfies the passing determination conditions, the training result determination unit 170 determines that the trainee 30 has safely passed over the passing object. The training result determination unit 170 stores the determination result of whether or not the trainee 30 has safely passed over the passing object in the storage unit 120 as the training log 123.

[0052] The function of each element shown in FIG. 4 can be realized, for example, by causing the processor 101 to execute a program module corresponding to that element. Next, the data stored in the storage unit 120 will be specifically described with reference to FIGS.

[0053] 5 is a diagram showing an example of 3D data. In the 3D data 121, a record is registered for each object placed in the virtual space 40. In each record, an object ID, a position, and shape model data are set. The object ID is an identifier of the object. The position is the position of the object in the virtual space. The shape model data is data that defines the shape of the object. The shape model data is, for example, a polygon model.

[0054] Various objects are placed in the virtual space 40 based on the 3D data 121. For example, a crosswalk 41, sidewalks 42 and 43, pedestrian traffic lights 44 and 45, etc. are placed. One or more objects in the virtual space 40 become objects to be passed that impede the actions of the trainee 30. For example, the sidewalks 42, 43 become objects to be passed that are of the type "obstacle" because the step between the sidewalks 42, 43 and the roadway becomes an obstacle. In addition, the combination of the crosswalk 41 and the pedestrian traffic lights 44, 45 on both ends of the crosswalk becomes an object to be passed that is of the type "crosswalk with traffic lights."

[0055] FIG. 6 is a diagram showing an example of object information. In the object information 122, a record is registered for each passing object. In each record, an object ID, a passing object type, a corresponding object ID, a difficulty level, and a passing determination condition are set. The object ID is an identifier of the passing object. The passing object type is a type of the passing object. Types of passing objects include an obstacle, a crosswalk with a traffic light, and the like. The corresponding object ID is an identifier of a three-dimensional object that constitutes the passing object. The identifiers of one or more objects are set in the corresponding object ID.

[0056] The difficulty level is information indicating the degree of difficulty for passing through an obstacle. The difficulty level is set according to the type of object to be passed. For example, if the object to be passed is an obstacle, the difficulty level is expressed by the height of the obstacle. Also, if the object to be passed is a pedestrian crossing with a traffic light, the difficulty level is expressed by the green light time (the time that the pedestrian signal is green) or the length of the pedestrian crossing.

[0057] The passing judgment condition is a condition for safely passing over an object to be passed. For example, if the object to be passed is an obstacle, the passing judgment condition is that the height of the heels of the trainee 30 is equal to or higher than the height of the obstacle. Also, if the object to be passed is a pedestrian crossing with a traffic light, the passing judgment condition is that the trainee can cross within the time when the light is green.

[0058] FIG. 7 is a diagram showing an example of a training log. In the training log 123, a record is set for each passing object set in front of the trainee 30 in the virtual space during walking training. Each record includes the training date and time, the type of passing object, the difficulty level, and the judgment result. The training date and time is the date and time when training to get over the corresponding passing object was conducted. The type of passing object is the type of the passing object that the trainee 30 attempted to get over. The difficulty level is the difficulty level of the passing object. The judgment result is the result of judgment as to whether or not the trainee 30 was able to pass through safely.

[0059] Walking training assistance is performed using the data shown in Figures 5 to 7. For example, when the trainee 30 wears the HMD 31 and starts walking training, the virtual space generation unit 130 generates an image from the trainee 30's viewpoint in the virtual space based on the 3D data 121, and causes the HMD 31 to display the image.

[0060] Fig. 8 is a diagram showing an example of an image displayed during walking training. For example, it is assumed that the position of the trainee 30 in the virtual space is near a crosswalk 41 on a sidewalk 42 (see Fig. 5). It is also assumed that the trainee 30's viewpoint faces the opposite side of the crosswalk 41. In this case, the crosswalk 41 is displayed in a display image 50 displayed on the HMD 31. A sidewalk 43 and a pedestrian traffic light 45 are displayed above the crosswalk 41.

[0061] The pedestrian traffic light 45 has a blue signal light 45a and a red signal light 45b. When the blue signal light 45a is displayed bright blue in the display image 50, this indicates that the pedestrian traffic light 45 is showing a green light. When the red signal light 45b is displayed bright red in the display image 50, this indicates that the pedestrian traffic light 45 is showing a red light.

[0062] Trainee 30 performs training to cross crosswalk 41 and move onto sidewalk 43 on the opposite side. In this case, the objects to be passed through include a crosswalk with a traffic light and sidewalk 43. The virtual space generation unit 130 switches the color of the lit signal of pedestrian traffic light 45. For example, the virtual space generation unit 130 displays pedestrian traffic light 45 as "lit red" for a certain period of time, and then switches it to "lit green." After the certain period of "lit green" has elapsed, the virtual space generation unit 130 changes the display of pedestrian traffic light 45 to "flashing green." Then, after the certain period of "flashing green" has elapsed, the virtual space generation unit 130 changes the display of pedestrian traffic light 45 to "lit red."

[0063] The trainee 30 walks (or steps on the spot) in the room while looking at the display image 50 displayed on the HMD 31. The movement of the trainee 30 is captured by the camera 107. Then, the posture of the trainee 30 is analyzed based on the captured image.

[0064] 9 is a diagram showing an example of a trainee's posture analysis process. A captured image 60 obtained from the camera 107 includes a trainee image 61. The posture analysis unit 140 extracts the trainee image 61 from the captured image 60. The posture analysis unit 140 generates a trainee model 62 based on the trainee image 61 using a technique such as a neural network. The trainee model 62 is, for example, a skeletal model of the trainee 30. The generated trainee model 62 reproduces the posture of the trainee 30 depicted in the trainee image 61.

[0065] The trainee model 62 analyzes the posture of the trainee 30 for each frame of image acquired from the camera 107, and transmits the generated trainee model 62 to the trainee position determining unit 150 and the training result determining unit 170. The trainee position determining unit 150 counts, for example, the number of steps per unit time from the change in posture of the trainee model 62. Then, the trainee position determining unit 150 multiplies the stride length of each step of the trainee 30 by the number of steps per unit time to determine the moving speed of the trainee 30 in the virtual space.

[0066] The trainee position determining unit 150 determines the position of the trainee 30 in the virtual space after a unit time has elapsed, based on the current position and movement speed of the trainee 30 in the virtual space. The trainee position determining unit 150 transmits the determined position to the training result determining unit 170.

[0067] The training result determination unit 170 determines whether or not the trainee 30 has safely passed over an object to be passed, based on the position and posture in the virtual space of the trainee 30. For example, when the trainee 30 enters a crosswalk with a traffic light, the training result determination unit 170 checks the color of the pedestrian traffic light at the crosswalk.

[0068] If the pedestrian traffic light is lit red at the time of intrusion, the training result determination unit 170 determines that the trainee 30 has failed to pass over the object to be passed over. Furthermore, if the pedestrian traffic light turns red while the trainee 30 is on the crosswalk, the training result determination unit 170 determines that the trainee 30 has failed to pass over the object to be passed over. If the trainee 30 passes over the crosswalk while the pedestrian traffic light is lit green or flashing green, the training result determination unit 170 determines that the trainee 30 has succeeded in safely passing over the crosswalk.

[0069] When the object to be passed is an obstacle, the training result determination unit 170 obtains the height of the heel of the raised foot from the ground based on the posture of the trainee 30. Then, the training result determination unit 170 compares the height of the heel with the height of the obstacle to determine whether or not the object to be passed has been passed safely.

[0070] FIG. 10 is a diagram showing an example of obstacle passage determination processing. For example, when the position of the trainee 30 on the crosswalk 41 is within the range of one stride from the sidewalk 43, the training result determination unit 170 performs a pass-through determination of the step area of ​​the sidewalk 43. In this case, the training result determination unit 170 obtains the height of the heel from the ground when the position of the heel reaches the position of the step of the sidewalk 43. Then, the training result determination unit 170 compares the height of the heel at the position of the step of the sidewalk 43 with the height of the sidewalk 43, and if the height of the heel is equal to or higher than the height of the sidewalk 43, it determines that the trainee 30 has safely passed the obstacle. On the other hand, if the height of the heel does not reach the height of the sidewalk 43, the training result determination unit 170 determines that the trainee 30 is in a situation where he or she would fall in the real world.

[0071] The training content setting section 160 changes the difficulty level of subsequent training based on the result of the determination by the training result determination section 170. 11 is a diagram showing an example of training content after safely passing an obstacle. In a graph 70, the horizontal axis represents time and the vertical axis represents the height of the heel of the trainee 30. The curve in the graph 70 indicates the change over time in the height of the heel of the trainee 30. The height of the obstacle to be overcome is shown at the time when the trainee 30 reaches the obstacle.

[0072] In the example of the graph 70, the trainee 30 passes the first obstacle safely without falling. In this case, the training content setting unit 160 sets the height of the next obstacle in the virtual space in the traveling direction of the trainee 30 to the same height as the obstacle that was passed. The training content setting unit 160 may also set the height of the next obstacle in the virtual space in the traveling direction of the trainee 30 to be higher than the obstacle that was passed. When increasing the height of the obstacle, the training content setting unit 160 sets the height to a height equal to or lower than a preset upper limit value.

[0073] By setting the object height that will be an obstacle, the object to be passed displayed on the HMD 31 is also displayed at the set height, allowing the trainee 30 to grasp the height of the next object to be passed.

[0074] 12 is a diagram showing an example of training content after failing to pass an obstacle. In a graph 71, the horizontal axis represents time and the vertical axis represents the height of the heel of the trainee 30. The curve in the graph 71 indicates the change over time in the height of the heel of the trainee 30. The height of the obstacle to be cleared is shown at the time when the trainee 30 reaches the obstacle.

[0075] In the example of graph 71, it is determined that the trainee 30 fails to get over the first obstacle and falls. In this case, the training content setting unit 160 sets the height of the next obstacle in the moving direction of the trainee 30 in the virtual space to be lower than the obstacle that the trainee 30 stumbled over. For example, if the next obstacle is a sidewalk, the training content setting unit 160 corrects the height of the object representing the sidewalk to a value lower than the height of the obstacle that the trainee 30 stumbled over.

[0076] Since the height of the obstacle object has been corrected, the passing object displayed on the HMD 31 is also displayed at the corrected height, allowing the trainee 30 to understand that the height of the next passing object is lower than the height of the previous passing object.

[0077] Next, the procedure for the process of determining the training for the trainee 30 will be described in detail. 13 is a flow chart showing an example of a procedure for a training determination process. The process shown in FIG. 13 will be described below in order of step numbers.

[0078] [Step S101] The training content setting unit 160 sets training content. For example, the training content setting unit 160 receives an input specifying a training item. The training item is, for example, obstacle passing training, crosswalk crossing training, and the like.

[0079] The training content setting unit 160 also accepts input for setting the difficulty level for the specified training item. For example, if the training item is obstacle passing training, the training content setting unit 160 accepts input of an initial value for the height of the obstacle. If the training item is crosswalk crossing training, the training content setting unit 160 also accepts input of initial values ​​for the length of time for a green light and the length of the crosswalk. The training content setting unit 160 sets the value indicating the difficulty level as the initial value of a numerical value indicating the difficulty level for the selected training.

[0080] In the case of the second or subsequent training of the trainee 30, the training content setting unit 160 sets the training content corrected in accordance with the previous training result as the current training content. In this case, the training content setting unit 160 may receive an input instructing to correct the training content, and correct the training content in accordance with the input correction instruction.

[0081] [Step S102] The training result determination unit 170 determines whether the specified training item is obstacle passing training. If it is obstacle passing training, the training result determination unit 170 proceeds to step S103. If it is not obstacle passing training, the training result determination unit 170 proceeds to step S104.

[0082] [Step S103] The training result determination unit 170 performs a determination process for the obstacle passing training. In the determination process for the obstacle passing training, it is determined whether or not the trainee 30 has passed through the set obstacles. The details of the determination process for the obstacle passing training will be described later (see FIG. 14). After that, the training result determination unit 170 advances the process to step S107.

[0083] [Step S104] The training result determination unit 170 determines whether the specified training item is crosswalk crossing training. If it is crosswalk crossing training, the training result determination unit 170 proceeds to step S105. If it is not crosswalk crossing training, the training result determination unit 170 proceeds to step S106.

[0084] [Step S105] The training result determination unit 170 performs a determination process for the crosswalk crossing training. In the determination process for the crosswalk crossing training, it is determined whether or not the trainee 30 was able to safely cross the set crosswalk. The details of the determination process for the crosswalk crossing training will be described later (see FIG. 15). After that, the training result determination unit 170 advances the process to step S107.

[0085] [Step S106] The training result determination unit 170 performs a process of collecting other training information. For example, the training result determination unit 170 has the trainee 30 walk on a flat place and collects the moving speed while walking.

[0086] [Step S107] The training result determination unit 170 judges whether the training has ended. The training result determination unit 170 judges that the training has ended when the training for all the set training items has ended. For example, if a sidewalk with a step is set as an obstacle beyond a crosswalk with a traffic signal, the training result determination unit 170 judges that the training has ended when the crosswalk crossing training and obstacle passing training have ended.

[0087] Furthermore, when repeated training is specified, the training result determination unit 170 determines that the training should be terminated when an input to terminate the training is received. The training result determination unit 170 may determine that the training is terminated when the trainee 30 has achieved the training objective for the training with the highest level of difficulty (e.g., the upper limit of the height of the obstacle) of the specified training item.

[0088] If the training result determination section 170 determines that the training has ended, the process proceeds to step S108. If the training result determination section 170 determines that the training should be continued, the process proceeds to step S102.

[0089] [Step S108] The training content setting section 160 performs a training content modification process based on the training result. The training content modification process will be described in detail later (see FIG. 16). After that, the training content setting section 160 ends the training assessment process.

[0090] In this way, the training is repeated while modifying the training content according to the training results. 14 is a flowchart showing an example of a procedure for a determination process in obstacle passing training. The process shown in FIG. 14 will be described below in order of step numbers.

[0091] [Step S201] The training result determination unit 170 causes the HMD 31 to display an image of the virtual space corresponding to the current position of the trainee 30. [Step S202] The training result determination unit 170 judges whether or not there is an obstacle that has not yet been passed ahead of the trainee 30 in the virtual space. If there is an obstacle that has not yet been passed, the training result determination unit 170 advances the process to step S203. If all of the set obstacles have been passed, the training result determination unit 170 ends the determination process of the obstacle passing training.

[0092] [Step S203] The training result determination unit 170 sets the height of the obstacle. For example, the training result determination unit 170 sets the height of the obstacle ahead to a height specified in the initial setting. Furthermore, if the height of the obstacle is corrected in the processing of step S211, the training result determination unit 170 sets the height of the obstacle ahead to the corrected height.

[0093] If it is determined that the obstacle has been passed safely, the height of the obstacle ahead may be revised to a higher height in the process of step S209. Even if the height of the obstacle is revised in the process of step S209, the training result determination section 170 sets the height of the obstacle ahead to the revised height.

[0094] [Step S204] The posture analysis unit 140 analyzes the posture of the trainee 30 based on the image of the trainee 30 captured by the camera 107. [Step S205] The trainee position determining unit 150 updates the position of the trainee 30 in the virtual space based on the state of the trainee 30's stepping.

[0095] [Step S206] The training result determination unit 170 judges whether or not the trainee 30 has reached the position of an obstacle. If the trainee 30 has reached the position of the obstacle, the training result determination unit 170 advances the process to step S208. If the trainee 30 has not reached the position of the obstacle, the training result determination unit 170 advances the process to step S207.

[0096] [Step S207] The training result determination unit 170 generates an image of the virtual space as seen by the trainee 30, based on the updated position of the trainee 30, and updates the display image of the HMD 31 to the generated image. After that, the training result determination unit 170 causes the process to proceed to step S204.

[0097] [Step S208] The training result determination unit 170 judges whether or not the trainee 30 has overcome the obstacle. For example, the training result determination unit 170 calculates the heel height from the ankle height of the skeletal model of the trainee 30. If the heel height when passing over the obstacle is equal to or greater than the height of the obstacle, the training result determination unit 170 judges that the trainee has overcome the obstacle. If the heel height is less than the height of the obstacle, the training result determination unit 170 judges that the trainee has failed to overcome the obstacle. If the trainee has overcome the obstacle, the training result determination unit 170 advances the process to step S209. If the trainee has failed to overcome the obstacle, the training result determination unit 170 advances the process to step S210.

[0098] [Step S209] The training result determination unit 170 determines that the trainee 30 has safely passed over the obstacle, and proceeds to step S201. At this time, the training result determination unit 170 may correct the height of the next obstacle to be higher than the current obstacle.

[0099] [Step S210] The training result determination section 170 detects that the trainee 30 stumbles over an obstacle and falls. [Step S211] The training result determination section 170 corrects the height of the next obstacle to be lower than the height of the current obstacle. After that, the training result determination section 170 causes the process to proceed to step S201.

[0100] In this way, it is possible to safely verify indoors whether the trainee 30 can overcome the obstacle without falling. Moreover, if the trainee 30 falls, the height of the obstacle is corrected to be lower. This makes it possible to set obstacles within a range that the trainee 30 can overcome, and prevents the trainee 30 from losing motivation for rehabilitation due to repeated fall determinations.

[0101] Fig. 15 is a flow chart showing an example of a procedure for a process of determining whether or not a person is crossing the crosswalk during training. The process shown in Fig. 15 will be described below in order of step numbers. [Step S301] The training result determination section 170 causes the HMD 31 to display an image of the virtual space corresponding to the current position of the trainee 30.

[0102] [Step S302] The training result determination unit 170 judges whether or not there is a pedestrian crossing with a traffic light that has not yet been crossed ahead of the trainee 30 in the virtual space. If there is a pedestrian crossing that has not yet been crossed, the training result determination unit 170 advances the process to step S303. If all set pedestrian crossings have been crossed, the training result determination unit 170 ends the pedestrian crossing training.

[0103] [Step S303] The training result determination unit 170 sets the length of the crosswalk ahead and the duration of the traffic light. For example, the training result determination unit 170 sets the length of the crosswalk ahead (the width of the road to be crossed) to a length specified in the initial setting. Furthermore, if the length of the crosswalk is corrected in the processing of step S312, the training result determination unit 170 sets the length of the crosswalk ahead to the corrected length. Similarly, the training result determination unit 170 sets the duration of the pedestrian traffic light (duration of green light and duration of red light) of the crosswalk ahead to a length specified in the initial setting. Furthermore, if the duration of the traffic light is corrected in the processing of step S312, the training result determination unit 170 sets the duration of the pedestrian traffic light for the crosswalk ahead to the corrected duration.

[0104] If it is determined that the user has successfully crossed the crosswalk, the length of the next crosswalk may be increased or the green light time may be shortened in the process of step S308. Even if the length of the crosswalk or the length of the green light is modified in the process of step S308, the training result determination unit 170 sets the length of the crosswalk or the length of the green light ahead to the modified value.

[0105] [Step S304] The posture analysis unit 140 analyzes the posture of the trainee 30 based on the image of the trainee 30 captured by the camera 107. [Step S305] The trainee position determining unit 150 updates the position of the trainee 30 in the virtual space based on the state of the trainee 30's stepping.

[0106] [Step S306] The training result determination unit 170 determines whether or not the trainee 30 is crossing a pedestrian crossing based on the position of the trainee 30. The training result determination unit 170 determines that the trainee 30 is crossing a pedestrian crossing during the period from when the trainee 30 enters the pedestrian crossing until it is determined whether or not the trainee has successfully crossed the pedestrian crossing. If the trainee 30 is crossing a pedestrian crossing, the training result determination unit 170 advances the process to step S307. If the trainee 30 is not crossing a pedestrian crossing, the training result determination unit 170 advances the process to step S310.

[0107] [Step S307] The training result determination unit 170 judges whether or not the trainee 30 has crossed the crosswalk while the light is green. For example, if the trainee 30's position in the virtual space is on the opposite side of the crosswalk in the traveling direction while the pedestrian signal is green, the training result determination unit 170 judges that the trainee 30 has crossed the crosswalk while the light is green. If the trainee 30 has crossed the crosswalk while the light is green, the training result determination unit 170 advances the process to step S308. If the trainee has not crossed the crosswalk yet, the training result determination unit 170 advances the process to step S309.

[0108] [Step S308] The training result determination unit 170 determines that the user has successfully crossed the crosswalk safely, and proceeds to step S301. At this time, the training result determination unit 170 may correct the length of the next crosswalk to be longer than the current crosswalk. The training result determination unit 170 may also correct the green light time of the next crosswalk to be shorter than the current crosswalk.

[0109] [Step S309] The training result determination unit 170 determines whether the traffic light is red. If the traffic light is red, the training result determination unit 170 proceeds to step S311. If the traffic light is green, the training result determination unit 170 proceeds to step S310.

[0110] [Step S310] The training result determination unit 170 generates an image of the virtual space as seen by the trainee 30, based on the updated position of the trainee 30, and updates the display image of the HMD 31 to the generated image. After that, the training result determination unit 170 proceeds to step S304.

[0111] [Step S311] The training result determination unit 170 determines that the trainee 30 has failed to cross the pedestrian crossing safely. [Step S312] The training result determination unit 170 corrects the length of the next crosswalk to be shorter, or corrects the time for which the pedestrian signal is green at the next crosswalk to be longer. After that, the training result determination unit 170 causes the process to proceed to step S301.

[0112] In this way, it is possible to safely verify indoors whether the trainee 30 can cross the crosswalk while the light is green. Moreover, if the light turns red while the trainee 30 is crossing, the length of the crosswalk is shortened or the green light time is lengthened. This makes it possible to set a crosswalk that the trainee 30 can cross, and prevents the trainee 30 from losing motivation for rehabilitation due to repeated failure to cross the crosswalk.

[0113] Once the set training is completed, the training content for the next session will be adjusted based on the results. 16 is a flow chart showing an example of a procedure for processing to modify training content. The processing shown in FIG. 16 will be described below in order of step number.

[0114] [Step S401] The training content setting unit 160 collects the training results of the trainee 30. For example, the training content setting unit 160 obtains from the training log 123 the contents and results of past training.

[0115] [Step S402] The training content setting section 160 selects a type of training item. [Step S403] The training content setting unit 160 determines whether the selected training item is obstacle passing training. If the selected training item is obstacle passing training, the training content setting unit 160 proceeds to step S404. If the selected training item is not obstacle passing training, the training content setting unit 160 proceeds to step S405.

[0116] [Step S404] Based on the results of the obstacle overcoming training, the training content setting unit 160 sets the obstacle height to be set in the next training within a range not exceeding a predetermined upper limit. For example, the training content setting unit 160 sets the obstacle height in the next training to a value that is a predetermined amount greater than the maximum obstacle height that has been safely overtaken. However, if the obstacle height determined by the training content setting unit 160 exceeds the upper limit, the upper limit is set as the obstacle height in the next training.

[0117] [Step S405] The training content setting unit 160 judges whether the selected training item is training for crossing a pedestrian crossing with a signal. If the selected training item is training for crossing a pedestrian crossing with a signal, the training content setting unit 160 advances the process to step S406. If the selected training item is not training for crossing a pedestrian crossing with a signal, the training content setting unit 160 advances the process to step S408.

[0118] [Step S406] Based on the results of training to cross a crosswalk with a signal, the training content setting unit 160 sets the green light time of the crosswalk to be set in the next training within a range equal to or greater than a predetermined lower limit. For example, the training content setting unit 160 sets the green light time in the next training to a value that is a predetermined amount smaller than the minimum green light time of a crosswalk that was safely crossed. However, if the obtained green light time falls below the lower limit, the training content setting unit 160 sets the green light time in the next training to the lower limit.

[0119] [Step S407] Based on the results of training to cross a crosswalk with a signal, the training content setting unit 160 sets the length of the crosswalk to be set in the next training within a range not exceeding a predetermined upper limit. For example, the training content setting unit 160 sets the length of the crosswalk in the next training to a value that is a predetermined amount larger than the maximum length of the crosswalk that was safely crossed. However, if the calculated length of the crosswalk exceeds the upper limit, the training content setting unit 160 sets the upper limit as the length of the crosswalk in the next training.

[0120] [Step S408] The training content setting unit 160 determines whether there are any unselected training items. If there are any unselected training items, the training content setting unit 160 proceeds to step S402. If there are no unselected training items, the training content setting unit 160 ends the training content correction process.

[0121] In this way, each time training is performed, the training content can be modified based on the training results. As a result, the trainee 30 can perform training at an appropriate level of difficulty, enabling efficient rehabilitation.

[0122] For example, the walking training support system shown in the second embodiment can provide effective walking training based on the patient's condition. The walking training support system can also modify the training content so as not to lower the patient's motivation for rehabilitation.

[0123] [Other embodiments] In the second embodiment, a VR image is displayed on the HMD 31, but an AR (Augmented Reality) or MR (Mixed Reality) image may be displayed. For example, the computer 100 can display on the HMD 31 an image in which an obstacle model generated by CG (Computer Graphics) or the like is superimposed on a real landscape that can be seen through the HMD 31.

[0124] Although the embodiments have been described above, the configurations of the parts shown in the embodiments can be replaced with other parts having similar functions. Any other components or steps may be added. Furthermore, any two or more configurations (features) of the above-mentioned embodiments may be combined. [Explanation of symbols]

[0125] 1 Trainer 2 Head-mounted display (HMD) 3. Camera 4a First object data 4b First Passage Criteria 5 Training images 6a Second object data 6b Second Passing Condition 7. Virtual Space 7a Trainer model 7b First Object 7c Second Object 10. Information processing device 11 Storage section 12 Processing section

Claims

1. displaying a training image including a first object arranged in a virtual space on a head mounted display worn by a trainee; Acquire an image of the trainee taken by a camera; determining whether or not the trainee has safely passed over the first object based on a posture change of the trainee shown in the captured image and a first passage determination condition indicating a condition for determining that the trainee has safely passed over the first object; determining a second object to be placed in the virtual space in a subsequent walking training based on a result of the determination as to whether the trainee has safely passed over the second object and a second passage determination condition indicating a condition for determining that the trainee has safely passed over the second object; A walking training support program that has processing executed by a computer.

2. In the process of determining the second object and the second passing determination condition, when it is determined that the vehicle has not passed safely over the first object, a height of the second object is determined to be lower than a height of the first object. The walking training support program according to claim 1.

3. In the process of determining the second object and the second passing determination condition, when it is determined that the object has safely passed over the first object, a height of the second object is determined to be higher than a height of the first object. The walking training support program according to claim 1.

4. In the process of determining the second object and the second passing determination condition, if it is determined that the trainee has not passed safely over the first object, a length of the second object in the moving direction of the trainee in the virtual space is determined to be shorter than a length of the first object. The walking training support program according to claim 1.

5. In the process of determining the second object and the second passage determination condition, when it is determined that the trainee has safely passed over the first object, a length of the second object in the moving direction of the trainee in the virtual space is determined to be longer than a length of the first object. The walking training support program according to claim 1.

6. the first passing determination condition indicates a first time allowed for safely passing over the first object; In the process of determining the second object and the second passing determination condition, when it is determined that the vehicle has not passed safely over the first object, the second time is determined to be longer than the first time. The walking training support program according to claim 1.

7. the first passing determination condition indicates a first time allowed for safely passing over the first object; In the process of determining the second object and the second passing determination condition, when it is determined that the vehicle is passing safely over the first object, the second time is determined to be shorter than the first time. The walking training support program according to claim 1.

8. displaying a training image including a first object arranged in a virtual space on a head mounted display worn by a trainee; Acquire an image of the trainee taken by a camera; determining whether or not the trainee has safely passed over the first object based on a posture change of the trainee shown in the captured image and a first passage determination condition indicating a condition for determining that the trainee has safely passed over the first object; determining a second object to be placed in the virtual space in a subsequent walking training based on a result of the determination as to whether the trainee has safely passed over the second object and a second passage determination condition indicating a condition for determining that the trainee has safely passed over the second object; A walking training support method in which processing is executed by a computer.

9. a processing unit that displays a training image including a first object placed in a virtual space on a head mounted display worn by a trainee, acquires a photographed image of the trainee taken by a camera, and determines whether or not the trainee has safely passed over the first object based on a change in the trainee's posture shown in the photographed image and a first passing determination condition indicating a condition for determining that the trainee has safely passed over the first object, and determines a second object to be placed in the virtual space in a subsequent walking training and a second passing determination condition indicating a condition for determining that the trainee has safely passed over the second object based on the determination result of whether or not the trainee has safely passed over the second object; An information processing device having the above configuration.

Images