Training systems, training methods, and programs

Abstract

To efficiently improve the level of skill of a training target of a user who wears a mounting fixture that assists the motion of a joint such as the knee and the lower back with a torque of a motor of a drive mechanism which reduces a load applied to a user and is provided in the vicinity of the joint of the user.SOLUTION: A training system displays an image of a virtual target on a display on the basis of a photographed image of a real space including the view point of a user generated by photography of a camera within an imaging range, a user position, a display condition of the virtual target and the output mode in the image according to the display condition of the virtual target, and applies operation control that disturbs the movement of the user to a drive mechanism which is provided in a mounting fixture that assists the movement of the user and assists the motion of a joint of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a training system, a training method, and a program. [Background technology]

[0002] There are harnesses worn by users to assist their movements. For example, the harness has a drive mechanism installed near the user's joints to reduce the load on the user, and the torque of a motor assists the movement of joints such as the knees and hips. Such harnesses are likely to be used in many situations in the future, but there is a demand for technology that allows users wearing such harnesses to train.

[0003] A related technology is disclosed in Patent Document 1. Patent Document 1 discloses a technology for acquiring data describing the actions of a person in a space, determining a relationship between the action and a physical feature, and evaluating whether the determined relationship satisfies a goal stored in a program plan. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2017-515531 Summary of the Invention [Problem to be solved by the invention]

[0005] Such harnesses, which have drive mechanisms for reducing the load on the user provided near the user's joints and which assist the movement of joints such as the knees and hips with the torque of a motor, are likely to be used in many situations in the future.Therefore, there is a demand for technology that efficiently improves the proficiency of trainees through training by users wearing such harnesses.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a training system, a training method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0007] According to a first aspect of the present invention, a training system includes a captured image of real space generated by a camera and including a user's viewpoint in the capture range, a mixed reality processing means for displaying an image of the virtual object on a display worn by the user based on the user's position, display conditions of the virtual object to be displayed on the display, and an output mode on the display according to the display conditions of the virtual object, and a training means for applying motion control that hinders the user's movement to a drive mechanism that is provided in a wearable device worn by the user and assists the movement of the user's joints.

[0008] According to a second aspect of the present invention, a training method displays an image of a virtual object on a display worn by the user based on a captured image of real space generated by a camera and including the user's viewpoint in the capture range, the user's position, display conditions for a virtual object to be displayed on the display, and an output mode on the display according to the display conditions for the virtual object, and applies motion control to a drive mechanism that is provided in a wearable device worn by the user and assists the movement of the user's joints, thereby hindering the movement of the user.

[0009] According to a third aspect of the present invention, the program causes a computer of a training system to function as a mixed reality processing means that displays an image of a virtual object on a display worn by the user based on a captured image of real space generated by a camera and including the user's viewpoint in the capture range, the user's position, display conditions for the virtual object to be displayed on the display, and an output mode on the display according to the display conditions for the virtual object, and as a training means that applies motion control to a drive mechanism that is provided in a harness worn by the user and assists the movement of the user's joints, thereby hindering the movement of the user. [Effects of the Invention]

[0010] According to the present invention, a drive mechanism that reduces the load on the user is provided near the user's joints, and the torque of the motor can be used to efficiently improve the proficiency of the training subject wearing a device that assists the movement of joints such as the knees and hips. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram of a training system according to an embodiment of the present invention. [Figure 2] 1 is a hardware configuration diagram of a training processing device according to an embodiment of the present invention. [Figure 3] FIG. 2 is a functional block diagram of a training processing device according to an embodiment of the present invention. [Figure 4] 1 is a diagram showing the configuration of a powered suit according to one embodiment of the present invention. [Figure 5] FIG. 2 is a diagram showing an example of a display displayed on a display of an HMD according to an embodiment of the present invention. [Figure 6] 1 is a flowchart illustrating a series of training steps using a training system according to an embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing a processing flow of a training processing device according to an embodiment of the present invention. [Figure 8] FIG. 1 is a diagram showing a minimum configuration of a training processing device according to an embodiment of the present invention. [Figure 9] FIG. 10 is a diagram showing a processing flow by a training processing device with a minimum configuration according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] A training system, a training method, and a program according to an embodiment of the present invention will be described below with reference to the drawings.

[0013] FIG. 1 is a schematic diagram of the training system according to this embodiment. The training system 100 includes at least an HMD (head mounted display) 2, which is a head-mounted device, a tool 3, a powered suit 400, and a training processing device 1. The HMD 2 is equipped with a camera that captures the user's field of view, a display, etc. The tool 3 may be any tool that can be used for training. The HMD 2, the tool 3, and the powered suit 400 are equipped with various sensors and communication devices, and are connected to the training processing device 1 via a communication network. The training processing device 1 may be a device attached to the HMD 2.

[0014] FIG. 2 is a hardware configuration diagram of the training processing device according to this embodiment. 2, the training processing device 1 is a computer equipped with various hardware components such as a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an HDD (Hard Disk Drive) 104, a communication module 105, and a database 106. As an example, the training processing device 1 is installed in a data center.

[0015] FIG. 3 is a functional block diagram of the training processing device according to this embodiment. The training processing device 1 executes a training program, thereby causing the training processing device 1 to fulfill the functions of a control unit 31, a mixed reality processing unit 32, a training unit 33, an acquisition unit 34, and a simulation unit 35. The control unit 31 controls each function of the training processing device 1. The mixed reality processing unit 32 displays an image of the virtual object on the display 222 of the HMD 2 based on the position of the user, the display conditions of the virtual object, and the output mode of the image according to the display conditions of the virtual object. The training unit 33 applies a load to the user by applying motion control that hinders the user's movement to a drive mechanism that is provided in a powered suit, which is a wearable device worn by the user, and that assists the movement of the user's joints. The acquisition unit 34 acquires sensing information of a predetermined biological movement of the user. For example, biological movements include eye movement and movement of a part of the body, and the acquisition unit 34 acquires sensing information such as the direction of eye movement and acceleration based on a predetermined axis of the arm or leg. The simulation unit 35 synchronizes the user's sensing information with the sensing information of other users based on the timing of the users' actions and outputs the information to a display device such as a terminal. The simulation unit 35 may have a function of performing various analytical processes using AI (Artificial Intelligence) or the like. The analytical processes may be performed by a computer device different from the training processing device 1.

[0016] FIG. 4 is a diagram showing the configuration of the powered suit according to this embodiment. The powered suit 400 is one aspect of a load reduction device. The powered suit 400 includes a skeleton 41, a belt 42, a waist actuator 43, a knee actuator 44, an ankle actuator 45, a sole plate 46, a foot attachment 47, a skeleton 91, a shoulder actuator 93, an elbow actuator 94, a wrist actuator 95, a skeleton 96, a glove 97, a control device 21, a battery 22, and the like. The skeleton 41 is broadly divided into a first skeleton 411, a second skeleton 412, and a third skeleton 413, for example. The skeleton 91 is broadly divided into a fourth skeleton 911, a fifth skeleton 912, and a sixth skeleton 913. The skeleton 96 is connected to a glove 97.

[0017] 4, the powered suit 400, as an example, is provided with a first skeleton 411, and left and right waist actuators 43 rotatably connect the first skeleton 411 to a corresponding second skeleton 412 that is located along the left or right thigh of the user wearing the powered suit 400. The left and right knee actuators 44 rotatably connect the corresponding left or right second skeleton 412 to a corresponding third skeleton 413 that is located along the left or right lower leg of the user wearing the powered suit 400. The ankle actuators 45 rotatably connect the corresponding left or right third skeleton 413 to a corresponding sole plate 46 that is located on the sole of the left or right foot attachment 47 of the user wearing the powered suit 400. The powered suit 400 also has a fourth skeletal section 911, and left and right shoulder actuators 93 rotatably connect the fourth skeletal section 911 to a corresponding fifth skeletal section 912 that is located along the left or right upper arm of the user wearing the powered suit 400. The left and right elbow actuators 94 rotatably connect the corresponding left or right fifth skeletal section 912 to a corresponding sixth skeletal section 913 that is located along the left or right forearm of the user wearing the powered suit 400. The wrist actuator 95 rotatably connects the corresponding left or right sixth skeletal section 913 to a skeletal section 96 that moves in conjunction with the movement of a glove 97 on the left or right hand of the user wearing the powered suit 400.

[0018] A user wearing the powered suit 400 places their left and right feet into the corresponding foot attachments 47 and secures the first skeleton 411 to their waist with the belt 42 so that it fits snugly. The powered suit 400 is structured to transfer most of the load of the powered suit 400 to the ground contact surface of the soles of the feet via the skeleton 41 and each actuator 43, 44, 45. The user turns on the control device 21 of the powered suit 400. The control device 21 controls each actuator 43, 44, 45 so that as much of the device weight of the powered suit 400 as possible is transferred to the walking surface via the skeleton 41 and each actuator 43, 44, 45. Similarly, the control device 21 controls the actuators 93, 94, 95 of the arms, shoulders, and wrists.

[0019] As shown in Figure 4, the user wears HMD2. HMD2 is equipped with a camera 221 and a transparent display 222. The camera 221 captures images of each subject within the user's field of view and generates an image of them. The display 222 is made of a material that allows external light to pass through toward the user's eyes, and displays an image on its inside. This allows the user to simultaneously view the outside scenery and the displayed image while wearing HMD2.

[0020] The training system 100 of this embodiment is used, for example, for shooting training. The user wears the powered suit 400 and carries a tool such as a mock gun to be used for shooting. The user performs shooting training at a predetermined training facility, training ground, or training area by visually recognizing an image of a virtual target displayed on the HMD 2 by processing by the training processing device 1. The user can also use the training system 100 to train in areas without facilities, such as urban areas, mountainous areas, or desert areas other than the training facility, training ground, or training area.

[0021] FIG. 5 is a diagram showing an example of a display displayed on the display of the HMD. As an example, a user performs shooting training in a training area. When the user arrives at a predetermined location in the training area, the training processing device 1 instructs the user's HMD 2 to display a first image 51 simulating a situation in which a virtual target vehicle is located at a predetermined distance, such as 100 meters, from the user's location. The training processing device 1 also instructs the user's HMD 2 to display a second image 52 depicting a virtual target person behind the vehicle at a predetermined timing. The training processing device 1 also acquires, as teacher data, from a database or the like, the target position and the aiming time from the time the second image 52 is displayed to the time the gun is aimed at the target position, which are included in the training results of other exemplary users. The training processing device 1 then instructs the user's HMD 2 to display a third image 53 depicting a target at the target position at a timing calculated by adding the aiming time from the time the second image 52 is displayed. The HMD 2 sequentially receives these instruction signals from the training processing device 1 and, based on the instructions, sequentially displays the first image 51, the second image 52, and the third image 53 on the display 222. These first image 51, second image 52, and third image 53 are superimposed on the space visually perceived by the actual user, and the training processing device 1 calculates and instructs the position, size, and display timing of these images to be displayed on the HMD 2. Therefore, the user can train with the sensation that virtual objects, people, etc. exist in the actual space just as they do in reality. In other words, the training processing device 1 is a device that allows the user to experience training using mixed reality (MR) processing. The example in FIG. 5 above is just one example, and the training processing device 1 may display only a person as an image of the virtual object. For example, the training processing device 1 may display an image of a virtual person behind a real car.

[0022] When a virtual target car shown as a first image 51 is displayed on the display 222 and then a virtual target person shown as a second image 52 is displayed, the user points the muzzle of the simulated gun, which is the tool 3, at the second image 52, which is the target. The simulated gun, which is the tool 3, is provided with acceleration sensors at multiple locations and also has a control device. The control device of the tool 3 transmits tool operation information, which includes values from the acceleration sensors and time, to the HMD 2. The HMD 2 receives the tool operation information from the tool 3. During this time, a third image 53, which indicates the aiming position of the model user, is displayed on the display 222. The HMD 2 displays a fourth image 54, which indicates the aiming position corresponding to the direction in which the user points the muzzle of the simulated gun, which is the tool 3, based on the relationship between the values from each acceleration sensor included in the tool operation information and the time. This allows the user to compare the position and timing at which third image 53, which indicates the aiming position of the exemplary user, is displayed with the position and timing at which fourth image 54, which indicates the position at which the user himself aims at the target using tool 3, and understand the difference between the position and timing at which he aims his own imitation gun at the target and the position and timing at which the exemplary user aims his imitation gun at the target.

[0023] The training processing device 1 generates a control signal to apply motion control to the drive mechanisms of the powered suit 400 that assist the user's joint movement, thereby hindering the user's movement, based on detected information such as the user's position, the time elapsed since the start of training, the user's distance traveled, and movements, as well as pre-stored information such as the virtual weight of luggage loaded on the loading platform 20. The training processing device 1 transmits the control signal to the powered suit 400. The control device 21 of the powered suit 400 receives the control signal. Based on the control signal, the control device 21 controls the drive of the waist actuator 43, knee actuator 44, ankle actuator 45, and other components of the powered suit 400. For example, the control device 21 applies a predetermined amount of torque to one or more of these actuators in a direction that hinders the user's joint movement. This allows the user to be subjected to a load simulating the load imposed by equipment carried or physical fatigue due to the user's actions or terrain. The training processing device 1 also controls the drive of actuators for the arms, shoulders, wrists, and other components so that a recoil force associated with the tool 3, such as a simulated gun, is applied to the user when a bullet is fired using the tool 3. This allows the training processing device 1 to apply a load to the user when using the tool 3.

[0024] In this way, the training processing device 1 allows the user to experience training using mixed reality processing, and during the training, the user can feel the load on the body based on the control of the training processing device 1 and compare the movements with those of other model users. This allows the user to perform training with a higher level of reality.

[0025] In this embodiment, an example will be described in which the training processing device 1 is used for shooting training, but the training processing device 1 may also be used for other training by the user. For example, the training processing device 1 may be used for training when the user trains in various situations such as operating machines or tools, playing sports, or providing nursing care.

[0026] FIG. 6 is a flowchart illustrating a series of steps in training using the training system. An administrator who manages the training processing device 1 creates a training program (step S101). The administrator installs the training program into the training processing device 1. The training processing device 1 stores teacher data of an exemplary user in advance (step S102). The user puts on the HMD 2, tool 3, and powered suit 400 and moves to the training area (step S103). The user turns on the HMD 2, tool 3, and powered suit 400 (step S104). This establishes a communication connection between the HMD 2, tool 3, and powered suit 400 and the training processing device 1. The HMD 2, tool 3, and powered suit 400 may also be communicatively connected to each other. The user views the surroundings through the display 222 of the HMD 2 (step S105). At this time, the user may be able to view an image of a virtual object displayed in the initial state based on instructions from the training processing device 1. The user sets the weight of the luggage and the torque to be applied to each actuator of the powered suit 400 from the setting screen of the powered suit 400 (step S106). Here, even if actual luggage is not loaded onto the carrier 20 of the powered suit 400, the control device 21 may control each actuator of the powered suit 400 based on the setting of the virtual weight of the user's luggage so that the user feels the weight of the luggage. This is an example of a processing mode in which the training processing device 1 applies operation control to the drive mechanism to hinder the user's movement based on the weight of a virtual object carried by the user.

[0027] Then, triggered by a predetermined operation by the user, the camera 221 of the HMD2 begins taking or recording images of the user's field of view (step S107). This starts the training. The HMD2 may send a signal to start the training to the tool 3, the powered suit 400, and the training processing device 1. The user moves around the training area (step S108). During this time, position detection devices provided in the HMD2, the tool 3, and the powered suit 400 sense the position (latitude, longitude, altitude), and one or more acceleration sensors provided in the HMD2, the tool 3, and the powered suit 400 sense the acceleration in three predetermined orthogonal axial directions. The HMD2, the tool 3, and the powered suit 400 repeatedly send sensing information including these sensed values to the training processing device 1.

[0028] The training processing device 1 generates an image of a virtual object based on an image captured by the camera 221, the user's position information included in the sensing information, and the elapsed time since the start of training. The training processing device 1 transmits the image of the virtual object to the HMD 2. The HMD 2 displays the image of the virtual object on the display 222 based on the position information, distance information, and size information of the virtual object included in the image (step S109). This processing is one aspect of mixed reality processing in which the training processing device 1 displays the image of the virtual object on the display based on the image captured by the camera 221, the user's position, display conditions of the virtual object to be displayed on the display worn by the user, and an output mode on the display according to the display conditions of the virtual object. The display conditions of the virtual object may be the user's position information, the elapsed time since the start of training, the user's visual range, the shooting direction of the camera 221, etc. The HMD 2 may display the image captured by the camera 221 on the display 222 and superimpose the image of the virtual object on the image, or may superimpose only the image of the virtual object on the display 222 so that the user can see the image of the virtual object at a predetermined position within the visual range. The training processing device 1 also generates control signals for each actuator of the powered suit 400. The training processing device 1 transmits the control signals to the powered suit 400.

[0029] The user visually recognizes the image of the virtual target displayed on the display 222 of the HMD2, and when a predetermined target is displayed, performs a shooting action using the tool 3 of the imitation gun (step S110). When the user inputs a firing operation, the tool 3 may emit a gunshot sound from a speaker. The tool 3 may also transmit sensing information, such as the direction of the shot, to the HMD2 based on the user's firing action of the imitation gun. The HMD2 may display on the display 222 the trajectory of the bullet fired from the imitation gun and an image of the bullet hole in the image of the virtual target based on the sensing information of the tool 3. The tool 3 also transmits information indicating that the shooting action has been performed to the control device 21 of the powered suit 400. In response to this, the control device 21 drives and controls each actuator of the powered suit 400 to apply vibrations to the user's body based on the firing of the imitation gun of the tool 3 (step S111).

[0030] The training processing device 1 records sensing information and the like obtained during the user's training in a database, linking it to the user ID (step S112). The training processing device 1 compares the user's movements during training based on the sensing information with the movements of a model user (step S113). The training processing device 1 may generate comparison data indicating the results of the comparison. The training processing device 1 also performs an analysis process on the user's past data (step S114). Using AI functions, the training processing device 1 improves or creates a new training program based on the sensing information, comparison data, and analysis results (step S115).

[0031] FIG. 7 is a diagram showing the processing flow of the training processing device. The acquisition unit 34 of the training processing device 1 associates the user ID with the device IDs of the HMD 2, tool 3, and powered suit 400 based on the user's operation on each input device of the HMD 2, tool 3, and powered suit 400, and records the associated IDs in the database 106 (step S201). This allows the training processing device 1 to associate these devices. The HMD 2, tool 3, and powered suit 400 communicate with each other based on the user's operation, and at least one of the devices transmits a registration request to the training processing device 1, including the device IDs of the associated HMD 2, tool 3, and powered suit 400 and their current location information. The acquisition unit 34 of the training processing device 1 associates the device IDs and location information of the HMD 2, tool 3, and powered suit 400 and records them (step S202). The training processing device 1 also receives a command to start processing from one of the HMD 2, tool 3, and powered suit 400 (step S203). In this embodiment, the control device 21 of the powered suit 400, which is communicatively connected to the HMD 2 and the tool 3, transmits a start signal indicating an instruction to start processing to the training processing device 1, and the training processing device 1 receives the signal. The training processing device 1 starts training processing for the user based on the start signal.

[0032] The control device 21 transmits user position information detected by a position sensor included in the control device 21 to the training processing device 1 at predetermined time intervals, such as every second. The control device 21 transmits sensing information, such as acceleration detected by each acceleration sensor of the HMD 2 and the tool 3, and orientation based on a predetermined direction detected by a direction sensor, to the training processing device 1. The control device 21 may transmit captured images acquired from the camera 221 of the HMD 2 to the training processing device 1. The control device 21 may transmit the position information, sensing information, captured images, and the like to the training processing device 1 at predetermined intervals. If sensors such as an IMU (Inertial Measurement Unit) or an encoder are attached to the powered suit, the control device 21 may transmit movement information of the wearer acquired from these sensors to the training processing device 1.

[0033] The acquisition unit 34 of the training processing device 1 acquires each piece of information (position information, sensing information, captured image) transmitted from the control device 21 (step S204). The acquisition unit 34 transfers each piece of information acquired from the control device 21 to the control unit 31. The control unit 31 outputs information to be used in mixed reality processing from the information acquired from the acquisition unit 34 to the mixed reality processing unit 32. The information to be used in mixed reality processing may be, for example, information such as the user's position information and the direction in which the camera is facing.

[0034] The mixed reality processing unit 32 acquires, for example, based on the user's position information, from the database 106, identification information of a virtual object to be displayed on the display 222 of the HMD 2 worn by the user (step S205). For example, the mixed reality processing unit 32 acquires one or more pieces of identification information of a virtual object to be stored in the database 106 in association with position information within a circle of a predetermined radius centered on the user's position information. The mixed reality processing unit 32 transmits the identification information of the virtual object to the control device 21 (step S206).

[0035] The control device 21 transmits identification information of the virtual target to the HMD2. Based on the identification information of the virtual target, the HMD2 acquires an image corresponding to the virtual target stored in advance and its position information. Based on the shooting direction of the camera 221, the position information, and the position information of the virtual target, the HMD2 determines whether the camera 221 of the HMD2 is facing in a direction that includes within its range the position information indicated by the virtual target. If the control device 21 determines that the camera 221 of the HMD2 is facing in a direction that includes within its range the position information indicated by the virtual target, it displays an image of the virtual target on the display of the HMD2. At this time, the HMD2 calculates the distance to the user based on the position information of the virtual target, and displays on the display 222 the image of the virtual target at a size corresponding to that distance. As an example, it is assumed that the HMD2 has acquired identification information of an image showing a person to be shot as a virtual target corresponding to a position 500 m north of the user's position. In this case, when the user points the camera 221 in the north direction, the HMD 2 displays on the display 222 a size that corresponds to the north direction and a distance of 500 m.

[0036] When the user moves the direction in which the camera 221 is pointed, the HMD 2 may control the display position of the virtual target image on the display 222 based on that movement so that the image appears to be located in the north direction relative to the user's position. This allows the user to confirm the target person in the north direction. Based on a predetermined program, the HMD 2 may display an image in which the target person is hiding behind a real four-wheeled vehicle that appears in the image captured by the camera 221.

[0037] In this way, the training processing device 1 performs control to display an image of the virtual object on the display 222 of the HDM 2 so that the virtual object can be viewed at a predetermined position based on sensing information such as the user's position information and the direction of the camera 221. The training processing device 1 may perform control so that an image of the virtual object suitable for training is displayed on the display 222 in accordance with the user's movement.

[0038] The acquisition unit 34 of the training processing device 1 acquires information on the weight of the luggage set by the user from the control device 21 and records the information in the database 106 in association with the user's ID. The training unit 33 of the training processing device 1 calculates the elapsed time from the start time of the user's training process (step S207). The training unit 33 also acquires the user's position information from the database 106 sequentially from the start time of the user's training process and calculates the user's movement distance during the training process (step S208). The training unit 33 identifies the user's motion based on the sensing information most recently acquired from the control device 21 (step S209). For example, the training unit 33 identifies the user's motion, such as walking, running, holding a gun, shooting, or lying prone. These motions may be identified based on sensing information such as the rotation angles of the actuators 43, 44, and 45 of the powered suit 400. The training unit 33 inputs information such as the weight of the luggage, the distance traveled, the elapsed time, and the movement into a predetermined load calculation program, and calculates the load on the corresponding actuator as a result (step S210). The training unit 33 may calculate the load on each actuator using a load calculation program corresponding to each actuator. The training unit 33 transmits a load control signal including an identifier of the actuator to be calculated and information indicating the load to the control device 21 of the powered suit 400 (step S211).

[0039] The control device 21 receives a load control signal. The control device 21 controls the corresponding actuator using the actuator identifier and the load information for the actuator included in the load control signal. This applies a load to each of the actuators 43, 44, and 45 of the powered suit 400, impeding the user's movement. The training processing device 1 controls the load of one of the actuators 43, 44, and 45 using a load calculation program based on at least one of the following information: luggage weight, travel distance, elapsed time, and movement. This allows a load to be applied according to the user's training status. By applying an appropriate load according to the user's status, the user's training skills can be improved. The processing of the training processing device 1 is one aspect of a process of applying motion control to a drive mechanism (actuator) that assists the user's joint movement and is provided in the harness (powered suit 400) worn by the user. The processing of the training processing device 1 is also one aspect of a process of applying motion control to a drive mechanism (actuator) that impedes the user's movement based on the weight of a virtual object (luggage) carried by the user. The processing of the training processing device 1 described above is one aspect of processing in which the drive mechanism (actuators) are subjected to operational control to hinder the user's movement based on the user's travel distance. The training processing device 1 may also have the control device 21 control the actuators of the powered suit 400 based on information about a virtual travel distance set by the user, even if the user does not actually travel, so that the user feels the weight of a part of their body, such as their legs, when traveling a long distance. This processing is also one aspect of processing in which the training processing device 1 applies operational control to the drive mechanism to hinder the user's movement based on the user's travel distance.

[0040] The training unit 33 outputs teacher data related to the user's movements, recorded based on the movements of other users, to the display of the HMD2 at a predetermined timing when the image of the virtual target is displayed (step S212). For example, when displaying an image of a person to be shot, which is a virtual target, on the display of the HMD2, the training unit 33 acquires teacher data of the model user from the database 106. The training unit 33 transmits the teacher data to the control device 21. The control device 21 transmits a teacher image based on the teacher data to the HMD2. The HMD2 displays the teacher image on the display 222. The teacher data may be a display of an image of the aiming position timed to align the aim, as described above, or an image showing the posture of the model user, instruction comments tailored to the movements of the model user, or the like. The user corrects their own movements based on the teacher image displayed on the display 222 of the HMD2. This allows the user to train themselves according to the movements of the model user.

[0041] The acquisition unit 34 acquires sensing information indicating the user's eye movements and the acceleration of the legs, arms, and hands in each axis direction in a predetermined three-dimensional space. The acquisition unit 34 associates the user's ID with the sensing information and records it in the database 106. The sensing information may be an image including the user's field of view generated by the camera 221. At a predetermined timing when the user finishes training or based on an operation instruction from the user, the simulation unit 35 uses the sensing information to display a video showing the user's actual movements and a video showing the movements of the model user on a display device such as a terminal device so that they can be compared (step S213). For example, the simulation unit 35 may superimpose an image captured by the camera 221 at the time when an image of a target to be shot is displayed on the display 222 of the HMD 2 on the display device such as the terminal device, showing the eye movements of the training user and the model user. This allows the user to check the terminal device after training and compare their own eye movements with those of the model user when the target to be shot appears. The simulation unit 35 may display on the terminal device moving images of the user's leg, arm, and hand movements, the movement of the tool 3, and the corresponding moving images of the model user. This processing is one aspect of processing in which the simulation unit 35 outputs the sensing information of the user and the sensing information of other users in synchronization with the timing related to the movements of those users.

[0042] According to the above-described process, the training processing device 1 uses MR technology to display an image of a virtual object required for training on the display 222 viewed by the user during training at an appropriate timing according to the user's position and viewing direction. The training processing device 1 also applies motion control to a drive mechanism that assists the user's joint movement and is provided in a harness worn by the user, at an appropriate timing to impede the user's movement. This provides a training system technology that improves the effectiveness of the user's training.

[0043] FIG. 8 is a diagram showing the minimum configuration of a training processing device. FIG. 9 is a diagram showing a processing flow by a training processing device with a minimum configuration. The training processing device 1 includes at least a mixed reality processing means 81 and a training means 82. The mixed reality processing means 81 displays an image of a virtual object on a display based on a captured image of a real space generated by a camera and including the user's viewpoint in the capture range, the user's position, the display conditions of the virtual object, and the output mode of the image according to the display conditions of the virtual object (step S901). The training means 82 applies motion control to a drive mechanism that is provided in the wearing equipment worn by the user and assists the movement of the user's joints, to hinder the movement of the user (step S902).

[0044] Each of the above-mentioned devices has a computer system built in. The steps of each of the above-mentioned processes are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the above-mentioned processes. Here, computer-readable recording medium refers to a magnetic disk, magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, etc. Alternatively, the computer program may be distributed to a computer via a communication line, and the computer that receives the program may execute the program.

[0045] The program may also be a program for realizing some of the functions described above, or may be a so-called differential file (differential program) that can realize the functions described above in combination with a program already recorded in the computer system. [Explanation of symbols]

[0046] 1. Training processing device 2. HMD 3. Tools 100... Training System 400...Powered Suit 11. Skeleton 12. Belt 13. Waist actuator 14. Knee actuator 15. Ankle Actuator 16···Shoe sole plate 17...foot attachment 20. Cargo bed 21. Control device 31 Control unit 32 Mixed reality processing unit 33 Training Department 34...Acquisition part 35 Simulation Section 221···Camera 222···Display

Claims

1. a mixed reality processing means for displaying an image of the virtual object on a display worn by the user based on a captured image of a real space generated by capturing an image of the real space including the user's viewpoint in the capture range, the position of the user, display conditions of the virtual object to be displayed on the display worn by the user, and an output mode on the display according to the display conditions of the virtual object; a training means for applying motion control to a drive mechanism that is provided in a wearing tool worn by the user and assists the movement of the user's joints, and that hinders the movement of the user; Equipped with The training means applies motion control to the drive mechanism to hinder the movement of the user based on the weight of the virtual object carried by the user. Training system.

2. a mixed reality processing means for displaying an image of the virtual object on a display worn by the user based on a captured image of a real space generated by capturing an image of the real space including the user's viewpoint in the capture range, the position of the user, display conditions of the virtual object to be displayed on the display worn by the user, and an output mode on the display according to the display conditions of the virtual object; a training means for applying motion control to a drive mechanism that is provided in a wearing tool worn by the user and assists the movement of the user's joints, and that hinders the movement of the user; Equipped with The training means applies motion control to the drive mechanism to hinder the movement of the user based on the moving distance of the user. Training system.

3. The training means outputs training information relating to the user's movements, which is recorded based on the movements of other users, at a predetermined timing when the image of the virtual object is displayed, to the head-mounted device provided with the display.

3. The training system according to claim 1 or 2.

4. an acquisition means for acquiring sensing information of a predetermined biological movement of the user; a simulation means for outputting the sensing information of the user and the sensing information of other users in synchronization with timing related to the actions of those users; The training system according to any one of claims 1 to 3, comprising:

5. The mixed reality processing means further displays an image of the virtual object on the display based on the position of the user and the field of view of the user. The training system according to any one of claims 1 to 4.

6. displaying an image of the virtual object on the display based on a captured image of a real space generated by capturing an image of the camera and including the user's viewpoint in the capture range, the position of the user, display conditions of the virtual object to be displayed on a display worn by the user, and an output mode on the display according to the display conditions of the virtual object; a motion control that hinders the movement of the user to a drive mechanism that is provided in a wearing device worn by the user and assists the movement of the user's joints, the motion control being applied to the drive mechanism based on the weight of a virtual object carried by the user; Training methods.

7. displaying an image of the virtual object on the display based on a captured image of a real space generated by capturing an image of the camera and including the user's viewpoint in the capture range, the position of the user, display conditions of the virtual object to be displayed on a display worn by the user, and an output mode on the display according to the display conditions of the virtual object; a motion control for hindering the movement of the user to a drive mechanism that is provided in a wearing tool worn by the user and assists the movement of the user's joints, the motion control being applied to the drive mechanism to hinder the movement of the user based on the moving distance of the user; Training methods.

8. The training system computer a mixed reality processing means for displaying an image of a virtual object on a display worn by the user based on a captured image of a real space generated by capturing an image of the real space including the user's viewpoint in the capturing range, the position of the user, display conditions of a virtual object to be displayed on the display worn by the user, and an output mode on the display according to the display conditions of the virtual object; a training means for applying motion control to a drive mechanism provided in a wearing device worn by the user and assisting the movement of the user's joints, the motion control being to hinder the movement of the user to the drive mechanism based on the weight of a virtual object carried by the user; A program that functions as a

9. The training system computer a mixed reality processing means for displaying an image of a virtual object on a display worn by the user based on a captured image of a real space generated by capturing an image of the real space including the user's viewpoint in the capturing range, the position of the user, display conditions of a virtual object to be displayed on the display worn by the user, and an output mode on the display according to the display conditions of the virtual object; a training means for applying motion control to a drive mechanism provided in a wearing tool worn by the user and assisting the movement of the user's joints, the motion control being to hinder the movement of the user to the drive mechanism based on the moving distance of the user; A program that functions as a

Images