Motion control method, motion control device, and motion control program

Abstract

In integrated body technology, the present invention provides a motion control method, motion control device, and motion control program that control the contribution of the learner's and teacher's movements to the avatar according to the learner's learning status. [Solution] In a virtual reality (VR) system including an motion control device, a head-mounted display (HMD), a tracking device, and a controller, a calculation unit 235 calculates the difference in motion between the learner and the teacher based on indicators of body parts related to the posture of the learner and the teacher for a specific motion, and a processing unit 238 controls the learner avatar in the VR space based on the calculated difference in motion and the indicators of body parts related to the learner's posture.

Description

Technical Field

[0001] The present invention relates to an operation control method, an operation control device, and an operation control program.

Background Art

[0002] In recent years, a technology called integrated body technology, which generates the motion of one avatar in a virtual reality (VR) space by synthesizing the motions of multiple people, has been attracting attention. For example, a method has been studied in which a trainee and an instructor who teaches the motion share one avatar, enabling the trainee to efficiently learn physical skills through training.

[0003] In training using integrated body technology, it is important to evoke a sense of agency (SoA) in the trainee, which is the feeling that one is the agent of the action. The sense of agency is also expressed as a sense of self-operation or simply a sense of being the agent. The sense of agency evoked in the trainee is stronger as the proportion of the trainee's contribution to the avatar's motion is larger. On the other hand, if the proportion of the trainee's contribution to the avatar's motion is larger, the proportion of the instructor's contribution to the avatar's motion becomes smaller, and it may become difficult to convey the correct motion to the trainee.

[0004] In integrated body technology, the avatar is controlled so that the motions of the trainee and the instructor each contribute 50% to the avatar's motion.

[0005] Also, a method has been proposed for controlling the contribution rates of both parties over time so that the trainee can evoke a sense of agency even when the instructor's contribution rate is high. In this method, control is performed such that the contribution rate of the trainee is decreased from a state where only the trainee can operate the avatar to a predetermined value over a predetermined time, and thereafter the contribution rate of the trainee is set to the predetermined value.

Prior Art Documents

Non-Patent Documents

[0006] [Non-Patent Document 1] Ryota Ito, Nami Ogawa, Takushi Narumi, and Michitaka Hirose, "Basic Survey on the Transmission of Physical Skills Using a Fusion Body," Proceedings of the 25th Annual Meeting of the Virtual Reality Society of Japan, Virtual Reality Society of Japan, September 2020, 3C3-7. [Non-Patent Document 2] Daiki Kodama, Takato Hashimoto, Yuji Hatada, Takushi Narumi, and Michitaka Hirose, "Improving the Sense of Agency Through Dynamic Control of Contribution Rate During Fusion Body Action," Proceedings of the 26th Annual Meeting of the Virtual Reality Society of Japan, Virtual Reality Society of Japan, September 2021, 1C3-4. [Overview of the project] [Problems that the invention aims to solve]

[0007] However, controlling the avatar based on predetermined time or contribution rates presents a problem: it is not possible to control it in accordance with the learner's training status, and the learning effect of integrated bodily technology is not realized. For example, if the teacher's movements contribute significantly to the avatar's movements when the learner is performing the correct movements, the learner's sense of agency may be diminished. As a result, even when there is a discrepancy between the learner's and teacher's movements, the learner may not perceive the avatar's movements as their own, their sense of agency diminishes, and the learning effect of integrated bodily technology is not achieved. The disclosed technology was developed in consideration of the above, and in one aspect, aims to control the contribution of the learner's and teacher's movements to the avatar's movements according to the learner's learning status. [Means for solving the problem]

[0008] In one aspect, this is a motion control method in which a computer controls the movement of an avatar in a VR space. The computer identifies a first indicator regarding the learner's body parts related to the learner's posture for a specific movement, and a second indicator regarding the teacher's body parts related to the teacher's posture for a specific movement. Based on the identified first and second indicators, the computer calculates the difference in movement between the learner and the teacher. The computer then controls the avatar based on the difference in movement and the first indicator. [Effects of the Invention]

[0009] In one aspect, the present invention allows for the control of the contributions of the learner's and teacher's movements to the avatar in integrated body technology, according to the learner's learning progress. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a schematic diagram showing an example configuration of a VR system according to the first embodiment. [Figure 2] Figure 2 is a block diagram showing an example of the functional configuration of the control device according to the first embodiment. [Figure 3] Figure 3 is a table showing an example of location information according to the first embodiment. [Figure 4] Figure 4 is a table showing an example of setting information according to the first embodiment. [Figure 5] Figure 5 is a table showing an example of object information according to the first embodiment. [Figure 6] Figure 6 shows an example of a function used to determine the learner's contribution rate and the teacher's contribution rate according to the first embodiment. [Figure 7] Figure 7 shows an example of a table used to determine the learner's contribution rate and the teacher's contribution rate according to the first embodiment. [Figure 8] Figure 8 is a flowchart showing an example of the initialization process according to the first embodiment. [Figure 9] Figure 9 is a flowchart showing an example of the control process according to the first embodiment. [Figure 10]FIG. 10 is a diagram for explaining the temporal changes in the distance, the contribution rate of the learner, and the contribution rate of the teacher according to the first embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of control processing according to the second embodiment. [Figure 12] FIG. 12 is a diagram for explaining the temporal changes in the distance, the contribution rate of the learner, and the contribution rate of the teacher according to the second embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of control processing according to the third embodiment. [Figure 14] FIG. 14 is a diagram for explaining the temporal changes in the distance, the contribution rate of the learner, and the contribution rate of the teacher according to the third embodiment. [Figure 15] FIG. 15 is a diagram showing an example of a function used for determining the contribution rate according to the fourth embodiment. [Figure 16] FIG. 16 is a flowchart showing an example of control processing according to the fourth embodiment. [Figure 17] FIG. 17 is a diagram showing an example of the states of the real space and the VR space in each embodiment. [Figure 18] FIG. 18 is a diagram showing an example of the hardware configuration of the operation control device.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the operation control method, operation control device, and operation control program according to the present embodiment will be described in detail with reference to the drawings. In the embodiments, components having the same function are denoted by the same reference numerals, and redundant descriptions are omitted. Note that the operation control method, operation control program, and operation control device described in the following embodiments are merely examples and do not limit the embodiments. Also, the following embodiments may be appropriately combined within a non - conflicting range. [First Embodiment]

[0012] FIG. 1 is a configuration diagram showing an example of a VR system according to the first embodiment. The VR system 1 includes an operation control device 2, a Head Mount Display (HMD) 101, a tracking device 102, and a controller 103.

[0013] The HMD 101 acquires the VR image for the right eye and the VR image for the left eye output from the operation control device 2, and displays a stereoscopic image on the display. The learner wears the HMD 101A, and the teacher wears the HMD 101B.

[0014] The tracking device 102 tracks the actions of the learner and the teacher, and outputs the acquired tracking data to the operation control device 2. The tracking device 102 can be realized by, for example, one or more infrared lidars and one or more markers. Each marker is attached to the bodies of the learner and the teacher. Each infrared lidar irradiates a laser, receives the laser reflected by each marker, and outputs tracking data including information on the irradiation direction of the laser and the distance to each marker to the operation control device 2.

[0015] Also, the tracking device 102 can be realized by, for example, one or more digital cameras. Each digital camera photographs the space including the learner or the teacher, and outputs tracking data including each captured image to the operation control device 2. The present invention is not limited to the described tracking method, the configuration of the tracking device 102, or the format of the tracking data, and the operation control device 2 can be realized according to various methods, configurations, and formats.

[0016] The tracking device 102A tracks the learner, and the tracking device 102B tracks the teacher. The tracking device 102A and the tracking device 102B each track the learner and the teacher in real time, and output the tracking data to the operation control device 2.

[0017] The controller 103 receives instructions from the teacher and outputs them to the motion control device 2. Specifically, for example, the controller 103 receives instructions from the teacher regarding whether or not to control the learner avatar using integrated body technology and outputs them to the motion control device 2.

[0018] Figure 2 is a block diagram showing an example of the functional configuration of the motion control device according to the first embodiment. Functionally, the motion control device 2 has a communication unit 210, a storage unit 220, and a control unit 230, as shown in Figure 2. The communication unit 210 communicates with connected external devices (HMD 101, tracking device 102, and controller 103, etc.) under the control of the control unit 230.

[0019] The memory unit 220 is a storage device such as an HDD (Hard Disk Drive) or a semiconductor memory storage device, and stores location information 221, setting information 222, spatial information 223, and object information 224.

[0020] The location information 221 includes information about the positions of the learner and the teacher. For example, the location information 221 includes tracking data output by the tracking device 102 and information about the positions of each joint of the learner and the teacher calculated by the estimation unit 233 based on the tracking data.

[0021] Figure 3 is a table showing an example of position information 221 in the first embodiment. For example, the position information 221 is recorded each time the motion control device 2 acquires tracking data from the tracking device 2. One record includes, for example, a number indicating the order in which the record was added. Another record includes, for example, position information of each joint of the learner and the teacher in VR space. Furthermore, another record includes information about the difference in motion between the learner and the teacher. This information about the difference in motion between the learner and the teacher is, for example, the distance in VR space between corresponding joints of the learner and the teacher. Another information about the difference in motion between the learner and the teacher is, for example, the sum of the distances between joints obtained by adding up the distances in VR space between corresponding joints of the learner and the teacher. The table shown in Figure 3 is an example of position information 221, and one record may include other data such as the velocity and angle of each joint of the learner and the teacher, the time the record was created, and the time each position was acquired.

[0022] The configuration information 222 is information pre-set by a teacher or other user, and indicates various settings for the VR system 1. Figure 4 is a table showing an example of the configuration information 222 in the first embodiment. The configuration information 222 has, for example, one record, and the record contains information about various settings. The configuration information 222 includes, for example, information about the standing position of each avatar in the VR space when the VR display is started. The configuration information 222 also includes, for example, information about the function used to calculate the contribution rate.

[0023] Furthermore, the configuration information 222 includes information output by the controller 103. Specifically, the configuration information 222 includes information on settings such as whether or not to control the learner avatar using fusion body technology. As an initial setting, the configuration information 222 includes a setting to not control the learner avatar using fusion body technology. The table shown in Figure 4 is an example of the configuration information 222, and the configuration information 222 can include various information related to settings, such as information on the display of the HMD 101B and information on connected devices.

[0024] Spatial information 223 is template information for defining the VR space. For example, spatial information 223 is a three-dimensional floor plan showing the arrangement of floors, walls, ceilings, etc., in the generated VR space.

[0025] Object information 224 is information indicating the shape, placement location, etc., of an object placed in the VR space, which is a structure within the VR space. For example, object information 224 may include 3D CAD data of the structure within the VR space.

[0026] Figure 5 is a table showing an example of object information 224 in the first embodiment. For example, object information 224 has a record for each structure in the VR space that is placed in the VR space, and one record includes the file name of the 3D CAD data of the structure in the VR space and the position information in the VR space of the reference position where the structure in the VR space is placed. The table shown in Figure 5 is an example of object information 224, and object information 224 can include various information about the structure in the VR space, such as the movement of the object and whether the object can be manipulated by the avatar.

[0027] The control unit 230 is a processing unit that controls the operation of the operation control device 2. The control unit 230 is implemented, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing a program stored in its internal memory using RAM as the working area. Alternatively, the control unit 230 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Unit) or an FPGA (Field Programmable Gate Array).

[0028] The control unit 230 includes a construction unit 231, an acquisition unit 232, an estimation unit 233, a determination unit 234, a calculation unit 235, a decision unit 236, a synthesis unit 237, a processing unit 238, and an output unit 239, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 230 is not limited to the configuration shown in Figure 2, and other configurations are also acceptable as long as they perform the information processing described later.

[0029] The construction unit 231 constructs the VR space. Specifically, the construction unit 231 constructs the VR space by arranging elements such as floors, walls, and ceilings based on the spatial information 223. Next, the construction unit 231 places structures within the VR space at predetermined locations. Furthermore, at the start of VR display, the construction unit 231 places the learner avatar based on the learner avatar's standing position information included in the setting information 222.

[0030] The acquisition unit 232 acquires tracking data from the tracking device 102. The acquisition unit 232 also acquires instructions output by the controller 103 and stores them in the setting information 222. Specifically, the acquisition unit 232 updates, for example, the information contained in the setting information 222 regarding whether or not to control the learner avatar using fusion body technology, according to the instructions acquired from the controller 103.

[0031] The estimation unit 233 estimates the posture of the learner and teacher based on tracking data acquired from the tracking device 102. For example, the estimation unit 233 estimates the skeletons of the learner and teacher based on the tracking data acquired from the tracking device 102 and adjusts the estimated skeletons of the learner and teacher to match the size of the avatars in the VR space. Then, the estimation unit 233 calculates the position information of each joint of the learner and teacher based on the adjusted skeleton. A joint is an example of a body part in the disclosed technology. The position information of each joint of the learner and teacher is, for example, position information indicating the three-dimensional position of each joint.

[0032] Furthermore, the estimation unit 233 identifies a reference joint from among the learner's joints and calculates positional information for each of the learner's joints with the reference joint as the origin. For example, the estimation unit 233 calculates the positional information for each of the learner's joints with the reference joint as the origin by subtracting the positional information for the reference learner's joint from the positional information for each of the learner's joints. The positional information for each of the learner's joints with the reference joint as the origin is an example of the first indicator of the disclosed technology.

[0033] Similarly, the estimation unit 233 identifies the same joints from among the teacher's joints that were identified for the learner, and calculates the position of each of the teacher's joints with the reference joint as the origin. For example, the estimation unit 233 calculates the position information of each of the teacher's joints with the reference joint as the origin by subtracting the position information of the reference teacher's joint from the position information of each of the teacher's joints. The position information of each of the teacher's joints with the identified joint as the origin is an example of a second indicator of the disclosure technique.

[0034] The determination unit 234 determines whether or not to use fused body technology to control the learner avatar, based on the information contained in the setting information 222 regarding whether or not to control the learner avatar using fused body technology.

[0035] If the determination unit 234 determines that the learner avatar will be controlled using fused body technology, the calculation unit 235 calculates the difference between the learner's and the teacher's movements based on the position information of each of the learner's joints with a reference joint as the origin, and the position information of each of the teacher's joints with a reference joint as the origin.

[0036] For example, the calculation unit 235 calculates the distance in VR space between corresponding joints of the learner and the teacher, based on the position information of each joint of the learner with a reference joint as the origin, and the position information of each joint of the teacher with a reference joint as the origin. Hereinafter, the distance in VR space between corresponding joints of the learner and the teacher may simply be referred to as distance. Furthermore, the calculation unit 235 calculates the sum of the distances between joints. The sum of the distances between joints is an example of the difference in operation of the disclosed technology.

[0037] Furthermore, the calculation unit 235 stores the calculated distance and the sum of the distances between joints in the setting information 222 of the storage unit 220.

[0038] The determination unit 236 determines the proportion by which the positional information of each joint of the learner, with the identified joint as the origin, contributes to the positional information of each corresponding joint of the learner avatar, based on the sum of the distances between joints calculated by the calculation unit 235. Hereinafter, the proportion by which the positional information of each joint of the learner, with the identified joint as the origin, contributes to the positional information of each corresponding joint of the learner avatar may also be called the learner contribution rate. The learner contribution rate is an example of the first proportion of the disclosed technology.

[0039] Furthermore, the determination unit 236 determines the proportion by which the positional information of each joint of the teacher, with the identified joint as the origin, contributes to the positional information of each corresponding joint of the learner avatar, based on the sum of the distances between joints calculated by the calculation unit 235. Hereafter, the proportion by which the positional information of each joint of the teacher, with the identified joint as the origin, contributes to the positional information of each corresponding joint of the learner avatar may also be called the teacher's contribution rate. The teacher's contribution rate is an example of the second contribution rate of the disclosed technology. Also, below, when each contribution rate is referred to, both the learner's contribution rate and the teacher's contribution rate will be shown.

[0040] The decision unit 236 determines the learner's contribution rate and the teacher's contribution rate so that their sum equals 100%. The specific methods for determining the learner's contribution rate and the teacher's contribution rate will be described later.

[0041] The synthesis unit 237 calculates the position information of each joint of the learner avatar based on the learner's contribution rate and the teacher's contribution rate determined by the determination unit 236.

[0042] Specifically, the synthesis unit 237 calculates the joint position information of the learner avatar relative to the reference joint by taking a weighted average of the learner's joint position information relative to the reference joint for each corresponding joint. The weights used in the weighted average are the learner's contribution rate for the learner side and the teacher's contribution rate for the teacher side.

[0043] Furthermore, the synthesis unit 237 refers to the position information 221 and obtains the current position information of the reference joint of the learner avatar. The synthesis unit 237 calculates the position information of each joint of the learner avatar in the VR space by adding the current position information of the reference joint of the learner avatar to the position information of the joints of the learner avatar with the reference joint as the origin.

[0044] The processing unit 238 controls the learner avatar based on the position information calculated by the estimation unit 233 or the synthesis unit 237. If the processing unit 238 decides to control the learner avatar using fused body technology based on the determination result of the determination unit 234, it controls the learner avatar based on the position information calculated by the synthesis unit 237. If the processing unit 238 decides not to control the learner avatar using fused body technology based on the determination result of the determination unit 234, it controls the learner avatar based on the position information calculated by the estimation unit 233.

[0045] Specifically, when the processing unit 238 controls the learner avatar using fused body technology, it controls each joint of the learner avatar in the VR space so that the corresponding joints of the learner avatar are positioned to match the position information of each joint of the learner avatar in the VR space calculated by the synthesis unit 237.

[0046] Alternatively, if the learning avatar is not controlled by the fused body technology, the processing unit 238 calculates the position information of each joint of the learning avatar in VR space by adding the current position information of the reference joint of the learning avatar to the position information of each joint of the learning avatar, which is calculated by the estimation unit 233 with the reference joint as the origin. Then, it controls each joint of the learning avatar in VR space so that the corresponding joint of the learning avatar is positioned at the calculated position information of each joint of the learning avatar in VR space.

[0047] The output unit 239 outputs a right-eye VR image and a left-eye VR image to the learner's and teacher's respective HMDs 101. Specifically, the output unit 239 generates output images so that the first-person perspective image of the learner's avatar is displayed on the learner's HMD 101A.

[0048] Next, we will explain how the determination unit 236 determines each contribution rate.

[0049] Figure 6 shows an example of a function used to determine the learner's contribution rate and the teacher's contribution rate according to the first embodiment. The determination unit 236 determines the learner's contribution rate and the teacher's contribution rate as a function of each contribution rate and the sum of the distances between joints calculated by the calculation unit 235, for example, as shown in the graph of Figure 6. According to the graph of Figure 6, the determination unit 236 determines the learner's contribution rate with β% as the upper limit when the sum of the distances between joints is 0 and θ% as the lower limit when the sum of the distances between joints is K, so that it becomes smaller as the sum of the distances between joints increases. Also, according to Figure 6, the determination unit 236 determines the teacher's contribution rate with α% as the upper limit when the sum of the distances between joints is K and γ% as the lower limit when the sum of the distances between joints is 0, so that it becomes larger as the sum of the distances between joints increases.

[0050] Figure 7 shows an example of a table used to determine the learner's contribution rate and the teacher's contribution rate according to the first embodiment. The determination unit 236 may determine the learner's contribution rate and the teacher's contribution rate based, for example, on the relationship between each contribution rate and the sum of the distances between joints calculated by the calculation unit 235, as shown in the table in Figure 7. According to the table in Figure 7, the determination unit 236 determines the learner's contribution rate and the teacher's contribution rate based on the range of the sum of the distances between joints. Also, according to the table in Figure 7, the determination unit 236 determines the learner's contribution rate to be smaller and the teacher's contribution rate to be larger as the sum of the distances between joints increases.

[0051] Figures 6 and 7 illustrate methods for determining the learner's contribution rate and the teacher's contribution rate based on the sum of distances between joints, and the present invention is not limited to these functions or tables. The determination unit 236 determines, using various functions, tables, data formats, etc., that the larger the sum of distances between joints, the smaller the learner's contribution rate and the larger the teacher's contribution rate.

[0052] Next, the operation of the motion control device 2 according to the first embodiment will be described. When the VR system 1 is started, the motion control device 2 first executes the initialization process shown in Figure 8. Then, the tracking device 102 starts tracking and outputs tracking data to the motion control device 2, and the motion control device 2 repeatedly executes the control process shown in Figure 9. Note that the control process is an example of the motion control method of the disclosed technology.

[0053] First, let's explain the initialization process shown in Figure 8.

[0054] Figure 8 is a flowchart showing an example of the initialization process according to the first embodiment. In the initialization process, the construction unit 231 generates a VR space based on spatial information 223 (S11), places structures in the VR space based on object information 224 (S12), and places each avatar based on setting information 222 (S13).

[0055] Next, the control process shown in Figure 9 will be explained.

[0056] Figure 9 is a flowchart showing an example of the control process according to the first embodiment. In step S21, the acquisition unit 232 acquires tracking data output by the tracking device 102, and based on the acquired tracking data, the estimation unit 233 calculates the position information of each joint of the learner and teacher in the VR space.

[0057] Next, in step S22, the determination unit 234 refers to the setting information 222. If the setting information 222 includes a setting that does not control the learner avatar using fused body technology (step S22: Yes), the process proceeds to step S23. If the setting information 222 includes information that does control the learner avatar using fused body technology (step S22: No), the process proceeds to step S27.

[0058] In step S23, the calculation unit 235 calculates the distance between corresponding joints between the learner and the teacher based on the position information of each joint in the VR space of the learner and the teacher calculated by the estimation unit 233.

[0059] In step S24, the determination unit 236 calculates the learner's contribution rate and the teacher's contribution rate based on the sum of the distances between joints calculated by the calculation unit 235, by referring to the function information of the setting information 222.

[0060] In step S25, the synthesis unit 237 calculates the position information of each joint of the learner avatar based on the contribution rates determined by the determination unit 236.

[0061] In step S26, the processing unit 238 controls the learner avatar so that each joint of the learner avatar in the VR space is positioned according to the position information of each joint calculated by the synthesis unit 237.

[0062] In step S27, the processing unit 238 controls the learner avatar so that each joint of the learner avatar in the VR space is located in the position information of each joint of the learner calculated by the estimation unit 233 within the VR space.

[0063] In step S28, the output unit 239 generates output images to be displayed on the learner's HMD 101A and the teacher's HMD 101B.

[0064] The communication unit 210 transmits the generated output image to each HMD 101. The control unit 230 then proceeds to step S21 and repeats the control process.

[0065] As described above, according to the VR system 1 in the first embodiment, the motion control device 2 controls the learner's contribution rate and the teacher's contribution rate to the learner avatar based on the difference between the learner's and teacher's movements. Specifically, the learner's contribution rate and the teacher's contribution rate to the learner avatar are determined based on the sum of the distances between the joints between the learner and the teacher. This makes it possible to control the contribution of the learner's and teacher's movements to the avatar in the integrated body technology according to the learner's learning status.

[0066] Figure 10 shows the relationship between distance, the learner's contribution rate, and the teacher's contribution rate when controlled by the VR system 1 of the first embodiment. According to the VR system 1 of the first embodiment, as shown in Figure 10, when the difference in movement is greater, the teacher's contribution rate can be increased, and when the difference in movement is smaller, the learner's contribution rate can be increased. This makes it possible to make the learner perceive the difference in movement between themselves and the teacher, while preventing a decrease in the learner's sense of agency. [Second Embodiment]

[0067] Next, a second embodiment will be described. The configuration of the VR system and the operation control device according to the second embodiment is the same as the configuration of the VR system 1 and operation control device 2 according to the first embodiment shown in Figure 1, so the same reference numerals will be used to describe the differences.

[0068] The setting information 222 of the second embodiment includes, in addition to the information included in the first embodiment, the sum of the distances between joints, which is the threshold for initiating the application of fused body technology to the learner avatar.

[0069] Furthermore, the setting information 222 includes a flag indicating, for example, whether the operation control device 2 has controlled the learner avatar based on the values ​​of each contribution rate calculated by the determination unit 236 based on distance. For example, the flag has an initial value of No, and if the operation control device 2 has already controlled the learner avatar based on the values ​​of each contribution rate calculated by the determination unit 236 based on distance, it has a value of Yes.

[0070] Furthermore, the setting information 222 includes, for example, values ​​for the learner's contribution rate and the teacher's contribution rate, which are specified in advance by the teacher or the like. The specified values ​​for each contribution rate are set in advance by the teacher or the like so that the learner's contribution rate is greater than the teacher's contribution rate. For example, the memory unit 220 stores the learner's contribution rate values ​​included in the setting information 222 by having the teacher or the like select in advance a range greater than 50% and less than or equal to 100%. Also, for example, the memory unit 220 stores the teacher's contribution rate value by having the teacher or the like select in advance a range greater than or equal to 0% and less than 50%. The memory unit 220 may also calculate the other contribution rate by subtracting the one-to-one contribution rate, which is set in advance, from 100%, and store it in the memory unit 220.

[0071] In the second embodiment, the determination unit 234 performs processing that branches the control of the learner avatar based on the distance calculated by the calculation unit 235, in addition to the processing performed by the determination unit 234 in the first embodiment.

[0072] Specifically, the determination unit 234 compares the sum of distances between joints calculated by the calculation unit 235 with the value of the sum of distances between joints, which is the threshold for initiating the application of fused body technology to the learner avatar, as included in the setting information 222. If, for example, the sum of distances between joints is greater than the value of the sum of distances between joints, which is the threshold for initiating the application of fused body technology to the learner avatar, the determination unit 234 performs control of the learner avatar based on the values ​​of each contribution rate specified in the setting information 222 beforehand.

[0073] Furthermore, if the sum of the distances between joints is less than or equal to the sum of the distances between joints which is the threshold for initiating the application of fused body technology to the learner avatar, the determination unit 234 controls the learner avatar based on the values ​​of each contribution rate calculated by the calculation unit 235 based on the distance. In addition, if the sum of the distances between joints is less than or equal to the sum of the distances between joints which is the threshold for initiating the application of fused body technology to the learner avatar, the determination unit 234 updates the flag in the setting information 222 to Yes.

[0074] Furthermore, the determination unit 234 determines whether the operation control device 2 has already performed control of the learner avatar based on the values ​​of each contribution rate calculated by the calculation unit 235 based on the distance. For example, the determination unit 234 refers to the flags in the setting information 222 and performs processing to branch the control of the learner avatar based on the reference result.

[0075] Specifically, if the flag in the setting information 222 is No, the determination unit 234 performs a process to branch the control of the learner avatar based on the distance calculated by the calculation unit 235. If the flag in the setting information 222 is Yes, the determination unit 234 does not perform a process to branch the control of the learner avatar, but proceeds with the process where the setting information 222 includes a setting to control the learner avatar using fused body technology.

[0076] In addition to the processing performed by the decision unit 236 in the first embodiment, the decision unit 236 in the second embodiment performs processing to obtain the values ​​of the learner's contribution rate and the teacher's contribution rate, which are included in the setting information 222 and have been specified in advance by the teacher or the like.

[0077] Specifically, if the determination unit 234 proceeds with processing where the setting information 222 includes a setting for controlling the learner avatar using integrated body technology, the determination unit 234 performs a process of referring to and obtaining the values ​​of the learner's contribution rate and the teacher's contribution rate, which are specified in advance by the teacher or the like and included in the setting information 222.

[0078] Next, the operation of the motion control device 2 according to the second embodiment will be described. When the VR system 1 is started, the motion control device 2 first performs the initialization process shown in Figure 8. Then, the tracking device 102 starts tracking and outputs tracking data to the motion control device 2, and the motion control device 2 performs the control process shown in Figure 11. Note that the control process is an example of the motion control method of the disclosed technology.

[0079] The initialization process shown in Figure 8 is the same as the initialization process in the first embodiment, so its explanation will be omitted.

[0080] Next, the control process shown in Figure 11 will be described. Figure 11 is a flowchart showing an example of the control process according to the second embodiment. Note that processes similar to those in the first embodiment are denoted by the same reference numerals, and detailed explanations are omitted.

[0081] When the process moves from step S21 through step S23 to step S31, the determination unit 234 determines whether or not it has already started controlling the learner avatar by synthesizing the actions of the learner and the teacher based on each contribution rate.

[0082] Specifically, the determination unit 234 refers to the flag in the setting information 222. If it is No, the process proceeds to step S32; if it is Yes, the process proceeds to step S24.

[0083] In step S32, the determination unit 234 compares the sum of the distances between joints with the threshold for initiating the application of fused body techniques to the learner avatar. If the sum of the distances between joints is less than or equal to the threshold for initiating the application of fused body techniques to the learner avatar (step S32: Yes), the determination unit 234 proceeds to step S24; if it is greater than the threshold (step S32: No), the process proceeds to step S33.

[0084] In step S33, the determination unit 236 refers to the setting information 222 and obtains the pre-specified values ​​for each contribution rate. In step S25, the synthesis unit 237 calculates the position information for each joint of the learner avatar based on the values ​​of each contribution rate obtained by the determination unit 236 in step S33 or calculated in step S24.

[0085] After step S25, in step S26, or in step S27, the processing unit 238 controls the learner avatar. In step S28, the output unit 239 generates output images to be transmitted to HMD101A and HMD101B, and the communication unit 210 transmits the respective output images to HMD101A and HMD101B. The control unit 230 then proceeds to step S21 and repeats the control process.

[0086] As described above, in the VR system 1 of the second embodiment, the motion control device 2 controls the learner's contribution rate and the teacher's contribution rate to the learner avatar based on the distance between the body parts of the learner and the teacher, similar to the first embodiment. Specifically, the learner's contribution rate and the teacher's contribution rate to the learner avatar are determined based on the sum of the distances between the joints between the learner and the teacher. This makes it possible to control the contribution of the learner's and teacher's movements to the avatar in the fused body technology according to the learner's learning status.

[0087] Figure 12 shows the relationship between distance and each contribution rate when control is performed by the VR system 1 of the second embodiment. The operation control device 2 of the second embodiment determines, based on the distance, whether to perform control using each contribution rate calculated based on the distance or to perform control using each contribution rate specified in advance.

[0088] Specifically, the motion control device 2 controls the learner avatar according to predetermined contribution rates during the period when the distance is greater than a threshold after the start of learning. More specifically, the motion control device 2 controls the learner avatar according to predetermined contribution rates such that the learner's contribution rate is greater than the teacher's contribution rate. This allows the integrated body technology to be used for learning movements with a greater learner contribution until the learner becomes accustomed to the movements, preventing a decrease in the learner's sense of self-control.

[0089] Furthermore, the motion control device 2 controls the learner avatar according to predetermined contribution rates during the period when the distance is greater than a threshold after learning begins, and calculates contribution rates based on distance and controls the learner avatar during the period when the distance falls below the threshold. This allows for control with a large learner contribution rate until the discrepancy in actions between the learner and the teacher falls below a threshold, and then calculates contribution rates according to distance and controls the learner avatar after the discrepancy in actions falls below a threshold. This prevents undermining the learner's sense of agency in the initial stages, and allows for more efficient learning once the learner has become accustomed to the actions being learned.

[0090] Furthermore, in the second embodiment, the contribution rates for the period from the start of learning when the distance is greater than the threshold may be, for example, 100% for the learner and 0% for the teacher. This allows the learner to operate the learner avatar by themselves until they become accustomed to the actions they are learning, thus preventing a decrease in the learner's sense of self-operation. [Third Embodiment]

[0091] Next, a third embodiment will be described. The configuration of the VR system and the operation control device according to the third embodiment is the same as the configuration of the VR system 1 and operation control device 2 according to the first embodiment shown in Figure 1, so the same reference numerals will be used to describe the differences.

[0092] The setting information 222 of the third embodiment includes, in addition to the information included in the first embodiment, the sum of the distances between joints, which is the threshold for initiating the application of fused body technology to the learner avatar. The setting information 222 also includes a flag indicating whether the motion control device 2 has already performed control of the learner avatar based on the values ​​of each contribution rate calculated by the determination unit 236 based on the distance. Specifically, the flag has, for example, an initial value of No, and a value of Yes if the motion control device 2 has performed control of the learner avatar based on the values ​​of each contribution rate calculated by the determination unit 236 based on the distance since the initialization process.

[0093] In the third embodiment, the determination unit 234 performs processing that branches the control of the learner avatar based on the distance calculated by the calculation unit 235, in addition to the processing performed by the determination unit 234 in the first embodiment.

[0094] Specifically, the determination unit 234 compares the sum of distances between joints calculated by the calculation unit 235 with the value of the sum of distances between joints, which is the threshold for initiating the application of fused body technology to the learner avatar, as included in the setting information 222. If, for example, the sum of distances between joints is greater than the value of the sum of distances between joints, which is the threshold for initiating the application of fused body technology to the learner avatar, the determination unit 234 proceeds with processing in the same way as when the setting information 222 includes a setting not to control the learner avatar using fused body technology.

[0095] Furthermore, the determination unit 234 proceeds with processing if the setting information 222 includes a setting to control the learner avatar using fused body technology, for example, if the sum of the distances between joints is less than or equal to the value of the sum of the distances between joints which is the threshold for initiating the application of fused body technology to the learner avatar. In addition, if the determination unit 234 updates the flag in the setting information 222 to Yes, for example, if the sum of the distances between joints is less than or equal to the value of the sum of the distances between joints which is the threshold for initiating the application of fused body technology to the learner avatar.

[0096] Furthermore, the determination unit 234 determines whether the operation control device 2 has already performed control of the learner avatar based on the values ​​of each contribution rate calculated by the determination unit 236 based on the distance. For example, the determination unit 234 refers to the flags in the setting information 222 and performs processing to branch the control of the learner avatar based on the reference result.

[0097] Specifically, if the flag in the setting information 222 is No, the determination unit 234 performs a process to branch the control of the learner avatar based on the distance calculated by the calculation unit 235. If the flag in the setting information 222 is Yes, the determination unit 234 does not perform a process to branch the control of the learner avatar, but proceeds with the process where the setting information 222 includes a setting to control the learner avatar using fused body technology.

[0098] Next, the operation of the motion control device 2 according to the third embodiment will be described. When the VR system 1 is started, the motion control device 2 first performs the initialization process shown in Figure 8. Then, the tracking device 102 starts tracking and outputs tracking data to the motion control device 2, and the motion control device 2 performs the control process shown in Figure 13. Note that the control process is an example of the motion control method of the disclosed technology.

[0099] The initialization process shown in Figure 8 is the same as the initialization process in the first embodiment, so its explanation will be omitted.

[0100] Next, the control process shown in Figure 13 will be described. Figure 13 is a flowchart showing an example of the control process according to the third embodiment. Note that processes similar to those in the first embodiment are denoted by the same reference numerals, and detailed explanations are omitted.

[0101] When the process moves from step S21 through step S23 to step S31, the determination unit 234 determines whether or not it has already started controlling the learner avatar by synthesizing the actions of the learner and the teacher based on each contribution rate.

[0102] Specifically, the determination unit 234 refers to the flag in the setting information 222. If it is No, the process proceeds to step S32; if it is Yes, the process proceeds to step S24.

[0103] In step S32, the determination unit 234 compares the sum of the distances between joints with the threshold for initiating the application of fused body techniques to the learner avatar. If the sum of the distances between joints is less than or equal to the threshold for initiating the application of fused body techniques to the learner avatar (step S32: Yes), the determination unit 234 proceeds to step S24; if it is greater than the threshold (step S32: No), the process proceeds to step S27.

[0104] After steps S24 and S25, in step S26 or step S27, the processing unit 238 controls the learner avatar. In step S28, the output unit 239 generates output images to be transmitted to HMD101A and HMD101B, and the communication unit 210 transmits the respective output images to HMD101A and HMD101B. The control unit 230 then proceeds to step S21 and repeats the control process.

[0105] As described above, in the VR system 1 of the third embodiment, the motion control device 2 controls the learner's contribution rate and the teacher's contribution rate to the learner avatar based on the distance between the body parts of the learner and the teacher, similar to the first embodiment. Specifically, the learner's contribution rate and the teacher's contribution rate to the learner avatar are determined based on the sum of the distances between the joints between the learner and the teacher. This makes it possible to control the contribution of the learner's and teacher's movements to the avatar in the fused body technology according to the learner's learning status.

[0106] Figure 14 shows the relationship between distance and each contribution rate when control is performed by the VR system 1 of the third embodiment. The motion control device 2 of the third embodiment determines whether or not to start controlling the avatar based on each contribution rate depending on the magnitude of the distance. Specifically, the motion control device 2 does not control the learner avatar using fused body technology during the period when the distance is greater than a threshold after learning has started. This allows the learner to operate the learner avatar by themselves until they become accustomed to the movements they are learning, preventing a decrease in the learner's sense of self-control.

[0107] Furthermore, the motion control device 2 does not control the learner avatar using integrated body technology during the period when the distance is greater than a threshold after the start of learning, and calculates each contribution rate based on the distance and controls the learner avatar during the period when the distance is below the threshold. As a result, the learner avatar is not controlled by integrated body technology until the discrepancy in movements between the learner and the teacher falls below a threshold, and control of the learner avatar using integrated body technology is performed after the discrepancy in movements falls below a threshold. This prevents the learner's sense of agency from being undermined in the initial stages, and allows for more efficient learning once the learner has become accustomed to the movements being learned. [Fourth Embodiment]

[0108] Next, a fourth embodiment will be described. The configuration of the VR system and the operation control device according to the second embodiment is the same as the configuration of the VR system 1 and operation control device 2 according to the first embodiment shown in Figure 1, so the same reference numerals will be used to describe the differences.

[0109] The setting information 222 of the fourth embodiment includes, in addition to the information included in the first embodiment, information regarding the time interval for updating the learner's contribution rate and the teacher's contribution rate. The time interval for updating each contribution rate is information indicating the period from when each contribution rate is updated until the next update of each contribution rate. The time interval for updating each contribution rate may be, for example, the number of times control processing is performed from when each contribution rate is updated until the next update of each contribution rate. Alternatively, the time interval for updating each contribution rate may be, for example, the number of times the HMD 101 updates the image from when each contribution rate is updated until the next update of each contribution rate. Alternatively, the time interval for updating each contribution rate may be, for example, the time from when each contribution rate is updated until the next update of each contribution rate.

[0110] In the fourth embodiment, the determination unit 234, in addition to the processing performed by the determination unit 234 in the first embodiment, refers to information regarding the time width for updating each contribution rate in the setting information 222 and determines whether a certain period of time has elapsed since the previous update of each contribution rate.

[0111] The determination unit 234 compares the number of times control processing is performed between updating each contribution rate included in the setting information 222 and updating each contribution rate again with the actual number of times control processing is performed. If the actual number of times control processing is performed is equal to the number of times control processing is performed between updating each contribution rate included in the setting information 222 and updating each contribution rate again, the count of the actual number of times control processing is performed is set to 0. If the actual number of times control processing is performed is not equal to the number of times control processing is performed between updating each contribution rate included in the setting information 222 and updating each contribution rate again, 1 is added to the count of the actual number of times control processing is performed.

[0112] The calculation unit 235 of the fourth embodiment calculates, in addition to the processing performed by the calculation unit 235 of the first embodiment, the sum of the distances between joints from the previous update of each contribution rate to the present. Hereinafter, the sum of the distances between joints from the previous update of each contribution rate to the present may also be called the sum of the distances between joints within the period.

[0113] In the fourth embodiment, the determination unit 236 determines each contribution rate based on the sum of the distances between joints within the period, which is calculated by the calculation unit 235.

[0114] Next, the method for determining each contribution rate in the fourth embodiment will be described.

[0115] Figure 15 shows an example of a function used to determine the contribution rate according to the fourth embodiment. For example, the determination unit 236 determines each contribution rate as a function of each contribution rate and the sum of the distances between joints during the period, as shown in the graph of Figure 15. According to the graph of Figure 15, the determination unit 236 determines the learner's contribution rate with β% as the upper limit when the sum of the distances between joints during the period is 0, and θ% as the lower limit when the sum of the distances between joints during the period is K, such that the learner's contribution rate becomes smaller as the sum of the distances between joints during the period increases. Also, according to Figure 15, the determination unit 236 determines the teacher's contribution rate with α% as the upper limit when the sum of the distances between joints during the period is K, and γ% as the lower limit when the sum of the distances between joints during the period is 0, such that the teacher's contribution rate becomes larger as the sum of the distances between joints during the period increases.

[0116] Figure 15 illustrates a method for determining the contribution rate based on the sum of distances between joints within a given period, and the present invention is not limited to these methods. The determination unit 236 determines, by various means, that the greater the sum of distances between joints within a given period, the smaller the learner's contribution rate and the larger the teacher's contribution rate.

[0117] Specifically, the determination unit 236 refers to the setting information 222 and determines the learner's contribution rate and the teacher's contribution rate based on a function of the sum of the distances between joints during the period and each contribution rate.

[0118] Next, the operation of the motion control device 2 according to the fourth embodiment will be described. When the VR system 1 is started, the motion control device 2 first performs the initialization process shown in Figure 8. Then, the tracking device 102 starts tracking and outputs tracking data to the motion control device 2, and the motion control device 2 performs the control process shown in Figure 16. Note that the control process is an example of the motion control method of the disclosed technology.

[0119] The initialization process shown in Figure 8 is the same as the initialization process in the first embodiment, so its explanation will be omitted.

[0120] Next, the control process shown in Figure 16 will be described. Figure 16 is a flowchart showing an example of the control process according to the fourth embodiment. Note that processes similar to those in the first embodiment are denoted by the same reference numerals, and detailed explanations are omitted.

[0121] If, after step S21, step S22 determines that the setting information 222 includes a setting for controlling the learner avatar using fused body technology (step S22: Yes), then in step S51, the determination unit 234 determines whether a certain period of time has elapsed since the previous update of each contribution rate. If it is determined that a certain period of time has elapsed (step S51: Yes), the process proceeds to step S23; if it is determined that a certain period of time has not elapsed (step S51: No), the process proceeds to step S25.

[0122] In step S23, the calculation unit 235 calculates the distance, and in step S52, the calculation unit 235 further calculates the sum of the distances between joints within the period.

[0123] In step S53, the determination unit 236 refers to the function information of the setting information 222 based on the sum of the distances between joints within the period, and determines and updates the learner's contribution rate and the teacher's contribution rate.

[0124] In step S25, the position of each joint of the learner avatar is calculated based on each contribution rate. When the learner avatar is controlled in step S26 or step S27, in step S28, the output unit 239 generates output images to be transmitted to HMD101A and HMD101B, and the communication unit 210 transmits the respective output images to HMD101A and HMD101B. The control unit 230 then proceeds to step S21 and repeats the control process.

[0125] As described above, in the VR system 1 of the fourth embodiment, the motion control device 2 controls the learner's contribution rate and the teacher's contribution rate to the learner avatar based on the distance between the body parts of the learner and the teacher, similar to the first embodiment. Specifically, the learner's contribution rate and the teacher's contribution rate to the learner avatar are determined based on the sum of the distances between the joints between the learner and the teacher. This makes it possible to control the contribution of the learner's and teacher's movements to the avatar in the fused body technology according to the learner's learning status.

[0126] Furthermore, the motion control device 2 determines whether or not to update each contribution rate based on the sum of the distances between joints over a certain period since the previous update of each contribution rate, and determines each contribution rate based on the sum of the distances between joints over a certain period since the previous update of each contribution rate. This allows the learner avatar to be controlled based on differences in movement within a certain period, and each contribution rate to be controlled based on the evaluation of movement within a certain period.

[0127] Next, we will describe specific learning actions for implementing each embodiment.

[0128] For example, this technology can be applied to learning movements that involve moving the body according to certain rules. More specifically, the guidance operation of a crane truck will be used as an example. Figure 17 shows the real space and VR space in each embodiment. In Figure 17, tracking devices 102A and 102B are described as using the Outside-In method, but for example, the Inside-Out method may also be used. To acquire the guidance operation of a crane truck, learner 3 repeatedly practices the guidance operation on the crane truck object 8 in the VR space 5. The guidance operation that the learner acquires is such that only the movement of the upper limbs among the body movements affects the success or failure of the operation.

[0129] Learner 3 wears the HMD 101A. Tracking device 102A tracks Learner 3, motion control device 2 processes the tracking data, and the HMD 101A displays the image generated by motion control device 2. As a result, Learner 3 operates as a Learner Avatar 6 in the VR space 5. Tracking device 102B also tracks Teacher 4. Each tracking device 102 tracks the entire body of Learner 3 and Teacher 4.

[0130] The motion control device 2 controls the learner avatar 6 according to the control processing in each embodiment, based on the tracking data of the learner 3 output by the tracking device 102A and the tracking data of the teacher 4 output by the tracking device 102B. For example, the acquisition unit 232 acquires tracking data of the entire body of the learner 3 and the teacher 4 from the tracking device 102. The estimation unit 233 identifies the right shoulder as the reference joint for the right upper limb and the left shoulder as the reference joint for the left upper limb, and calculates joint position information for the upper limbs of the learner 3 and the teacher 4 with the reference joint as the origin, based on the acquired tracking data. Based on the joint position information with the reference joint as the origin, the motion control device 2 controls the part of the learner avatar 6 corresponding to each joint of the upper limb of the learner 3, that is, the position of each joint of the upper limb of the learner avatar 6.

[0131] According to the first embodiment, as shown in Figure 17, when the position of the learner's hand 3 is low and there is a large discrepancy with the teacher's movement, the distance between each joint of the learner's hand 3 and the teacher's hand 4 increases. Therefore, the motion control device 2 determines that the teacher's contribution rate is larger. The motion control device 2 also controls the learner's avatar 6 based on the determined contribution rate and outputs an image reflecting the control result to the HMD 101A.

[0132] This allows learner 3 to recognize that they are raising their hands higher in VR space 5 compared to their actual movements, and to improve their own movements.

[0133] Furthermore, as learner 3's actions improve, the discrepancy between learner 3's actions and teacher 4's actions decreases. As a result, the action control device 2 determines learner 3's contribution rate to be larger and controls the learner avatar 6 accordingly. This allows learner 3 to progress in learning actions without losing their sense of agency.

[0134] Furthermore, according to the second embodiment, during the period when the distance from the start of learning is greater than a threshold, the motion control device 2 controls the learner avatar 6 using pre-set contribution rates such that the learner 3's contribution rate is greater than the teacher 4's contribution rate. This allows the learner 3 to maintain a sense of agency even when they are unable to perform actions properly for reasons such as not being accustomed to operating in the VR space 5.

[0135] Furthermore, during the period after the distance falls below a threshold, the motion control device 2 performs the same processing as in the first embodiment. This makes it possible to more effectively learn movements using integrated bodily technology while maintaining the learner's sense of agency.

[0136] Furthermore, according to the third embodiment, during the period when the distance from the learner 3 to the learning space is greater than a threshold, the motion control device 2 does not control the learner avatar 6 using fused body technology, but instead controls the learner avatar 6 based on the learner 3's actions. This allows the learner 3 to maintain a sense of agency even when they are unable to perform actions properly for reasons such as not being accustomed to operating in the VR space 5.

[0137] Furthermore, during the period after the distance falls below a threshold, the motion control device 2 performs the same processing as in the first embodiment. This makes it possible to more effectively learn movements using integrated bodily technology while maintaining the learner's sense of agency.

[0138] Furthermore, according to the fourth embodiment, the motion control device 2 determines whether or not to update each contribution rate based on the sum of the distances between joints over a certain period of time since the previous update of each contribution rate, and determines each contribution rate based on the sum of the distances between joints over a certain period of time since the previous update of each contribution rate. This makes it possible to control the learner avatar 6 based on the distance over a certain period of time, and to control each contribution rate based on the evaluation of the movements over a certain period of time.

[0139] Each component of each part illustrated in each embodiment does not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each part are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. For example, the calculation unit 235 and the determination unit 236 may be integrated. Furthermore, each process illustrated in each embodiment is not limited to the order shown above, and may be performed simultaneously or in a different order, as long as the processing content is not contradictory.

[0140] In addition, in each embodiment, the teacher 4 may wear the HMD 101B. In this case, the teacher 4 can recognize the state of the VR space 5 and the movements of the learner avatar 6 in the VR space 5 as a teacher avatar 7 that operates based on the tracking data output by the tracking device 102B. In this case, the motion control device 2 may generate output images such that the teacher avatar 7 is not displayed in the images output to the HMD 101A.

[0141] Furthermore, the motion control device 2 may display the learner avatar 6's movements within the VR space 5 on a connected monitor or the like. In this case, the teacher 4 can recognize the movements of the learner avatar 6 by looking at the monitor or the like. At this time, the motion control device 2 may, for example, communicate with a camera or the like installed around the learner 3 in the real space to display the learner 3's movements in the real space on the monitor. This allows the teacher 4 to intervene in the learner avatar 6's movements while also being aware of the learner 3's movements in the real space.

[0142] Furthermore, the tracking device 102 may track the entire bodies of the learner 3 and teacher 4, or it may track only the necessary parts. For example, if the learner 3 is learning the guidance operation of a crane, the tracking device 102 may track only the upper limbs, which are relevant to the success or failure of the operation. Alternatively, the tracking device 102 may track the entire bodies of the learner 3 and teacher 4, and the motion control device 2 may select the body parts to process as appropriate during the process of performing control processing. For example, when the estimation unit 233 estimates the position information of the joints of the learner 3 and teacher 4, it may be configured to estimate the position information of only the pre-set joints. In this case, the motion control device 2 may apply the processing of each embodiment to the pre-set joints of the learner avatar 6, and process the other joints of the learner avatar 6 so that only the movements of the learner 3 are reflected.

[0143] Furthermore, in each embodiment, tracking device 102A tracks learner 3 and tracking device 102B tracks teacher 4, but this is not limited to this configuration. For example, one tracking device 102 may track both learner 3 and teacher 4 while identifying them.

[0144] Furthermore, in each embodiment, the motion control device 2 controls the learner avatar 6 based on the tracking data output by the tracking device 102, but this is not limited to this. For example, the motion control device 2 may store pre-recorded tracking data of the teacher and calculate the positions of the teacher's body parts based on the stored tracking data. Alternatively, for example, the motion control device 2 may store pre-created ideal motion information and use the positions of the body parts calculated based on the stored motion information as the positions of the teacher's body parts.

[0145] As described above, when the positions of the teacher's body parts are calculated based on pre-recorded or created data, the motion control device 2 may perform a process to indicate to learner 3 the timing to start the action in order to synchronize the timing of the teacher's and learner 3's actions. For example, the motion control device 2 may superimpose a count or the like indicating the timing to start the action onto the image output to the HMD 101A.

[0146] Furthermore, while each embodiment describes a case where the positions of the joints of learner 3 and teacher 4 are estimated as body parts to control the avatar's posture, the body parts of the disclosed technology are not limited to these. For example, the positions of body parts of a person such as hands, fingertips, or head may be estimated as body parts, or the posture of a person may be estimated from the position and orientation of a controller worn by the person to control the avatar's posture.

[0147] Furthermore, while the sum of distances between joints was described as an example of the difference in motion in each embodiment, the motion control device 2 may calculate the difference in motion based on something other than distance. For example, the motion control device 2 may estimate the velocity and angle of each joint of the learner 3 and the teacher 4, and calculate the difference in motion based on the estimated velocity and angle.

[0148] Specifically, for example, the estimation unit 233 refers to the past position information of each joint of learner 3 and teacher 4 stored in the position information 221 of the memory unit 220. Next, the estimation unit 233 estimates the velocity of each joint of learner 3 and teacher 4 by finding the difference between the estimated position information of each joint of learner 3 and teacher 4 and the corresponding past position information of each joint of learner 3 and teacher 4. The velocity of each joint of learner 3 is an example of the first indicator of the disclosed technology, and the velocity of each joint of teacher 4 is an example of the second indicator of the disclosed technology. The calculation unit 235 calculates the difference in velocity between corresponding joints of learner 3 and teacher 4 estimated by the estimation unit 233, and calculates the velocity difference in VR space 5 between corresponding joints of learner 3 and teacher 4. Hereinafter, the velocity difference in VR space 5 between corresponding joints of learner 3 and teacher 4 may simply be called the velocity difference. The calculation unit 235 then calculates the sum of the calculated velocity differences for each joint, and the motion control device 2 may process the sum of the velocity differences for each joint as the difference in motion. Alternatively, the calculation unit 235 may calculate the sum of the magnitudes of the calculated speed differences for each joint, and the motion control device 2 may process the sum of the magnitudes of the speed differences for each joint as the difference in motion.

[0149] Furthermore, for example, the estimation unit 233 estimates the angles of each joint of learner 3 and teacher 4 based on the tracking data acquired by the acquisition unit 232. The angles of each joint of learner 3 are an example of the first indicator of the disclosed technology, and the angles of each joint of teacher 4 are an example of the second indicator of the disclosed technology. The calculation unit 235 calculates the difference in angles between corresponding joints of learner 3 and teacher 4, which was estimated by the estimation unit 233, and calculates the angle difference in VR space 5 between corresponding joints of learner 3 and teacher 4. Hereinafter, the angle difference in VR space 5 between corresponding joints of learner 3 and teacher 4 may simply be referred to as the angle difference. The calculation unit 235 then calculates the sum of the calculated angle differences for each joint, and the motion control device 2 may process the sum of the angle differences for each joint as the difference in motion.

[0150] Furthermore, for example, the estimation unit 233 estimates the rotation angles of each body part, such as the head and arms, based on the tracking data acquired by the acquisition unit 232. The rotation angles of each body part of learner 3 are an example of the first indicator of the disclosure technique, and the rotation angles of each body part of teacher 4 are an example of the second indicator of the disclosure technique. The calculation unit 235 calculates the difference in rotation angles between corresponding body parts of learner 3 and teacher 4, which was estimated by the estimation unit 233, and calculates the difference in rotation angles between corresponding body parts of learner 3 and teacher 4 in the VR space 5. Hereinafter, the difference in rotation angles between corresponding body parts of learner 3 and teacher 4 in the VR space 5 may simply be referred to as the rotation angle difference. The calculation unit 235 then calculates the sum of the calculated rotation angle differences for each body part, and the motion control device 2 may process the sum of the rotation angle differences for each body part as the difference in motion.

[0151] Furthermore, the calculation unit 235 may combine the calculated distance, speed difference, angle difference, rotation angle difference, etc., to calculate the difference in motion. For example, the calculation unit 235 may calculate the weighted sum of the sum of the distances at each joint and the sum of the magnitudes of the speed differences at each joint, and use this as the difference in motion.

[0152] Furthermore, in each embodiment, the motion control device 2 controlled the position of parts of the learner avatar 6, but it is not limited to this, and for example, the motion control device 2 may also control the angles, rotation angles, etc., of the body parts of the learner avatar 6. For example, the estimation unit 233 may estimate the rotation angles of the heads of the learner 3 and teacher 4 based on tracking data, and the synthesis unit 237 may calculate the rotation angle of the head of the learner avatar 6 based on the estimated rotation angles. Then, the processing unit 238 may control the rotation angle of the head of the learner avatar 6 based on the calculation result of the synthesis unit 237.

[0153] In addition to the configurations described in each embodiment, the motion control device 2 may also control the VR space 5 and generate output images when the learner 3 or the learner avatar 6 is performing the correct actions.

[0154] Specifically, for example, the determination unit 234 may refer to the position information 221 and determine whether the learner's movements are correct based on whether the sum of the distances between joints over a certain period of time is less than or equal to a threshold included in the pre-configured information 222. Alternatively, for example, the determination unit 234 may compare the movements of learner 3 over a certain period of time obtained by referring to the position information 221 with examples of correct movements included in the pre-configured information 222 and determine whether the movements of learner 3 are correct. Furthermore, for example, the determination unit 234 may determine whether the movements of learner avatar 6 are correct based on whether the joints of learner avatar 6 are located in the position information included in the pre-configured information 222.

[0155] If the determination unit 234 determines that the actions of the learner or the learner avatar 6 are correct, the processing unit 238 controls and operates the structure in the VR space 5. For example, if learner 3 is learning the guidance operation of a crane, if the determination unit 234 determines that the actions of learner 3 or the learner avatar 6 are correct, the operation control device 2 may operate the crane object 8 in the VR space 5 according to the guidance operation. Also, if the determination unit 234 determines that the actions of learner 3 are correct, the output unit 239 may superimpose an indication on the output image that the actions are correct.

[0156] As described above, the motion control device 2 may perform VR space control and output image generation when the learner 3 or learner avatar 6 is performing the correct actions. This allows the learner to perceive the correct actions more efficiently, thereby improving the efficiency of learning.

[0157] Furthermore, while we have shown a specific example of applying each embodiment to a learner learning the guidance operation of a crane, the actions that learners learn are not limited to this. For example, it could be actions that the learner performs on an object that they can manipulate.

[0158] For example, learners may learn to operate a machine. For example, learners and teachers may operate a machine included in object information 224 placed in the VR space, and an operation panel included in object information 224 placed in the VR space. The motion control device 2 may calculate the difference in actions from the actions of the learner and teacher, and control the learner avatar in the VR space based on the calculated difference in actions and the learner's actions. Alternatively, the motion control device 2 may control the machine and operation panel in the VR space based on the actions of the learner avatar.

[0159] Furthermore, for example, the learner may learn how to drive a car. The learner and the teacher perform actions to operate the steering wheel, pedals, etc., which constitute the car and are included in the object information 224, placed in the VR space. The motion control device 2 may calculate the difference in actions from the actions of the learner and the teacher, and control the learner avatar based on the calculated difference and the learner's actions. The motion control device 2 may also control the position of the car in the VR space, as well as the steering wheel, pedals, etc., which constitute the car, based on the actions of the learner avatar.

[0160] The present invention is not limited to the movements that learners are meant to learn, and can be applied to learning various movements using integrated body technology, allowing learners to learn various movements through the present invention.

[0161] Furthermore, the various processing functions performed by each device may be executed in whole or in part on a CPU (or a microcomputer such as an MPU or MCU (Micro Controller Unit)). It goes without saying that the various processing functions may also be executed in whole or in part on a program analyzed and executed by the CPU (or a microcomputer such as an MPU or MCU), or on wired logic hardware.

[0162] Incidentally, the various processes described in each of the above embodiments can be realized by executing a pre-prepared program on a computer. Therefore, below, an example of a computer that executes a program having the same functions as those in each of the above embodiments will be described. Figure 18 is a block diagram showing an example of a computer that executes a program.

[0163] As shown in Figure 18, the computer 300 includes a CPU 301 that performs various calculations, an input device 302 that accepts data input, and a monitor 303. The computer 300 also includes a media reader 304 that reads programs and the like from a storage medium, an interface device 305 for connecting to various devices, and a communication device 306 for connecting to other information processing devices via wired or wireless connections. The computer 300 also includes a RAM 307 for temporarily storing various information and a hard disk drive 308. Each of the devices 301 to 308 is connected to a bus 309.

[0164] The hard disk drive 308 stores a program 308A that implements the same functions as the construction unit 231, acquisition unit 232, estimation unit 233, determination unit 234, calculation unit 235, decision unit 236, synthesis unit 237, processing unit 238, and output unit 239 shown in Figure 1. The hard disk drive 308 also stores various data such as location information 221, setting information 222, spatial information 223, and object information 224. The input device 302 receives input of various information, such as changes to the setting information 222, from the user of the computer 300. The monitor 303 displays various screens, such as the display screen, to the user of the computer 300. The communication device 306 is connected to a network (not shown) and exchanges various information with other devices.

[0165] The CPU 301 reads program 308A stored in the hard disk drive 308, loads it into RAM 307, and executes it, thereby running a process that performs various functions of the computer 300. In other words, this process performs functions similar to those of each processing unit in the computer 300. Specifically, the CPU 301 reads program 308A from the hard disk drive 308 to realize functions similar to those of the construction unit 231, acquisition unit 232, estimation unit 233, determination unit 234, calculation unit 235, decision unit 236, synthesis unit 237, processing unit 238, and output unit 239. Then, the CPU 301 executes a process that performs the same processing as the construction unit 231, acquisition unit 232, estimation unit 233, determination unit 234, calculation unit 235, decision unit 236, synthesis unit 237, processing unit 238, and output unit 239.

[0166] Note that the above program 308A does not necessarily have to be stored in the hard disk drive 308. For example, the computer 300 may read and execute the program 308A stored on a storage medium that the computer 300 can read. Examples of storage media that the computer 300 can read include portable recording media such as CD-ROMs, DVDs (Digital Versatile Discs), and USB (Universal Serial Bus) memory, semiconductor memory such as flash memory, and hard disk drives. Alternatively, the program 308A may be stored on a device connected to a public network, the internet, or a LAN, and the computer 300 may read and execute the program 308A from there. [Explanation of Symbols]

[0167] 1 VR system 2. Operation control device 3. Learners 4 teacher 5 VR space 6. Learner Avatars 7 Teacher Avatar 8 Crane Truck Object 101, 101A, 101B HMD 102, 102A, 102B Tracking device 103 Controller 210 Communications Department 220 Storage section 221 Location information 222 Configuration Information 223 Spatial Information 224 Object Information 230 Control Unit 231 Construction Department 232 Acquisition Department 233 Estimation Department 234 Judgment section 235 Calculation Unit 236 Decision Section 237 Synthesis section 238 Processing Unit 239 Output section 300 Computers 301 CPU 302 Input device 303 Monitor 304 Media Reader 305 Interface device 306 Communication equipment 307 RAM 308 Hard Disk Drives 308A Program Bus 309

Claims

1. In a motion control method for controlling the movement of an avatar in a VR space, Identify a first indicator concerning the learner's body parts related to the learner's posture during a specific movement, and a second indicator concerning the teacher's body parts related to the teacher's posture during the specific movement. Based on the first indicator and the second indicator, the difference in behavior between the learner and the teacher is calculated. Based on the difference in the operation and the first indicator, the avatar is controlled. An operation control method characterized by causing a computer to perform the following action.

2. Based on the differences in the aforementioned operations, a first percentage is determined in which the first indicator contributes to the third indicator. The third indicator is determined based on the first ratio, the first indicator, and the second indicator. Based on the third indicator, the part of the avatar corresponding to the body part is controlled. The operation control method according to claim 1.

3. The greater the difference in the aforementioned operation, the larger the determination of the first ratio. The operation control method according to claim 2.

4. Until the difference in the aforementioned operation falls below a threshold, the first ratio is determined to be greater than the second ratio to which the second indicator contributes to the third indicator. The operation control method according to claim 2.

5. After the difference in the aforementioned operation falls below the threshold, the larger the difference in the aforementioned operation, the larger the first ratio is determined to be. The operation control method according to claim 4.

6. Until the difference in the aforementioned operation falls below a threshold, the avatar is controlled based only on the first indicator among the difference in the aforementioned operation and the first indicator. The operation control method according to claim 1.

7. After the difference in the operation falls below the threshold, a first percentage of the first indicator's contribution to the third indicator is determined based on the difference in the operation. The third indicator is determined based on the first ratio, the first indicator, and the second indicator. Based on the third indicator, the part of the avatar corresponding to the body part is controlled. The operation control method according to claim 6.

8. After the difference in the aforementioned operation falls below the threshold, the greater the difference in the aforementioned operation, the larger the first ratio is determined to be. The operation control method according to claim 7.

9. The difference in the aforementioned actions is the distance between the learner's body part and the teacher's body part. The operation control method according to any one of claims 1 to 8.

10. In a motion control device that controls the movement of an avatar in a VR space, A calculation unit calculates the difference in movement between the learner and the teacher based on a first index where a specified body part of the learner is located and a second index where a specified body part of the teacher is located. A processing unit that controls the position of the avatar part corresponding to the body part based on the difference in the aforementioned operation and the first indicator, An operating control device equipped with the necessary components.

11. In a motion control program that controls the movement of an avatar in a VR space, Identify a first indicator where the learner's body part is located for a specific movement, and a second indicator where the teacher's body part is located for a specific movement. Based on the first indicator and the second indicator, the difference in behavior between the learner and the teacher is calculated. Based on the difference in the aforementioned movements and the first indicator, the position of the avatar part corresponding to the body part is controlled. An operation control program characterized by causing a computer to perform a process that includes the following.

Images