Information processing apparatus, system of information processing apparatus, method for controlling information processing apparatus, and program

Abstract

To reduce a situation where not all work operations of a learner can be recognized depending on the position or the direction of a body part of the learner.SOLUTION: An information processing apparatus has: first acquisition means that acquires, from recording means, first information on the position or the posture of a virtual object corresponding to an instructor; second acquisition means that acquires second information on the position or the posture of an operating device supported by a learner; and control means that controls playback of images of the virtual object corresponding to the instructor on the basis of the difference between the first information acquired by the first acquisition means and the second information acquired by the second acquisition means.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] In conventional cross reality (XR) systems that allow users to experience virtual reality, a head-mounted display (HMD) is a head-mounted display device equipped with a small display that the user wears on the head. HMDs are used for learning work processes in facilities such as factories and for learning sports. In this case, the HMD displays the instructor during the work process or sports learning as a virtual object, and the wearer of the HMD can view the virtual object in conjunction with their own movements.

[0003] Patent document 1 discloses a method for controlling the playback speed of a model video based on the work movements of a learner contained in a field of view image captured by an imaging unit, so that the speed of the instructor's work movements contained in the model video matches the speed of the learner's work movements. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-144233 A Summary of the Invention [Problem to be solved by the invention]

[0005] The conventional technology disclosed in the above-mentioned Patent Document 1 is based on the work movements of the learner contained in the visual field image captured by the imaging unit, so depending on the position or orientation of the learner's body parts, the work movements of the learner may be hidden and may not be fully recognized. Therefore, the present invention aims to reduce the inability to fully recognize the work movements of the learner due to the position or orientation of the learner's body parts. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device comprising: a first acquisition means for acquiring first information relating to a position or posture of a virtual object corresponding to an instructor from a recording means; a second acquisition means for acquiring second information relating to a position or posture of an operating device supported by a learner; and a control means for controlling playback of an image of the virtual object corresponding to the instructor based on a difference between the first information acquired by the first acquisition means and the second information acquired by the second acquisition means. Effect of the Invention

[0007] According to the present invention, it is possible to reduce the occurrence of a situation where a learner's working movements are not fully recognized due to the positions and orientations of parts of the learner's body. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1A is a diagram for explaining an information processing system according to a first embodiment, and FIG. 1B is a diagram for explaining a block diagram according to the first embodiment. [Diagram 2] FIG. 1A is a diagram showing an example in which an instructor is displayed as a virtual object in the first embodiment, and FIG. 1B is a diagram showing an example in which the virtual object is displayed semi-transparently when the wearer is moving in the first embodiment. [Diagram 3] 13A is a table showing movement information of a virtual object in the first embodiment, and FIG. 13B is a table showing movement information of a wearer in the first embodiment. [Figure 4] 5 is a flowchart for determining whether a wearer is moving in the first embodiment. [Diagram 5] FIG. 1A is a diagram showing an example of displaying a virtual object superimposed on the hand of a wearer in the first embodiment, and FIG. 1B is a diagram showing an example of displaying a virtual object so that the hand of the wearer is displayed on top of the virtual object in the first embodiment. [Figure 6]FIG. 13A is a schematic diagram of an HMD wearer photographed by an external camera in the second embodiment, and FIG. 13B is a system configuration diagram of an HMD wearer photographed by an external camera in the second embodiment. [Figure 7] FIG. 10A is a diagram showing an example of an image displayed when the difference is within the wearer's field of view in the second embodiment, and FIG. 10B is a diagram showing an example of an image displayed when the difference is outside the wearer's field of view in the second embodiment. [Figure 8] 6 is a flowchart for comparing movement information of a virtual object and a wearer in the first embodiment. [Figure 9] FIG. 1A is a diagram showing an example of starting video playback of a virtual object in the first embodiment; FIG. 1B is a diagram showing an example of stopping video playback of a virtual object in the first embodiment; and FIG. 1C is a diagram showing an example of resuming video playback of a virtual object in the first embodiment. [Figure 10] 6 is a flowchart for stopping playback of a virtual object in the first embodiment. [Figure 11] 13 is a flowchart for determining whether to change the display during second or subsequent learning in the third embodiment. [Figure 12] 4A to 4C are diagrams illustrating a movement trajectory of a virtual object in a predetermined section, a movement trajectory of a wearer, and a difference therebetween in the first embodiment. [Figure 13] 13 is a flowchart for determining whether to play the next series of movements or to play the same series of movements again after playing a series of movements according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiment described below is one example of a means for realizing the present invention, and may be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. Also, each embodiment may be appropriately combined.

[0010] (First embodiment) <System configuration> An information processing system according to a first embodiment will be described with reference to Fig. 1(a). The information processing system includes an HMD 100 including a display device 101 and a control device 102, and a controller 200. The controller 200 is a device for performing various operations of the HMD 100.

[0011] The internal configuration of an HMD and a controller, which are examples of an information processing device, will be described with reference to FIG.

[0012] <Configuration of the display device 101> The display device 101 includes an imaging unit 111 , a display unit 112 , a position and orientation detection unit 113 , and an operation unit 114 .

[0013] The display device 101 is configured, for example, as a glasses-type display portion of an HMD, and detects the position and orientation of a user wearing the display device 101. Then, a composite image is displayed in which a captured image capturing an area in front of the user and a virtual object shown in CG (computer graphics) in a form corresponding to the detected position and orientation are combined.

[0014] This allows a user wearing the display device 101 to observe a virtual reality image in which CG in a virtual space is superimposed according to the line of sight in real space. In order to generate a stereo image, two display units for the right eye and the left eye may be implemented. In the following description, the display device 101 is assumed to be a glasses-type display part of an HMD, but it may also be a display device such as a tablet terminal or a smartphone. In other words, any display device that is portable and can display an image corresponding to the user's field of view can be used.

[0015] The imaging unit 111 is composed of an objective optical system that captures a real image of the outside world as light, and an image sensor that converts an optical signal into an electrical signal. The imaging unit 111 includes two cameras (imaging devices). The two cameras are for capturing captured images used for compositing with an image of a virtual space and generating position and orientation information, and have an imaging unit for a left eye and an imaging unit for a right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the person wearing the display device 101, and an image (captured image) of each frame in the moving image is output from the imaging unit for the left eye. The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the person wearing the display device 101, and an image (captured image) of each frame in the moving image is output from the imaging unit for the right eye. In other words, the imaging unit 111 acquires captured images as stereo images having parallax that approximately matches the positions of the left eye and right eye of the person wearing the display device 101. By measuring distances using the stereo cameras, information on the distances from the two cameras to the subject can be acquired as distance information. In an HMD for an MR (Mixed Reality) system, the imaging unit is preferably disposed so that the central optical axis of the imaging range of the imaging unit is approximately aligned with the line of sight of the wearer of the HMD. Alternatively, the imaging unit may be disposed at a position where the central optical axis of the imaging range of the imaging unit does not match the line of sight of the wearer of the HMD, and the position of the viewpoint may be converted to match the line of sight of the wearer of the HMD (a so-called view conversion method).

[0016] Each of the left eye imaging section and the right eye imaging section has an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the entering light as a captured image. The images of a subject (the area in front of the user) captured by the two cameras are output to the control device 102. Note that the imaging section 202 may capture and output a video instead of a captured image.

[0017] The display unit 112 displays an image generated by the control device 102. The display unit 112 has a liquid crystal panel or an organic EL panel. When the user wears the display device 101, the display unit 112 is disposed in front of each eye of the user. It is also possible to use a device using a semi-transmissive half mirror for the image display unit 203. In this case, for example, the display unit 112 may display an image by using a technology generally called AR (Augmented Reality) so that the CG is superimposed directly on the real space seen through the half mirror. The image display unit 203 may also display an image of a complete virtual space without using a captured image by using a technology generally called VR (Virtual Reality).

[0018] The position and orientation detection unit 113 is a functional unit for acquiring the position and orientation of the display device 101, and can calculate the change in orientation, relative direction, and position in real space of the display device 101. For example, an IMU (inertial measurement unit; inertial sensor) capable of detecting inertial information (amount of spatial movement and angle), a direction sensor using geomagnetism, and an orientation detection sensor using GPS can be used.

[0019] It is assumed that the position and orientation detection unit 113 is a functional unit that acquires at least one of the position and orientation of the display device 101 .

[0020] The operation unit 114 is a device for the user to operate the HMD 100. For example, it may be an operation member such as a button as shown in Fig. 4, or a mouse or a keyboard. By operating the operation unit 114, the user can switch between displaying and not displaying the display unit 112 and set the interpupillary distance, which will be described later.

[0021] <Configuration of the control device 102> The control device 102 includes a CPU 120, a ROM 130, a RAM 140, an inertial information receiving unit 150, a marker position information receiving unit 160, a skeleton information receiving unit 170, and a communication unit 180. An image acquiring unit 121, a position and orientation extracting unit 122, a control unit 123, and a movement information acquiring unit 124 are control blocks that operate on the CPU 120.

[0022] It is assumed here that the control device 102 is a device having high-performance arithmetic processing functions and graphic display functions, such as a smartphone, a PC (personal computer), or a workstation.

[0023] By connecting the control device 102 and the display device 101, a user wearing the display device 101 can view VR images, which are images of a virtual space. Note that instead of images of a virtual space, MR images, which are images of a mixed reality world in which the real world and the virtual world are seamlessly integrated in real time, may be used.

[0024] The image acquisition unit 121 acquires an image of the real space (real image) captured by the imaging unit 111 .

[0025] The position and orientation extraction unit 122 acquires the position and orientation information detected by the position and orientation detection unit 113. Furthermore, a configuration may be adopted in which a marker arranged in the real space is detected from the real image acquired by the image acquisition unit 121, and the position and orientation are calculated.

[0026] The control unit 123 generates an image in which the real image acquired by the image acquisition unit 121 is combined with CG, and transmits the combined image to the display unit 112. Therefore, the user can view the combined image displayed on the display unit 112 by wearing the display device 101. The user can experience various mixed realities, such as CG blended into real space. Note that instead of the control unit 211 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing.

[0027] The control unit 123 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the display device 101. For example, when the control unit 123 places a virtual object represented by the CG near a specific object existing in real space in the space represented by the composite image, the control unit 123 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 111 is closer. By controlling the position, orientation, and size of the CG in this manner, the control unit 123 can generate a composite image in which a CG object that is not placed in real space appears as if it were placed in real space.

[0028] Furthermore, in the control unit 123, the communication unit 150 receives information about a change in the position or attitude of the controller 200 from the communication unit 205 of the controller 200. The control unit 123 superimposes on the combined image an indication position according to the information about the change in the position or attitude of the controller 200. Note that the control unit 123 may also superimpose on the combined image an indication position according to the information about the change in the position and attitude of the controller 200.

[0029] The movement information acquisition unit 124 acquires movement information regarding the movements of the hands, feet, and fingers of the HMD wearer from information received from at least one of the inertial information receiving unit 150, the marker position information receiving unit 160, and the skeletal information receiving unit 170 described below.

[0030] The ROM 130 is an electrically erasable and recordable non-volatile memory that stores information such as CG. The control unit 123 generates the instructor's hands, feet, and fingers as CG, which are virtual objects, from the movement information of the instructor's hands, feet, and fingers and the human body shape information stored in the ROM 130, and reproduces the generated virtual object as a video according to the movement information and displays it on the display unit 112. The movement information of the instructor's hands, feet, and fingers stored here may be movement information of another person who will be a teacher in a work process in a facility such as a factory or when learning a sport, or may be movement information of the instructor's own movements in the past. The control unit 123 can switch the CG to be read from the ROM 130 (i.e., the CG used to generate a composite image). In order to display the instructor's hands, feet, and fingers as virtual objects, the instructor's body movements may be acquired in advance. In this case, the body movements may be acquired from a captured image or video, or may be acquired by a method of acquiring movement information using a marker or a tracking device, which will be described later. The student's movement information may be acquired by a method different from the method used to acquire the instructor's movements.

[0031] The RAM 140 is used as a buffer memory for temporarily storing image data captured by the imaging unit 111, an image display memory for the display unit 112, a working area for the control unit 123, etc. It also temporarily stores data on the positions, postures, and movement trajectories of the learner's body parts.

[0032] The inertial information receiving unit 150 receives inertial information from tracking devices attached to the wearer's hands, feet, or fingers. The movement information acquiring unit 124 calculates movement information related to the movements of the wearer's hands, feet, or fingers from the inertial information received from the inertial information receiving unit 150.

[0033] The marker position information receiving unit 160 receives marker position information from a device that captures images of the markers on the wearer's hands, feet, and fingers with an external camera and measures the positions of the markers. The movement information acquiring unit 124 calculates movement information related to the movements of the wearer's hands, feet, and fingers from the marker position information received from the marker position information receiving unit 160.

[0034] The skeletal information receiving unit 170 is a means for receiving skeletal information from a device that captures an image of the wearer with an external camera and generates skeletal information of the wearer from the captured image. The movement information acquiring unit 124 calculates movement information related to the movements of the wearer's limbs and fingers from the skeletal information received from the skeletal information receiving unit 170.

[0035] The control unit 123 controls the HMD 100 based on values ​​input from the controller 200 acquired via the communication unit 180 .

[0036] The communication unit 180 is an interface for communicating with an external device via so-called wireless LAN or Bluetooth (registered trademark) in accordance with the IEEE802.11 standard. However, the communication method is not limited to wireless LAN or Bluetooth, and any communication protocol may be used, regardless of whether it is wireless or wired, as long as communication can be performed. The HMD 100 communicates with the controller 200 via the communication unit 180. The communication unit 180 can also communicate with devices other than the controller 200, such as a smartphone or a tablet terminal.

[0037] The display device 101 and the control device 102 are connected to each other so that they can communicate data with each other. Therefore, the connection between the display device 101 and the control device 102 may be wired or wireless. The display device 101 and the control device 102 may be integrated and configured so as to be portable by the user. For example, they may be a so-called stand-alone HMD.

[0038] The position and orientation of the display device 101 may be estimated from an image captured by the display device 101. SLAM (Simultaneous Localization and Mapping) is known as a method for estimating a self-position using a captured image. SLAM is disclosed as a method for detecting the positions of feature points in a camera image and determining a self-position in a three-dimensional space from the amount of movement of the feature point positions between frames. It is known that various disclosed algorithms of SLAM are used to estimate a self-position in a three-dimensional space from these feature points.

[0039] <Configuration of controller 200> 1(b), the internal configuration of the controller 200 will be described. The controller 200 has a CPU 201, a position and orientation detection unit 202, an operation unit 203, a vibration unit 204, and a communication unit 205.

[0040] The CPU 201 is a control unit that controls each component of the controller 200. Note that instead of the CPU 201 controlling the entire device, the entire device may be controlled by having a plurality of hardware devices share the processing.

[0041] The position and orientation detection unit 202 is a functional unit for acquiring the position and orientation of the controller 200, and can calculate the orientation change, relative direction, and position of the controller 200 in real space. For example, an IMU (inertial measurement unit; inertial sensor) capable of detecting inertial information (spatial movement amount and angle), a direction sensor using geomagnetism, and an orientation detection sensor using GPS can be used. The position and orientation detection unit 202 may include any device that does not hinder the miniaturization of the controller 200 and can detect inertial information (information such as positional displacement, speed, or acceleration).

[0042] It is assumed that the position and orientation detection unit 202 is a functional unit that acquires at least one of the position and orientation of the controller 200.

[0043] The operation unit 203 may include any of a button, a touch pad, a touch panel, a cross key, a joystick, and a track pad device. For example, the user presses and holds a button to display a menu including a pointer on the HMD 100. After that, the user can move the pointer to a desired item by pressing the cross key in any direction. Then, the user can perform a decision action such as deciding the selection of the item by pressing the button. For example, it is possible to switch between displaying and not displaying a ray 1000, which will be described later in FIG. 10, by displaying and selecting the ray 1000 in a menu. Operation information in the operation unit 203 is transmitted to the HMD 100 via the communication unit 205.

[0044] The vibration unit 204 vibrates the controller 200. For example, when the ray 1000 touches the CG 1001, the CPU 201 may receive a vibration instruction from the HMD 100 via the communication unit 205, and control the vibration unit 204 to vibrate the controller 200. When the controller 200 vibrates, the user knows that the ray 1000 has touched the CG 1001.

[0045] The communication unit 205 performs wireless communication with the communication unit 105 of the HMD 100. When there are multiple controllers, each of them performs wireless communication with the communication unit 105.

[0046] The controller 200 may have an output unit. The output unit is configured with an LED light source, a speaker, and the like.

[0047] Note that a camera for estimating the controller's own position may be configured in the controller 200. The controller's own position in the three-dimensional space may be estimated by various disclosed algorithms of SLAM described above.

[0048] <Display: Virtual objects are displayed semi-transparently when the wearer is moving> FIG. 2(a) is a diagram showing an example in which an instructor is displayed as a virtual object. A virtual object 201 is the instructor's left hand, and a virtual object 202 is the instructor's right hand. A left hand 211 and a right hand 212 are the left and right hands of the person wearing the HMD photographed by the photographing unit 111. A member A (221), a member B (222), and a member C (223) are real members, and these are also real objects photographed by the photographing unit 111. The control unit 123 obtains images of the left hand 211, the right hand 212, the member A (221), the member B (222), and the member C (223) from the image acquisition unit 121, and confirms the positions of the member A (221), the member B (222), and the member C (223). Then, the virtual objects 201 and 202 are displayed at positions that appear to operate these members, and a composite image of the real object and the virtual object is generated and displayed on the display unit 112. Furthermore, the control unit 123 reproduces the virtual objects 201 and 202 as a moving image in accordance with the movement information stored in the ROM 130 .

[0049] FIG. 2(b) is a diagram showing an example in which a virtual object is displayed semi-transparently when the wearer is moving.

[0050] The control unit 123 determines whether the wearer is moving from the movement information of the wearer acquired by the movement information acquisition unit 124, and if the wearer is moving, displays the virtual objects 201 and 202 semi-transparently or in another color such as gray. Here, the virtual object 201 (left hand) is gripping member A (221) and the virtual object 202 (right hand) is pushing member C (223), but when the virtual objects 201 and 202 become semi-transparent, the wearer can easily visually recognize the members and can move in accordance with the movement of the virtual objects.

[0051] Note that the virtual object may be controlled so as to be semi-transparent at all times, not just when it is moving.

[0052] <Acquisition of movement information using tracking devices> The inertial information receiving unit 150 receives inertial information from a tracking device supported by the wearer, who is a learner, with his / her hands, feet, or fingers. The tracking device may be in a shape that can be held, or may be in a shape that can be attached to the hands, feet, or fingers. The tracking device may be a controller as shown in FIG. 1(a) and FIG. 1(b), or may be a glove-type device. The tracking device is not limited to a device that tracks hands, feet, or fingers, and may be attached to a tool so as to track the tool, or a tool with a built-in tracking device may be used. The movement information acquiring unit 124 calculates movement information related to the movement of the wearer's hands, feet, or fingers from the inertial information received from the inertial information receiving unit 150. The amount of movement of the position of the tracking device may be directly detected, but only the speed of movement of the tracking device or the acceleration of the movement may be directly detected. When the speed of movement is acquired, the speed can be integrated to calculate the amount of movement. When the acceleration of movement is acquired, the acceleration can be integrated twice to calculate the amount of movement.

[0053] In addition to the amount of movement of the position, the amount of movement of the attitude or the inclination may be acquired. The amount of movement of the attitude or the inclination may be calculated by integrating the angular velocity or integrating the angular acceleration twice.

[0054] Note that the movement information may be either a change in position or a change in attitude, or may include both the position and attitude.

[0055] 3(a) is a table showing movement information of virtual objects, and indicates how far virtual objects 201 and 202 have moved in each of the x, y, and z directions per second. The first line indicates time, and indicates movement information from 10:10:1 to 10:10:8. This information is stored in RAM 140. Sections 301 and 302, which indicate time periods, will be described later.

[0056] 3(b) is a table showing the movement information of the wearer, indicating how far the wearer's left hand 211 and right hand 212 have moved in each of the x, y, and z directions per second. The first line indicates the time, and represents the movement information from 10:10:01 to 10:10:08. The control unit 123 acquires this movement information from the movement information acquisition unit 124.

[0057] The control unit 123 determines whether the wearer is making the same movement as the virtual object based on whether the amount of movement in Figures 3(a) and (b) is the same. Note that even if the movement of the wearer differs from the time of the movement of the virtual object, it may be determined that the movement is the same if the movement trajectory is the same.

[0058] <Process to determine whether the wearer is moving> FIG. 4 is a flowchart for determining whether the wearer is moving.

[0059] In step S401, the control unit 123 acquires movement information of the wearer from the movement information acquisition unit 124, and the process proceeds to step S402.

[0060] In step S402, the control unit 123 determines whether the wearer is moving from the acquired movement information. The control unit 123 checks whether the amount of movement per second is less than a predetermined threshold, and if the time when the amount of movement is less than the threshold continues for a certain period of time, it determines that the wearer is not moving. Note that, although the determination here is based on whether the amount of movement per second is less than the threshold, the amount of movement per any number of seconds may be acquired, not limited to one second. If the control unit 123 determines that the wearer is moving, it proceeds to step S403, and if it determines that the wearer is not moving, it proceeds to step S404.

[0061] In step S403, the control unit 123 changes the display of the virtual object, displays the virtual object semi-transparently or in a different color such as gray, and then completes the process.

[0062] In step S404, the control unit 123 does nothing and ends the process.

[0063] In this way, when the wearer is moving, the virtual objects 201 and 202 become semi-transparent, making it easier for the wearer to visually recognize the components, and enabling the wearer to move in accordance with the movement of the virtual objects.

[0064] <Obtaining movement information using markers> Markers may be attached to the wearer's hands, feet, or fingers to acquire position information of the wearer's hands, feet, or fingers. In this case, the marker position information receiving unit 160 is a means for capturing images of the markers on the wearer's hands, feet, or fingers with a camera and receiving marker position information from a device that measures the positions of the markers. The movement information acquiring unit 124 calculates movement information related to the movements of the wearer's hands, feet, or fingers from the marker position information received from the marker position information receiving unit 160.

[0065] The camera that captures the marker may be an external device that is placed at a predetermined position in real space, and the position of the marker measured by the external device may be received by the marker position information receiving unit 160. The camera that captures the marker may be the imaging unit 111 of the display device 101, and the position information of the marker received by the marker position information receiving unit 160 may be measured by the control unit 123. The number of cameras that capture the marker is not limited to one, and the marker may be captured by multiple cameras, and the position of the marker may be measured from each captured image or video. The marker is not limited to being attached to the wearer, and may also be attached to a device or tool that the wearer holds or wears.

[0066] <Acquisition of movement information based on images> The wearer may be photographed by a camera, and position information of the wearer's hands, feet, and fingers may be obtained from the photographed image information. In this case, the skeletal information receiving unit 170 is a means for receiving skeletal information from a device that photographs the wearer by an external camera and generates skeletal information of the wearer from the photographed image. The movement information acquiring unit 124 calculates movement information related to the movement of the wearer's hands, feet, and fingers from the skeletal information received from the skeletal information receiving unit 170.

[0067] Here, an example is given in which inertial information, marker position information, and skeletal information are received and movement information of the wearer's limbs and fingers is calculated. Movement information is the amount of movement from one position to another of the hands, fingers, feet, joint points, bone angles, etc., and movement information of the wearer's limbs and fingers may be obtained by other methods.

[0068] <Processing to display the wearer's hand on top of the virtual object when the wearer is moving> 5(a) is a diagram showing an example in which virtual objects are displayed superimposed on the hands of a wearer. Here, virtual objects 201 and 202 are displayed superimposed on a left hand 211 and a right hand 212.

[0069] 5(b) is a diagram showing an example of displaying virtual objects so that the wearer's hands are displayed on the virtual objects. In this example, a left hand 211 and a right hand 212 are displayed on virtual objects 201 and 202.

[0070] Usually, the virtual object is displayed superimposed on the wearer's hand as shown in Fig. 5(a), but the display may be changed when the wearer is moving. For example, when virtual object 202 is pushing member C (223) and the wearer moves his / her right hand 212 to the same position, the virtual object is displayed so that the wearer's hand is displayed above the virtual object as shown in Fig. 5(b). In other words, the wearer's hand is displayed in front of the virtual object. This allows the wearer to see his / her hand clearly when aligning it with the position of the virtual object, making it easier for him / her to move his / her hand.

[0071] In this case, in step S403 in the flow described with reference to FIG. 4, the control unit 123 changes the display of the virtual object so that the virtual object appears to be under the wearer's hand.

[0072] The display of the virtual object may be always displayed with the same transparency without being changed, or may be displayed so that the positional relationship between the user's body and the virtual object is always the same. Here, transparency is an index that indicates that the more transparent it is, i.e., the more the object in the back can be seen through, the higher the transparency, and the less transparent it is, i.e., the less the object in the back can be seen through, the lower the transparency.

[0073] <Playback control: Start playback when it reaches the start position> FIG. 9(a) is a diagram showing an example when playback of a moving image of a virtual object is started.

[0074] 9(a) shows a scene in which a virtual object 201 representing a left hand is grasping a member A (221), a virtual object 202 representing a right hand is within the field of view of the display device 101, and the wearer aligns the left hand 211 and the right hand 202 to the same positions as the virtual objects. As shown in FIG. 9(a), when the wearer aligns the same body part of the wearer as the body part of the virtual object corresponding to the instructor to the same position as the virtual object, playback of the video starts.

[0075] Explaining this by applying it to the flowchart of FIG. 8 for comparing the movement information of the virtual object and the wearer, in step S804, the control unit 123 judges whether the wearer's hand has come to the same position as the virtual object. In step S804, if the control unit 123 judges that the wearer's hand is in the same position as the virtual object's hand, the process proceeds to step S805, and if the control unit 123 judges that the wearer's hand is not in the same position as the virtual object's hand, the process proceeds to step S806. At this time, the control unit 123 checks the positions of the real objects, member A (221), member B (222), and member C (223), and displays the virtual objects 201 and 202 at positions that appear to operate these members. The control unit 123 checks whether the difference between the positions of the virtual object and the wearer is less than a certain threshold and whether the amount of movement of the wearer per second is less than a certain threshold, and if the time that is less than the threshold continues for a certain period of time, it judges that the wearer has come to the same position. Here, it is confirmed that the wearer has not passed the same position as the virtual object, but is stationary at the correct position.

[0076] In step S805, the control unit 123 performs control so as to start playing the image of the virtual object.

[0077] In step S806, the control unit 123 does nothing, that is, performs control so as not to start playback, and then ends the process.

[0078] In this case, the wearer can start playing an image of the virtual object by aligning a part of his or her body with the virtual object.

[0079] It should be noted that all parts of the instructor's body that are visible within the field of view of the display device 101 may be displayed as virtual objects, and only the necessary parts may be aligned with the positions of the specified parts of the learner who is the wearer, that is, when they match, playback of the video may be started. When playback of the video is started, the body parts displayed as virtual objects and the body parts of the wearer may all match, or may only partially match. When playback is started when only a part of the displayed virtual objects match, the display of the displayed virtual objects that should be matched may be changed, such as by highlighting them. For example, when they are surrounded by a colored frame, it becomes possible for the learner who is the wearer to know which parts are important to match.

[0080] <Playback control: Stop playback> FIG. 9(b) is a diagram showing an example when the video playback of a virtual object is stopped. When playing back a video of a virtual object, the end of a series of movements is set as a division, and the video may be paused when the video has been played up to the division. FIG. 9(b) shows a scene in which a virtual object 201 and a left hand grasp a member A (221), a virtual object 202 moves its hand toward member C (223) to push member C (223), and the wearer moves his right hand to place his hand on the virtual object 202. At this time, a series of movements is made up of the period until the virtual object 202 pushes member C (223) and stops moving, and when the series of movements is completed, the playback of the video of the virtual object is paused. Note that the pause may be made when a predetermined time has elapsed.

[0081] In addition, when generating the instructor's hands, feet, and fingers as CG virtual objects from the movement information of the instructor's hands, feet, and fingers and the human body shape information stored in ROM 130, the control unit 123 may generate information on a series of these movements and add it to the movement information of the virtual object.

[0082] 3(a), each of sections 301 and 302 represents a series of movements of a virtual object. The section between sections 301 and 302 is a boundary, and when the control unit 123 plays back a video of the generated virtual object in accordance with the movement information, it pauses the playback of the video of the virtual object after playing back the series of movements enclosed by section 301.

[0083] With reference to FIG. 10, a flowchart for performing a process for stopping the reproduction of a virtual object corresponding to an instructor will be described.

[0084] In step S1001, the control unit 123 acquires movement information of a virtual object from the ROM 130, and the process proceeds to step S1002.

[0085] In step S1002, the control unit 123 determines whether the virtual object has completed a series of movements based on the acquired movement information. If the control unit 123 determines that the series of movements has ended, the process proceeds to step S1003, and if the control unit 123 determines that the series of movements has not ended, the process proceeds to step S1004.

[0086] In step S1003, when the series of movements ends, the control unit 123 stops playing the image of the virtual object and ends the process.

[0087] In step S1004, the control unit 123 determines whether a certain time has elapsed since the start of playback of the image of the virtual object. If the control unit 123 determines that a certain time has elapsed since the start of playback of the image of the virtual object, the process proceeds to step S1005.

[0088] Moreover, if the control unit 123 determines that a certain period of time has not elapsed since the start of playback of the image of the virtual object, the process proceeds to step S1001.

[0089] In step S1005, the control unit 123 stops playing the image of the virtual object and completes the process.

[0090] In this way, the wearer can stop the video playback when the virtual object performs a series of movements without performing any manual operation. Also, if the series of movements is long, the wearer can stop the video playback even in the middle of the series of movements after a certain period of time has passed.

[0091] <Playback control: Processing to resume video playback> Fig. 9(c) is a diagram showing an example of resuming playback of a moving image of a virtual object. Here, the right hand 212 of the wearer is pressing member C (223) in the same manner as the virtual object 202 representing the right hand in Fig. 9(b).

[0092] Explaining this by applying it to the flowchart of FIG. 8 for comparing the movement information of the virtual object and the wearer, in step S804, the control unit 123 judges whether a predetermined part of the wearer's body has made the same movement as the virtual object. At this time, the wearer views a series of movements of the virtual object as a video in the section surrounded by section 301 in FIG. 3(a) and tries to make the same movement. If the difference between the movement amount of the virtual object and the wearer is less than a predetermined threshold, the control unit 123 judges that the virtual object and the wearer have made the same movement. That is, in step S804, if the difference between the movement amount of the virtual object and the wearer is less than a predetermined threshold, the control unit 123 judges that the virtual object and the wearer have made the same movement, and proceeds to step S805. Also, if the difference between the movement amount of the virtual object and the wearer is equal to or greater than a predetermined threshold, the control unit 123 judges that the virtual object and the wearer have made different movements, and proceeds to step S806.

[0093] In step S805, the control unit 123 resumes playing the video of the virtual object. If the video of section 301 was being played, the control unit 123 plays the video of section 302 in step S805.

[0094] In step S806, the control unit 123 completes the process without doing anything, that is, performs control so as not to resume playback of the image of the virtual object, and completes the process.

[0095] Note that when proceeding to step S806, playback of the image of the virtual object is not started. Therefore, in order to start playback of the image of the virtual object, the wearer may be notified that the virtual object and the wearer have made different movements, and prompted to play back the series of movements again.

[0096] As described above, when the wearer aligns a predetermined part of the body with the same position as the virtual object and holds it still, the following series of movements may be played back.

[0097] Here, with reference to Fig. 12(a) and Fig. 12(b), a method of calculating the difference between the movement amount of the virtual object and the wearer and a method of determining whether they have made the same movement will be described. Fig. 12(a) shows a movement trajectory 1211 of the X coordinate of a virtual object corresponding to an instructor, a movement trajectory 1212 of a learner, and a change 1213 in the difference between the two movement trajectories. Also, a dotted line 1214 represents a threshold. Also, the movement trajectories in Fig. 12(a) are shown only in terms of the X coordinate. Fig. 12(a) shows a case where the difference between the movement trajectory of the virtual object and the movement trajectory of the learner is less than the threshold. Fig. 12(b) shows a movement trajectory 1211 of a virtual object corresponding to an instructor, a movement trajectory 1222 of a learner, and a change 1223 in the difference between the two movement trajectories. Also, a dotted line 1214 represents a threshold. Also, the movement trajectories in Fig. 12(b) are shown only in terms of the X coordinate. Fig. 12(b) shows a case where there is a period in which the difference between the movement trajectory of the virtual object and the movement trajectory of the learner is equal to or greater than a threshold. As in Fig. 12(a), when the difference between the movement trajectory of the virtual object and the learner is less than the threshold during a series of movements, the control unit 123 determines that the virtual object and the wearer have made the same movements. Also, as in Fig. 12(b), when there is a period in which the difference between the movement trajectory of the virtual object and the learner is equal to or greater than a threshold during a series of movements, the control unit 123 determines that the virtual object and the wearer have made different movements.

[0098] Alternatively, the difference may be integrated only during a series of movements to calculate the total area, and then the total area may be compared with the threshold area to determine whether it is less than the threshold. Alternatively, it may be determined whether each of the differences in the positions of the virtual object and the wearer at each time in FIG. 3 is less than the threshold.

[0099] In this way, when the wearer performs the same movement as the virtual object, video playback of the virtual object is resumed, allowing the wearer to view the next movement without having to perform manual operations.

[0100] <Playback control: Processing to go back and play back when a different movement is made> So far, the starting, stopping, and resuming of video playback have been described, but if the wearer is unable to make the same movements as the virtual object, the video of the virtual object may be replayed by going back through the video.

[0101] With reference to FIG. 13, a process of tracing back and replaying a series of movements when the wearer makes a movement different from that of the virtual object will be described.

[0102] Explanation of steps S801 to S803 will be omitted.

[0103] In step S1304, the control unit 123 determines whether a predetermined part of the wearer's body has made the same movement as the virtual object. At this time, the wearer views a series of movements of the virtual object as a video in the section surrounded by section 301 in FIG. 3(a) and tries to make the same movement. If the difference between the movement amounts of the virtual object and the wearer is less than a predetermined threshold, the control unit 123 determines that the virtual object and the wearer have made the same movement. That is, in step S1304, if the difference between the movement amounts of the virtual object and the wearer is less than a predetermined threshold, the control unit 123 determines that the virtual object and the wearer have made the same movement, and proceeds to step S1305. Also, if the difference between the movement amounts of the virtual object and the wearer is equal to or greater than a predetermined threshold, the control unit 123 determines that the virtual object and the wearer have made different movements, and proceeds to step S1306.

[0104] In step S1305, the control unit 123 resumes playing the video of the virtual object. If the video of section 301 in Fig. 3(a) has been played, in step S1305 the control unit 123 plays the video of section 302 in Fig. 3(a).

[0105] In step S1306, the control unit 123 plays back the series of movements of the virtual object again. That is, if the video of section 301 in Fig. 3(a) has been played back, in step S1306, the video of section 301 in Fig. 3(a) is played back again.

[0106] In this way, if the wearer is unable to perform the same movements as the virtual object, the wearer can recheck the movements of the virtual object by replaying the video of the virtual object.

[0107] In the first embodiment, an example in mixed reality has been described, but the present invention may be implemented in a virtual space. In the case of a virtual space, a virtual object corresponding to the wearer is displayed on the display unit 112. At least one of the position, posture, and movement trajectory of the wearer may be acquired as described above, and a difference may be calculated and compared.

[0108] In the first embodiment, a so-called video see-through method has been shown as an example in which a virtual object is superimposed on a captured image acquired by the imaging unit 111 and displayed on the display unit 112, but the present invention is not limited to this. A so-called optical see-through method may be used in which a virtual object corresponding to an instructor is displayed on the display unit 112 and the real space behind the display unit 112 is visible through the display unit 112.

[0109] Second embodiment <Processing to switch avatar display method> 6 is a schematic diagram of an external camera capturing an image of a person wearing the display device 101. The external camera 600 captures a full-body image of the person wearing the display device 101, and transmits the captured image to the HMD 100. The control unit 123 of the control device 102 receives the image captured by the external camera 600 from the external camera 600 via the communication unit 180.

[0110] Fig. 7(a) shows an image displayed on the display unit 112, and the range displayed here is the range of the field of view of the HMD. Fig. 7(a) shows an example of an image displayed when there is a difference between the wearer and a virtual object within the field of view of the wearer. In this example, virtual objects 201, 202, left hand 211, and right hand 212 can be displayed within the field of view. The wearer can compare the positions of the virtual objects with those of his or her own hands, and align his or her hands or arms with the positions of the virtual objects.

[0111] Fig. 7(b) is a diagram showing an example of an image displayed when there is a difference between the wearer and a virtual object outside the wearer's field of view. Fig. 7(b) is an image displayed on the display unit 112, in which a full-body image of the instructor and a full-body image of the wearer of the HMD 100 captured by the external camera 600 are displayed as virtual objects. Virtual object 701 is the right foot of the instructor, virtual object 702 is the left foot of the instructor, and right foot 711 and left foot 712 are the right foot and left foot of the wearer of the HMD captured by the external camera.

[0112] In this example, it is impossible to display virtual objects 701 and 702, a right foot 711, and a left foot 712 within the field of view. Here, virtual objects 701 and right foot 711 are different and are outside the field of view. For this reason, it is necessary to display a whole-body image on display unit 112 as shown in FIG. 7(b).

[0113] The control unit 123 acquires a full-body image of the wearer captured by the external camera 600 via the communication unit 180, and displays the acquired full-body image of the wearer and the full-body image of the virtual object as a composite image on the display unit 112. The full-body image of the wearer captured by the external camera 600 may be displayed on the entire display unit 112, or may be displayed on a part of the display unit 112 together with an image within the field of view as shown in FIG. 7(a). The image may be transmitted to an external display device so that the image displayed on the display unit 112 is also displayed on the external display device. Since the display device 101 can only be viewed by the learner, the image may be shown to an instructor who is instructing the learner in another location.

[0114] Furthermore, the control unit 123 controls so that a whole-body image as a virtual object is reproduced in accordance with the movement information stored in the ROM 130 .

[0115] A flowchart for comparing movement information of a virtual object and a wearer will be described with reference to FIG.

[0116] In step S801, the control unit 123 acquires movement information of a virtual object from the ROM 130, and the process proceeds to step S802.

[0117] In step S802, the control unit 123 acquires movement information of the wearer from the movement information acquisition unit 124, and the process proceeds to step S803.

[0118] In step S803, the control unit 123 compares the movement information of the virtual object and the wearer, and proceeds to step S804.

[0119] In step S804, the control unit 123 judges whether or not it is necessary to change the display based on the result of the comparison in step S803. If the control unit 123 judges in step S804 that it is necessary to change the display, the process proceeds to step S805. If the control unit 123 judges that it is necessary to change the display, the process proceeds to step S806. At this time, taking FIG. 7(a) and FIG. 7(b) as an example, if the part where there is a difference between the movement information of the virtual object and the wearer is the virtual object 201 and the left hand 211, or the virtual object 202 and the right hand 212, it is the hand part, which is within the visual field of the display device 101. Also, if the part where there is a difference between the movement information of the virtual object and the wearer is the virtual object 701 and the right foot 711, or the virtual object 702 and the left foot 712, it is outside the visual field of the display device 101, so it is judged that it is necessary to change the display of the virtual object and display a whole body image of the virtual object.

[0120] In step S805, the control unit 123 changes the display of the virtual object to display a full-body image of the virtual object, and then completes the process.

[0121] In step S806, the control unit 123 completes the process without doing anything.

[0122] In this way, when the wearer's movement differs from that of the virtual object and is outside the field of view, the display of the virtual object is changed to display a full-body image, making it possible for the wearer to notice that the movement of his or her right leg is different in the example of Figure 7(b).

[0123] (Third embodiment) <Playback control: From the second time onwards, playback and stop processing are performed for periods where different movements occurred> During second or subsequent learning, since the wearer has already viewed the video of the virtual object, playing the entire video may be cumbersome. In this case, in order to learn the period in which the wearer does not move correctly or makes a mistake, the image of the virtual object may be displayed and played only during the period in which the wearer moves differently from the virtual object. During second or subsequent learning, when changing the display of the virtual object in step S805 of FIG. 8, only the playback process and playback stop process are performed when the wearer moves differently. With reference to FIG. 11, a flowchart for determining whether to display the image of the virtual object during second or subsequent learning will be described.

[0124] In step S1101, the control unit 123 determines whether the wearer is viewing a video of a virtual object for the second or subsequent time. If the control unit 123 determines that the wearer is viewing a video of a virtual object for the first time, that is, if the control unit 123 determines that the wearer is not viewing a video of a virtual object for the second or subsequent time, the process proceeds to step S1102. If the control unit 123 determines that the wearer is viewing a video of a virtual object for the second or subsequent time, the process proceeds to step S1103.

[0125] In step S1102, the control unit 123 performs control so as to display an image of a virtual object.

[0126] In step S1103, the control unit 123 determines whether the display change process is a video playback start process. If the control unit 123 determines that the display change process is a video playback start process, the control unit 123 proceeds to step S1104, and if the control unit 123 determines that the display change process is not a video playback start process, the control unit 123 proceeds to step S1105.

[0127] In step S1104, the control unit 123 completes the process without doing anything.

[0128] In step S1105, the control unit 123 determines whether the display change process is a video playback restart process. If the control unit 123 determines that the display change process is a video playback restart process, the control unit 123 proceeds to step S1106, and if the control unit 123 determines that the display change process is not a video playback restart process, the control unit 123 proceeds to step S1107.

[0129] In step S1106, control unit 123 completes the process without doing anything.

[0130] In step S1107, the control unit 123 performs control so as to display an image of the virtual object.

[0131] By following the flow chart shown in FIG. 11, from the second time onwards, the process of starting the video playback by matching a specific part of the learner to the start position and the process of stopping the video playback of the virtual object due to the difference in the movement trajectory can be omitted. For example, from the second time onwards, instead of starting the video playback by matching a specific part of the learner to the start position, the learner starts the series of movements in section 301 without following the movement of the virtual object. Also, if the movement trajectory matches the non-displayed virtual object in the series of movements in section 301, the virtual object is not displayed and the series of movements in section 302 is continued without stopping at the series of movements in section 301. If the movement trajectory of the learner matches the movement trajectory of the virtual object corresponding to the instructor, the virtual object can be left hidden while sections 301 and 302 can be ended. Also, if the movement trajectory of the learner does not match the movement trajectory of the virtual object corresponding to the instructor, the virtual object is displayed. If the movement trajectory of the learner and the movement trajectory of the virtual object corresponding to the instructor do not match and the virtual object is displayed, the virtual object may be hidden again at the timing when the series of movements ends. If the movement trajectory of the learner and the movement trajectory of the virtual object corresponding to the instructor do not match and the virtual object is displayed, the virtual object may be hidden again at the timing when the movement trajectory of the learner and the movement trajectory of the virtual object corresponding to the instructor match.

[0132] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0133] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0134] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0135] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0136] [Configuration 1] An information processing device, a first acquisition means for acquiring first information on a position or a posture of a virtual object corresponding to the instructor from a recording means; a second acquisition means for acquiring second information related to a position or a posture of the operation device supported by the learner; a control means for controlling playback of an image of a virtual object corresponding to the instructor based on a difference between the first information acquired by the first acquisition means and the second information acquired by the second acquisition means; 13. An information processing device comprising:

[0137] [Configuration 2] When the difference is less than a predetermined threshold, the control means controls to start playing an image of a virtual object corresponding to the instructor. 2. The information processing device according to configuration 1.

[0138] [Configuration 3] the control means controls to start playing a video of a virtual object corresponding to the instructor when a difference between the first information acquired by the first acquisition means at a predetermined time point and the second information acquired by the second acquisition means is less than the predetermined threshold value; 3. The information processing device according to configuration 2.

[0139] [Configuration 4] the control means controls to start playing a video of a virtual object corresponding to the instructor when a difference between a movement trajectory based on the first information for a predetermined period of time and a movement trajectory based on the second information for a predetermined period of time is less than the predetermined threshold value. 4. The information processing device according to configuration 2 or 3.

[0140] [Configuration 5] the control means controls to play back an image of the virtual object corresponding to the instructor from a first time point to a second time point and to pause the image at the second time point; when a difference between a movement trajectory based on the first information from the first time point to the second time point and a movement trajectory based on the second information from the first time point to the second time point is less than the predetermined threshold, the control means starts playing a video of a virtual object corresponding to the instructor from the second time point. 5. The information processing device according to configuration 4.

[0141] [Configuration 6] the control means controls to play back an image of the virtual object corresponding to the instructor from a first time point to a second time point and to pause the image at the second time point; when a difference between a movement trajectory based on the first information from the first time point to the second time point and a movement trajectory based on the second information from the first time point to the second time point is equal to or greater than the predetermined threshold, the control means starts playing a video of the virtual object corresponding to the instructor from the first time point. 5. The information processing device according to configuration 4.

[0142] [Configuration 7] when there is a period from a first time point to a second time point during which a difference between a movement trajectory based on the first information and a movement trajectory based on the second information is equal to or greater than the predetermined threshold, the control means stops playing a video of the virtual object corresponding to the instructor at the second time point. 5. The information processing device according to configuration 4.

[0143] [Configuration 8] The method further includes a notification unit that notifies the user that the difference is equal to or greater than a predetermined threshold value when the difference is equal to or greater than a predetermined threshold value. 8. The information processing device according to any one of configurations 1 to 7.

[0144] [Configuration 9] the notifying means notifies a display means that displays a virtual object corresponding to the instructor that the difference is equal to or greater than the threshold value. 9. The information processing device according to configuration 8.

[0145] [Configuration 10] the control means controls to change a display method of the virtual object corresponding to the instructor depending on whether the difference is less than a predetermined threshold value. 10. The information processing device according to any one of configurations 1 to 9.

[0146] [Configuration 11] When the difference is less than the predetermined threshold, the control means controls the transparency of the virtual object corresponding to the instructor so as to be higher than when the difference is equal to or greater than the predetermined threshold. 11. The information processing device according to configuration 10.

[0147] [Configuration 12] the control means controls so that, when the difference is less than the predetermined threshold, the virtual object corresponding to the instructor is displayed behind the learner, and, when the difference is equal to or greater than the predetermined threshold, the virtual object corresponding to the instructor is displayed in front of the learner. 11. The information processing device according to configuration 10.

[0148] [Configuration 13] The imaging device further includes a third acquisition means for acquiring a captured image from the imaging means, the control means displays a virtual object corresponding to the instructor in a superimposed manner on the captured image acquired by the third acquisition means. 13. The information processing device according to any one of configurations 1 to 12.

[0149] [Configuration 14] The display device further includes a transmitting means for transmitting an image of the virtual object corresponding to the instructor to an optical see-through type display device having a display means. 15. The information processing device according to any one of configurations 1 to 14.

[0150] [Configuration 15] the control means controls to display, in a virtual space, a virtual object corresponding to the learner based on the second information. 15. The information processing device according to any one of configurations 1 to 14.

[0151] [Configuration 16] Further comprising a fourth acquisition means for acquiring the first information based on a position or a posture of the operation device supported by the instructor. 16. The information processing device according to any one of configurations 1 to 15.

[0152] [Configuration 17] When the reproduction of the image of the virtual object corresponding to the instructor is the second or subsequent time, the control means controls so as to display the image of the virtual object corresponding to the instructor during a period in which the difference was equal to or greater than a predetermined threshold when the image of the virtual object corresponding to the instructor was reproduced for the first time, and not to display the image of the virtual object corresponding to the instructor during a period in which the difference was less than the predetermined threshold. 17. The information processing device according to any one of configurations 1 to 16.

[0153] [Control method] a first acquisition step of acquiring first information on a position or a posture of a virtual object corresponding to an instructor from a recording means; a second acquisition step of acquiring second information regarding a position or a posture of the operating device supported by the learner; a control step of controlling playback of an image of a virtual object corresponding to the instructor based on a difference between the first information acquired in the first acquisition step and the second information acquired in the second acquisition step; 13. A method for controlling an information processing apparatus comprising the steps of:

[0154] [program] A program for causing a computer to function as each of the means of the information processing device according to configuration 1.

[0155] [system] a first acquisition device that acquires first information on a position or a posture of a virtual object corresponding to an instructor from a recording device; A learner-supported control device; a second acquisition device that acquires second information related to a position or an attitude of the operation device; a control device that controls playback of an image of a virtual object corresponding to the instructor based on a difference between the first information acquired by the first acquisition device and the second information acquired by the second acquisition device; An information processing system comprising:

Claims

1. An information processing device, a first acquiring means for acquiring first information on a position or a posture of a virtual object corresponding to an instructor from a recording means; a second acquisition means for acquiring second information related to a position or a posture of the operation device supported by the learner; a control means for controlling playback of an image of a virtual object corresponding to the instructor based on a difference between the first information acquired by the first acquisition means and the second information acquired by the second acquisition means; 13. An information processing device comprising:

2. When the difference is less than a predetermined threshold, the control means controls to start playing an image of a virtual object corresponding to the instructor.

2. The information processing apparatus according to claim 1,

3. the control means controls to start playing a video of a virtual object corresponding to the instructor when a difference between the first information acquired by the first acquisition means at a predetermined time point and the second information acquired by the second acquisition means is less than the predetermined threshold value; 3. The information processing apparatus according to claim 2.

4. the control means controls to start playing a video of a virtual object corresponding to the instructor when a difference between a movement trajectory based on the first information for a predetermined period of time and a movement trajectory based on the second information for a predetermined period of time is less than the predetermined threshold value.

3. The information processing apparatus according to claim 2.

5. the control means controls to play back an image of the virtual object corresponding to the instructor from a first time point to a second time point and to pause the image at the second time point; when a difference between a movement trajectory based on the first information from the first time point to the second time point and a movement trajectory based on the second information from the first time point to the second time point is less than the predetermined threshold, the control means starts playing a video of a virtual object corresponding to the instructor from the second time point.

5. The information processing apparatus according to claim 4.

6. the control means controls to play back an image of the virtual object corresponding to the instructor from a first time point to a second time point and to pause the image at the second time point; the control means starts playing a video of the virtual object corresponding to the instructor from the first time point when a difference between a movement trajectory based on the first information from the first time point to the second time point and a movement trajectory based on the second information from the first time point to the second time point is equal to or greater than the predetermined threshold value.

5. The information processing apparatus according to claim 4.

7. when there is a period from a first time point to a second time point during which a difference between a movement trajectory based on the first information and a movement trajectory based on the second information is equal to or greater than the predetermined threshold, the control means stops playing a video of the virtual object corresponding to the instructor at the second time point.

5. The information processing apparatus according to claim 4.

8. The information processing device further includes a notification unit configured to notify a user that a difference between the first information acquired by the first acquisition unit and the second information acquired by the second acquisition unit is equal to or greater than a predetermined threshold value when the difference is equal to or greater than a predetermined threshold value.

2. The information processing apparatus according to claim 1,

9. the notifying means notifies a display means that displays a virtual object corresponding to the instructor that the difference is equal to or greater than the threshold value.

9. The information processing apparatus according to claim 8,

10. the control means controls to change a display method of the virtual object corresponding to the instructor depending on whether the difference is less than a predetermined threshold value.

2. The information processing apparatus according to claim 1,

11. When the difference is less than the predetermined threshold, the control means controls the transparency of the virtual object corresponding to the instructor so as to be higher than when the difference is equal to or greater than the predetermined threshold.

11. The information processing apparatus according to claim 10,

12. the control means controls so that, when the difference is less than the predetermined threshold, the virtual object corresponding to the instructor is displayed behind the learner, and, when the difference is equal to or greater than the predetermined threshold, the virtual object corresponding to the instructor is displayed in front of the learner.

11. The information processing apparatus according to claim 10,

13. The imaging device further includes a third acquisition means for acquiring a captured image from the imaging means, the control means displays a virtual object corresponding to the instructor in a superimposed manner on the captured image acquired by the third acquisition means.

2. The information processing apparatus according to claim 1,

14. The display device further includes a transmission means for transmitting an image of the virtual object corresponding to the instructor to an optical see-through type display device having a display means.

2. The information processing apparatus according to claim 1,

15. the control means controls to display, in a virtual space, a virtual object corresponding to the learner based on the second information.

2. The information processing apparatus according to claim 1,

16. The apparatus further includes a fourth acquisition means for acquiring the first information based on a position or a posture of the operation device supported by the instructor.

2. The information processing apparatus according to claim 1,

17. When the reproduction of the image of the virtual object corresponding to the instructor is the second or subsequent time, the control means controls so as to display the image of the virtual object corresponding to the instructor during a period in which the difference was equal to or greater than a predetermined threshold when the image of the virtual object corresponding to the instructor was reproduced for the first time, and not to display the image of the virtual object corresponding to the instructor during a period in which the difference was less than the predetermined threshold.

2. The information processing apparatus according to claim 1,

18. a first acquisition step of acquiring first information on a position or a posture of a virtual object corresponding to an instructor from a recording means; a second acquisition step of acquiring second information regarding a position or a posture of the operation device supported by the learner; a control step of controlling playback of an image of a virtual object corresponding to the instructor based on a difference between the first information acquired in the first acquisition step and the second information acquired in the second acquisition step; 13. A method for controlling an information processing apparatus comprising the steps of:

19. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

20. a first acquisition device that acquires first information on a position or a posture of a virtual object corresponding to an instructor from a recording device; A learner-supported control device; a second acquisition device that acquires second information related to a position or an attitude of the operation device; a control device that controls playback of an image of a virtual object corresponding to the instructor based on a difference between the first information acquired by the first acquisition device and the second information acquired by the second acquisition device; An information processing system comprising:

Images