Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

To provide an information processing apparatus, a control method for the information processing apparatus, and a program which, when a user practices a sport, enable the user to feel as if the user is actually practicing the sport through at least the sense of touch among the five senses and to efficiently practice the sport.SOLUTION: The information processing apparatus 340 includes an HMD position and orientation information processing means 352 serving as an obtaining means for obtaining the position and orientation information of the user, a virtual object position and orientation calculation means 355 serving as a detection means for detecting the degree of contact between the virtual object and the user in the mixed reality image and the positional relationship between the virtual object and the user in the mixed reality image, and a training continuation determination means 358 for determining the suitability of the positional relationship for the user using a training continuation condition setting table stored in a ROM or HDD and changing the state of the virtual object in the mixed reality image in accordance with the determination result.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In recent years, mixed reality (MR), which combines real space and virtual space, has become well known. In MR, for example, a mixed reality in which a virtual object is superimposed on real space is displayed on a head-mounted display (HMD) using computer graphics (CG). This allows users to easily experience, for example, events that are difficult to repeat repeatedly in real space or events that are difficult to cause in the real world. In recent years, MR has also been used for physical training aimed at improving physical abilities and physical functions. For example, Patent Document 1 discloses a baseball batting practice support system using MR. The system described in Patent Document 1 can notify a trainee of the impact position (impact position) between a real bat and a ball, which is a virtual object contained in a virtual space. Patent Document 2 also discloses a rehabilitation device using MR. The device described in Patent Document 2 can superimpose a video of a moving body part on an actually paralyzed body part. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6467698 [Patent Document 2] Japanese Patent Application Laid-Open No. 2015-39522 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the system described in Patent Document 1, the trainee cannot feel the vibrations or stimulation that occur when a real bat collides with a ball as a virtual object. Furthermore, with the device described in Patent Document 2, even if a paralyzed body part moves on an image, the body part does not actually move, so the trainee does not feel that the body part has moved. Furthermore, with the system described in Patent Document 1, there are cases where the ball is hit with the center of the bat and cases where the ball is hit off the center of the bat, and it is not possible to perform batting practice that corresponds to each case.

[0005] The present invention has been made in consideration of the above-mentioned problems. It is an object of the present invention to provide an information processing device that allows, for example, when practicing a sport, to feel as if the practice is actually being performed through at least the sense of touch of the five senses, and to perform the practice efficiently. Similarly, it is an object of the present invention to provide a control method and a program for an information processing device that allows, for example, when practicing a sport, to feel as if the practice is actually being performed through at least the sense of touch of the five senses, and to perform the practice efficiently. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the information processing device of the present invention is characterized by comprising: a generation means for generating a mixed reality image by combining an image of real space with an image of virtual space including a movable virtual object; an acquisition means for acquiring position and orientation information regarding the position and orientation of a user viewing the mixed reality image; a detection means for detecting the degree of contact between the virtual object and the user in the mixed reality image and the positional relationship between the virtual object and the user in the mixed reality image based on position information regarding the position of the virtual object in the mixed reality image and the position and orientation information; a control means for performing control to give the user the sensation that the virtual object is in contact with the user based on the degree of contact; a judgment means for judging whether the positional relationship is appropriate for the user; and a change means for changing the state of the virtual object in the mixed reality image based on the result of the judgment by the judgment means. [Effects of the Invention]

[0007] According to the present invention, when practicing a sport, for example, a person can feel as if he or she is actually practicing through at least the sense of touch, one of the five senses, and can perform the practice efficiently. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a configuration diagram showing an outline of an MR system according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing the positional relationship between a user, an HMD, a wearable device, and a camera in a lifting practice system (MR system). [Figure 3] FIG. 1 is a block diagram showing an example of the hardware configuration of an HMD, an information processing device, a wearable device, and a camera included in an MR system. [Figure 4] 10 is a training continuation condition setting table stored in the training continuation conditions of the information processing device. [Figure 5] 10 is a flowchart showing a process executed by the information processing device. [Figure 6]FIG. 10 is a block diagram showing an example of the hardware configuration of an HMD, an information processing device, a wearable device, and a camera included in an MR system according to a second embodiment. [Figure 7] 10 is a training correction setting table stored in the training continuation conditions of the information processing device. [Figure 8] 10 is a flowchart showing a process executed by the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in each embodiment. For example, each component constituting the present invention can be replaced with any configuration that can perform the same function. Also, any component may be added. Furthermore, any two or more configurations (features) of each embodiment can be combined.

[0010] First Embodiment The first embodiment will be described below with reference to Fig. 1 to Fig. 5. Fig. 1 is a configuration diagram showing an outline of an MR system according to the first embodiment. As shown in Fig. 1, an MR system (information processing system) 1000 includes an HMD (head mounted display) 101, an information processing device (image processing device) 103, a wearable device 106, and a camera 107. The information processing device 103 is communicably connected to the HMD 101, the wearable device 106, and the camera 107. The communication method is not particularly limited, and is preferably, for example, a wireless communication method that configures a small-scale network such as WLAN or WPAN, but is not limited to this and may be a wired communication method. In addition, although the HMD 101 and the information processing device 103 are configured as separate entities in the configuration shown in FIG. 1, this is not limited thereto, and for example, the HMD 101 may have all the functions of the information processing device 103 built in. The MR system 1000 is used, for example, for simulated practice of ball games using a ball. In this embodiment, the MR system 1000 is a juggling practice system that can be used for practicing soccer juggling.

[0011] The HMD 101 has an imaging device 105 and is worn on the head when used. The HMD 101 captures an image of a subject viewed by a user (hereinafter simply referred to as "user") using the HMD 101 using the imaging device 105, and obtains digital image data of the subject. The HMD 101 is also configured to display a mixed reality image (MR image) that combines (superimposes) an image of real space with an image of virtual space (CG image) on a display unit 312 (see FIG. 3). This allows the user to view the mixed reality image. The mixed reality image is generated by an information processing device 103. The information processing device 103 has a display unit 102 and an operation unit 104. The operation unit 104 is configured with a keyboard or the like and can input various data, commands, etc. The display unit 102 is configured with a liquid crystal display or the like and can display data input from the operation unit 104, results of commands, etc.

[0012] The wearable device 106 is a shoe-type device worn on each of the user's feet. A plurality of markers 106a are provided on the outer side (surface) of the wearable device 106 for acquiring position information related to the user's position in real space. A haptic device (not shown) is worn on the inner side of the wearable device 106. The haptic device is a device that gives the user a tactile sensation by, for example, vibration, electrical stimulation, ultrasonic waves, etc. Note that the haptic device is not limited to a device that generates vibration, electrical stimulation, ultrasonic waves, etc., as long as it is configured to give the user a tactile sensation. In addition to the haptic device, a motion sensor (not shown) is worn on the inner side of the wearable device 106. The motion sensor is a device that acquires tilt and acceleration as posture information related to the user's posture in real space. In the wearable device 106, one haptic device and one motion sensor are arranged on the back side of each marker 106a, but this is not limiting. For example, the number and arrangement of the markers 106a, tactile devices, and motion sensors may be changed depending on various conditions such as the intended use of the MR system 1000. The camera 107 is a device that captures an image of a user in real space. This camera 107 is used to measure the position of the wearable device 106 worn by the user in real space. The camera 107 is not particularly limited as long as it is capable of measuring the position of the wearable device 106, and may be, for example, a 3D camera that captures 3D images or a 2D camera that captures 2D images. For example, if the camera 107 is a 2D camera, the position information of the wearable device 106 from the 2D image may be inferred using a network model or acquired using a rule-based image processing algorithm.

[0013] As described above, in this embodiment, the MR system 1000 is a system for practicing lifting balls. FIG. 2 is a diagram showing the positional relationship between the user, HMD, wearing device, and camera in the lifting ball training system (MR system). The X-axis in FIG. 2 indicates one horizontal direction (floor surface) in the real space, the Y-axis indicates one horizontal direction in the real space that is perpendicular to the X-axis, and the Z-axis indicates the vertical direction (up and down direction) in the real space, but does not exist in the real space. The intersection of the X-axis, Y-axis, and Z-axis is the center position when the user practices lifting balls. A practice range 201 in which lifting balls can be practiced is set by a circle centered at this center position. In the lifting ball training system, the user practices lifting balls while wearing the HMD 101 on his head and the wearing device 106 on each foot. In the MR system 1000 as a lifting practice system, a plurality of cameras 107 (for example, a dozen cameras) are installed hanging from the ceiling so that they can capture the practice area 201 without any blind spots, i.e., so that they can capture the user from various angles. Each camera 107 accurately acquires position information of the marker 106a of the wearable device 106 in real time as position information of the user in real space. Furthermore, in the MR system 1000, tilt and acceleration, which are posture information related to the user's posture in real space, can be acquired in real time from the motion sensor of the wearable device 106. Therefore, in the MR system 1000, by arranging the marker 106a and the motion sensor in pairs, position and posture information related to the user's position and posture in real space can be acquired in real time.

[0014] The HMD 101 in the lifting practice system is a head-mounted display device, and in this embodiment, a video see-through device is used. However, the present invention is not limited to this. For example, a non-transparent device that can input external video information and audio information may also be used. In the lifting practice system, the surrounding environment (scenery), such as the floor and walls that the user is viewing, is analyzed via the HMD 101, and the user's position and posture in the real world can be acquired. The virtual space in the lifting practice system also includes a virtual object that can be moved within the virtual space. In this embodiment, the virtual object is a soccer ball (ball). Gravity is applied to the virtual space, just like in real space. Information on the position and movement (speed, acceleration, and direction of movement) of the virtual object under the action of gravity is acquired. The virtual object, the soccer ball, is generated as a CG image based on the user's position and posture information. In the lifting practice system, a mixed reality image that combines an image of the real space with an image of the virtual space including the virtual object is displayed on the HMD 101. This allows the user to experience virtual objects that do not exist in real space. The MR system 1000 is not limited to the above configuration, and may have a configuration including, for example, a wearable terminal or a remote sensor, as long as it can perform the same functions as the configuration. The wearable terminal is not particularly limited, and examples thereof include a gyro sensor, an acceleration sensor, a GPS, and a position and orientation measurement means using wireless communication. The remote sensor is not particularly limited, and examples thereof include a position and orientation measurement camera using visible light or invisible light, a laser measurement device, and the like.

[0015] FIG. 3 is a block diagram showing an example of the hardware configuration of an HMD, an information processing device, a wearable device, and a camera included in an MR system. The HMD 310 shown in FIG. 3 is the internal configuration of the HMD 101. The HMD 310 includes an imaging unit 311, a display unit 312, and an information processing device communication unit 313. The imaging unit 311 captures an image of a subject viewed by a user using the imaging device 105 to obtain digital image data of the subject. The display unit 312 displays a mixed reality image to the user. The display unit 312 includes an optical system disposed in front of each of the user's eyes. The information processing device communication unit 313 communicates with the HMD communication unit 341 of the information processing device 340. The communication protocol used for this communication is not particularly limited as long as it enables communication of image data, video data, text, control data, and the like. Examples of communication protocols include HTTP, FTP, PTP, USB, LAN, HL7, DICOM, and the like.

[0016] The wearing device 320 shown in Fig. 3 is an internal configuration of the wearing device 106. The wearing device 320 has a posture information acquisition means 321, a tactile output means 322, and an information processing device communication means 323. The posture information acquisition means 321 is a means for acquiring posture information such as tilt and acceleration from a motion sensor attached to the inside of the wearing device 106. The tactile output means 322 is a device for operating a tactile device attached to the inside of the wearing device 106 to give the user a tactile sensation similar to that of kicking a soccer ball, for example, by vibration or the like. The information processing device communication means 323 is a means for communicating with the wearing device communication means 342 of the information processing device 340.

[0017] 3 is the internal configuration of the camera 107. The camera 330 has an imaging means 331 and an information processing device communication means 332. The imaging means 331 is a means for obtaining digital image data of a subject. The information processing device communication means 332 is a means for communicating with a camera communication means 343 of an information processing device 340.

[0018] The information processing device 340 shown in FIG. 3 is an internal configuration of the information processing device 103. The information processing device 340 includes an HMD communication means 341, a wearable device communication means 342, a camera communication means 343, and a storage means 351. The information processing device 340 also includes an HMD position and orientation information processing means (acquisition means) 352, a wearable device orientation information processing means 353, an image processing means 354, and a virtual object position and orientation calculation means (detection means) 355. The information processing device 340 also includes a CG generation means 356, an MR image generation means (generation means) 357, a training continuation determination means 358, and a tactile output information processing means 359. The HMD communication means 341 is a means for communicating with the information processing device communication means 313 of the HMD 310. The wearable device communication means 342 is a means for communicating with the information processing device communication means 323 of the wearable device 320. The camera communication means 343 is a means for communicating with the information processing device communication means 332 of the camera 330.

[0019] The storage means 351 stores digital data obtained by capturing images using the imaging means 311 and the imaging means 331, and stores posture information acquired from the posture information acquisition means 321. The storage means 351 can also store information as a working memory used in executing a program. This program includes, for example, a program for causing a CPU (not shown), which is a computer, to execute the information processing device 340 (information processing device 103) and each unit and each unit (control method of the information processing device) of the information processing device 103. These programs are stored in a ROM (not shown) or HDD (not shown) of the information processing device 340. The storage means 351 is also assigned HMD information 361, virtual object information 362, wearing device information 363, and training continuation conditions 364. The HMD information 361 is an area for storing the position and orientation of the HMD 310. The virtual object information 362 is an area for storing information related to a virtual object. In this embodiment, the virtual object is a soccer ball. Examples of the initial information about this soccer ball include a spherical shape with a diameter of 20 cm and an initial position 1 m above the center of the practice area 201. Other examples include a speed of 0 m / s, a speed direction (velocity direction vector) of (0,0,0), a rotation speed of 0 m / s, and a rotation direction of (0,0,0). Therefore, in the mixed reality image, if a soccer ball with a diameter of 20 cm is positioned 1 m above the center of the practice area 201 and a juggling practice (training) is started, the soccer ball will fall due to gravity. The user can then first perform juggling practice by kicking the falling soccer ball. The virtual object information 362 is also an area for storing the number of juggling attempts. The initial value of the number of juggling attempts is 0. The wearing device information 363 is an area for storing position information of the marker 106a in real space and posture information from the motion sensor. The training continuation condition 364 is an area for storing a training continuation condition setting table 400 (see FIG. 4) described later.

[0020] The HMD position and orientation information processing means 352 is a means for acquiring information on the position and orientation of the HMD 310 in real space as position and orientation information related to the user's position and orientation in real space, using digital data captured by the imaging means 311 (acquisition step). Upon receiving the digital data captured by the imaging means 311, the HMD position and orientation information processing means 352 sequentially acquires information on the position and orientation of the HMD 310 and stores it in HMD information 361. The wearing device orientation information processing means 353 acquires position information of the marker 106a of the wearing device 106 in real space, using digital data captured by the imaging means 331. In addition, it receives orientation information (tilt and acceleration of the motion sensor) acquired from the orientation information acquisition means 321, and stores the position information of the marker 106a in real space and the orientation information of the motion sensor (position, tilt, and acceleration of the motion sensor) in the wearing device information 363. These pieces of information can also be used as position and orientation information related to the user's position and orientation in real space. The image processing means 354 is a means for converting the digital data captured by the imaging means 311 into an image in real space and transferring it to the MR image generating means 357 .

[0021] The virtual object position and orientation calculation means 355 is a means for calculating the position and movement amount of a virtual object (soccer ball) in a virtual space. Information obtained by calculations in the virtual object position and orientation calculation means 355 includes, in addition to the position and movement amount of the virtual object, the virtual object's speed, movement direction (directional vector), rotation speed, and rotation direction, for example. This information is stored in virtual object information 362. The virtual object position and orientation calculation means 355 also sequentially reads information from the mounting device information 363 and the mounting device information 363 to determine whether the soccer ball and the mounting device 106 collide with each other in the mixed reality image due to a juggling. In this embodiment, the virtual object position and orientation calculation means 355 can detect the following two pieces of information based on position information regarding the position of the virtual object in the mixed reality image and position and orientation information of the mounting device 106 (user position and orientation information) (detection process). The first piece of information is the degree of contact between the virtual object and the user in the mixed reality image. The second piece of information is the positional relationship between the virtual object and the user in the mixed reality image. If the virtual object position and orientation calculation means 355 determines that the soccer ball has collided with the wearing device 106, it calculates the position, speed, and orientation of the soccer ball due to the collision. Furthermore, the virtual object position and orientation calculation means 355 determines that the juggling was successful and adds 1 to the number of juggling attempts in the virtual object information 362 and stores the result. The virtual object position and orientation calculation means 355 (control means) also performs control to give the user the sensation that the soccer ball has come into contact with the wearing device 106 (user) based on the degree of contact (control step). Specifically, the virtual object position and orientation calculation means 355 identifies the haptic device of the wearing device 106 that is located at the position in the mixed reality image where the soccer ball has collided, and instructs the haptic device to vibrate via the haptic output information processing means 359.

[0022] The CG generation means 356 is a means for visualizing an image in which a virtual object is visible using computer graphics. The CG generation means 356 sequentially reads information from the HMD information 361 and the virtual object information 362 and generates a CG image as an image of the virtual space. The CG image is transferred to the MR image generation means 357. The MR image generation means 357 is a means for generating a mixed reality image by combining the image of the real space transferred from the image processing means 354 with the image of the virtual space transferred from the CG generation means 356 (generation step). The mixed reality image is transferred to the display means 312. The training continuation determination means 358 is a means for determining whether to continue or stop training based on the training continuation condition 364. The training continuation determination means 358 can also resume training based on the training continuation condition 364. The tactile output information processing means 359 is a means for operating the tactile device via the tactile output means 322 based on an instruction from the virtual object position and orientation calculation means 355.

[0023] FIG. 4 shows a training continuation condition setting table stored in the training continuation conditions of the information processing device. The training continuation condition setting table 400 shown in FIG. 4 is stored in the ROM (not shown) or HDD (not shown) of the information processing device 340. This training continuation condition setting table 400 is composed of five items: "Application Order," "Training Continuation Condition," "Continue / Stop / End," "Resume Condition," and "Virtual Object at Resumption." The "Application Order" indicates the order in which the training continuation conditions are applied. The training continuation conditions are applied in ascending order of the application order numbers. The "Training Continuation Condition" defines the conditions for continuing the training, which is lifting practice, and the conditions for stopping the training. The "Continue / Stop / End" defines whether to continue, stop, or end the training depending on the training conditions. The "Resume Condition" defines the conditions for restarting the training if the training has been stopped. The "Virtual Object at Resumption" defines the state in which the virtual object is to be restarted when training is resumed. When using the training continuation condition setting table 400, immediately before that, information is read from the virtual object information 362 and the wearing device information 363 to obtain the position of the virtual object, the position of the wearing device 106 (shoes), and the number of lifts.

[0024] When using the training continuation condition setting table 400, the "training continuation condition" with the first "application order" is evaluated first. This "training continuation condition" is a condition that "the virtual object (soccer ball) has contacted the floor (ground)." In this case, "Continue / Stop / End" is set to "Training Stop." After that, if the "Resume condition" is set to "The wearing device 106 (shoes) has not moved for three seconds," the user is deemed ready for training, and training is resumed. Then, the "Virtual object at time of resumption" is set to "Resume from the position where the soccer ball was kicked last time." In this way, with the "application order" set to first, if the user fails to lift the soccer ball and the soccer ball falls to the floor, training can be resumed from the state immediately before the failed lift. Then, if the condition with the first "application order" is met, conditions with second and subsequent "application order" are not evaluated. On the other hand, if the condition with the first "application order" is not met, the condition with the second "application order" is evaluated.

[0025] The "Training Continuation Condition" with the "Application Order" set to 2 is a condition that "the virtual object (soccer ball) has moved outside the practice range" or "the wearing device 106 (shoes) has moved outside the practice range." In this case, "Continue / Stop / End" is set to "Training Stop." After that, if the "Resume Condition" is set to "The wearing device 106 (shoes) has stopped moving for three seconds and the wearing device 106 (shoes) is in the center position of the practice range," the user is deemed ready for training, and training is resumed. Then, the "Virtual Object at Resumption" is set to "Resume from the initial position of the soccer ball" for this restart. This "initial position" refers to the initial position included in the initial information of the soccer ball described above. In this way, with the "Application Order" set to 2, if the user kicks the soccer ball too many times and cannot continue juggling unless they chase the soccer ball outside the practice range 201, the user can resume training from the beginning. Then, if the condition with the "Application Order" set to 2 is met, conditions 3 and onward are not evaluated. On the other hand, if the condition with the "application order" of 2 is not met, the condition with the "application order" of 3 is evaluated.

[0026] The "Training Continuation Condition" with an "Application Order" of 3 is the condition "100 successful lifts." In this case, "Continue / Stop / End" is used to determine "End of Training." In this way, with an "Application Order" of 3, the user can end training if they have practiced lifting sufficiently. If the condition with an "Application Order" of 3 is met, the conditions from 4 onwards are not evaluated. In contrast, if the condition with an "Application Order" of 3 is not met, the condition with an "Application Order" of 4 is evaluated. The "Training Continuation Condition" with an "Application Order" of 4 is the condition "All (Unconditional)." In this way, "Continue / Stop / End" is used to determine "Continue of Training." In this way, with an "Application Order" of 4, the user can continue training if they have successfully lifted.

[0027] With this type of training continuation condition setting table 400, the "training continuation condition (determination means)" can determine whether the positional relationship with the soccer ball in the mixed reality image is suitable for the user to practice juggling (suitability) (determination process). Then, depending on the result of this determination, the "continue / stop / end (change means)" can change the state of the soccer ball in the mixed reality image (change process). For example, if the soccer ball in the mixed reality image has touched the ground (moved to a pre-set position), it is determined that the positional relationship with the soccer ball is not suitable for the user to practice juggling (see "application order" as number 1). Looking at it the other way around, if the soccer ball in the mixed reality image is not touching the ground, it is determined that the positional relationship with the soccer ball is suitable for the user to practice juggling.

[0028] Furthermore, if the soccer ball in the mixed reality image moves outside the practice range 201 (a pre-set range), it is also determined that the user's positional relationship with the soccer ball is not suitable for juggling practice (see "application order" as number 2). Looking at it the other way around, if the soccer ball in the mixed reality image has not moved outside the practice range 201, that is, is located within the practice range 201, it is also determined that the user's positional relationship with the soccer ball is suitable for juggling practice. Also, if the user (wearable device 106) moves outside the practice range 201 in the mixed reality image, it is also determined that the user's positional relationship with the soccer ball is not suitable for juggling practice (see "application order" as number 2). Looking at it the other way around, if the user has not moved outside the practice range 201 in the mixed reality image, that is, is located within the practice range 201, it is also determined that the user's positional relationship with the soccer ball is suitable for juggling practice.

[0029] Then, if it is determined that the positional relationship between the user and the soccer ball is not suitable (not suitable) for juggling practice, the movement state of the soccer ball, i.e., the training, is forcibly stopped (see "application order" numbers 1 and 2). Note that instead of stopping the movement state of the soccer ball, the soccer ball may be hidden. On the other hand, if it is determined that the positional relationship between the user and the soccer ball is suitable (suitable) for juggling practice, the movement state of the soccer ball, i.e., the training, can be continued (see "application order" number 4). Furthermore, after the movement state of the soccer ball has been forcibly stopped or the soccer ball has been hidden, the movement state of the soccer ball, i.e., the training, can be resumed (see "application order" numbers 1 and 2).

[0030] The training continuation condition setting table 400 may set, for example, conditions for the moment the user kicks the soccer ball. Under these conditions, for example, the speed, direction, height, rotation, and other conditions of the soccer ball at the moment the user kicks the soccer ball may be calculated, and if it is determined that the user cannot continue juggling, the training may be stopped. The training continuation condition setting table 400 may also set, for example, conditions for the range within which the soccer ball can move. Under these conditions, for example, if the soccer ball rises to a height of 1 meter or more, the training may be stopped. The training continuation condition setting table 400 may also set conditions such as the user's profile (age, experience, etc.). Under these conditions, for example, if the user is an elementary school student with no experience, the training may end after five successful juggling attempts. In this way, the information processing device 340 can define and change conditions for transitioning to various training states. This allows, for example, the user to undergo intensive training under more difficult conditions to overcome their sense of weakness. Furthermore, it is possible to provide the user with appropriate training conditions, position the soccer ball in an appropriate position, and resume juggling.

[0031] FIG. 5 is a flowchart showing processing executed by the information processing device. The program based on the flowchart shown in FIG. 5 is started when the user wears the HMD 101 and the wearing device 106, moves to the center position of the practice area 201, and holds the position for three seconds. As shown in FIG. 5, in step S501, the information processing device 340 (CPU) performs initial setting. Specifically, the information processing device 340 initializes the position, speed, direction, and number of jugglings of the soccer ball (virtual object) included in the virtual object information 362, and stores the initialized state in the virtual object information 362. Furthermore, the training continuation condition setting table 400 (see FIG. 4) is set in the training continuation condition 364. After step S501 is executed, the process proceeds to step S502.

[0032] In step S502, the information processing device 340 activates the HMD position and orientation information processing means 352. The HMD position and orientation information processing means 352 constantly saves the position and orientation of the HMD 310 in the HMD information 361. This makes it possible to detect the current position and orientation of the HMD 310 by referring to the HMD information 361. After step S502 is executed, the process proceeds to step S503.

[0033] In step S503, the information processing device 340 activates the wearing device attitude information processing means 353. The wearing device attitude information processing means 353 constantly saves the position, tilt, and acceleration of the wearing device 320 in the wearing device information 363. This makes it possible to detect the current position, tilt, and acceleration of the wearing device 320 by referring to the wearing device information 363. After step S503 is executed, the process proceeds to step S504.

[0034] In step S504, information processing device 340 activates virtual object position / orientation calculation means 355. Virtual object position / orientation calculation means 355 reads the previous motion state of the soccer ball (virtual object), such as the position, speed, and orientation, from virtual object information 362, and calculates the position, speed, and orientation of the soccer ball as needed. The calculation results are saved in virtual object information 362. This makes it possible to detect the position, speed, and orientation of the soccer ball by referring to virtual object information 362. After step S504 is executed, the process proceeds to step S505.

[0035] In step S505, the information processing device 340 activates the image processing means 354. The image processing means 354 sequentially transfers images of the real space captured by the imaging means 311 of the HMD 310 to the MR image generation means 357. After step S505 is executed, the process proceeds to step S506.

[0036] In step S506, the information processing device 340 activates the CG generation means 356. The CG generation means 356 generates CG images of a soccer ball visible from the position of the HMD 310, and sequentially transfers the CG images to the MR image generation means 357. After executing step S506, the process proceeds to step S507.

[0037] In step S507, the information processing device 340 activates the MR image generation means 357. The MR image generation means 357 generates a mixed reality image (MR image) by combining the image of real space transferred from the image processing means 354 with the CG image transferred from the CG generation means 356. This mixed reality image is transferred to the display means 312 of the HMD 310. As a result, the mixed reality image is displayed on the display means 312. Note that the CG image is updated as needed by the CG generation means 356. As a result, when a soccer ball is kicked by lifting, a mixed reality image of the soccer ball moving in the kicked direction is displayed on the display means 312. After step S507 is executed, the process proceeds to step S509.

[0038] In step S508, the information processing device 340 activates the training continuation determination means 358. The training continuation determination means 358 uses the training continuation condition setting table 400 to determine whether to continue, stop, or end the training (lifting), and the process proceeds to step S509.

[0039] In step S509, the information processing device 340 determines whether or not training has been completed in step S508. If it is determined in step S509 that training has been completed, the process proceeds to step S510. On the other hand, if it is determined in step S509 that training has not been completed, the process proceeds to step S511.

[0040] In step S510, the information processing device 340 stops the operation of the HMD position and orientation information processing means 352, the wearable device orientation information processing means 353, the virtual object position and orientation calculation means 355, the image processing means 354, the CG generation means 356, and the MR image generation means 357. After step S510 is executed, the processing ends.

[0041] In step S511, the information processing device 340 determines whether or not training was determined to be continued in step S508. If the determination in step S511 indicates that training should be continued, the process returns to step S508, and subsequent steps are executed in order. On the other hand, if the determination in step S511 indicates that training should not be continued, i.e., that training should be stopped, the process proceeds to step S512.

[0042] In step S512, the information processing device 340 stops the operation of the CG generating means 356. This stops the soccer ball from being displayed in the mixed reality image. After step S512 is executed, the process proceeds to step S513.

[0043] In step S513, the information processing device 340 stops the operation of the virtual object position and orientation calculation means 355. This stops the storage of the calculation results of the virtual object position and orientation calculation means 355 for the virtual object information 362. After step S513 is executed, the process proceeds to step S514.

[0044] In step S514, the information processing device 340 activates the training continuation determination means 358. The training continuation determination means 358 uses the training continuation condition setting table 400 to determine whether the "resume conditions" in the same "application order" as the "training continuation conditions" met in step S508 are satisfied. If it is determined that the "resume conditions" are satisfied, the process proceeds to step S515. On the other hand, if it is determined that the "resume conditions" are not satisfied, the process remains in standby at step S514.

[0045] In step S515, the training continuation determination means 358 uses the training continuation condition setting table 400 to change the motion state of the soccer ball stored in the virtual object information 362. This change is made according to the "virtual object at restart" with the same "application order" as the "training continuation condition" matched in step S508. Specifically, if the "application order" matches the first "training continuation condition," the position, speed, and orientation of the soccer ball when it was kicked the previous time are copied from the motion state of the soccer ball stored in the virtual object information 362, and the copied state is additionally saved in the virtual object information 362. Furthermore, if the "application order" matches the second "training continuation condition," the initialized state of the soccer ball set in step S501 is copied from the motion state of the soccer ball stored in the virtual object information 362, and the copied state is additionally saved in the virtual object information 362. After step S515 is executed, the process proceeds to step S516.

[0046] In step S516, the information processing device 340 activates the virtual object position / orientation calculation means 355. The virtual object position / orientation calculation means 355 reads the motion state of the soccer ball, which was saved in the virtual object information 362 in step S515, from the virtual object information 362, and calculates the position, speed, and direction of the soccer ball as needed. The calculation results are saved in the virtual object information 362. After step S516 is executed, the process proceeds to step S517.

[0047] In step S517, similarly to step S506, the information processing device 340 operates the CG generating means 356. After step S517 is executed, the process returns to step S507, and the subsequent steps are executed in order.

[0048] As described above, in this embodiment, when practicing, for example, juggling (sports) in a mixed reality image, the haptic device (haptic output means 322) of the wearing device 106 allows the user to feel as if the practice is actually being performed through at least the sense of touch, one of the five senses. Furthermore, during juggling practice, the soccer ball may fall to the ground or fall outside the practice range 201, resulting in a failed juggling attempt. Even in this case, the user can resume juggling practice without having to pick up the soccer ball from the ground or chase the soccer ball outside the practice range 201 to retrieve it. As a result, the user can efficiently practice juggling. Note that, although the wearing device 106 is shoe-type in this embodiment, this is not limiting. For example, the wearing device 106 may be a supporter-type device that covers the thighs for juggling with the thighs, or a shirt-type device that allows for trapping with the chest. Furthermore, the wearing device 106 may be a shoe-type device, a supporter-type device, and a shirt-type device integrated into one. In this case, the user can practice juggling more efficiently, with an improved sense of realism.

[0049] Second Embodiment The second embodiment will be described below with reference to FIGS. 6 to 8. Differences from the previous embodiment will be mainly described, and similar details will not be described again. This embodiment is similar to the first embodiment, except that training appropriate for improving a user's lifting technique is inferred and executed based on the user's lifting proficiency. FIG. 6 is a block diagram showing an example of the hardware configuration of the HMD, information processing device, wearing device, and camera included in the MR system according to the second embodiment. The information processing device 340 shown in FIG. 6 further includes a training accumulation information analysis unit 371, and training accumulation information 372 is allocated to the storage unit 351. The training accumulation information analysis unit 371 analyzes the training accumulation information 372 to acquire the user's lifting proficiency. The training accumulation information analysis unit 371 then infers training appropriate for improving the user's lifting technique based on the user's lifting proficiency. In this way, the training accumulation information analysis unit 371 functions as a recommendation unit that recommends practice sessions to improve the user's lifting skills. The user can then perform these practice sessions. Furthermore, the training accumulated information analysis means 371 reads information from the virtual object information 362 as needed, and when the maximum number of liftings is updated, stores the updated maximum number of liftings in the training accumulated information 372 .

[0050] FIG. 7 shows a training correction setting table stored in the training continuation conditions of the information processing device. The training correction setting table 700 shown in FIG. 7 is stored in the information processing device 340 together with the training continuation condition setting table 400 (see FIG. 4). This training correction setting table 700 is composed of four items: "Application Order," "Training Correction Condition," "Virtual Object Correction at Restart," and "Practice Type." The "Application Order" indicates the order in which the training correction conditions are applied. The training correction conditions are applied in ascending order of the application order numbers. The "Training Correction Condition" defines the conditions for executing the training correction. The "Virtual Object Correction at Restart" defines the method for correcting the virtual object when the training correction conditions are satisfied. The "Practice Type" defines the practice content after the training correction. The training correction setting table 700 is used when "Continue / Stop / End" in the training continuation condition setting table 400 is set to "Stop." When using the training correction setting table 700, information is read from the training accumulation information 372 immediately beforehand to obtain the maximum number of lifts.

[0051] To use the training correction setting table 700, the "training correction condition" with the first "application order" is evaluated first. This "training correction condition" is the condition that "the maximum number of juggling attempts is less than 10." In this case, the "correction of virtual object at restart" is to "increase the size of the virtual object (soccer ball) by 1 cm in diameter at restart." The "training type" is also set to "practice to overcome weaknesses." If the condition with the first "application order" is met, then the conditions with the second and subsequent "application order" are not evaluated. On the other hand, if the condition with the first "application order" is not met, then the condition with the second "application order" is evaluated.

[0052] The "Training Correction Condition" with "Application Order" set to 2 is the condition "Maximum number of juggling attempts is between 10 and 30." In this case, the "Virtual Object Correction at Resumption" performs the correction "Reduce the size of the virtual object (soccer ball) by 1 cm in diameter at the time of resumption." The "Practice Type" is set to "Intensive Training." If the condition with "Application Order" set to 2 is met, the conditions from 3 onwards are not evaluated. In contrast, if the condition with "Application Order" set to 2 is not met, the condition with "Application Order" set to 3 is evaluated. The "Training Correction Condition" with "Application Order" set to 3 is the condition "All (Unconditional)." In this case, the "Virtual Object Correction at Resumption" is not performed.

[0053] 8 is a flowchart showing the processing executed by the information processing device. As shown in FIG. 8, in step S801, the information processing device 340 (CPU) executes initial settings. In this step S801, in addition to the same processing as step S501 in the flowchart shown in FIG. 5, the maximum number of liftings in the training accumulation information 372 is set to "0 times." After executing step S801, the processing executes steps S802 to S804. Steps S802 to S804 are the same as steps S502 to S504. After executing step S804, the processing proceeds to step S805.

[0054] In step S805, the information processing device 340 activates the training accumulated information analysis means 371. The training accumulated information analysis means 371 reads information from the virtual object information 362 as needed, and stores the maximum number of liftings in the training accumulated information 372. This makes it possible to detect the user's maximum number of liftings by referring to the training accumulated information 372. After step S805 is executed, the process executes steps S806 to S816. Steps S806 to S816 are similar to steps S505 to S515. After step S816 is executed, the process proceeds to step S817.

[0055] In step S817, the information processing device 340 activates the training continuation determination means 358. The training continuation determination means 358 uses the training correction condition setting table 700 to correct the size of the soccer ball in the virtual object information 362 in accordance with "Correction of virtual object at time of resumption." After step S817 is executed, the process executes steps S818 and S819. Steps S818 and S819 are similar to steps S516 and S517.

[0056] As described above, in this embodiment, the training correction setting table 700 can be used for simulated practice to improve the user's lifting skills, i.e., to appropriately change the size (condition) of the soccer ball depending on the user's level of proficiency. For example, if the maximum number of lifts is less than 10, the size of the soccer ball can be increased to facilitate continued experience in lifting. Furthermore, if the maximum number of lifts is between 10 and 30, the size of the soccer ball can be decreased to allow the user to master the technique of kicking the soccer ball accurately and be able to lift the soccer ball 30 or more times continuously. The user's training history may be presented, and the aforementioned correction method may be suggested or recommended for more efficient training, allowing the user to select whether or not to make the correction. Furthermore, background information, such as the cumulative number of lifts and years of soccer experience, may be stored in the training accumulation information 372, and multiple pieces of information stored in the training accumulation information 372 may be combined.

[0057] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments and various modifications and variations are possible within the spirit and scope of the present invention. The present invention can also be realized by providing a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium. One or more general-purpose processors (ASICs) in the computer of the system or device then read and execute the program. The present invention can also be realized by a dedicated processor (e.g., an ASIC or FPGA) that implements one or more functions. Furthermore, the present invention can also be realized by a combination of a general-purpose processor and a dedicated processor. Note that the term "processor" here refers to a processor in a broad sense and includes both general-purpose and dedicated processors. Furthermore, the processing that implements the present invention may be performed by a single processor alone, or may be performed by multiple processors located in physically separate locations in cooperation with each other. In the above-described embodiments, the MR system 1000 is used for juggling practice. However, this is not limited to this. It can also be used for other purposes besides juggling practice, such as volleyball tossing and receiving practice, or tennis serving and receiving practice.

[0058] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) A generating means for generating a mixed reality image by combining an image of a real space with an image of a virtual space including a movable virtual object; an acquisition means for acquiring position and orientation information relating to the position and orientation of a user viewing the mixed reality image; a detection means for detecting a degree of contact between the virtual object in the mixed reality image and the user and a positional relationship between the virtual object in the mixed reality image and the user based on position information regarding a position of the virtual object in the mixed reality image and the position and orientation information; a control means for performing control to give the user a sensation that the virtual object is in contact with the user based on the degree of contact; a determination means for determining whether the positional relationship is appropriate for the user; and a change unit that can change a state of the virtual object in the mixed reality image in accordance with a result of the determination by the determination unit. (Configuration 2) The information processing device described in Configuration 1, characterized in that the determination means determines that the positional relationship is incorrect when the virtual object in the mixed reality image has moved to a predetermined position or has moved outside a predetermined range. (Configuration 3) The information processing device described in Configuration 2, characterized in that the judgment means judges that the positional relationship is appropriate if the virtual object in the mixed reality image has not moved to the predetermined position or has not moved outside the predetermined range. (Configuration 4) The information processing device described in any one of configurations 1 to 3, characterized in that the judgment means judges that the positional relationship is invalid when the user moves outside a range preset in the mixed reality image. (Configuration 5) The information processing device according to Configuration 4, wherein the determining means determines that the positional relationship is appropriate if the user has not moved outside the preset range. (Configuration 6) The information processing device according to any one of configurations 1 to 3, wherein, when the determination by the determination means determines that the positional relationship is not correct, the change means stops the movement of the virtual object or makes the virtual object invisible. (Configuration 7) The information processing device according to configuration 6, wherein the change means is capable of forcibly stopping the movement of the virtual object and then restarting the movement. (Configuration 8) The information processing device according to configuration 6, wherein the change means is capable of re-displaying the virtual object after hiding the virtual object. (Configuration 9) An information processing device according to any one of configurations 1 to 8, characterized in that, when the determination by the determination means determines that the positional relationship is appropriate, the change means continues the moving state of the virtual object. (Configuration 10) The information processing device according to any one of configurations 1 to 9, wherein the virtual object is a ball and the information processing device is used for practicing a ball game using the ball. (Configuration 11) The information processing device according to configuration 10, wherein the change means is capable of changing the state of the virtual object for the practice session to improve the user's ball game skills. (Configuration 12) The information processing device according to configuration 10 or 11, wherein the change means is capable of changing the state of the virtual object in accordance with the user's level of proficiency in the ball game. (Configuration 13) The information processing device according to any one of configurations 10 to 12, further comprising a recommendation unit that recommends the practice simulation to improve the user's ball game skills. (Configuration 14) The information processing device according to any one of configurations 1 to 13, characterized in that it is communicably connected to a head-mounted display including a display means for displaying the mixed reality image. (Method 1) A method for controlling an information processing device, comprising: a generating step of generating a mixed reality image by combining an image of real space with an image of virtual space including a movable virtual object; an acquisition step of acquiring position and orientation information relating to the position and orientation of a user viewing the mixed reality image; a detection step of detecting a degree of contact between the virtual object in the mixed reality image and the user based on position information regarding the position of the virtual object in the mixed reality image and the position and orientation information acquired in the acquisition step; a control step of performing control to give the user a sensation that the virtual object has come into contact with the user, based on a detection result from the detection step; a determining step of determining whether the detection result in the detecting step is appropriate for the user; a change step of changing a state of the virtual object in the mixed reality image in accordance with a result of the determination in the determination step. (Program 1) A program for causing a computer to execute each means of the information processing device according to any one of configurations 1 to 14. [Explanation of symbols]

[0059] 103 Information processing equipment 340 Information Processing Equipment 352 HMD position and orientation information processing means 353 Mounting device attitude information processing means 355 Virtual object position and orientation calculation means 357 MR image generation means 359 Tactile output information processing means

Claims

1. a generating means for generating a mixed reality image by combining an image of real space with an image of virtual space including a movable virtual object; an acquisition means for acquiring position and orientation information relating to the position and orientation of a user viewing the mixed reality image; a detection means for detecting a degree of contact between the virtual object in the mixed reality image and the user and a positional relationship between the virtual object in the mixed reality image and the user based on position information regarding the position of the virtual object in the mixed reality image and the position and orientation information; a control means for performing control to give the user a sensation that the virtual object is in contact with the user based on the degree of contact; a determination means for determining whether the positional relationship is appropriate for the user; and a change unit that can change a state of the virtual object in the mixed reality image in accordance with a result of the determination by the determination unit.

2. 2. The information processing device according to claim 1, wherein the determination means determines that the positional relationship is incorrect when the virtual object in the mixed reality image has moved to a predetermined position or has moved outside a predetermined range.

3. 3. The information processing device according to claim 2, wherein the determination means determines that the positional relationship is appropriate if the virtual object in the mixed reality image has not moved to the predetermined position or has not moved outside the predetermined range.

4. The information processing apparatus according to claim 1 , wherein the determining means determines that the positional relationship is not correct when the user moves outside a range that is preset in the mixed reality image.

5. 5. The information processing apparatus according to claim 4, wherein said determining means determines that said positional relationship is appropriate when said user has not moved outside said preset range.

6. 2. The information processing device according to claim 1, wherein, when the determination by the determination means determines that the positional relationship is not correct, the change means stops the movement of the virtual object or makes the virtual object invisible.

7. 7. The information processing apparatus according to claim 6, wherein the change means is capable of forcibly stopping the movement of the virtual object and then restarting the movement.

8. 7. The information processing apparatus according to claim 6, wherein the change means is capable of re-displaying the virtual object after hiding the virtual object.

9. 2. The information processing apparatus according to claim 1, wherein, when the determining means determines that the positional relationship is appropriate, the changing means continues the moving state of the virtual object.

10. 2. The information processing device according to claim 1, wherein the virtual object is a ball, and the information processing device is used for practicing ball games using the ball.

11. The information processing device according to claim 10 , wherein the change means is capable of changing the state of the virtual object for the practice session to improve the user's ability in the ball game.

12. The information processing device according to claim 10 , wherein the change means is capable of changing the state of the virtual object in accordance with the user's level of proficiency in the ball game.

13. The information processing device according to claim 10, further comprising a recommendation unit that recommends the practice session to improve the user's ball game skills.

14. The information processing apparatus according to claim 1 , wherein the information processing apparatus is communicably connected to a head-mounted display having a display means for displaying the mixed reality image.

15. A method for controlling an information processing device, comprising: a generating step of generating a mixed reality image by combining an image of real space with an image of virtual space including a movable virtual object; an acquisition step of acquiring position and orientation information relating to the position and orientation of a user viewing the mixed reality image; a detection step of detecting a degree of contact between the virtual object in the mixed reality image and the user based on position information regarding the position of the virtual object in the mixed reality image and the position and orientation information acquired in the acquisition step; a control step of performing control to give the user a sensation that the virtual object has come into contact with the user, based on a detection result from the detection step; a determining step of determining whether the detection result in the detecting step is appropriate for the user; a change step of changing a state of the virtual object in the mixed reality image in accordance with a result of the determination in the determination step.

16. 2. A program for causing a computer to execute each means of the information processing apparatus according to claim 1.

Images