Information processing system, information processing method and program

Abstract

To provide an information processing system, method, and program for improving a function of grasping a body position by changing a combination state of intracerebral regions.SOLUTION: In an information processing system 1000, an information processing device 100 displays target positions of the hands or legs of a user in a virtual space, on a display unit of an HMD 110 worn on the head of the user. The positions of the hands or legs of a user in a virtual space are obtained on the basis of motion information about the motion of the hands or legs of the user in an actual space. The difference between the obtained positions of the hands or legs of the user and the target positions is obtained, and information on the difference is displayed on the display unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technology relating to a brain function training device that performs learning on a target event. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-91359 [Non-Patent Document 1] Natsue Yoshimura, 6 others, "Brain Sciences", [online], December 10, 2020, [searched on December 10, 2021], Internet <ULR: Brain Sciences | Free Full-Text | Age-Related Decline of Sensorimotor Integration Influences Resting-State Functional Brain Connectivity (mdpi.com)> Summary of the Invention [Problem to be solved by the invention]

[0004] The brain's ability to grasp the body's position can decline due to aging, etc. The technology in Patent Document 1 cannot improve the function of grasping the body's position by changing the connectivity of brain regions. [Means for solving the problem]

[0005] According to one aspect of the present invention, an information processing system is provided. This information processing system displays a target position of a user's hand or foot in a virtual space on a display unit worn on the user's head. The system determines the position of the user's hand or foot in the virtual space based on motion information related to the user's hand or foot motion in real space. The system determines the difference between the determined position of the user's hand or foot and the target position. Information related to the difference is displayed on the display unit. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system 1000. As shown in FIG. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing device 100. As shown in FIG. [Figure 3] FIG. 3 is a diagram illustrating an example of the hardware configuration of the HMD 110. As shown in FIG. [Figure 4] FIG. 4 is a diagram showing an example of information processing related to a game in the information processing system 1000. [Figure 5] FIG. 5 is a diagram showing an example of a screen on which a target position is displayed in a virtual space. [Figure 6] FIG. 6 is a diagram (part 1) showing an example of a screen 600 on which rewards and the like are displayed. [Figure 7] FIG. 7 is a diagram showing an example of a screen 700 displaying the position of the hand in the virtual space. [Figure 8] FIG. 8 is a diagram (part 2) showing an example of a screen 800 on which rewards and the like are displayed. [Figure 9] FIG. 9 is a diagram showing an example of a game result screen 900. As shown in FIG. [Figure 10] FIG. 10 is a diagram showing an example of brain functional connectivity that changes with continued use of the information processing system 1000. [Figure 11] FIG. 11 is a diagram for explaining an example of information processing related to deviation detection. [Figure 12]FIG. 12 is a diagram showing an example of brain functional connections that are correlated with game performance (the magnitude of the angle at which the deviation was noticed) and age when playing the game of Variation 5, focusing on the movement-related regions. [Figure 13] FIG. 13 is a diagram showing the angle between the line connecting the starting point and the end point and the line connecting each position (1 / 2 and 1 / 4 of the total length) of the trajectory points of the hand from the starting point to the end point. [Figure 14] FIG. 14 is a diagram showing an example of brain functional connections that are correlated with age and the magnitude of the angle at positions 25%, 40%, 50%, and 60% from the starting point in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0007] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings. Various features shown in the following embodiments can be combined with each other.

[0008] Incidentally, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium, or may be provided so that it can be downloaded from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0009] In this embodiment, the term "unit" may also include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In addition, various types of information are handled in this embodiment, and this information may be represented by, for example, physical values ​​of signal values ​​representing voltages and currents, high and low signal values ​​as a binary bit set consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculations may be performed on a circuit in the broad sense.

[0010] In addition, a circuit in the broad sense is a circuit realized by at least appropriately combining a circuit, circuitry, a processor, a memory, etc. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0011] <Embodiment 1> 1. System Configuration FIG. 1 is a diagram illustrating an example of the system configuration of an information processing system 1000. As illustrated in FIG. 1, the information processing system 1000 includes, as a system configuration, an information processing device 100, an HMD (Head Mounted Display) 110, a controller 120, and a controller 130. The information processing device 100 and the HMD 110 are connected to each other so as to be able to communicate with each other via wireless communication or the like. The information processing device 100 is also connected to each of the controllers 120 and 130 so as to be able to communicate with each other via wireless communication or the like. The information processing device 100 is, for example, a game console main unit or the like, and performs processing of the present embodiment, which will be described later. The HMD 110 is a head-mounted display worn on a user's head. The HMD 110 may be a transparent type or a non-transparent type. However, in the present embodiment, the HMD 110 will be described as a non-transparent type unless otherwise specified. Note that this does not limit the present embodiment. The controller 120 is a controller held in the user's right hand. The controller 130 is a controller held in the user's left hand. The shapes of the controllers 120 and 130 shown in FIG. 1 are merely examples, and the controllers 120 and 130 may be glove-shaped or the like and may be fixed to the user's hands, for example, by being worn on the hands. Here, the information processing device 100 is an example of an information processing system as set forth in the claims, and the information processing system 1000 is another example of an information processing system as set forth in the claims.

[0012] (Processing Overview) The information processing device 100 controls the progress of the game as described below. As part of the game, the information processing device 100 displays a target position of the user's hand in a virtual space on the display unit of the HMD 110 worn on the user's head. The information processing device 100 determines the position of the user's hand in the virtual space based on movement information related to the user's hand movement in real space. Here, the user's hand movement is a movement of the user attempting to align the user's hand with the target position displayed in the virtual space. The information processing device 100 determines the difference between the determined position of the user's hand and the target position. Then, the information processing device 100 displays information related to the difference on the display unit of the HMD 110. In this way, it is possible to change and train the coordinated activity of brain areas involved in motor and visual functions.

[0013] 2. Hardware Configuration (1) Hardware Configuration of Information Processing Device 100 FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing device 100. The information processing device 100 includes, as its hardware configuration, a control unit 201, a storage unit 202, and a communication unit 203. The control unit 201 is a processor such as a CPU (Central Processing Unit) and controls the entire information processing device 100. The storage unit 202 is a hard disk drive (HDD), a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or any combination thereof, and stores programs and data used when the control unit 201 executes processing based on the programs. The control unit 201 executes processing based on the programs stored in the storage unit 202, thereby realizing processing such as that shown in the flowchart in FIG. 4 (described later). The communication unit 203 controls communication between the information processing device 100 and other devices (e.g., the HMD 110, the controller 120, and the controller 130). In this embodiment, the communication unit 203 is described as controlling wireless communication with other devices, but it may also control wired communication. Furthermore, the information processing device 100 may have a communication unit that controls wireless communication and a communication unit that controls wired communication. The storage unit 202 is an example of a storage medium.

[0014] (2) HMD110 hardware configuration FIG. 3 is a diagram illustrating an example of the hardware configuration of the HMD 110. The HMD 110 includes, as its hardware configuration, a control unit 301, a storage unit 302, an imaging unit 303, a display unit 304, and a communication unit 305. The control unit 301 is a processor such as a CPU, and controls the entire HMD 110. The storage unit 302 is a ROM, RAM, SSD, or any combination thereof, and stores programs and data used when the control unit 301 executes processing based on the programs. The control unit 301 executes processing based on the programs stored in the storage unit 302, thereby realizing the functions of the HMD 110. The imaging unit 303 is a camera or the like, and captures an image of a subject. The display unit 304 is a display or the like, and displays a virtual space image and objects displayed in the virtual space image. The communication unit 305 manages communication between the HMD 110 and other devices (e.g., the information processing device 100, etc.). In this embodiment, the communication unit 305 is described as controlling wireless communication with other devices, but it may also control wired communication. Also, the HMD 110 may have a communication unit that controls wireless communication and a communication unit that controls wired communication. The storage unit 302 is an example of a storage medium.

[0015] 3. Information Processing FIG. 4 is a diagram showing an example of information processing related to a game in the information processing system 1000. In step S401, the information processing device 100 determines a target position and displays the target position on the display unit 304 of the HMD 110. FIG. 5 shows an example of a screen including the target position. FIG. 5 is a diagram showing an example of a screen on which the target position is displayed in a virtual space. A screen 500 is displayed on the display unit 304. The screen 500 shows the virtual space. A left target object 501 is an object indicating the target position of the user's left hand in the virtual space. A right target object 502 is an object indicating the target position of the user's right hand in the virtual space. Note that the objects displayed in the foreground on the screen are the initial positions of the user's hands. Before starting the game, the user holds the controller 120 and the controller 130, aligns their hands with the initial positions, and then extends their hands. The information processing device 100 determines the positions that the user can reach based on information about the positions of the outstretched hands, and displays the left target object 501 and the right target object 502 within the user's reach. In step S402, the information processing device 100 turns off the display of the hand positions so that the positions of the user's hands holding the controllers 120 and 130 are not displayed on the display unit 304 of the HMD 110.

[0016] In step S403, the information processing device 100 grasps, tracks, and records the positions of the user's hands in the virtual space based on information from the controllers 120 and 130. The controllers 120 and 130 include sensors such as an acceleration sensor and a gyro sensor. Information acquired by these sensors is transmitted from the controllers 120 and 130 to the information processing device 100. The information acquired by these sensors is an example of motion information related to the user's hands. Sensors such as an acceleration sensor and a gyro sensor are examples of motion sensors. The information processing device 100 grasps, tracks, and records the positions of the user's hands in the virtual space based on the received information. In step S404, the information processing device 100 determines whether or not predetermined information has been received from the controller 120 and the controller 130. If the information processing device 100 determines that predetermined information has been received from the controller 120 and the controller 130, it determines that there has been a user input, and proceeds to S405. If the information processing device 100 determines that the predetermined information has not been received from the controller 120 and the controller 130, it determines that there has been no user input, and returns to S403. The user holds the controller 120 in their right hand and the controller 130 in their left hand, and when it determines that their right hand has reached the position of the right target object 502 and their left hand has reached the position of the left target object 501, they perform an operation such as pressing a predetermined button on the controller. If it determines that a predetermined operation has been performed, the controllers 120 and 130 transmit predetermined information indicating that the predetermined operation has been performed to the information processing device 100. Here, the predetermined operation is not limited to pressing a predetermined button, but may be any predetermined method, such as shaking the controller 120 and the controller 130. However, in this embodiment, for the sake of simplicity, the predetermined operation will be described as pressing a predetermined button.

[0017] In step S405, the information processing device 100 calculates the distance between the position of the right hand in the virtual space and the target position of the right hand, based on the position of the user's right hand in the virtual space when the predetermined information was received and the position of the right target object 502 in the virtual space. Similarly, the information processing device 100 calculates the distance between the position of the left hand in the virtual space and the target position of the left hand, based on the position of the user's left hand in the virtual space when the predetermined information was received and the position of the left target object 501 in the virtual space.

[0018] In step S406, the information processing device 100 calculates a reward for the right hand based on the distance between the position of the right hand in the virtual space and the target position of the right hand. Similarly, the information processing device 100 calculates a reward for the left hand based on the distance between the position of the left hand in the virtual space and the target position of the left hand. The reward may be such that the closer the distance, the higher the score, or such that better words are displayed as the distance increases. The information processing device 100 determines the reward, for example, as "GREAT!!", "GOOD!!", "FAIR", "POOR", "BAD", etc., in order of proximity between the hand position and the target position based on a plurality of predetermined distance thresholds.

[0019] In step S407, the information processing device 100 displays the determined reward, etc. on a screen displayed on the display unit 304 of the HMD 110. FIG. 6 is a diagram (part 1) showing an example of a screen 600 on which the reward, etc. is displayed. A left hand object 601 included in the screen 600 indicates the position of the user's left hand when it is determined that the user has reached the left target object 501 and performed a predetermined operation. A right hand object 602 included in the screen 600 indicates the position of the user's right hand when it is determined that the user has reached the right target object 502 and performed a predetermined operation. A left evaluation object 603 included in the screen 600 is an object indicating a reward for the left hand. In the example of FIG. 6, the left evaluation object 603 includes information of "FAIR." A right evaluation object 604 included in the screen 600 is an object indicating a reward for the right hand. In the example of FIG. 6, the right evaluation object 604 includes information of "GOOD!!." Furthermore, a character string 605 includes numerical values ​​indicating the direction and magnitude of the deviation between the left hand and the target position of the left hand. The direction of deviation is an example of information indicating whether the position of the user's hand has deviated forward, backward, left, or right from the target position. The magnitude of deviation is an example of a numerical value indicating the magnitude of the difference. Similarly, character string 606 includes a numerical value indicating the direction of deviation between the right hand and the target position of the right hand and the magnitude of the deviation. From step S402 to step S407, the user cannot see the position of his or her own hand until a predetermined operation is performed. The display processing from step S402 to step S407 is an example of display processing in the first display mode.

[0020] After the processing of step S407 is completed and a predetermined time has elapsed, in step S408, the information processing device 100 turns on the display of the hand position so that the positions of the user's hands holding the controllers 120 and 130 are displayed on the display unit 304 of the HMD 110. In step S409, the information processing device 100 grasps, tracks, and records the positions of the user's hands in the virtual space based on information from the controllers 120 and 130. Because the display of hand positions is ON, the information processing device 100 controls the display unit 304 of the HMD 110 to display objects indicating the positions of the user's hands based on the grasped positions of the user's hands in the virtual space. FIG. 7 is a diagram showing an example of a screen 700 displaying the positions of the hands in the virtual space. A left hand object 701 included in the screen 700 is an object indicating the position of the user's left hand in the virtual space. The left hand object 701 is displayed on the screen 700 until the user moves their left hand and performs a predetermined operation. A right hand object 702 included in the screen 700 is an object indicating the position of the user's right hand in the virtual space. Similarly, the right hand object 702 is displayed on the screen 700 until the user moves their right hand and performs a predetermined operation.

[0021] In step S410, the information processing device 100 determines whether or not predetermined information has been received from the controller 120 and the controller 130. If the information processing device 100 determines that predetermined information has been received from the controller 120 and the controller 130, it proceeds to S411, assuming that there has been input by the user. If the information processing device 100 determines that there has not been input by the user, it returns to S409.

[0022] In step S411, the information processing device 100 calculates the distance between the position of the right hand in the virtual space and the target position of the right hand, based on the position of the user's right hand in the virtual space when the predetermined information is received and the position of the right target object 502 in the virtual space. Similarly, the information processing device 100 calculates the distance between the position of the left hand in the virtual space and the target position of the left hand, based on the position of the user's left hand in the virtual space when the predetermined information is received and the position of the left target object 501 in the virtual space.

[0023] In step S412, the information processing device 100 calculates a reward for the right hand based on the distance between the position of the right hand in the virtual space and the target position of the right hand. Similarly, the information processing device 100 calculates a reward for the left hand based on the distance between the position of the left hand in the virtual space and the target position of the left hand.

[0024] In step S413, the information processing device 100 displays the determined reward, etc. on a screen displayed on the display unit 304 of the HMD 110. FIG. 8 is a diagram (part 2) showing an example of a screen 800 on which the reward, etc. is displayed. A left evaluation object 801 included in the screen 800 is an object indicating a reward related to the left hand. In the example of FIG. 8, the left evaluation object 801 includes information of "GREAT!!". A right evaluation object 802 included in the screen 800 is an object indicating a reward related to the right hand. In the example of FIG. 8, the right evaluation object 802 includes information of "GREAT!!". Since objects indicating the positions of the user's hands have been displayed on the display unit 304 of the HMD 110 since step S408, the user can move their hands to the left target object 501 and the right target object 502 while checking the current positions of their hands. This increases the likelihood that "GREAT!!" will be displayed as a reward.

[0025] In step S414, the information processing device 100 determines whether the processes from step S401 to step S413 have been repeated a specified number of times. If the information processing device 100 determines that the processes have been repeated a specified number of times, the process proceeds to step S415. If the information processing device 100 determines that the processes have not been repeated a specified number of times, the process returns to step S401. In this embodiment, the specified number of times is, for example, 30 times. In step S415, the information processing device 100 displays the game result on the screen displayed on the display unit 304 of the HMD 110. Fig. 9 is a diagram showing an example of a game result screen 900. As shown in Fig. 9, the game result screen 900 includes information on the number of times related to the reward or evaluation for the left hand and information on the number of times related to the reward or evaluation for the right hand in the game.

[0026] The user cannot see the position of his or her own hand from step S402 to step S407. The display process from step S408 to step S413 is an example of the display process in the second display mode.

[0027] As described above, in the first display mode, the determined position of the user's hand is not displayed on the display unit. In the second display mode, the determined position of the user's hand is displayed on the display unit. In this embodiment, the information processing device 100 transitions from the first display mode to the second display mode. In this way, by repeating the action of first moving the hand to a target position while the position of the user's hand is not visible, and then moving the hand to the target position while the position of the user's hand is visible, changes occur in the functional connections related to movement and vision, and attention and vision, and the coordination of movement, vision, and attention functions can be affected.

[0028] FIG. 10 is a diagram showing an example of changes in brain activity due to continuous use of the information processing system 1000. Four men and four women in their 60s were asked to play the above-mentioned game twice a day for 20 days. Brain activity was measured at rest before and after the 20 days, and changes in brain functional connections (networks) were confirmed. The main functional connections that showed changes are as follows: Left supplementary motor area (SMA) - Right cerebellar lobe 10 (↓) Part of the salience network - visual-related areas (↓) Cerebellar lobe 3 - part of the secondary somatosensory cortex (↑) Secondary somatosensory cortex - visual-related area (↑) · Visual-related areas - Language-related areas (↓)

[0029] As described above, according to this embodiment, an HMD or the like is used to create a situation where the user's hands are not visible, and the user is instructed of the discrepancy between the hand position and the target position by moving the hand to a target position displayed on the screen, and by prompting the user to make corrections, the connectivity state of the brain areas correlated with movement can be changed, thereby improving motor function.

[0030] (Variation 1) A description will be given of a variation of embodiment 1. In the above-described embodiment, an example has been given in which the controller 120 and the controller 130 are held in the hands. However, the controllers 120 and 130 may be attached to the feet with leg bands or the like. In such a configuration, the same effect can be obtained by replacing "hands" in the first embodiment with "feet." In other words, according to variant example 1, an HMD or the like is used to create a situation where the user's toes are not visible, and the user is made to move their feet to a target position displayed on the screen, thereby instructing the user about the discrepancy between the foot position and the target position and prompting them to make corrections, thereby changing the connectivity of areas in the brain that are correlated with movement and improving motor function.

[0031] (Variation 2) A modification of the first embodiment will be described. The information processing device 100 of the second modification may determine whether to transition to the second display mode after the first display mode, depending on the difference between the position of the user's hand and the target position of the hand. For example, if there is almost no deviation between the position of the user's hand and the target position, it is not necessary to transition to the second display mode in which the user approaches the target position while viewing their own hand. Therefore, if the difference between the position of the user's left hand and the target position of the left hand is within a threshold, and the difference between the position of the user's right hand and the target position of the right hand is within a threshold, the information processing device 100 does not transition from the first display mode to the second display mode, but transitions to the next game. On the other hand, if the difference between the position of the user's left hand and the target position of the left hand is not within the threshold, or the difference between the position of the user's right hand and the target position of the right hand is not within the threshold, the information processing device 100 transitions from the first display mode to the second display mode. By adopting such a configuration, unnecessary processing can be reduced.

[0032] (Variation 3) A modification of the first embodiment will be described. The information processing device 100 of Modification 3 displays an image generated based on the measurement results of the user's electroencephalogram on the display unit 304 of the HMD 110. The information processing device 100 receives the measurement results of the blood flow volume in the brain before and after playing the above-mentioned game from a communicable MRI (Magnetic Resonance Imaging) or the like, and generates an image that indicates, based on the measurement results, functional areas in which changes in the user's brain activity have occurred in color or text, and displays the image on the display unit 304 of the HMD 110. Furthermore, if the HMD 110 has a function for measuring the blood flow volume in the brain, the information processing device 100 may receive the measurement results of the blood flow volume in the brain before and after playing the game from the HMD 110. By using this configuration, the user can understand, based on changes in blood flow in the brain, which parts of the brain have changed in connectivity before and after playing the game.

[0033] (Variation 4) A modification of the first embodiment will be described. The information processing device 100 of the fourth modification determines the target position of the next hand based on the magnitude of the difference, and displays the target position on the display unit 304 of the HMD 110. For example, if the difference between the hand position and the target position is equal to or greater than a threshold, the information processing device 100 may determine the target position of the next game to be within a predetermined range close to the target position of the previous game, and if the difference between the hand position and the target position is less than the threshold, the information processing device 100 may determine the target position of the next game randomly. By adopting such a configuration, it is possible to gradually correct the positional deviation.

[0034] (Variation 5) We measured the angle at which misalignment was noticed using a musculoskeletal robot capable of precisely measuring the position of the hand, and then performed functional magnetic resonance imaging (fMRI) at rest to investigate the network correlated with age and misalignment sensitivity in motor-related areas. FIG. 11 is a diagram illustrating an example of information processing related to misalignment detection. Mirror in FIG. 11 is a screen displayed on the display unit 304 of the HMD 110. Origin indicates the initial position. Target is the target position to which the hand is to be moved. When a user moves their hand from the origin position to the target position, the cursor may move in accordance with the user's hand movement (no visual rotation) or may move out of sync with the user's hand movement (cursor visual rotation). After moving their hand from the origin position to the target position, the user answers Yes or No as to whether the cursor was out of sync with their hand movement. FIG. 12 is a diagram illustrating an example of brain activity correlated with game performance (the angle at which the misalignment was noticed) and age when playing the game of Variation 5. For FIGS. 11 and 12, see Non-Patent Document 1.

[0035] In addition, using a system that can easily measure hand position using a VR-HMD, we investigated whether networks correlated with age and misalignment sensitivity could be detected using resting-state fMRI, as in the case of musculoskeletal robots, including in areas other than those related to movement. Focusing on the timing at which the deviation is recognized, we calculated the deviation for each phase of the hand movement from the start point to the end point, and identified a network that correlates with each. FIG. 13 is a diagram showing the angle between the line connecting the starting point and the end point and the line connecting each position (1 / 2 and 1 / 4 of the total length) of the trajectory points of the hand from the starting point to the end point. FIG. 14 is a diagram showing an example of brain functional connections that are correlated with age and the magnitude of the angle at positions 25%, 40%, 50%, and 60% from the starting point in FIG. Movements within 25-40% of the starting point involve connections centered on the visual information binding area and areas related to attention. It is said that movements 50% of the way from the starting point involve connections centered on the visual cortex (motor-related areas) and connections centered on movement and attention. It is said that connections centered on the cerebellar vermis (such as movement-related visual fields) are involved in movements 60% of the distance from the starting point.

[0036] (Variation 6) A modification of the first embodiment will be described. In the above-described first embodiment and the like, the HMD 110 and the information processing device 100 are described as separate entities. However, the functions of the information processing device 100 may be included in the HMD 110, and the HMD 110 may be configured as an integrated device. Even with the configuration of variant example 6, a situation can be created in which the user's hands are not visible using an HMD or the like, and by having the user move their hand to a target position displayed on the screen, the discrepancy between the hand position and the target position can be instructed to the user, and by prompting them to make corrections, the connectivity state of the brain areas correlated with movement can be changed, thereby improving motor function.

[0037] (Variation 7) A modification of the first embodiment will be described. In the above-described first embodiment and the like, it has been described that motion information regarding the user's hand motions is acquired from motion sensors included in the controllers 120, 130, and the like. However, the information processing system 1000 of the seventh modification is configured to include a webcam and the like. The information processing device 100 may obtain the user's hand motions and the like from video information (a plurality of consecutive captured images) including the user captured by the webcam. Then, the information processing device 100 may obtain the position of the user's hand in the virtual space based on the obtained user's hand motions and the like. According to the seventh modification, the movement, position, etc. of the user's hand can be identified even if the user does not hold a controller or the like.

[0038] (Variation 8) A modification of the first embodiment will be described. In the above-described first embodiment and the like, it has been described that the first display mode is switched to the second display mode after a predetermined time has elapsed. The information processing device 100 of the eighth modification is configured to switch between the first display mode and the second display mode when a predetermined command is received from the controller 120 or the controller 130 based on the selection of a predetermined button on the controller 120 or the controller 130. According to the eighth modification, the user can switch, at his / her own will, whether or not to display an object representing his / her hand in the virtual space during the game.

[0039] <Additional Notes> The invention may be provided in the following aspects: The information processing system receives the movement information from a motion sensor related to the user, and determines the position of the user's hands or feet in the virtual space based on the movement information. In the information processing system, a numerical value indicating the magnitude of the difference is displayed on the display unit as information regarding the difference. In the information processing system, the information relating to the difference is displayed on the display unit as information indicating whether the determined position of the user's hand or foot has shifted forward, backward, left or right from the target position. In the information processing system, in a first display mode, the determined positions of the user's hands or feet are not displayed on the display unit, and in a second display mode, the determined positions of the user's hands or feet are displayed on the display unit. In the information processing system, after the first display mode, the information processing system transitions to the second display mode. In the information processing system, whether or not to transition to the second display mode after the first display mode is determined according to the difference during the first display mode. The information processing system switches between the first display mode and the second display mode based on a command input. In the information processing system, an image generated based on the measurement results of the user's brain waves is displayed on the display unit, and the image includes information regarding changes in the user's brain activity. In the information processing system, the next target position is determined based on the magnitude of the difference, and the target position is displayed. An information processing method executed by an information processing system, comprising: displaying a target position of a user's hand or foot in a virtual space on a display unit worn on the user's head; determining the position of the user's hand or foot in the virtual space based on movement information regarding the movement of the user's hand or foot in real space; determining the difference between the determined position of the user's hand or foot and the target position; and displaying information regarding the difference on the display unit. A program for causing a computer to function as the information processing system. Of course, this is not the case.

[0040] For example, the above-mentioned program may be provided as a computer-readable non-transitory storage medium that stores the program. Furthermore, the above-described embodiments and modifications may be combined in any desired manner. There may be multiple processors such as CPUs. Data may be stored in a storage unit of each device, or in a storage unit of another device that can communicate with each device.

[0041] Finally, while various embodiments of the present invention have been described, these are presented by way of example only and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. The embodiments and their modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as defined in the appended claims. [Explanation of symbols]

[0042] 3: Cerebellum 10: Right cerebellum 100: Information processing device 110: HMD 120: Controller 130: Controller 201: Control unit 202: Storage section 203: Communications Department 301: Control section 302: Storage section 303: Imaging unit 304:Display section 305: Communications Department 500:screen 501: Left target object 502: Right target object 600:screen 601: Left hand object 602: Right hand object 603: Left evaluation object 604: Right evaluation object 605 : String 606: String 700:screen 701: Left hand object 702: Right hand object 800:screen 801: Left evaluation object 802: Right evaluation object 900: Game result screen 1000: Information Processing Systems

Claims

1. An information processing system, At least one control unit is provided, The control unit is configured to execute the following steps by reading a program: In the position display step, a target position of the hand or foot of the user in the virtual space is displayed on a display unit attached to the user's head; In the position calculation step, a position of the user's hand or foot in the virtual space is calculated based on movement information relating to a movement of the user's hand or foot in the real space; In the difference calculation step, a difference between the obtained position of the hand or foot of the user and the target position is calculated, In the difference display step, information about the difference is displayed on the display unit; In the position display step, In a first display mode, the positions of the user's hands or feet calculated in the position calculation step are not displayed on the display unit, In the second display mode, the positions of the user's hands or feet calculated in the position calculation step are displayed on the display unit. Information processing system.

2. 2. The information processing system according to claim 1, In the position calculation step, the motion information is received from a motion sensor related to the user, and the positions of the user's hands or feet in the virtual space are calculated based on the motion information. Information processing system.

3. 3. The information processing system according to claim 1, In the difference display step, a numerical value indicating the magnitude of the difference is displayed on the display unit as information about the difference. Information processing system.

4. 4. The information processing system according to claim 1, In the difference display step, information on whether the obtained position of the user's hand or foot has shifted forward, backward, left, or right with respect to the target position is displayed on the display unit as information on the difference. Information processing system.

5. 5. The information processing system according to claim 1, In the position display step, the display mode is switched from the first display mode to the second display mode. Information processing system.

6. 6. The information processing system according to claim 5, In the position display step, it is determined whether or not to transition to the second display mode after the first display mode, depending on the difference in the first display mode. Information processing system.

7. 6. The information processing system according to claim 5, the position display step switches between the first display mode and the second display mode based on a command input. Information processing system.

8. 8. The information processing system according to claim 1, In the image display step, an image generated based on the measurement result of the electroencephalogram of the user is displayed on the display unit; the image includes information about changes in the user's brain activity. Information processing system.

9. 9. The information processing system according to claim 1, In the position display step, the next target position is determined based on the magnitude of the difference, and the target position is displayed. Information processing system.

10. An information processing method, comprising: The information processing system according to any one of claims 1 to 9 includes the steps executed by the information processing system. Information processing methods.

11. A program, Computer, A program for causing the information processing system according to any one of claims 1 to 9 to function.

Images