Game program, game processing method, and game device

Abstract

To guide contents that a user should operate in a simple configuration and in a way that the user can play more intuitively.SOLUTION: A game program allows a computer that outputs video such that a user can visually recognize it and executes a process of a game playable by using a video output device allowing actual space to be recognized visually by fitting it onto the user's head to execute the steps of: acquiring captured video of the actual space; generating virtual space corresponding to the actual space from the captured video; disposing a designation object for designating a user's operation in a position based on a reference position corresponding to the user within the virtual space so as to be visually recognized by the user; displaying the virtual space in which at least the designation object is disposed in accordance with the actual space; detecting at least partial operation of the user's body from the captured video; and evaluating the detected operation on the basis of timing and a position based on the designation object disposed in the virtual space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a game program, a game processing method, and a game device. [Background technology]

[0002] Among music games, there is a dance game that detects the user's body movements and evaluates the quality of the dance. For example, Patent Document 1 discloses a dance game in which the path and timing that the user (player) should trace by moving their hands and feet to music are displayed on a game screen displayed opposite the user, and the user moves their hands and feet while looking at the display. This dance game can be played, for example, on a home game console.

[0003] Patent Document 2 also discloses a dance game in which a user steps on an operation panel arranged in real space in accordance with instructions displayed on a game screen in sync with music. This dance game requires an operation panel to be placed at the user's feet to determine where the user's feet step in real space, and is an example of a so-called arcade game installed in amusement facilities such as game arcades. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-196286 A [Patent Document 2] JP 2016-193006 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the game described in the above Patent Document 1, although the game screen can guide the user as to what kind of body movement (trajectory) and at what timing, it is not suitable for a game that instructs the user on the position in real space where the user should move (for example, the position where the foot should step). For example, if an attempt is made to realize a game such as that described in the above Patent Document 2 with a simple configuration such as a home game machine without installing an operation panel at the user's feet, it is difficult for the user to know to which position in real space the foot should be moved, and intuitive play may be difficult.

[0006] One objective of some aspects of the present invention is to provide a game program, a game processing method, and a game device that guides the user on the actions they should take, allowing for more intuitive play with a simple configuration.

[0007] Another aspect of the present invention has as an object to provide a game program, a game processing method, and a game device that are capable of achieving the effects described in the embodiments described below. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, one aspect of the present invention is a game program for causing a computer that executes processing for a game that can be played using a video output device that is worn on a user's head and outputs video visible to the user and allows real space to be viewed, to execute the following steps: acquiring an image of a real space; generating a virtual space corresponding to the real space from the image; placing an instruction object that instructs the user to move at a position in the virtual space based on a reference position corresponding to the user so that the instruction object is visible to the user; displaying the virtual space in which at least the instruction object is placed in correspondence with the real space; detecting a movement of at least a part of the user's body from the image; and evaluating the detected movement based on a timing and position based on the instruction object placed in the virtual space.

[0009] Moreover, one aspect of the present invention is a game processing method executed by a computer that executes processing of a game that can be played using a video output device that is worn on a user's head and outputs an image visible to the user and allows a real space to be viewed, the game processing method including the steps of: acquiring an image of the real space; generating a virtual space corresponding to the real space from the image; placing an instruction object that instructs the user to move at a position in the virtual space based on a reference position corresponding to the user so that the instruction object is visible to the user; displaying the virtual space in which at least the instruction object is placed in correspondence with the real space; detecting a movement of at least a part of the user's body from the imaged video; and evaluating the detected movement based on a timing and position based on the instruction object placed in the virtual space.

[0010] Moreover, one aspect of the present invention is a gaming device that is worn on a user's head and that executes processing of a game that can be played using a video output device that outputs an image visible to the user and allows a real space to be viewed, the gaming device comprising: an acquisition unit that acquires an image of the real space; a generation unit that generates a virtual space corresponding to the real space from the image acquired by the acquisition unit; a placement unit that places an instruction object that instructs the user to perform an action in a position based on a reference position corresponding to the user, within the virtual space generated by the generation unit, so that the instruction object is visible to the user; a display control unit that displays the virtual space in which at least the instruction object is placed, in correspondence with the real space; a detection unit that detects a movement of at least a part of the user's body from the image acquired by the acquisition unit; and an evaluation unit that evaluates the movement detected by the detection unit based on a timing and position based on the instruction object placed in the virtual space. [Brief description of the drawings]

[0011] [Figure 1] 3 is a diagram showing an overview of game processing by the game device according to the first embodiment. [Diagram 2] FIG. 4 is a diagram showing the definition of spatial coordinates in a virtual space according to the first embodiment. [Diagram 3] 1 is a block diagram showing an example of a hardware configuration of a game device according to a first embodiment. [Figure 4] FIG. 1 is a block diagram showing an example of the functional configuration of a game device according to a first embodiment. [Diagram 5] 6 is a flowchart showing an example of a designation object arrangement process according to the first embodiment. [Figure 6] 10 is a flowchart showing an example of a designated object display process according to the first embodiment. [Figure 7] 11 is a flowchart showing an example of a play evaluation process according to the first embodiment. [Figure 8] FIG. 11 is a diagram showing an overview of game processing by a game device according to a second embodiment. [Figure 9] 13 is a diagram showing the definition of spatial coordinates in a virtual space and the position of a user image according to the second embodiment. [Figure 10] FIG. 11 is a block diagram showing an example of the functional configuration of a game device according to a second embodiment. [Figure 11] 10 is a flowchart showing an example of a designation object arrangement process according to the second embodiment. [Figure 12] FIG. 13 is a block diagram showing an example of a hardware configuration of a game system according to a third embodiment. [Figure 13] FIG. 11 is a block diagram showing an example of the functional configuration of a game device according to a third embodiment. [Figure 14] FIG. 13 is a diagram showing an overview of game processing by a game device according to a fourth embodiment. [Figure 15] FIG. 13 is a block diagram showing an example of a hardware configuration of a game device according to a fourth embodiment. [Figure 16] FIG. 13 is a block diagram showing an example of the functional configuration of a game device according to a fourth embodiment. [Figure 17] 13 is a flowchart showing an example of a designation object arrangement process according to the fourth embodiment. [Figure 18] 13 is a flowchart showing an example of a designated object display process according to the fourth embodiment. [Figure 19] 13 is a flowchart showing an example of a play evaluation process according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [First embodiment] First, a first embodiment of the present invention will be described.

[0013] [Overview of the game device] First, an overview of an example of the game processing executed by the game device according to the present embodiment will be described. The game device according to the present embodiment can be a typical home game machine, but may also be used in amusement facilities such as game arcades.

[0014] FIG. 1 is a diagram showing an overview of game processing by a game device according to this embodiment. This diagram shows an overview of a play situation in which a user U plays a dance game (an example of a music game) using a game device 10. The game device 10 includes a video output device. The video output device may display video on a display, or may project video. For example, the game device 10 is configured as a head mounted display (HMD) that outputs video visible to the user and enables the user to view real space by wearing the game device on the user's head.

[0015] In the illustrated example of a dance game, the user U moves at least a part of his / her body in response to the timing and position of an instruction object displayed on the HMD in time with the music. The instruction object is an object displayed to guide the user U on the timing and position at which the user should move in real space. In this embodiment, the instruction object arranged in the virtual space is displayed on the HMD in association with the real space, allowing the user to play intuitively.

[0016] For example, the game device 10 is configured as an HMD that allows the real space to be optically viewed (so-called optically transparent (optically see-through) type HMD). The game device 10 displays a pointing object arranged in a virtual space on a transparent display located in front of the user's eyes while being worn on the user's head. This allows the user to view an image in which the pointing object displayed on the display is superimposed on the real space that is visible through the display.

[0017] The game device 10 may be configured as a retinal projection type optically transmissive HMD. In the case of the retinal projection type, the game device 10 includes an image projection device that projects an image directly onto the user's retina instead of a display. A pointing object arranged in the user's virtual space is displayed so as to be visible by being directly projected onto the user's retina.

[0018] The game device 10 may be configured as an HMD that displays an image captured in real space in real time (a so-called video see-through type HMD). In this case, the game device 10 displays a real-time image of the real space on a display located in front of the user's eyes while being worn on the user's head, and also displays a pointing object located in a virtual space superimposed on the real-time image.

[0019] The game device 10 is worn on the head of the user U, and generates a virtual space from an image captured in the line of sight of the user U in real space. For example, the virtual space is defined as a three-dimensional coordinate space of XYZ with mutually orthogonal X-axis and Y-axis parallel to the floor surface (plane) and a vertical Z-axis orthogonal to the floor surface (plane). The generated virtual space includes positions corresponding to at least some of the objects in the real space (e.g., the user U, the floor, a wall, etc.). In the following description, the direction of the Z-axis toward the ceiling is also referred to as the upward direction, and the direction toward the floor surface is also referred to as the downward direction.

[0020] The game device 10 sets the position of the user U in this virtual space as a reference position and places an instruction object that instructs the user's action at a position based on the reference position (for example, a predetermined position around the reference position). For example, the instruction object includes a judgment object and a moving object. The judgment object is an instruction object that is placed at a judgment position that is a judgment criterion when evaluating the user's action. For example, the judgment object is placed at a position (height) corresponding to the floor surface in the Z coordinate in the virtual space, and around the reference position (the position of the user U) in the XY coordinate (for example, a range that the user U can reach by taking a step). In the illustrated example, a judgment object HF is placed in front of the reference position (the position of the user U), a judgment object HB is placed behind, a judgment object HR is placed to the right, and a judgment object HL is placed to the left. Here, the reference position (the position of the user U) and the front, rear, right, and left of the reference position are directions initialized at the start of playing this dance game, and are fixed even if the orientation of the user U changes during play.

[0021] The moving object appears from the ceiling side in the Z coordinate in the virtual space, and gradually moves downward toward a determination object (determination position) that is placed at a position (height) corresponding to the floor surface. The appearance position may be set in advance based on, for example, the position of the user U's head (the position of the game device 10), or may change according to a predetermined rule. The moving object NF is a moving object that moves toward a determination object HF (determination position of the moving object NF). The moving object NB is a moving object that moves toward a determination object HB (determination position of the moving object NB). The moving object NR is a moving object that moves toward a determination object HR (determination position of the moving object NR). The moving object NL is a moving object that moves toward a determination object HL (determination position of the moving object NL).

[0022] The timing and position at which each moving object gradually moves and reaches each judgment object (judgment position) are the timing and position at which the user U should operate. For example, the user is required to step on the judgment object HF with his / her foot when the moving object NF reaches the judgment object HF. The user's action is evaluated based on the timing and position at which the moving object reaches the judgment object, and the score is updated according to the evaluation. For example, if it is determined that the timing and position at which the moving object reaches the judgment object and the timing and position of the user's action match, a score is added, and if it is determined that they do not match, a score is not added. For example, whether or not the timing and position match is determined by whether or not the user steps on at least a part of the judgment area (e.g., the area of ​​the judgment object HR) corresponding to the position at which the moving object reaches the judgment object within a predetermined time (e.g., within 0.5 seconds before and after the arrival time) corresponding to the timing at which the moving object reaches the judgment object. The score to be added may change depending on the degree of match between the timing and position at which the moving object reaches the judgment object and the timing and position of the user's action.

[0023] FIG. 1 shows the correspondence between the real space including the user U and the virtual space including the instruction objects in one diagram, and is different from the play screen visible to the user U while playing. Each instruction object does not exist in the real space but exists only in the virtual space and is visible through the game device 10. The instruction objects visible to the user U while actually playing are present within a field of view (Fov) visible through the display part of the game device 10. The instruction objects included in this field of view are displayed on the game device 10 (HMD) and superimposed on the real space, making them visible to the user U. The game device 10 also displays display information related to the game other than the instruction objects (such as the score and information about the music being played).

[0024] FIG. 2 is a diagram showing the definition of spatial coordinates of the virtual space according to this embodiment. As described above, in this embodiment, the vertical axis is the Z axis, and the axes perpendicular to each other on a horizontal plane perpendicular to the Z axis are the X axis and the Y axis. In addition, at the initialization at the start of playing this dance game, a reference position K1 (an example of a first reference position based on the position of the game device 10) corresponding to the position of the user U is defined as the coordinate origin, and the X axis is defined as the axis of the line of sight of the user U. During play, the reference position K1 (coordinate origin), the X axis, the Y axis, and the Z axis are fixed. A change in the rotation direction about the Z axis is also called a change in the yaw direction (left-right direction), a change in the rotation direction about the Y axis is also called a change in the pitch direction (up-down direction), and a change in the rotation direction about the X axis is also called a change in the roll direction.

[0025] When the orientation of the head of the user U wearing the game device 10 changes, the game device 10 detects this as a change in the rotation direction of each axis (yaw direction, pitch direction, roll direction) using a built-in acceleration sensor or the like. The game device 10 changes the field of view (Fov) shown in FIG. 1 based on the detected change in the rotation direction of each axis, and changes the display of the instruction object included in the virtual space. In this way, even if the orientation of the head of the user U changes, the game device 10 can display the instruction object included in the virtual space on the display in accordance with the change in the field of view. Note that changes in the yaw direction are sometimes called changes in the left-right direction, and changes in the pitch direction are sometimes called changes in the up-down direction.

[0026] The reference position K1 shown in the figure is merely an example and is not limited to this position. Also, although the reference position K1 is defined as the coordinate origin of the spatial coordinates, the coordinate origin may be defined at another position.

[0027] [Hardware Configuration of Game Device 10] Next, an overview of the hardware configuration of the game device 10 according to this embodiment will be described. 3 is a block diagram showing an example of a hardware configuration of a game device 10 according to this embodiment. The game device 10 is an optically transmissive HMD and includes an imaging unit 11, a display unit 12, a sensor 13, a storage unit 14, a CPU (Central Processing Unit) 15, a communication unit 16, and a sound output unit 17.

[0028] The imaging unit 11 is a camera that captures an image in the line of sight of a user U who wears the game device 10 (HMD) on his / her head. That is, the imaging unit 11 is provided in the game device 10 (HMD) such that the optical axis corresponds to the line of sight when the game device 10 is worn on the head. The imaging unit 11 may be a monocular camera or a dual camera. The imaging unit 11 outputs the captured image.

[0029] The display unit 12 is, for example, a transmissive display in an optically transmissive HMD. For example, the display unit 12 displays at least a pointing object. The display unit 12 may be configured to include two displays for the right and left eyes, or may be configured to include one display that can be viewed by both eyes without distinguishing between the right and left eyes. In addition, when the game device 10 is a retinal projection type optically transmissive HMD, the display unit 12 is an image projection device that projects an image directly onto the user's retina.

[0030] When the game device 10 is a video see-through HMD, the display unit 12 is an opaque display that does not allow the real space to be optically recognized.

[0031] The sensor 13 is a sensor that outputs a detection signal related to the direction of the game device 10. For example, the sensor 13 is a gyro sensor that detects the angle, angular velocity, angular acceleration, etc. of an object. The sensor 13 may be a sensor that detects a change in direction, or a sensor that detects the direction itself. For example, the sensor 13 may include an acceleration sensor, a tilt sensor, a geomagnetic sensor, etc. instead of or in addition to a gyro sensor.

[0032] The memory unit 14 includes, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory), a ROM (Read-Only Memory), a Flash ROM, a RAM (Random Access Memory), etc., and stores the program and data for the dance game, data on the generated virtual space, etc.

[0033] The CPU 15 functions as a control center that controls each unit of the game device 10. For example, the CPU 15 executes a game program stored in the storage unit 14 to execute game processing, and executes processes such as generating a virtual space corresponding to a real space from captured video, arranging a designated object in the generated virtual space, and detecting a user's action and evaluating it based on the timing and position of the designated object, as described with reference to FIG.

[0034] The communication unit 16 includes a communication device that performs wireless communication such as Bluetooth (registered trademark) or Wi-Fi (registered trademark). The communication unit 16 may include a digital input / output port such as a Universal Serial Bus (USB), a video input / output port, and the like.

[0035] The sound output unit 17 outputs the playing sound of the music played in the dance game, the sound effects of the game, etc. For example, the sound output unit 17 may be configured to include a speaker, earphones, headphones, or a terminal connectable thereto. Note that the sound output unit 17 may output various sounds to an external speaker, earphones, headphones, etc. via wireless communication such as Bluetooth (registered trademark).

[0036] The hardware components of the game device 10 described above are connected to each other via a bus so as to be able to communicate with each other.

[0037] [Functional configuration of game device 10] Next, the functional configuration of the game device 10 will be described with reference to FIG. Fig. 4 is a block diagram showing an example of a functional configuration of the game device 10 according to this embodiment. The illustrated game device 10 includes a control unit 150 as a functional configuration realized by the CPU 15 executing a program stored in the storage unit 14. The control unit 150 executes the process of the dance game described with reference to Figs. 1 and 2. For example, the control unit 150 includes an image acquisition unit 151, a virtual space generation unit 152, an object placement unit 154, a gaze direction detection unit 155, a display control unit 156, a motion detection unit 157, and an evaluation unit 158.

[0038] The video acquisition unit 151 (an example of an acquisition unit) acquires captured video of the real space captured by the imaging unit 11. For example, before starting to play a dance game, the game device 10 instructs the user U to look in a predetermined direction (for example, an instruction to look around up, down, left and right). The game device 10 causes this instruction to be displayed, for example, on the display unit 12. As a result, the video acquisition unit 151 acquires captured video of the surroundings of the user U in the real space captured by the imaging unit 11.

[0039] The virtual space generating unit 152 (an example of a generating unit) generates a virtual space corresponding to the real space from the captured image acquired by the image acquiring unit 151. For example, the virtual space generating unit 152 detects the position of an object (such as a floor or a wall) existing in the real space from the captured image, and generates data of a three-dimensional coordinate space including at least part of the position information of the detected object (such as a floor or a wall) as data of the virtual space. As an example, a reference position K1 (see FIG. 2) corresponding to the user U based on the position of the game device 10 itself mounted on the head of the user U is defined as the coordinate origin of the virtual space (three-dimensional coordinate space). The virtual space generating unit 152 generates virtual space data including position information corresponding to an object (such as a floor or a wall) existing in the real space in the virtual space (three-dimensional coordinate space) with the reference position K1 corresponding to the user U as the coordinate origin. The virtual space generating unit 152 stores the generated virtual space data in the storage unit 14.

[0040] Here, any known technology can be applied to a detection method for detecting the position of an object (floor, wall, etc.) existing in real space from a captured image. For example, when the imaging unit 11 is a dual camera (stereo camera), the position of the object (floor, wall, etc.) may be detected by analyzing the captured image using the parallax between the left and right cameras. When the imaging unit 11 is a monocular camera, detection using parallax is possible in the same way as with a dual camera by using captured images captured from two locations with the monocular camera shifted by a specified distance. Also, instead of or in addition to this image analysis, the position of an object (floor, wall, etc.) existing in real space may be detected using laser light, sound waves, etc.

[0041] The object placement unit 154 (an example of a placement unit) places an instruction object that instructs the user U to perform an action in a position based on a reference position K1 corresponding to the user U in the virtual space so that the user U can see it. Specifically, the object placement unit 154 places a judgment object (see judgment objects HF, HB, HR, and HL in FIG. 1) at a judgment position in the virtual space corresponding to the position of the floor. In addition, the object placement unit 154 places a moving object (see moving objects NF, NB, NR, and NL in FIG. 1) at an appearance position in the virtual space at a timing set in advance in accordance with the music, and moves the moving object (changes the placement position) toward the judgment object. When placing the instruction object (judgment object and moving object), the object placement unit 154 updates the virtual space data stored in the storage unit 14 based on coordinate information of the placement position in the virtual space.

[0042] The gaze direction detection unit 155 detects the orientation of the game device 10, i.e., the gaze direction of the user U, based on the detection signal output from the sensor 13. The gaze direction detection unit 155 may detect the orientation of the game device 10, i.e., the gaze direction of the user U, by analyzing the captured image of the real space captured by the imaging unit 11. For example, the gaze direction detection unit 155 may detect the position and inclination of an object or an edge of an object by analyzing the captured image, and detect the orientation of the game device 10, i.e., the gaze direction of the user U, based on the detection result. In addition, the gaze direction detection unit 155 may detect the difference in the position and inclination of an object or an edge of an object between each frame by detecting the position and inclination of an object or an edge of an object in each frame of the captured image, and detect the change in the orientation of the game device 10, i.e., the gaze direction of the user U, based on the detection result. The gaze direction detection unit 155 may detect the orientation of the game device 10, i.e., the gaze direction of the user U, based on both the detection signal output from the sensor 13 and the analysis of the captured image of the real space.

[0043] The display control unit 156 refers to the virtual space data stored in the storage unit 14, and causes the display unit 12 to display the virtual space in which at least the instruction object is arranged in association with the real space. Here, associating the virtual space with the real space includes associating the coordinates of the virtual space generated based on the real space with the coordinates of the real space. When displaying the virtual space, the display control unit 156 determines the viewpoint position and line of sight direction in the virtual space based on the position and orientation of the game device 10 (HMD) in the real space, that is, the position and direction of the user U. For example, the display control unit 156 causes the display unit 12 to display the instruction object arranged in a range of the virtual space corresponding to the range of the field of view (Fov) (range of the real space) determined by the line of sight direction of the user U detected by the line of sight direction detection unit 155 (see FIG. 1).

[0044] The motion detection unit 157 (an example of a detection unit) detects motion of at least a part of the body of the user U from the captured video. For example, the motion detection unit 157 detects the motion of the feet of the user U playing a dance game. Any known technology can be applied as a recognition technology for recognizing at least a part of the body of the user U (i.e., the recognition target) from the captured video. For example, the motion detection unit 157 recognizes the image area of ​​the recognition target from the captured video using feature information of the recognition target (e.g., feature information of the feet). The motion detection unit 157 detects the motion of the recognition target (e.g., the motion of the feet) by extracting and tracking the image area of ​​the recognition target from each frame of the captured video.

[0045] The evaluation unit 158 ​​evaluates the motion of at least a part of the body of the user U detected by the motion detection unit 157, based on the timing and position based on the instruction object arranged in the virtual space. For example, the evaluation unit 158 ​​compares the timing and position at which the moving object reaches the judgment object with the timing and position of the foot motion of the user U (motion of stepping on the judgment object), and evaluates the play based on the motion of the user U. If the evaluation unit 158 ​​can determine that the timing and position of both match based on the comparison result, it adds a score, and if it can determine that they do not match, it does not add a score.

[0046] The evaluation unit 158 ​​may evaluate the play based on the actions of the user U by comparing the positions of the user U's feet at the time when the moving object reaches the judgment object with the position of the judgment object.

[0047] [Operation of Pointer Object Placement Processing] Next, a description will be given of the operation of a designated object placing process for generating a virtual space and placing a designated object in the dance game process executed by the CPU 15 of the game device 10. Fig. 5 is a flowchart showing an example of the designated object placing process according to this embodiment.

[0048] First, the CPU 15 acquires a captured image of the real space captured by the imaging unit 11 (step S101). For example, before starting to play a dance game, the CPU 15 causes the display unit 12 to display an instruction for the user U to look in a predetermined direction (e.g., an instruction to look up, down, left and right), and acquires a captured image of the surroundings of the user U in the real space.

[0049] Next, the CPU 15 generates a virtual space corresponding to the real space from the captured image acquired in step S101 (step S103). For example, the CPU 15 detects the positions of objects (floors, walls, etc.) existing in the real space from the captured image. The CPU 15 generates virtual space data of a three-dimensional coordinate space including position information of at least a part of the detected objects (floors, walls, etc.) in a virtual space (three-dimensional coordinate space) having a reference position K1 corresponding to the user U as the coordinate origin. Then, the CPU 15 stores the generated virtual space data in the storage unit 14.

[0050] Next, at or before the start of play of the dance game, the CPU 15 places a judgment object (see judgment objects HF, HB, HR, and HL in FIG. 1) at a judgment position based on a reference position K1 in the virtual space corresponding to the position of the floor (step S105). When placing the judgment object, the CPU 15 adds position information of the placed judgment object to the virtual space data stored in the storage unit 14.

[0051] When the dance game is started, the CPU 15 judges whether or not there is an appearance trigger for a moving object (step S107). The appearance trigger occurs at a timing set in advance in accordance with the music. When the CPU 15 judges in step S107 that there is an appearance trigger (YES), the CPU 15 proceeds to the process of step S109.

[0052] In step S109, the CPU 15 places a moving object (one or more of the moving objects NF, NB, NR, and NL in FIG. 1) at an appearance position based on the reference position K1 in the virtual space, and starts moving the moving object toward a judgment position (the position of a judgment object corresponding to each moving object). When placing the moving object, the CPU 15 adds position information of the placed moving object to the virtual space data stored in the storage unit 14. Also, when moving the placed moving object, the CPU 15 updates the position information of the moving object added to the virtual space data stored in the storage unit 14. Then, the process proceeds to step S111. On the other hand, when the CPU 15 determines in step S107 that there is no appearance trigger (NO), the process proceeds to step S111 without performing the process of step S109.

[0053] In step S111, the CPU 15 judges whether or not the moving object has reached the judgment position. The CPU 15 erases the moving object that has been judged to have reached the judgment position (YES) in step S111 from the virtual space (step S113). When erasing the moving object from the virtual space, the CPU 15 deletes the position information of the moving object to be erased from the virtual space data stored in the storage unit 14.

[0054] On the other hand, the CPU 15 continues to gradually move the moving object determined in step S111 that the moving object has not reached the determination position (NO) toward the determination position (step S115). When moving the placed moving object, the CPU 15 updates the position information of the moving object to be moved among the virtual space data stored in the storage unit 14.

[0055] Next, the CPU 15 determines whether or not the dance game has ended (step S117). For example, the CPU 15 determines that the dance game has ended when the music being played has ended. When the CPU 15 determines that the dance game has not ended (NO), the CPU 15 returns to the process of step S107. On the other hand, when the CPU 15 determines that the dance game has ended (YES), the CPU 15 ends the instruction object placement process.

[0056] In addition, the order of placing the judgment object and the first appearing moving object may be simultaneous, the judgment object may be placed first, or conversely, the judgment object may be placed later (until the first appearing moving object reaches the judgment position).

[0057] [Operation of the Designated Object Display Processing] Next, a description will be given of the operation of an instruction object display process for displaying an instruction object arranged in the virtual space in the dance game process executed by the CPU 15 of the game device 10. Fig. 6 is a flowchart showing an example of the instruction object display process according to this embodiment.

[0058] The CPU 15 detects the line of sight of the user U (the direction of the game device 10) based on the detection signal output from the sensor 13 (step S201).

[0059] The CPU 15 refers to the virtual space data stored in the storage unit 14 and causes the display unit 12 to display a virtual space corresponding to the range of the field of view (Fov) based on the line of sight direction detected in step S201 (range of real space). For example, the CPU 15 causes the display unit 12 to display instruction objects (determination object and moving object) arranged in the range of the virtual space corresponding to the range of the field of view (Fov) based on the line of sight direction (step S203). As a result, the moving objects are displayed on the display unit 12 at a timing set in advance in accordance with the music.

[0060] Next, the CPU 15 determines whether or not the dance game has ended (step S205). For example, the CPU 15 determines that the dance game has ended when the music being played has ended. When the CPU 15 determines that the dance game has not ended (NO), the CPU 15 returns to the process of step S201. On the other hand, when the CPU 15 determines that the dance game has ended (YES), the CPU 15 ends the instruction object display process.

[0061] [Play evaluation process] Next, a description will be given of the operation of a play evaluation process for evaluating a play based on a movement of at least a part of the body of the user U in the processing of the dance game executed by the CPU 15 of the game device 10. Fig. 7 is a flowchart showing an example of the play evaluation process according to this embodiment.

[0062] The CPU 15 acquires an image of the real space captured by the imaging unit 11 (step S301). Next, the CPU 15 detects a motion of at least a part of the body of the user U from the image acquired in step S301 (step S303). For example, the CPU 15 detects the motion of the feet of the user U playing a dance game.

[0063] Then, CPU 15 evaluates the motion of at least a part of the body of user U (e.g., a foot) detected in step S303 based on the timing and position based on the instruction object arranged in the virtual space (step S305). For example, CPU 15 compares the timing and position at which the moving object reaches the determination object with the timing and position of the foot motion of user U (motion of stepping on the determination object), and evaluates the play based on the foot motion of user U.

[0064] Furthermore, CPU 15 updates the game score based on the evaluation result in step S305 (step S307). For example, CPU 15 adds a score when it can determine that the timing and position of the moving object reaching the judgment object coincides with the timing and position of the foot action (action of stepping on the judgment object) of user U, and does not add a score when it can determine that they do not coincide.

[0065] Next, the CPU 15 determines whether the dance game has ended (step S309). For example, the CPU 15 determines that the dance game has ended when the music being played has ended. When the CPU 15 determines that the dance game has not ended (NO), the CPU 15 returns to the process of step S301. On the other hand, when the CPU 15 determines that the dance game has ended (YES), the CPU 15 ends the play evaluation process.

[0066] [Summary of the first embodiment] As described above, the game device 10 according to the present embodiment is worn on the head of the user U, and outputs an image visibly recognizable to the user U, and executes processing of a game that can be played using the game device 10 (an example of a video output device) that can visually recognize the real space. For example, the game device 10 acquires an image of the real space, and generates a virtual space corresponding to the real space from the acquired image. Then, the game device 10 places an instruction object that instructs the user U to move at a position based on a reference position K1 corresponding to the user U in the virtual space so that the user can see it, and displays the virtual space in which at least the instruction object is placed in association with the real space. In addition, the game device 10 detects a movement of at least a part of the body of the user U from the acquired image, and evaluates the detected movement based on the timing and position based on the instruction object placed in the virtual space.

[0067] As a result, in game processing in which the user U's actions are evaluated based on the timing and position of an instruction object that instructs the user U's actions, the game device 10 is worn on the head, and the instruction object is associated with real space and made visible to the user U, so that the game device 10 can guide the user on the actions he or she should take, allowing for more intuitive play with a simple configuration.

[0068] For example, the reference position K1 is a first reference position in the virtual space corresponding to the position of the user U wearing the game device 10 (an example of a video output device) and is based on the position of the see-through HMD in the virtual space. For example, the reference position K1 is a position in the virtual space corresponding to the position of the user U in the real space (the position of the see-through HMD) and is defined as the coordinate origin of the virtual space (three-dimensional coordinate space).

[0069] This allows the game device 10 to display instruction objects in correspondence with real space based on the position of the user U playing the game, thereby making the instructions to the user U feel more real and enabling more intuitive gameplay.

[0070] Furthermore, the game device 10 moves an instruction object (e.g., a moving object) placed at a predetermined position (appearance position) in the virtual space toward a predetermined judgment position (e.g., the position of a judgment object). Then, the game device 10 evaluates the movement of at least a part of the body (e.g., a foot) of the user U detected from the captured video based on the timing at which the instruction object (e.g., the moving object) moving in the virtual space reaches the judgment position and the judgment position.

[0071] This allows the game device 10 to evaluate, using the captured video, whether or not the user U was able to perform the action as instructed.

[0072] In addition, since the user U can only view instruction objects within a range of the field of view based on the user's line of sight in the game device 10, he / she cannot simultaneously view instruction objects in the front, back, left, and right (360° around the user U). Therefore, the game device 10 may limit the position where the instruction object is placed to a part of the virtual space according to the orientation of the user U wearing the game device 10 (an example of a video output device). For example, the game device 10 may place only the front, right, and left instruction objects and not place any instruction object behind based on the orientation of the user U (reference position K1) at the time of initialization.

[0073] As a result, the game device 10 does not issue instructions to perform actions outside the range of the user U's field of view (for example, behind), so the user U can play without worrying about the range outside the range of the field of view (for example, behind) during play. Therefore, the game device 10 can prevent the difficulty level of play from becoming too high.

[0074] Furthermore, when the game device 10 limits the position where the instruction object is placed to a part of the virtual space according to the orientation of the user U, the game device 10 may change the direction of the restriction during play according to the orientation of the user U. For example, when the user U faces forward, the game device 10 may place only instruction objects in front, right, and left with respect to the user U (reference position K1), and may not place an instruction object behind. Furthermore, when the user U faces right, the game device 10 may place only instruction objects in front, right, and left (right, front, and back before facing right) with respect to the user U (reference position K1) after facing right, and may not place an instruction object behind (left before facing right). Similarly, when the user U faces left or backward, the game device 10 may not place an instruction object in the opposite direction (right or front before changing the orientation).

[0075] As a result, the game device 10 follows the change in the orientation of the user U and does not always issue instructions to perform actions outside the range of the user U's field of vision, thereby making it possible to reduce the difficulty of playing the game.

[0076] [Second embodiment] Next, a second embodiment of the present invention will be described. In the example described in the first embodiment, the instruction objects that the user U can actually see are limited to the instruction objects that are arranged in the virtual space and are arranged within the range of the field of view based on the line of sight of the user U. Therefore, for example, when an instruction object exists behind the user U, it may be difficult to recognize it. This difficult element can be used as a game feature, but on the other hand, there is a concern that it may be difficult for beginners to play. In the first embodiment, it was described that the difficulty may be suppressed by limiting the position where the instruction object is arranged to a part in the virtual space depending on the direction of the user U, but in this case, the number of types of actions to be instructed to the user U during play will be reduced. In addition, when the user U actually plays, it is necessary to play while looking at the feet and instruction object of the user U that are present below the user U, so there is a concern that it will have a negative effect on the body movements of the user U and make it difficult to dance. Therefore, in this embodiment, the above concern is solved by using a mirror.

[0077] 8 is a diagram showing an overview of game processing by the game device according to this embodiment. This diagram shows an overview of a play situation in which a user U plays a dance game using the game device 10A according to this embodiment. Like FIG. 1, this diagram shows the correspondence between the real space in which the user U is included and the virtual space in which the instruction object is included, all in one diagram, and is different from the play screen that the user U can see while playing.

[0078] In the illustrated example, the user U is playing a dance game in a position facing a mirror MR. As in FIG. 1, instruction objects (determination objects and moving objects) are arranged around the user U in the virtual space. Furthermore, the user U is reflected in the mirror MR opposite the user U. Here, the virtual image of the user U reflected in the mirror MR is referred to as a "user image UK." The game device 10A detects a user image UK corresponding to the user U from an image captured in the direction of the mirror MR, and also places instruction objects around the detected user image UK as if the instruction objects arranged around the user U were reflected in the mirror MR.

[0079] FIG. 9 is a diagram showing the definition of spatial coordinates of the virtual space according to this embodiment and the position of the user image UK. This diagram is a diagram in which the position of the user image UK detected from the captured image is added to the definition of spatial coordinates of the virtual space shown in FIG. 2. A reference position K2 (an example of a second reference position) corresponding to the position of the user image UK in the virtual space is detected at a position at the end (rear) of the mirror MR in the X-axis direction (line of sight direction) with respect to a reference position K1 (for example, the coordinate origin) corresponding to the position of the user U. For example, if the position where the mirror surface of the mirror MR intersects with the X-axis is the mirror surface position M1, the reference position K2 is detected at a position in the X-axis direction where the distance from the reference position K1 to the mirror surface position M1 is the same as the distance from the mirror surface position M1 to the reference position K2. The reference position K2 may be a position corresponding to the center of the head of the user image UK or a position corresponding to the center of gravity of the user image UK, and can be defined at any position.

[0080] Returning to FIG. 8, the game device 10A detects the image area (contour) and distance of the user image UK from the captured image, and detects a reference position K2 corresponding to the position of the user image UK in the virtual space, separate from a reference position K1 corresponding to the position of the user U. The game device 10A then places instruction objects around the reference positions K1 and K2, respectively, based on the reference positions K1 and K2. At this time, since the user image UK is a virtual image of the user U reflected in the mirror MR, the front-to-back orientation is reversed with respect to the user U. Therefore, the game device 10A places the instruction objects to be placed around the reference position K2 (position of the user image UK) with the front-to-back orientation (front-to-back positional relationship in spatial coordinates) reversed from that of the instruction objects to be placed around the reference position K1 (position of the user U).

[0081] For example, if the direction on the X-axis from reference position K1 to reference position K2 is defined as the positive direction, then the determination object HF and moving object NF arranged forward with respect to reference position K1 (position of user U) are arranged in the positive direction of the X-axis with respect to reference position K1. In contrast, the determination object HF' and moving object NF' arranged forward with respect to reference position K2 (position of user image UK) are arranged in the negative direction of the X-axis with respect to reference position K2. Furthermore, the determination object HB and moving object NB arranged behind reference position K1 (position of user U) are arranged in the negative direction of the X-axis with respect to reference position K1. In contrast, the determination object HB' and moving object NB' arranged behind reference position K2 (position of user image UK) are arranged in the positive direction of the X-axis with respect to reference position K2.

[0082] On the other hand, the determination object HR and moving object NR arranged to the right of the reference position K1 (the position of the user U), and the determination object HR' and moving object NR' arranged to the right of the reference position K2 (the position of the user image UK) are arranged in the same direction (e.g., the positive direction) of the Y axis with respect to the respective reference positions. Furthermore, the determination object HL and moving object NL arranged to the left of the reference position K1 (the position of the user U), and the determination object HL' and moving object NL' arranged to the left of the reference position K2 (the position of the user image UK) are arranged in the same direction (e.g., the negative direction) of the Y axis with respect to the respective reference positions. Furthermore, the upward and downward positional relationships between the instruction object arranged with respect to the reference position K1 and the instruction object arranged with respect to the reference position K2 are also the same.

[0083] [Configuration of game device 10A] The game device 10A according to this embodiment may be a device including an optically transparent HMD, or may be a device including a video transparent HMD, similar to the game device 10 described in the first embodiment. Here, similar to the first embodiment, the game device 10A will be described as an optically transparent HMD. The hardware configuration of the game device 10A is the same as the configuration example shown in FIG. 3, and therefore the description thereof will be omitted.

[0084] FIG. 10 is a block diagram showing an example of a functional configuration of the game device 10A according to this embodiment. The illustrated game device 10A includes a control unit 150A as a functional configuration realized by the CPU 15 executing a program stored in the storage unit 14. The control unit 150A includes an image acquisition unit 151, a virtual space generation unit 152, a user image detection unit 153A, an object placement unit 154A, a gaze direction detection unit 155, a display control unit 156, a motion detection unit 157, and an evaluation unit 158. In this figure, the same reference numerals are used for the components corresponding to those in FIG. 4, and the description thereof will be omitted as appropriate. The functional configuration of the game device 10A is mainly different from the functional configuration of the game device 10 shown in FIG. 4 in that a user image detection unit 153A for detecting a reference position corresponding to a user image UK reflected in the mirror MR is added.

[0085] The user image detection unit 153A detects a user image UK (an example of an image) corresponding to the user U from the captured image acquired by the image acquisition unit 151. For example, the user image UK is detected as a virtual image of the user U reflected in a mirror MR located opposite the user U. This detection requires recognition that the user image UK is a virtual image of the user U playing a dance game. As a recognition method, for example, an identifiable marker (such as a mark or a label) may be attached to the body of the user U or the game device 10A (HMD) worn on the head of the user U, and the user image detection unit 153A may recognize that it is a virtual image of the user U by detecting this marker from the captured image. In addition, the user image detection unit 153A may recognize that it is a virtual image of the user U by instructing the user U to perform a specific action (for example, raising and lowering the right hand) from the captured image, and detecting a person performing an action according to the instruction.

[0086] The virtual space generation unit 152 generates data of a three-dimensional coordinate space including position information of the user image UK as virtual space data in addition to position information of at least a part of the object (floor, wall, etc.) detected from the captured image. For example, the virtual space generation unit 152 detects the position of an object (floor, wall, etc.) existing in the real space from the captured image. In addition, the virtual space generation unit 152 detects the position of the user image UK (reference position K2) detected by the user image detection unit 153A. The method of detecting the position of the user image UK may be a detection method using the parallax of the camera (imaging unit) similar to the above-mentioned method of detecting the position of an object (floor, wall, etc.) existing in the real space. Then, the virtual space generation unit 152 generates data of a three-dimensional coordinate space including position information of at least a part of the detected object (floor, wall, etc.) and position information of the reference position K2 as virtual space data. Note that, as an example, the coordinate origin of the virtual space (three-dimensional coordinate space) is set to the reference position K1 corresponding to the user U as in the first embodiment. The virtual space generating unit 152 stores the generated virtual space data in the storage unit 14 .

[0087] The object placement unit 154A places an instruction object in the virtual space at a position based on a reference position K1 corresponding to the user U, and also places an instruction object in a position based on a reference position K2 corresponding to the user image UK (see FIGS. 8 and 9). When placing the instruction object in the virtual space at a position based on the reference position K2, the object placement unit 154A also reverses the front-to-back orientation with respect to the reference position K2.

[0088] Here, the object placement unit 154A may determine whether the detected user image UK is an image reflected in the mirror MR by detecting a person performing the above-mentioned specific action (for example, raising and lowering the right hand) from the captured image, by instructing the person to perform the action. Also, the object placement unit 154A may determine that the detected user image UK is an image reflected in the mirror MR by selecting a preselectably set mirror mode (a mode in which the player plays while watching the player's image reflected in the mirror MR).

[0089] For example, when the line of sight of the user U is in the direction of the mirror MR, the display control unit 156 causes the display unit 12 to display an instruction object arranged in a range of the virtual space corresponding to the range of the field of view in the direction of the mirror MR. That is, the display control unit 156 can display an instruction object arranged in a position based on the reference position K2 corresponding to the user image UK reflected in the mirror MR so that the user U can view it from above.

[0090] The movement detection unit 157 detects a movement of at least a part of the body of the user U by detecting a movement of at least a part of the body of the user image UK reflected in the mirror MR from the captured image.

[0091] The evaluation unit 158 ​​evaluates the motion of at least a part of the body of the user image UK (user image UK reflected in the mirror MR) detected by the motion detection unit 157, using the instruction object arranged at a position based on the reference position K2 corresponding to the user image UK. Specifically, the evaluation unit 158 ​​evaluates the motion of at least a part of the body of the user image UK (user image UK reflected in the mirror MR) based on the timing and position based on the instruction object arranged at a position based on the user image UK reflected in the mirror MR. That is, the user U can play while looking in the direction of the mirror MR without looking at the feet of the user U and the instruction object present below.

[0092] In the example shown in FIG. 8, the instruction object is arranged at both the position based on the reference position K1 corresponding to the user U and the position based on the reference position K2 corresponding to the user image UK, but the present invention is not limited to this. For example, when the instruction object is arranged at the position based on the reference position K2 corresponding to the user image UK, the object arrangement unit 154A does not need to arrange the instruction object at the position based on the reference position K1 corresponding to the user U. In other words, when the instruction object is displayed at the position based on the reference position K2, the instruction object at the position based on the reference position K1 may be hidden. This prevents the instruction object displayed in the mirror MR from being hidden by the instruction objects displayed around the user U, and improves the visibility of the instruction object.

[0093] When the instruction object is placed at a position based on the reference position K2, the object placement unit 154A may display the instruction object at a position based on the reference position K1 in an inconspicuous manner with reduced visibility, for example, by making the object semi-transparent or reducing its size. Note that the process of changing the display mode of the instruction object may be performed by the display control unit 156.

[0094] Furthermore, the object arrangement unit 154A (or the display control unit 156) may make the instruction objects around the user U invisible or semi-transparent only when the mirror MR is within the field of view of the user U, and may display the instruction objects around the user U as usual when the mirror MR is out of the field of view. This allows the instruction objects to be visible even when the mirror MR is out of the field of view (for example, when the user U faces the opposite direction to the mirror MR).

[0095] [Operation of Pointer Object Placement Processing] Next, the operation of the instruction object arranging process for generating a virtual space and arranging an instruction object in the dance game process executed by the CPU 15 of the game device 10A will be described. Here, the instruction object arranging process corresponding to the user U (reference position K1) is the same as the process shown in Fig. 5, so its description will be omitted, and the instruction object arranging process for arranging an instruction object corresponding to the user image UK (reference position K2) reflected in the mirror MR will be described. Fig. 11 is a flowchart showing an example of the instruction object arranging process according to this embodiment.

[0096] First, the CPU 15 acquires a captured image of the real space captured by the imaging unit 11 (step S401). For example, the CPU 15 acquires a captured image including a user image UK (see FIG. 8) reflected in a mirror MR in the line of sight of a user U playing a dance game.

[0097] Next, the CPU 15 detects a virtual image (user image UK) of the user U playing the dance game from the captured video acquired in step S401 (step S403).

[0098] Further, the CPU 15 generates a virtual space corresponding to the real space from the captured image acquired in step S401 (step S405). For example, the CPU 15 detects the positions of objects (floors, walls, etc.) existing in the real space from the captured image and the position (reference position K2) of the user image UK detected in step S403, and generates data of a three-dimensional coordinate space including position information of at least a part of the detected objects (floors, walls, etc.) and position information of the reference position K2 as data of the virtual space. As an example, the CPU 15 generates virtual space data including position information of at least a part of the detected objects (floors, walls, etc.) and position information of the reference position K2 in a virtual space (three-dimensional coordinate space) with the reference position K1 corresponding to the user U as the coordinate origin. Then, the CPU 15 stores the generated virtual space data in the storage unit 14.

[0099] Next, at or before the start of play of the dance game, the CPU 15 places a judgment object (see judgment objects HF', HB', HR', and HL' in FIG. 8) at a judgment position based on a reference position K2 in the virtual space corresponding to the position of the floor (step S407). When placing the judgment object, the CPU 15 adds position information of the placed judgment object to the virtual space data stored in the storage unit 14.

[0100] When the dance game is started, the CPU 15 judges whether or not there is an appearance trigger for a moving object (step S409). The appearance trigger occurs at a timing set in advance in accordance with the music. When the CPU 15 judges in step S409 that there is an appearance trigger (YES), the CPU 15 proceeds to the process of step S411.

[0101] In step S411, the CPU 15 places a moving object (one or more of the moving objects NF', NB', NR', and NL' in FIG. 8) at an appearance position based on the reference position K2 in the virtual space, and starts moving the moving object toward a judgment position (the position of a judgment object corresponding to each moving object). When placing the moving object, the CPU 15 adds position information of the placed moving object to the virtual space data stored in the storage unit 14. Furthermore, when moving the placed moving object, the CPU 15 updates the position information of the moving object added to the virtual space data stored in the storage unit 14. Then, the process proceeds to step S413. On the other hand, if the CPU 15 determines in step S409 that there is no appearance trigger (NO), the process proceeds to step S413 without performing the process of step S411.

[0102] In step S413, the CPU 15 judges whether the moving object has reached the judgment position. The CPU 15 erases the moving object that has been judged to have reached the judgment position (YES) from the virtual space (step S415). When erasing the moving object from the virtual space, the CPU 15 deletes the position information of the moving object to be erased from the virtual space data stored in the storage unit 14.

[0103] On the other hand, if the CPU 15 determines that the moving object has not reached the determination position (NO), the CPU 15 continues to move the moving object gradually toward the determination position (step S417). When moving the moving object, the CPU 15 updates the position information of the moving object to be moved among the virtual space data stored in the storage unit 14.

[0104] Next, the CPU 15 determines whether or not the dance game has ended (step S419). For example, the CPU 15 determines that the dance game has ended when the music being played has ended. When the CPU 15 determines that the dance game has not ended (NO), the CPU 15 returns to the process of step S409. On the other hand, when the CPU 15 determines that the dance game has ended (YES), the CPU 15 ends the instruction object placement process.

[0105] In addition, the order of placing the judgment object and the first appearing moving object may be simultaneous, the judgment object may be placed first, or conversely, the judgment object may be placed later (until the first appearing moving object reaches the judgment position).

[0106] [Summary of the second embodiment] As described above, the game device 10A according to the present embodiment further detects a user image UK, which is a virtual image (one example of an image) corresponding to the user U, from an image captured of a real space. Then, the game device 10A places an instruction object, which instructs the user U to perform an action, in a position based on the reference position K2 of the user image UK corresponding to the user U in the virtual space so that the instruction object is visible to the user.

[0107] Thus, by wearing the game device 10A on the head, the instruction object instructing the user U to move can be displayed around the virtual image (user image UK) of the user U reflected in the mirror MR, for example, so that the user can be guided to move more intuitively with a simple configuration. For example, the game device 10A can simultaneously view the instruction objects displayed around the user image UK (for example, front, back, left and right) from above without limiting the position where the instruction object is placed to a part of the virtual space, so that the types of movements to be instructed to the user U during play can be diversified. In addition, the game device 10A can prevent the user U from having difficulty dancing because the user U can play while looking at the direction of the mirror MR facing the user U without having to look at his / her own feet and the instruction object present below. In addition, the game device 10A can display the instruction objects around the user U playing the game and around the virtual image (user image UK) of the user reflected in the mirror MR, for example, so that the user can play while arbitrarily selecting which of the instruction objects is easier to play.

[0108] The mirror MR may be something other than a mirror as long as it has the effect of a mirror (mirror reflection). For example, the user U may play the game in a bright room facing a window at night (when it is dark outside), and the window may be used as the mirror MR to utilize the virtual image of the user U reflected on the window.

[0109] Furthermore, when placing the instruction object at a position based on the reference position K2 in the virtual space (around the user image UK), the game device 10A inverts the front-to-back orientation with respect to the reference position K2. This allows the game device 10A to display the instruction object in accordance with the orientation of the user image UK reflected in the mirror MR, and therefore allows the user to be guided as to the actions to be taken so that the user can play the game more intuitively.

[0110] Note that, when placing an instruction object at a position based on the reference position K2 in the virtual space (around the user image UK), the game device 10A may reduce the visibility of the instruction object to be placed at a position based on the reference position K1 (around the user U). For example, the game device 10A may display the instruction object at a position based on the reference position K1 in an inconspicuous manner with reduced visibility, such as by making the object semi-transparent or shrinking it. Furthermore, when placing an instruction object at a position based on the reference position K2 in the virtual space (around the user image UK), the game device 10A may not need to place the instruction object at a position based on the reference position K1 (around the user U).

[0111] This allows the game device 10A to prevent the instruction object displayed in the mirror MR from being hidden by the instruction objects displayed around the user U, thereby improving the visibility of the instruction object.

[0112] In the present embodiment, the user U places the instruction object in the virtual space in association with the user image UK (his / her own virtual image) reflected in the mirror MR. However, instead of the mirror MR, the instruction object may be placed in the virtual space in association with the image of the user U displayed on a monitor (display device). For example, the game device 10A further includes a camera (imaging device) that captures the user U in the real space and a monitor (display device) that displays the captured image in real time on the side facing the user U, and displays the image of the user U captured by the camera on the monitor. Then, instead of the user image UK (his / her own virtual image) reflected in the mirror MR, the game device 10A may detect the image of the user U from the image displayed on the monitor, and place the instruction object in the virtual space in association with the detected image of the user U. In this case, the position of the image of the user U displayed on the monitor becomes the reference position. The image of the user U displayed on the monitor is oriented in the opposite direction to the user image UK of the user U reflected in the mirror MR. Therefore, the game device 10A inverts the direction of the instruction object arranged in association with the image of the user U displayed on the monitor, in addition to the direction of the front-back direction, with respect to the instruction object arranged in association with the user U, left-right direction.

[0113] In addition, the game mode in this embodiment in which the instruction object is arranged using the mirror MR, the game mode in which the instruction object is arranged using the monitor, and the game mode in which the instruction object is arranged without using either the mirror or the monitor (the game processing mode by the game device 10) described in the first embodiment in which the instruction object is arranged are different from each other in the display manner of the instruction object (such as the reference position when the instruction object is arranged, whether or not to invert the front and back or the left and right, etc.). Therefore, when there are two or more of these game modes (for example, when the configuration of the game device 10 and the configuration of the game device 10A are combined), the configuration may be such that the user can select which mode to use before starting the dance game. In this way, the user image UK reflected in the mirror MR and the image of the user U displayed on the monitor can be detected smoothly, and erroneous recognition can be reduced.

[0114] [Third embodiment] Next, a third embodiment of the present invention will be described. In the above first and second embodiments, an example was described in which the game device 10 (10A) is configured as a complete device serving as a see-through HMD, but it may also be configured as a separate device connected to the see-through HMD via a wired or wireless connection.

[0115] 12 is a block diagram showing an example of a hardware configuration of a game system including a game device 10C according to this embodiment. The game device 10C does not include a video output device. The illustrated game system 1C includes the game device 10C and an HMD 20C as a video output device. For example, the HMD 20C is a see-through HMD.

[0116] The HMD 20C includes an imaging unit 21C, a display unit 22C, a sensor 23C, a storage unit 24C, a CPU 25C, a communication unit 26C, and a sound output unit 27C. The imaging unit 21C, the display unit 22C, the sensor 23C, and the sound output unit 27C correspond to the imaging unit 11, the display unit 12, the sensor 13, and the sound output unit 17 shown in FIG. 3, respectively. The storage unit 24C temporarily stores data of the captured image captured by the imaging unit 21C and display data acquired from the game device 10C, etc. The storage unit 24C also stores programs and the like required for controlling the HMD 20C. The CPU 25C functions as a control center that controls each unit included in the HMD 20C. The communication unit 26C communicates with the game device 10C using wired or wireless communication. The HMD 20C transmits the captured image captured by the imaging unit 21C, the detection signal of the sensor 23C, and the like to the game device 10C via the communication unit 26C. In addition, the HMD 20C acquires display data, sound data, and the like of the dance game from the game device 10C via the communication unit 26C.

[0117] The game device 10C includes a storage unit 14C, a CPU 15C, and a communication unit 16C. The storage unit 14C stores a program and data of a dance game, data of a generated virtual space, and the like. The CPU 15C functions as a control center that controls each unit of the game device 10C. For example, the CPU 15C executes a game program stored in the storage unit 14C to execute game processing, and executes processing of generating a virtual space corresponding to a real space from an imaged video, processing of placing an instruction object in the generated virtual space, processing of detecting a user's motion and evaluating the motion based on the timing and position of the instruction object, and the like. The communication unit 16C communicates with the HMD 20C using wired or wireless communication. The game device 10C acquires an imaged video captured by the imaging unit 21C of the HMD 20C and a detection signal of the sensor 23C, and the like, via the communication unit 16C. The game device 10C also transmits display data and sound data of the dance game to the HMD 20C via the communication unit 16C.

[0118] Fig. 13 is a block diagram showing an example of a functional configuration of a game device 10C according to this embodiment. The illustrated game device 10C includes a control unit 150C as a functional configuration realized by a CPU 15C executing a program stored in a storage unit 14C. The control unit 150C has the same configuration as the control unit 150 shown in Fig. 4 or the control unit 150A shown in Fig. 10, except that the control unit 150C exchanges data with each unit (such as an imaging unit 21C, a display unit 22C, a sensor 23C, and a sound output unit 27) included in the HMD 20C via a communication unit 16C.

[0119] In this way, the game device 10C may be configured as a separate device that communicates with the HMD 20 as an external device. Note that the game device 10C may be, for example, a smartphone, a PC (Personal Computer), a home game console, or the like.

[0120] [Fourth embodiment] Next, a fourth embodiment of the present invention will be described. In the above first to third embodiments, the aspects using an HMD worn on the head have been described, but in this embodiment, an aspect not using an HMD will be described.

[0121] FIG. 14 is a diagram showing an overview of game processing by the game device according to this embodiment. This diagram shows an overview of a play situation in which a user U plays a dance game using a game device 10D. The illustrated game device 10D is an example in which a smartphone is applied as an example. In this embodiment, a command object arranged in a virtual space is displayed on the display unit 12D or monitor 30D of the game device 10D in association with an image of the user U captured by the front camera 11DA of the game device 10D, thereby enabling the user to play intuitively. The monitor 30D is an external display unit (display device) that can be connected to the game device 10D by wire or wirelessly. For example, the monitor 30D is one having a display screen larger than the display unit 12D provided in the game device 10D.

[0122] The game device 10D recognizes the image area of ​​the user U from the captured image of the user U. The game device 10D then defines a reference position K3 corresponding to the position of the user U in the virtual space, generates virtual space (XYZ three-dimensional space) data in which an instruction object is placed at a position based on the reference position K3, and displays the virtual space data superimposed on the captured image. Note that the reference position K3 may be a position corresponding to the center of the user U's head or a position corresponding to the center of gravity of the user U, and can be defined at any position.

[0123] In the illustrated example, the instruction object is also displayed on the display unit 12D of the game device 10D, but by also displaying it on the monitor 30D, which has a larger screen than the game device 10D, the visibility of the instruction object for the user U is increased. A description will be given with reference to the display screen of the monitor 30D. The user image UV shows an image of the user U included in the captured image. Around the user image UV, an image is displayed in which the instruction object, which is placed at a position based on the reference position K3 of the user U in the virtual space, is superimposed on the captured image.

[0124] For example, the image captured by the front camera 11DA can be an image that is inverted left and right like a mirror. A determination object HR and a moving object NR that instruct the user U to move to the right are displayed on the right side of the user image UV as viewed from the screen of the monitor 30D, and a determination object HL and a moving object NL that instruct the user U to move to the left are displayed on the left side of the user image UV. In addition, a determination object HF and a moving object NF that instruct the user U to move forward are displayed on the front side of the user image UV as viewed from the screen of the monitor 30D, and a determination object HB and a moving object NB that instruct the user U to move backward are displayed on the rear side of the user image UV.

[0125] In this manner, in this embodiment, instruction objects placed in the virtual space can be displayed in correspondence with an image of the user U, similar to the case where instruction objects are displayed around the user image UK reflected in the mirror MR shown in FIG. 8, so that the user can be guided as to the actions he or she should take, enabling intuitive play.

[0126] The instruction object may be displayed on either the game device 10D or the monitor 30D.

[0127] [Hardware configuration of game device 10D] The hardware configuration of the game device 10D will be described with reference to FIG. 15 is a block diagram showing an example of a hardware configuration of a game device 10D according to this embodiment. The game device 10D includes two imaging units, a front camera 11DA and a back camera 11DB, a display unit 12D, a sensor 13D, a storage unit 14D, a CPU 15D, a communication unit 16D, a sound output unit 17D, and a video output unit 18D.

[0128] Front camera 11DA is provided on the surface (front side) of game device 10D on which display unit 12D is provided, and captures an image in the direction facing display unit 12D. Back camera 11DB is provided on the surface (rear side) of game device 10D opposite to the surface on which display unit 12D is provided, and captures an image in the direction facing the rear side.

[0129] Display unit 12D includes a liquid crystal display, an organic EL display, etc. For example, display unit 12D may be configured as a touch panel that detects a touch operation on the display screen.

[0130] The sensor 13D is a sensor that outputs a detection signal related to the direction of the game device 10D. For example, the sensor 13D may include one or more of a gyro sensor, an acceleration sensor, a tilt sensor, a geomagnetic sensor, and the like.

[0131] The storage unit 14D includes, for example, an EEPROM, a ROM, a Flash ROM, a RAM, and the like, and stores the program and data for this dance game, data on the generated virtual space, and the like.

[0132] The CPU 15D functions as a control center that controls each unit of the game device 10D. For example, the CPU 15D executes a game program stored in the storage unit 14D to execute game processing, and executes processing such as superimposing and displaying an instruction object arranged in a virtual space on an image captured by the user U, as described with reference to FIG.

[0133] The communication unit 16D includes a communication device that performs wireless communication such as Bluetooth (registered trademark) or Wi-Fi (registered trademark).

[0134] The sound output unit 17D outputs the performance sounds of the music played in the dance game, the game's sound effects, etc. For example, the sound output unit 17 includes a speaker, and a phone terminal for connecting earphones, headphones, etc.

[0135] Video output unit 18D is configured to include a video output terminal that outputs a video to be displayed on display unit 12D to an external display device (for example, monitor 30D shown in FIG. 14). The video output terminal may be a multipurpose terminal including outputs other than video output, or may be a terminal dedicated to video output.

[0136] [Functional configuration of game device 10D] Next, the functional configuration of the game device 10D will be described with reference to FIG. 16 is a block diagram showing an example of a functional configuration of a game device 10D according to this embodiment. The illustrated game device 10D includes a control unit 150D as a functional configuration realized by a CPU 15D executing a program stored in a storage unit 14D. The control unit 150D includes an image acquisition unit 151D, a virtual space generation unit 152D, a user detection unit 153D, an object placement unit 154D, a display control unit 156D, a motion detection unit 157D, and an evaluation unit 158D.

[0137] The image acquiring unit 151D (an example of an acquiring unit) acquires a captured image of a real space captured by the front camera 11DA. For example, as shown in FIG. 14, the image acquiring unit 151D acquires a captured image including a user U playing a dance game.

[0138] The virtual space generating unit 152D (an example of a generating unit) generates a virtual space corresponding to the real space from the captured image acquired by the image acquiring unit 151D. For example, the virtual space generating unit 152D detects the position of an object (such as a floor or a wall) existing in the real space from the captured image acquired, and generates data of a three-dimensional coordinate space including at least part of the position information of the detected object (such as a floor or a wall) as data of the virtual space. As an example, the virtual space generating unit 152D defines a reference position K3 corresponding to the user U detected from the captured image by the user detecting unit 153D as the coordinate origin of the virtual space (three-dimensional coordinate space of XYZ) at the initialization at the start of playing this dance game, and generates virtual space data. During play, the reference position K3 (coordinate origin) and the X-axis, Y-axis, and Z-axis are fixed. The virtual space generating unit 152D stores the generated virtual space data in the storage unit 14D.

[0139] The user detection unit 153D detects an image of the user U from the captured image acquired by the image acquisition unit 151D. This detection requires recognizing that the image of a person detected from the captured image is an image of the user U playing the dance game. As a method of recognizing that the image is of the user U, for example, an identifiable marker (such as a mark or a label) may be attached to the body of the user U, and the user detection unit 153D may recognize that the image is of the user U by detecting this marker from the captured image. Also, by instructing the user U to perform a specific action (such as raising and lowering the right hand), the user detection unit 153D may recognize that the image is of the user U by detecting a person performing an action according to the instruction from the captured image.

[0140] The object placement unit 154D (an example of a placement unit) places an instruction object in a position based on a reference position K3 corresponding to the user U in the virtual space so that the instruction object can be viewed by the user U. Specifically, the object placement unit 154D places a determination object (see the determination objects HF, HB, HR, and HL in FIG. 14) in a determination position in the virtual space corresponding to the position of the floor. The object placement unit 154D also places a moving object (see the moving objects NF, NB, NR, and NL in FIG. 14) in an appearance position in the virtual space at a timing set in advance in accordance with the music, and moves the moving object (changes the placement position) toward the determination object. When placing the instruction object (the determination object and the moving object), the object placement unit 154D updates the virtual space data stored in the storage unit 14D based on the coordinate information of the placement position in the virtual space.

[0141] The display control unit 156D generates a composite image by combining the captured image acquired by the image acquisition unit 151D with the image of the instruction object arranged in the virtual space by the object arrangement unit 154D. Then, the display control unit 156D causes the generated composite image to be displayed on the display unit 12D. Furthermore, the display control unit 156D outputs the generated composite image from the image output unit 18D. For example, the display control unit 156D causes the generated composite image to be displayed on the display unit 12D after being horizontally inverted. Similarly, the display control unit 156D causes the generated composite image to be output from the image output unit 18D after being horizontally inverted.

[0142] The motion detection unit 157D (an example of a detection unit) detects motion of at least a part of the body of the user U from the captured video acquired by the video acquisition unit 151D. For example, the motion detection unit 157D detects the motion of the feet of the user U playing a dance game. The motion detection unit 157D detects the motion of the feet by extracting and tracking the image area of ​​the feet from each frame of the captured video.

[0143] The evaluation unit 158D evaluates the motion of at least a part of the body of the user U detected by the motion detection unit 157D based on the timing and position based on the instruction object arranged in the virtual space. For example, the evaluation unit 158D compares the timing and position at which the moving object reaches the judgment object with the timing and position of the foot motion of the user U (motion of stepping on the judgment object), and evaluates the play based on the motion of the user U. If the evaluation unit 158D can determine that the timing and position of both match based on the comparison result, it adds a score, and if it can determine that they do not match, it does not add a score.

[0144] The evaluation unit 158D may evaluate the play based on the action of the user U by comparing the position of the user U's foot at the time when the moving object reaches the judgment object with the position of the judgment object.

[0145] [Operation of Pointer Object Placement Processing] Next, an operation of an instruction object placing process for generating a virtual space and placing an instruction object in the dance game process executed by the CPU 15D of the game device 10D will be described. Fig. 17 is a flowchart showing an example of the instruction object placing process according to this embodiment.

[0146] First, the CPU 15D acquires a captured image of the real space captured by the front camera 11DA (step S501). For example, the CPU 15 acquires a captured image including a user U playing a dance game, as shown in FIG.

[0147] Next, the CPU 15D detects an image of the user U playing the dance game from the captured image acquired in step S501 (step S503).

[0148] Next, the CPU 15D generates a virtual space corresponding to the real space from the captured image acquired in step S501 (step S505). For example, the CPU 15D detects the position of an object (floor, wall, etc.) existing in the real space from the captured image, and generates data of a three-dimensional coordinate space including at least part of the position information of the detected object (floor, wall, etc.) as data of the virtual space. As an example, the CPU 15D generates virtual space data including at least part of the position information of the detected object (floor, wall, etc.) in a virtual space (three-dimensional coordinate space) having a reference position K3 corresponding to the user U detected from the captured image by the user detection unit 153D as the coordinate origin. Then, the CPU 15D stores the generated virtual space data in the storage unit 14D.

[0149] Next, at or before the start of play of the dance game, the CPU 15D places a judgment object (see judgment objects HF, HB, HR, and HL in FIG. 14) at a judgment position based on a reference position K3 in the virtual space corresponding to the position of the floor (step S507). When placing the judgment object, the CPU 15D adds position information of the placed judgment object to the virtual space data stored in the storage unit 14D.

[0150] When the dance game is started, the CPU 15D determines whether or not a moving object appearance trigger occurs (step S509). The appearance trigger occurs at a timing set in advance in accordance with the music. When the CPU 15D determines in step S509 that an appearance trigger occurs (YES), the CPU 15D proceeds to the process of step S511.

[0151] In step S511, the CPU 15D places a moving object (one or more of the moving objects NF, NB, NR, and NL in FIG. 14) at an appearance position based on the reference position K3 in the virtual space, and starts moving the moving object toward a judgment position (the position of a judgment object corresponding to each moving object). When placing the moving object, the CPU 15D adds position information of the placed moving object to the virtual space data stored in the storage unit 14D. Also, when moving the placed moving object, the CPU 15D updates the position information of the moving object added to the virtual space data stored in the storage unit 14D. Then, the CPU 15D proceeds to the process of step S513. On the other hand, when the CPU 15D determines in step S509 that there is no appearance trigger (NO), the CPU 15D proceeds to the process of step S513 without performing the process of step S511.

[0152] In step S513, the CPU 15D determines whether the moving object has reached the determination position. The CPU 15D erases the moving object that has been determined to have reached the determination position (YES) from the virtual space (step S515). When erasing the moving object from the virtual space, the CPU 15D deletes the position information of the moving object to be erased from the virtual space data stored in the storage unit 14D.

[0153] On the other hand, the CPU 15D continues to gradually move the moving object that has been determined not to have reached the determination position (NO) toward the determination position (step S517). When moving the moving object, the CPU 15D updates the position information of the moving object to be moved among the virtual space data stored in the storage unit 14D.

[0154] Next, the CPU 15D determines whether the dance game has ended (step S519). For example, the CPU 15D determines that the dance game has ended when the music being played has ended. If the CPU 15D determines that the dance game has not ended (NO), the CPU 15D returns to the process of step S509. On the other hand, if the CPU 15D determines that the dance game has ended (YES), the CPU 15D ends the instruction object placement process.

[0155] In addition, the order of placing the judgment object and the first appearing moving object may be simultaneous, the judgment object may be placed first, or conversely, the judgment object may be placed later (until the first appearing moving object reaches the judgment position).

[0156] [Operation of the Designated Object Display Processing] Next, an operation of a command object display process for displaying a command object arranged in a virtual space in the dance game process executed by the CPU 15D of the game device 10D will be described. In this embodiment, the command object is displayed as a composite image in which the command object is superimposed on an image captured by capturing an image of the user U. FIG. 18 is a flowchart showing an example of the designated object display process according to the present embodiment.

[0157] The CPU 15D acquires a captured image of the real space captured by the front camera 11DA, and acquires virtual space data from the storage unit 14D (step S601).

[0158] Then, the CPU 15D generates a composite image by combining the acquired captured image with the instruction object included in the virtual space data, and displays the generated composite image on the display unit 12D (step S603). The CPU 15D also outputs the generated composite image to the image output unit 18D, and displays it on the monitor 30D connected to the image output unit 18D (step S603). As a result, a composite image in which the instruction object is superimposed on the captured image of the user U is displayed on the display unit 12D and the monitor 30D in real time. The CPU 15D may display the composite image on either the display unit 12D or the monitor 30D.

[0159] Next, the CPU 15D determines whether the dance game has ended (step S605). For example, the CPU 15D determines that the dance game has ended when the music being played has ended. If the CPU 15D determines that the dance game has not ended (NO), the CPU 15D returns to the process of step S601. On the other hand, if the CPU 15D determines that the dance game has ended (YES), the CPU 15D ends the instruction object display process.

[0160] [Play evaluation process] Next, a description will be given of the operation of a play evaluation process for evaluating a play based on a movement of at least a part of the body of the user U in the process of the dance game executed by the CPU 15D of the game device 10D. Fig. 19 is a flowchart showing an example of the play evaluation process according to this embodiment.

[0161] The CPU 15D acquires an image of the real space captured by the front camera 11DA (step S701). Next, the CPU 15D detects a motion of at least a part of the body of the user U from the image acquired in step S701 (step S703). For example, the CPU 15D detects the motion of the feet of the user U playing a dance game.

[0162] Then, the CPU 15D evaluates the motion of at least a part of the body (e.g., a foot) of the user U detected in step S703 based on the timing and position based on the instruction object arranged in the virtual space (step S705). For example, the CPU 15D compares the timing and position at which the moving object reaches the determination object with the timing and position of the user U's foot motion (motion of stepping on the determination object) to evaluate the play based on the user U's foot motion.

[0163] Furthermore, CPU 15D updates the game score based on the evaluation result in step S705 (step S707). For example, CPU 15D adds a score when it can determine that the timing and position of the moving object reaching the judgment object coincides with the timing and position of the foot action (action of stepping on the judgment object) of user U, and does not add a score when it can determine that they do not coincide.

[0164] Next, the CPU 15D determines whether the dance game has ended (step S709). For example, the CPU 15D determines that the dance game has ended when the music being played has ended. If the CPU 15D determines that the dance game has not ended (NO), the CPU 15D returns to the process of step S701. On the other hand, if the CPU 15D determines that the dance game has ended (YES), the CPU 15D ends the play evaluation process.

[0165] [Summary of the Fourth Embodiment] As described above, the game device 10D according to the present embodiment acquires an image of a real space, and generates a virtual space corresponding to the real space from the acquired image. Then, the game device 10D places an instruction object that instructs the user U to move at a position based on a reference position K3 corresponding to the user in the generated virtual space so that the user U can see it, and displays a composite image obtained by combining the image of the instruction object placed in the virtual space with the image of the instruction object placed in the virtual space on the display unit 12D (an example of a display unit). Note that the game device 10D may display the composite image on the monitor 30D (an example of a display unit). In addition, the game device 10D detects a movement of at least a part of the body of the user U from the acquired image, and evaluates the detected movement based on the timing and position based on the instruction object placed in the virtual space.

[0166] As a result, in game processing in which the actions of user U are evaluated based on the timing and position of an instruction object that instructs the user U's actions, the game device 10D causes a composite image in which the instruction object is combined with an image captured by the user U to be visually displayed on the game device 10D (e.g., a smartphone) or an externally connected monitor 30D (e.g., a home television), so that the user can be guided as to what actions to take with a simple configuration, enabling more intuitive play.

[0167] For example, the game device 10D displays the above composite image on the display unit 12D or the monitor 30D after flipping the image horizontally.

[0168] This allows the user U to play the game while looking at the display unit 12D or the monitor 30D in the same way as if he or she were looking in a mirror.

[0169] Furthermore, the game device 10 moves an instruction object (e.g., a moving object) placed at a predetermined position (appearance position) in the virtual space toward a predetermined judgment position (e.g., the position of a judgment object). Then, the game device 10 evaluates the movement of at least a part of the body (e.g., a foot) of the user U detected from the captured video based on the timing at which the instruction object (e.g., the moving object) moving in the virtual space reaches the judgment position and the judgment position.

[0170] This allows the game device 10D to evaluate, using the captured video, whether or not the user U was able to perform the action as instructed.

[0171] [Variations] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above embodiment, and the present invention also includes designs that do not deviate from the gist of the present invention. For example, the configurations described in the above embodiment can be combined in any combination.

[0172] The instruction object described in each of the above-mentioned embodiments is an example, and various modes can be used as long as it instructs the user U to perform an action. For example, the content of the action instructed to the user U varies depending on the type (mode) of the instruction object. For example, by changing the thickness (width in the Z-axis direction) of the moving object, the time from when the bottom of the moving object reaches the judging object to when the top of the moving object reaches the judging object changes, so the thickness of the moving object may be used to indicate the time to continue stepping on the judging object. The moving object does not necessarily appear in the vertical direction of the destination judging object, and may appear from a position deviating from the vertical direction. In addition, the moving direction of the moving object and the position of the judging object can also be set arbitrarily.

[0173] Also, the judgment object may not be displayed at the judgment position. For example, when the position of the floor surface is the judgment position, the timing and position at which the moving object reaches the floor surface become the instruction contents for instructing the user U's action. For example, when a moving object having a certain thickness (for example, a length about the same as the height of the user U) is moved vertically toward the floor surface (judgment position) in a diagonal direction (for example, a direction inclined at 45° from the vertical direction) instead of the vertical direction, the position at which the moving object reaches the floor surface changes over time from the position on the XY plane when the bottom of the moving object reaches the floor surface to the position on the XY plane when the top of the moving object reaches the floor surface. Therefore, an instruction to move the position to be stepped on by the foot may be given using a moving object having a certain thickness in a diagonal direction.

[0174] Furthermore, the determination position is not limited to the floor surface, and can be set at any position between the floor surface and the ceiling, for example. The height of the determination position may be set according to the height of the user U by detecting the height. Furthermore, the displayed moving object itself may indicate the movement of the user U without setting a determination position. For example, the position of the moving object when it appears or when it is moving and the timing of the position may indicate the movement of the user U. For example, the trajectory of the movement of the moving object may indicate the trajectory of the movement of the user U (for example, the trajectory of the hand movement).

[0175] In the first and second embodiments, the game device 10 and the game device 10A configured as a see-through HMD are provided with the imaging unit 11. However, the imaging unit 11 may be installed in a different location from the game device 10 and the game device 10A, where the imaging unit 11 can image the user U playing the dance game. In this case, the device including the imaging unit 11 installed in the different location is connected to the game device 10 and the game device 10A by wired or wireless communication. In the game system 1C including the game device 10C and the HMD 20C described in the third embodiment, the imaging unit 21C corresponding to the imaging unit 11 is provided in the HMD 20C. However, the imaging unit 21C may be installed in a different location from the HMD 20C, where the imaging unit 21C can image the user U playing the dance game. In this case, the device including the imaging unit 21C installed in the different location is connected to the HMD 20C or the game device 10C by wired or wireless communication. The imaging unit 21C may be provided in the game device 10C.

[0176] In addition, in the game device 10D described in the fourth embodiment, a configuration has been described in which the front camera 11DA provided as an imaging unit is used to capture an image of the user U playing the dance game, but a configuration may be adopted in which an image of the user U is captured using a device including an imaging unit installed in a different location from the game device 10D. In this case, the device including the imaging unit installed in a different location is connected for wired or wireless communication with the game device 10D.

[0177] In addition, a program for realizing the functions of the above-mentioned control unit 150 (150A, 150C, 150D) may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform processing as the control unit 150 (150A, 150C, 150D). Here, "reading a program recorded in a recording medium into a computer system and executing it" includes installing the program into a computer system. The "computer system" here includes hardware such as an OS and peripheral devices. The "computer system" may also include a plurality of computer devices connected via a network including a communication line such as the Internet, a WAN, a LAN, or a dedicated line. The "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, and a storage device such as a hard disk built into a computer system. In this way, the recording medium storing the program may be a non-transient recording medium such as a CD-ROM. The recording medium also includes a recording medium provided inside or outside and accessible from a distribution server to distribute the program. The code of the program stored in the recording medium of the distribution server may be different from the code of the program in a format executable by the terminal device. In other words, the format in which the program is stored in the distribution server does not matter as long as it can be downloaded from the distribution server and installed in a format executable by the terminal device. The program may be divided into multiple parts, downloaded at different times, and then combined on the terminal device, or each of the divided programs may be distributed by a different distribution server. Furthermore, the "computer-readable recording medium" includes a memory that holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted via a network. The above program may also be for realizing part of the above-mentioned functions.Furthermore, the above-mentioned functions may be realized in combination with a program already recorded in the computer system, that is, a so-called differential file (differential program).

[0178] In addition, some or all of the functions of the control unit 150 (150A, 150C, 150D) described above may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each of the above-mentioned functions may be individually processed, or some or all of them may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit technology that can replace LSI appears due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0179] In the above embodiment, at least a part of the data stored in the storage unit 14 (14C, 14D) of the game device 10 (10A, 10C, 10D) may be stored in an externally connected storage device. The externally connected storage device is a storage device connected to the game device 10 (10A, 10C, 10D) by wire or wirelessly. For example, the externally connected storage device may be a storage device connected by a Universal Serial Bus (USB), a wireless Local Area Network (LAN), a wired LAN, or the like, or may be a storage device (data server) connected via the Internet or the like. This storage device (data server) connected via the Internet or the like may be used by using cloud computing.

[0180] Furthermore, a configuration corresponding to at least a part of each unit included in the control unit 150 (150A, 150C, 150D) may be included in a server connected via the Internet, etc. For example, the above embodiment can be applied to a so-called cloud game in which the processing of a game such as a dance game is executed by a server.

[0181] In the above embodiment, a dance game, which is an example of a music game, has been described as an example, but the present invention is not limited to dance games. For example, the present invention can be applied to music games in general, in which an operation is performed on an object that appears in time with a piece of music. In addition to music games, the present invention can also be applied to games in which an operation is performed on an object that appears at a predetermined timing, such as punching, kicking, knocking down, or hitting with a weapon.

[0182] [Note] From the above description, the present invention can be understood, for example, as follows: In order to facilitate understanding of the present invention, reference numerals in the accompanying drawings are conveniently written in parentheses, but the present invention is not limited to the illustrated embodiments.

[0183] (Appendix A1) A game program according to one aspect of the present invention is a game program for a computer that executes processing of a game that can be played using a video output device (10, 10A, 20C) that is worn on the head of a user (U) and outputs a video that is visible to the user and that can visually recognize a real space, the game program including steps of acquiring captured video of the real space (S101, S301, S401), generating a virtual space corresponding to the real space from the captured video (S103, S405), and calculating a position of the virtual space based on a reference position (K1, K2) corresponding to the user. The virtual machine executes the steps of: placing an instruction object that instructs the user to move at a position so that it is visible to the user (S105, S109, S407, S411); displaying the virtual space in which at least the instruction object is placed in correspondence with the real space (S203); detecting a movement of at least a part of the user's body from the captured image (S303); and evaluating the detected movement based on a timing and position based on the instruction object placed in the virtual space (S305).

[0184] According to the configuration of Appendix A1, in a game program, in a game process in which a user's actions are evaluated based on the timing and position of an instruction object that instructs the user's actions, by wearing a video output device such as an HMD on the head, the instruction object is associated with real space and made visible to the user, so that with a simple configuration, it is possible to guide the user on the actions he or she should take, enabling more intuitive play.

[0185] (Appendix A2) Also, one aspect of the present invention is a game program as described in Appendix A1, wherein the reference position includes a first reference position (K1) in the virtual space corresponding to the position of the user (U) wearing the video output device (10, 10A, 20C), and the first reference position is based on the position of the video output device in the virtual space.

[0186] According to the configuration of Appendix A2, the game program can display instruction objects in correspondence with real space based on the position of the user playing the game, thereby making the instructions to the user feel more real and enabling more intuitive gameplay.

[0187] (Appendix A3) Also, one aspect of the present invention is a game program as described in Appendix A2, wherein in the placing steps (S105, S109, S407, S411), the position at which the instruction object is placed is restricted to a portion of the virtual space depending on the orientation of the user (U) wearing the video output device (10, 10A, 20C).

[0188] According to the configuration of Appendix A3, the game program does not issue instructions to perform actions outside the user's field of view (e.g., behind), so that the user can play without worrying about what is outside the field of view (e.g., behind) while playing, and the difficulty of the game can be prevented from becoming too high.

[0189] (Appendix A4) Also, one aspect of the present invention is a game program as described in Appendix A1, further comprising causing the computer to execute a step (S403) of detecting an image (UK) corresponding to the user (U) from the captured image, and the reference position includes a second reference position (K2) in the virtual space of the detected image corresponding to the user.

[0190] According to the configuration of Appendix A4, the game program can display instruction objects instructing the user's actions around a virtual image (user image UK) of the user reflected in a mirror or the like by wearing a video output device such as an HMD on the head, and can guide the user to the actions to be taken so that the user can play more intuitively with a simple configuration. For example, the game program can simultaneously view instruction objects displayed around the user image UK (for example, front, back, left and right) from above without limiting the position of the instruction object to a part of the virtual space, so that the types of actions to be instructed to the user during play can be diversified. In addition, the game program can evaluate the user's actions without the user having to look at his or her own feet and instruction objects located below, so that dancing can be made easier.

[0191] (Appendix A5) Also, one aspect of the present invention is a game program as described in Appendix A2 or Appendix A3, further comprising causing the computer to execute a step (S403) of detecting (UK) corresponding to the user (U) from the captured image, and the reference position includes a second reference position (K2) in the virtual space of an image corresponding to the detected user.

[0192] According to the configuration of appended note A5, the game program can display instruction objects instructing the user's actions around a virtual image (user image UK) of the user reflected in a mirror or the like by wearing a video output device such as an HMD on the head, and can guide the user to the actions to be taken so that the user can play more intuitively with a simple configuration. For example, the game program can simultaneously view instruction objects displayed around the user image UK (for example, front, back, left and right) from above without limiting the position where the instruction object is placed to a part of the virtual space, so that the types of actions instructed to the user during play can be diversified. In addition, the game program can evaluate the user's actions without the user having to look at his or her own feet and instruction objects present below, so that dancing is not difficult. In addition, the game program can display instruction objects around the user playing the game and around the virtual image of the user reflected in a mirror or the like, so that the user can play the game while arbitrarily selecting which of the instruction objects is easier to play.

[0193] (Appendix A6) Also, one aspect of the present invention is a game program as described in Appendix A5, wherein in the placing steps (S105, S109, S407, S411), when placing the instruction object at a position based on the second reference position (K2) in the virtual space, the visibility of the instruction object to be placed at a position based on the first reference position (K1) is reduced, or the instruction object is not placed at a position based on the first reference position (K1).

[0194] According to the configuration of Appendix A6, the game program can prevent an instruction object displayed at a position (around the user U) based on a first reference position (e.g., reference position K1) from obscuring an instruction object displayed at a position (around the virtual image of the user reflected in the mirror MR) based on a second reference position (e.g., reference position K2), thereby improving the visibility of the instruction object.

[0195] (Appendix A7) Also, one aspect of the present invention is a game program as described in any one of Appendices A4 to A6, wherein the image (UK) corresponding to the detected user (U) is an image of the user reflected in a mirror (MR) located opposite, and in the placing steps (S105, S109, S407, S411), when placing the instruction object at a position based on the second reference position (K2) in the virtual space, the front-to-back orientation with respect to the second reference position is reversed.

[0196] According to the configuration of Appendix A7, the game program can display instruction objects corresponding to the orientation of the virtual image of the user (user image UK) reflected in the mirror, and can guide the user as to what actions to take so that the user can play intuitively while looking in the mirror.

[0197] (Appendix A8) Also, one aspect of the present invention is a game program as set forth in any one of Appendices A1 to A7, wherein in the placing steps (S105, S109, S407, S411), the instruction object placed at a predetermined position in the virtual space is moved toward a predetermined judgment position, and in the evaluating step (S305), the detected action is evaluated based on the judgment position and the timing at which the instruction object moving in the virtual space reaches the judgment position.

[0198] According to the configuration of Appendix A8, the game program can use the captured video to evaluate whether or not the user has performed the action as instructed.

[0199] (Supplementary Note A9) Furthermore, one aspect of the present invention is the game program according to any one of Supplementary Note A1 to Supplementary Note A8, in which the content of the action instructed to the user (U) varies depending on the type of the instruction object.

[0200] According to the configuration of appended claim A9, the game program can diversify the actions that the user takes while playing, making it possible to provide a highly entertaining game.

[0201] (Appendix A10) A game processing method according to one aspect of the present invention is a game processing method executed by a computer that executes processing of a game that can be played using a video output device (10, 10A, 20C) that is worn on a head of a user (U) to output a video that is visible to the user and that allows the user to view a real space, the game processing method including steps of acquiring a captured video of the real space (S101, S301, S401), generating a virtual space corresponding to the real space from the captured video (S103, S405), and calculating a reference position in the virtual space corresponding to the user. The method includes steps (S105, S109, S407, S411) of placing an instruction object instructing the user to perform a movement at a position based on a position (K1, K2) of the instruction object so that the instruction object is visible to the user, a step (S203) of displaying the virtual space in which at least the instruction object is placed in correspondence with the real space, a step (S303) of detecting a movement of at least a part of the body of the user from the captured image, and a step (S305) of evaluating the detected movement based on a timing and position based on the instruction object placed in the virtual space.

[0202] According to the configuration of Appendix A10, in a game processing method in which a user's action is evaluated based on the timing and position based on an instruction object that instructs the user's action, by wearing a video output device such as an HMD on the head, the instruction object is associated with real space and made visible to the user, so that the user can be guided as to what action to take with a simple configuration, enabling more intuitive play.

[0203] (Appendix A11) A game device (10, 10A, 10C) according to one aspect of the present invention is a game device that executes processing of a game that can be played by using a video output device (10, 10A, 20C) that outputs a video that is visible to a user (U) by wearing it on the head of the user and that can visually recognize a real space, and includes an acquisition unit (151, S101, S301, S401) that acquires a captured video of the real space, a generation unit (152, S103, S405) that generates a virtual space corresponding to the real space from the captured video acquired by the acquisition unit, and a reference position (K1) corresponding to the user in the virtual space generated by the generation unit. the positioning unit (154, S105, S109, S407, S411) that positions an instruction object instructing the user to perform an action so as to be visible to the user at a position based on a position (K2) of the instruction object; a display control unit (156, S203) that displays the virtual space in which at least the instruction object is placed in correspondence with the real space; a detection unit (157, S303) that detects a movement of at least a part of the user's body from the captured image acquired by the acquisition unit; and an evaluation unit (158, S305) that evaluates the movement detected by the detection unit based on a timing and position based on the instruction object placed in the virtual space.

[0204] According to the configuration of Appendix A11, in a game process in which a user's actions are evaluated based on the timing and position of an instruction object that instructs the user's actions, by attaching a video output device such as an HMD to the head, the instruction object can be associated with real space and made visible to the user, so that the game device can guide the user on the actions he or she should take with a simple configuration, allowing for more intuitive play.

[0205] (Appendix B1) Furthermore, a game program according to one embodiment of the present invention causes a computer to execute the steps of acquiring an image captured of a real space (S501, S701), generating a virtual space corresponding to the real space from the captured image (S505), placing an instruction object instructing a movement of the user at a position in the virtual space based on a reference position (K3) corresponding to the user (U) so that the instruction object is visible to the user (U) (S507, S511), displaying a composite image obtained by combining the captured image with an image of the instruction object placed in the virtual space on a display unit (12D, 30D) (S603), detecting a movement of at least a part of the user's body from the captured image (S703), and evaluating the detected movement based on a timing and position based on the instruction object placed in the virtual space (S705).

[0206] According to the configuration of Appendix B1, in a game process in which a user's actions are evaluated based on the timing and position of an instruction object that instructs the user's actions, the game program causes a composite image in which the instruction object is combined with an image captured by the user to be visually displayed on a display unit of a smartphone, a home television, or the like, so that the user can be guided as to what actions to take with a simple configuration, enabling more intuitive play.

[0207] (Additional Note B2) Furthermore, one aspect of the present invention is the game program according to Additional Note B1, wherein in the displaying step (S603), the composite image is displayed on the display unit (12D, 30D) with the composite image flipped left and right.

[0208] According to the configuration of Appendix B2, the game program can be played while the user is looking at the display unit (monitor) with the same sensation as if he or she were looking in a mirror.

[0209] (Appendix B3) Also, one aspect of the present invention is a game program as described in Appendix B1 or Appendix B2, wherein in the placing step (S507, S511), the instruction object placed at a predetermined position in the virtual space is moved toward a predetermined judgment position, and in the evaluating step (S705), the detected action is evaluated based on the judgment position and the timing at which the instruction object moving in the virtual space reaches the judgment position.

[0210] According to the configuration of Appendix B3, the game program can use the captured video to evaluate whether or not the user has performed the action as instructed.

[0211] (Supplementary Note B4) Furthermore, one aspect of the present invention is the game program according to any one of Supplementary Note B1 to Supplementary Note B3, in which the content of the action instructed to the user (U) varies depending on the type of the instruction object.

[0212] According to the configuration of Appendix B4, the game program can diversify the actions that the user takes while playing, making it possible to provide a highly entertaining game.

[0213] (Appendix B5) Furthermore, a game processing method according to one embodiment of the present invention is a game processing method executed by a computer, and includes the steps of acquiring an image captured of a real space (S501, S701), generating a virtual space corresponding to the real space from the captured image (S505), arranging an instruction object instructing a movement of the user at a position in the virtual space based on a reference position (K3) corresponding to the user (U) so that the instruction object is visible to the user (U) (S507, S511), displaying a composite image obtained by combining the captured image and an image of the instruction object arranged in the virtual space on a display unit (12D, 30D) (S603), detecting a movement of at least a part of the user's body from the captured image (S703), and evaluating the detected movement based on a timing and a position based on the instruction object arranged in the virtual space (S705).

[0214] According to the configuration of Appendix B5, in a game processing method in which a user's action is evaluated based on the timing and position based on an instruction object that instructs the user's action, a composite image in which the instruction object is combined with an image captured by the user is displayed on a display unit of a smartphone, a home television, or the like so that the composite image can be viewed by the user, and therefore the user can be guided as to what actions he or she should take with a simple configuration, enabling more intuitive play.

[0215] (Appendix B6) A game device (10D) according to one aspect of the present invention includes an acquisition unit (151D, S501, S701) that acquires an image of a real space, a generation unit (152D) that generates a virtual space corresponding to the real space from the image acquired by the acquisition unit, and a placement unit (154D, S507, S511) that places an instruction object that instructs the user to perform an action in a position based on a reference position (K3) corresponding to the user (U) in the virtual space generated by the generation unit so that the instruction object is visible to the user. a display control unit (156D, S603) that causes a display unit (12D, 30D) to display a composite image obtained by combining the captured image with the image of the instruction object placed in the virtual space; a detection unit (157D, S703) that detects a movement of at least a part of the user's body from the captured image acquired by the acquisition unit; and an evaluation unit (158D, S705) that evaluates the movement detected by the detection unit based on a timing and position based on the instruction object placed in the virtual space.

[0216] According to the configuration of Appendix B6, in a game process in which a user's actions are evaluated based on the timing and position of an instruction object that instructs the user's actions, the game device causes a composite image in which the instruction object is combined with an image captured by the user to be visually displayed on a display unit of a smartphone, home television, or the like, so that the user can be guided as to what actions to take with a simple configuration, enabling more intuitive play. [Explanation of symbols]

[0217] 1C game system, 10, 10A, 10C, 10D game device, 11 imaging unit, 11DA front camera, 11DB back camera, 12, 12D display unit, 13, 13D sensor, 14, 14C, 14D memory unit, 15, 15C, 15D CPU, 16, 16C, 16D communication unit, 17, 17D sound output unit, 18D video output unit, 20C HMD, 21C imaging unit, 22C display unit, 23C sensor, 24C memory unit, 25C CPU, 26C communication unit, 27C sound output unit, 150, 150A, 150C, 150D control unit, 151, 151D video acquisition unit, 152, 152D virtual space generation unit, 153A user image detection unit, 153D User detection unit, 154, 154A, 154D object placement unit, 155 gaze direction detection unit, 156, 156D display control unit, 157, 157D action detection unit, 158, 158D evaluation unit

Claims

1. a computer that executes processing for a game that can be played using a video output device that is worn on a user's head and outputs a video that can be visually recognized by the user and allows the user to visually recognize a real space; acquiring an image of the real space; generating a virtual space corresponding to the real space from the captured image; detecting an image corresponding to the user from the captured image; placing an instruction object that instructs the user to perform an action at a position based on a reference position corresponding to the user within the virtual space so as to be visible to the user while playing the game; displaying the virtual space, in which at least the pointing object is disposed, in association with the real space so that the pointing object is superimposed on the real space and visible; detecting a motion of at least a part of the user's body from the captured image; evaluating the detected motion based on a timing and a position based on the pointing object disposed in the virtual space; Run the command, the reference position includes a second reference position in the virtual space of an image corresponding to the detected user; Game program.

2. the detected image corresponding to the user is an image of the user reflected in a mirror located opposite the user, In the step of placing, when placing the instruction object at a position based on the second reference position in the virtual space, a front-to-back orientation of the instruction object with respect to the second reference position is inverted. The game program according to claim 1 .

3. A game processing method executed by a computer that executes processing of a game that can be played using a video output device that is worn on a user's head to output a video that can be viewed by the user and that can view a real space, comprising: acquiring an image of the real space; generating a virtual space corresponding to the real space from the captured image; detecting an image corresponding to the user from the captured image; placing an instruction object that instructs the user to perform an action at a position based on a reference position corresponding to the user within the virtual space so as to be visible to the user while playing the game; displaying the virtual space, in which at least the pointing object is disposed, in association with the real space so that the pointing object is superimposed on the real space and visible; detecting a motion of at least a part of the user's body from the captured image; evaluating the detected motion based on a timing and a position based on the pointing object disposed in the virtual space; Including, the reference position includes a second reference position in the virtual space of an image corresponding to the detected user; Game processing method.

4. A game device that is worn on a user's head and that executes processing of a game that can be played using a video output device that outputs a video image that can be viewed by the user and enables the user to view a real space, comprising: An acquisition unit that acquires an image of the real space; a generation unit that generates a virtual space corresponding to the real space from the captured image acquired by the acquisition unit; a detection unit that detects an image corresponding to the user from the captured video acquired by the acquisition unit; a placement unit that places an instruction object, which instructs the user to perform an action, at a position based on a reference position corresponding to the user within the virtual space generated by the generation unit so as to be visible to the user during play of the game; a display control unit that displays the virtual space, in which at least the instruction object is placed, in association with the real space so that the instruction object is superimposed on the real space and visible; a detection unit that detects a motion of at least a part of the body of the user from the captured image acquired by the acquisition unit; an evaluation unit that evaluates the action detected by the detection unit based on a timing and a position based on the instruction object arranged in the virtual space; Equipped with the reference position includes a second reference position in the virtual space of an image corresponding to the detected user; Gaming device.

Images