Non-transitory computer-readable storage medium, game processing method, and

Abstract

To guide contents to be operated by a user so as to enable a more intuitive play with a simple configuration.SOLUTION: Acquiring a captured image of a real space, generating a virtual space corresponding to the real space from the captured image, arranging an instructing object for instructing an action of a user at a position based on a reference position corresponding to the user in the virtual space so that the instructing object is visible to the user, and displaying the virtual space in which at least the instructing object is arranged in association with the real space. The program causes the computer to execute a step of detecting a motion of at least a part of a body of a user from a captured video, and a step of evaluating the detected motion on the basis of a timing and a position based on an indicating object disposed in a virtual space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a game program, a game processing method, and a game device. [Background technology]

[0002] Among music games, there are dance games that detect the user's body movements and evaluate the quality of the dance. For example, Patent Document 1 discloses a dance game in which the trajectory and timing that the user (player) should trace by moving their hands and feet in time with the music are displayed on a game screen opposite the user, and the user moves their hands and feet while looking at the display. This dance game can be played, for example, on a home game console.

[0003] Patent Document 2 also discloses a dance game in which a user steps on an operation panel placed in real space in accordance with instructions displayed on a game screen in time with music. This dance game requires operation panels to be placed at the user's feet to determine where the user's feet should step in real space, and is an example of a so-called arcade game installed in amusement facilities such as game arcades. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-196286 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-193006 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the game described in Patent Document 1, although the game screen can guide the user as to what kind of body movement (trajectory) to make and when, it is not suitable for games that instruct the user as to where in real space the user should move (for example, where to step). For example, if an attempt is made to realize a game such as that described in Patent Document 2 above with a simple configuration such as a home game console without an operation panel at the user's feet, it is difficult for the user to know where in real space the user should move their feet, making it difficult to play intuitively.

[0006] One of the objectives of some aspects of the present invention is to provide a game program, a game processing method, and a game device that have a simple configuration and guides the user through the actions they should take, allowing for more intuitive play.

[0007] Another aspect of the present invention aims to provide a game program, a game processing method, and a game device that are capable of achieving the effects described in the embodiments described below. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, one aspect of the present invention is a game program for causing a computer that executes processing for a game that can be played using a video output device that is worn on a user's head and outputs video visible to the user and allows the user to view real space to execute the following steps: acquiring captured video of the real space; generating a virtual space corresponding to the real space from the captured video; placing instruction objects that instruct the user to perform actions in the virtual space at positions based on a reference position corresponding to the user so that the objects are visible to the user; displaying the virtual space in which at least the instruction objects are placed in correspondence with the real space; detecting movement of at least a part of the user's body from the captured video; and evaluating the detected movement based on timing and position based on the instruction objects placed in the virtual space.

[0009] Another aspect of the present invention is a game processing method executed by a computer that executes processing for a game that can be played using a video output device that is worn on a user's head and outputs video visible to the user while allowing the user to view a real space, the game processing method including the steps of: acquiring captured video of the real space; generating a virtual space corresponding to the real space from the captured video; arranging, in the virtual space, an instruction object that instructs the user to perform an action, at a position based on a reference position corresponding to the user, so that the instruction object is visible to the user; displaying the virtual space in which at least the instruction object is arranged, in correspondence with the real space; detecting a movement of at least a part of the user's body from the captured video; and evaluating the detected movement based on a timing and position based on the instruction object arranged in the virtual space.

[0010] Another aspect of the present invention is a gaming device that is worn on a user's head and that executes processing for a game that can be played using a video output device that outputs video visible to the user and allows the user to view a real space, the gaming device comprising: an acquisition unit that acquires captured video of the real space; a generation unit that generates a virtual space corresponding to the real space from the captured video acquired by the acquisition unit; a placement unit that places instruction objects that instruct the user to perform actions in a position that is based on a reference position corresponding to the user, within the virtual space generated by the generation unit, so that the instruction objects are visible to the user; a display control unit that displays the virtual space in which at least the instruction objects are placed, in correspondence with the real space; a detection unit that detects a movement of at least a part of the user's body from the captured video acquired by the acquisition unit; and an evaluation unit that evaluates the movement detected by the detection unit based on a timing and position based on the instruction object placed in the virtual space. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram showing an outline of game processing by the game device according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing definitions of spatial coordinates in a virtual space according to the first embodiment. [Figure 3] 1 is a block diagram showing an example of the hardware configuration of a game device according to a first embodiment. [Figure 4] FIG. 1 is a block diagram showing an example of the functional configuration of a game device according to a first embodiment. [Figure 5] 10 is a flowchart showing an example of a designation object placement process according to the first embodiment. [Figure 6] 10 is a flowchart showing an example of a designated object display process according to the first embodiment. [Figure 7] 10 is a flowchart showing an example of a play evaluation process according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing an outline of game processing by a game device according to a second embodiment. [Figure 9] 10A and 10B are diagrams showing definitions of spatial coordinates in a virtual space and positions of a user image according to the second embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of the functional configuration of a game device according to a second embodiment. [Figure 11] 10 is a flowchart showing an example of an instruction object placement process according to the second embodiment. [Figure 12] FIG. 10 is a block diagram showing an example of the hardware configuration of a game system according to a third embodiment. [Figure 13] FIG. 10 is a block diagram showing an example of the functional configuration of a game device according to a third embodiment. [Figure 14] FIG. 10 is a diagram showing an outline of game processing by a game device according to a fourth embodiment. [Figure 15] FIG. 10 is a block diagram showing an example of the hardware configuration of a game device according to a fourth embodiment. [Figure 16] FIG. 10 is a block diagram showing an example of the functional configuration of a game device according to a fourth embodiment. [Figure 17] 13 is a flowchart showing an example of an instruction object placement process according to the fourth embodiment. [Figure 18] 13 is a flowchart showing an example of an instruction object display process according to the fourth embodiment. [Figure 19] 10 is a flowchart showing an example of a play evaluation process according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [First embodiment] First, a first embodiment of the present invention will be described.

[0013] [Outline of the game device] First, an overview of an example of game processing executed by a game device according to this embodiment will be described. The game device according to this embodiment can be exemplified by a home game machine, but may also be used in amusement facilities such as game arcades.

[0014] FIG. 1 is a diagram showing an overview of game processing by a game device according to this embodiment. This diagram shows an overview of a play situation in which a user U plays a dance game (an example of a music game) using a game device 10. The game device 10 includes a video output device. The video output device may display video on a display or may project video. For example, the game device 10 is configured as an HMD (Head Mounted Display) that is worn on the user's head and outputs video that can be viewed by the user and enables the user to view real space.

[0015] In the illustrated example of a dance game, the user U moves at least a part of their body in accordance with the timing and position of instruction objects displayed on the HMD in time with the music. The instruction objects are displayed to guide the user U in the timing and position at which they should move in real space. In this embodiment, the instruction objects placed in the virtual space are displayed on the HMD in association with the real space, allowing the user to play intuitively.

[0016] For example, the game device 10 is configured as an HMD (a so-called optically transparent (optical see-through) type HMD) that allows a real space to be optically viewed. The game device 10 is worn on the user's head and displays a pointing object located in a virtual space on a transparent display positioned in front of the user's eyes. This allows the user to view an image in which the pointing object displayed on the display is superimposed on the real space that is visible through the display.

[0017] The game device 10 may be configured as a retinal projection optically transmissive HMD. In the case of a retinal projection type, the game device 10 includes an image projection device that projects an image directly onto the user's retina instead of a display. A pointing object placed in the user's virtual space is displayed so as to be visible by being directly projected onto the user's retina.

[0018] Furthermore, game device 10 may be configured as an HMD that displays captured images of real space in real time (so-called video see-through HMD). In this case, game device 10 is worn on the user's head and displays real-time images of the real space on a display positioned in front of the user's eyes, and also displays a pointing object located in a virtual space superimposed on the real-time images.

[0019] The game device 10 is worn on the head of the user U, and generates a virtual space from captured images of the user U's line of sight in real space. For example, the virtual space is defined as an XYZ three-dimensional coordinate space with mutually orthogonal X and Y axes parallel to the floor surface (plane) and a vertical Z axis perpendicular to the floor surface (plane). The generated virtual space includes positions corresponding to at least some of the objects in the real space (e.g., the user U, the floor, a wall, etc.). In the following description, the direction of the Z axis toward the ceiling is also referred to as the upward direction, and the direction toward the floor is also referred to as the downward direction.

[0020] The game device 10 uses the position of the user U in this virtual space as a reference position and places instruction objects that instruct the user to perform actions at positions based on the reference position (e.g., predetermined positions around the reference position). For example, instruction objects include a determination object and a moving object. The determination object is an instruction object that is placed at a determination position that serves as a determination standard for evaluating the user's actions. For example, the determination object is placed in the virtual space at a position (height) that corresponds to the floor surface in Z coordinates and around the reference position (the position of the user U) in XY coordinates (e.g., within the range that the user U can reach with one step). In the example shown in the figure, a determination object HF is placed in front of the reference position (the position of the user U), a determination object HB is placed behind the reference position, a determination object HR is placed to the right of the reference position, and a determination object HL is placed to the left of the reference position. Here, the reference position (the position of the user U) and the forward, backward, right, and left of the reference position are directions that are initialized at the start of play of the dance game and remain fixed even if the orientation of the user U changes during play.

[0021] The moving object appears from the ceiling side in the Z coordinate in the virtual space and gradually moves downward toward a determination object (determination position) that is placed at a position (height) corresponding to the floor surface. The appearance position may be set in advance based on, for example, the position of the user U's head (the position of the game device 10), or may change according to a predetermined rule. The moving object NF is a moving object that moves toward the determination object HF (determination position of the moving object NF). The moving object NB is a moving object that moves toward the determination object HB (determination position of the moving object NB). The moving object NR is a moving object that moves toward the determination object HR (determination position of the moving object NR). The moving object NL is a moving object that moves toward the determination object HL (determination position of the moving object NL).

[0022] The timing and position at which each moving object gradually moves and reaches its respective judgment object (judgment position) are the timing and position at which the user U should operate. For example, when the moving object NF reaches the judgment object HF, the user is required to step on the judgment object HF with their foot. The user's action is evaluated based on the timing and position at which the moving object reaches the judgment object, and the score is updated according to the evaluation. For example, if it is determined that the timing and position at which the moving object reaches the judgment object and the timing and position of the user's action match, a score is added, and if it is determined that they do not match, no score is added. For example, whether or not this timing and position match is determined by whether or not the user steps on at least a part of the judgment area (e.g., the area of ​​the judgment object HR) corresponding to the position at which the moving object reaches the judgment object within a predetermined time period (e.g., within 0.5 seconds before or after the arrival time) corresponding to the timing at which the moving object reaches the judgment object. Note that the score added may vary depending on the degree of match between the timing and position at which the moving object reaches the judgment object and the timing and position of the user's action.

[0023] Note that FIG. 1 illustrates, in a single diagram, the correspondence between the real space in which the user U is located and the virtual space in which the instruction objects are located, and is different from the play screen that the user U can view while playing. Each instruction object does not exist in the real space but exists only in the virtual space and is viewable via the game device 10. The instruction objects that are viewable by the user U while actually playing the game exist within a field of view (Fov) that can be viewed through the display portion of the game device 10. The instruction objects included in this field of view are displayed on the game device 10 (HMD) and are superimposed on the real space, making them viewable to the user U. Note that the game device 10 also displays display information related to the game other than the instruction objects (such as the score and information about the song being played).

[0024] FIG. 2 is a diagram illustrating the definition of spatial coordinates in the virtual space according to this embodiment. As described above, in this embodiment, the vertical axis is defined as the Z axis, and axes perpendicular to each other in a horizontal plane perpendicular to the Z axis are defined as the X axis and the Y axis. Furthermore, at the initialization when the dance game starts to be played, a reference position K1 (an example of a first reference position based on the position of the game device 10) corresponding to the position of the user U is defined as the coordinate origin, and the X axis is defined as the axis of the user U's line of sight. During play, the reference position K1 (coordinate origin), the X axis, the Y axis, and the Z axis are fixed. A change in the rotation direction about the Z axis is also referred to as a change in the yaw direction (left-right direction), a change in the rotation direction about the Y axis is also referred to as a change in the pitch direction (up-down direction), and a change in the rotation direction about the X axis is also referred to as a change in the roll direction.

[0025] When the orientation of the head of the user U wearing the game device 10 changes, the game device 10 detects this as a change in the rotation direction of each axis (yaw direction, pitch direction, roll direction) using a built-in acceleration sensor or the like. The game device 10 changes the field of view (Fov) shown in FIG. 1 based on the detected change in the rotation direction of each axis, and changes the display of the instruction object included in the virtual space. In this way, even if the orientation of the head of the user U changes, the game device 10 can display the instruction object included in the virtual space on the display in accordance with the change in field of view. Note that a change in the yaw direction may also be referred to as a change in the left-right direction, and a change in the pitch direction may also be referred to as a change in the up-down direction.

[0026] The reference position K1 shown in the figure is an example and is not limited to this position. Also, although the reference position K1 is defined as the coordinate origin of the spatial coordinates, the coordinate origin may be defined at another position.

[0027] [Hardware Configuration of Game Device 10] Next, an outline of the hardware configuration of the game device 10 according to this embodiment will be described. 3 is a block diagram showing an example of the hardware configuration of a game device 10 according to this embodiment. The game device 10 is an optically transmissive HMD and includes an imaging unit 11, a display unit 12, a sensor 13, a storage unit 14, a CPU (Central Processing Unit) 15, a communication unit 16, and a sound output unit 17.

[0028] The imaging unit 11 is a camera that captures an image in the line of sight of a user U who wears the game device 10 (HMD) on their head. That is, the imaging unit 11 is provided in the game device 10 (HMD) so that its optical axis corresponds to the line of sight when the game device 10 (HMD) is worn on the head. The imaging unit 11 may be a monocular camera or a dual camera. The imaging unit 11 outputs the captured image.

[0029] The display unit 12 is, for example, a transmissive display in an optically transmissive HMD. For example, the display unit 12 displays at least a pointing object. The display unit 12 may be configured to include two displays, one for the right eye and one for the left eye, or may be configured to include a single display that can be viewed by both eyes, regardless of whether it is for the right eye or the left eye. Furthermore, when the game device 10 is a retinal projection optically transmissive HMD, the display unit 12 is an image projection device that projects an image directly onto the user's retina.

[0030] When the game device 10 is a video see-through HMD, the display unit 12 is an opaque display that does not allow the real space to be optically recognized.

[0031] The sensor 13 is a sensor that outputs a detection signal related to the direction of the game device 10. For example, the sensor 13 is a gyro sensor that detects the angle, angular velocity, angular acceleration, etc. of an object. The sensor 13 may be a sensor that detects a change in direction, or a sensor that detects the direction itself. For example, the sensor 13 may include an acceleration sensor, a tilt sensor, a geomagnetic sensor, etc. instead of or in addition to a gyro sensor.

[0032] The memory unit 14 includes, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory), a ROM (Read-Only Memory), a Flash ROM, a RAM (Random Access Memory), etc., and stores the program and data for this dance game, data for the generated virtual space, etc.

[0033] CPU 15 functions as a control center that controls each unit included in game device 10. For example, CPU 15 executes a game program stored in storage unit 14 to perform game processing, and as described with reference to FIG. 1, performs processing such as generating a virtual space corresponding to real space from captured video, placing a designated object in the generated virtual space, and detecting a user's action and evaluating it based on the timing and position of the designated object.

[0034] The communication unit 16 includes a communication device that performs wireless communication using Bluetooth (registered trademark), Wi-Fi (registered trademark), etc. The communication unit 16 may also include a digital input / output port such as a USB (Universal Serial Bus), a video input / output port, etc.

[0035] Sound output unit 17 outputs the sounds of the music played in the dance game, sound effects of the game, etc. For example, sound output unit 17 may be configured to include a speaker, earphones, headphones, or a terminal connectable to any of these. Note that sound output unit 17 may output various sounds to an external speaker, earphones, headphones, etc. via wireless communication such as Bluetooth (registered trademark).

[0036] The hardware components of the game device 10 described above are connected to each other via a bus so that they can communicate with each other.

[0037] [Functional configuration of game device 10] Next, the functional configuration of the game device 10 will be described with reference to FIG. 4 is a block diagram showing an example of the functional configuration of game device 10 according to this embodiment. Game device 10 shown in the figure includes control unit 150 as a functional configuration realized by CPU 15 executing a program stored in storage unit 14. Control unit 150 executes the processing of the dance game described with reference to FIGS. 1 and 2. For example, control unit 150 includes image acquisition unit 151, virtual space generation unit 152, object placement unit 154, gaze direction detection unit 155, display control unit 156, action detection unit 157, and evaluation unit 158.

[0038] The video acquisition unit 151 (an example of an acquisition unit) acquires captured video of the real space captured by the imaging unit 11. For example, before starting to play a dance game, the game device 10 instructs the user U to look in a predetermined direction (for example, to look up, down, left, and right). The game device 10 displays this instruction on, for example, the display unit 12. As a result, the video acquisition unit 151 acquires captured video of the surroundings of the user U in the real space captured by the imaging unit 11.

[0039] The virtual space generation unit 152 (an example of a generation unit) generates a virtual space corresponding to the real space from the captured video acquired by the video acquisition unit 151. For example, the virtual space generation unit 152 detects the positions of objects (floors, walls, etc.) existing in the real space from the acquired captured video, and generates three-dimensional coordinate space data including position information of at least a portion of the detected objects (floors, walls, etc.) as virtual space data. As an example, a reference position K1 (see FIG. 2) corresponding to the user U based on the position of the game device 10 itself worn on the head of the user U is defined as the coordinate origin of the virtual space (three-dimensional coordinate space). The virtual space generation unit 152 generates virtual space data including position information corresponding to objects (floors, walls, etc.) existing in the real space within the virtual space (three-dimensional coordinate space) with the reference position K1 corresponding to the user U as the coordinate origin. The virtual space generation unit 152 stores the generated virtual space data in the storage unit 14.

[0040] Here, any known technology can be applied to a detection method for detecting the position of an object (floor, wall, etc.) existing in real space from captured images. For example, if the imaging unit 11 is a dual camera (stereo camera), the position of the object (floor, wall, etc.) may be detected by analyzing the captured images using the parallax between the left and right cameras. Also, if the imaging unit 11 is a monocular camera, detection using parallax is possible as with a dual camera by using captured images captured from two locations with the monocular camera shifted by a specified distance. Also, instead of or in addition to this image analysis, the position of an object (floor, wall, etc.) existing in real space may be detected using laser light, sound waves, etc.

[0041] The object placement unit 154 (an example of a placement unit) places an instruction object that instructs the user U to perform an action in a position in the virtual space based on a reference position K1 corresponding to the user U so that the instruction object is visible to the user U. Specifically, the object placement unit 154 places a determination object (see determination objects HF, HB, HR, and HL in FIG. 1) at a determination position in the virtual space that corresponds to the position of the floor. The object placement unit 154 also places a moving object (see moving objects NF, NB, NR, and NL in FIG. 1) at an appearance position in the virtual space at a timing set in advance in accordance with the music, and moves the moving object toward the determination object (changes the placement position). When placing the instruction object (determination object and moving object), the object placement unit 154 updates the virtual space data stored in the storage unit 14 based on coordinate information of the placement position in the virtual space.

[0042] The gaze direction detection unit 155 detects the orientation of the game device 10, i.e., the gaze direction of the user U, based on the detection signal output from the sensor 13. The gaze direction detection unit 155 may detect the orientation of the game device 10, i.e., the gaze direction of the user U, by analyzing the captured video of real space captured by the imaging unit 11. For example, the gaze direction detection unit 155 may analyze the captured video to detect the position and inclination of an object or an edge of an object, and detect the orientation of the game device 10, i.e., the gaze direction of the user U, based on the detection result. Alternatively, the gaze direction detection unit 155 may detect the position and inclination of an object or an edge of an object in each frame of the captured video to detect differences in the position and inclination of the object or the edge of the object between frames, and detect a change in the orientation of the game device 10, i.e., the gaze direction of the user U, based on the detection result. The gaze direction detection unit 155 may detect the orientation of the game device 10, i.e., the gaze direction of the user U, based on both the detection signal output from the sensor 13 and an analysis of the captured video of real space.

[0043] The display control unit 156 refers to the virtual space data stored in the storage unit 14 and causes the display unit 12 to display a virtual space in which at least an instruction object is placed, in association with the real space. Here, associating the virtual space with the real space includes associating coordinates of the virtual space generated based on the real space with coordinates of the real space. When displaying the virtual space, the display control unit 156 determines the viewpoint position and line of sight direction in the virtual space based on the position and orientation of the game device 10 (HMD) in the real space, i.e., the position and direction of the user U. For example, the display control unit 156 causes the display unit 12 to display an instruction object placed in a range of the virtual space corresponding to the range of the field of view (Fov) (range of the real space) determined by the line of sight direction of the user U detected by the line of sight direction detection unit 155 (see FIG. 1 ).

[0044] The motion detection unit 157 (an example of a detection unit) detects the motion of at least a part of the body of the user U from the captured video. For example, the motion detection unit 157 detects the motion of the feet of the user U playing a dance game. Note that any known technology can be applied as the recognition technology for recognizing at least a part of the body of the user U (i.e., the recognition target) from the captured video. For example, the motion detection unit 157 recognizes the image area of ​​the recognition target from the captured video using feature information of the recognition target (e.g., feature information of the feet). The motion detection unit 157 detects the motion of the recognition target (e.g., foot motion) by extracting and tracking the image area of ​​the recognition target from each frame of the captured video.

[0045] The evaluation unit 158 ​​evaluates the movement of at least a part of the body of the user U detected by the movement detection unit 157 based on the timing and position based on the instruction object arranged in the virtual space. For example, the evaluation unit 158 ​​compares the timing and position at which the moving object reaches the judging object with the timing and position of the user U's foot movement (movement of stepping on the judging object), and evaluates the play based on the movement of the user U. If the evaluation unit 158 ​​can determine that the timing and position of both match based on the comparison result, it adds a score, and if it can determine that they do not match, it does not add a score.

[0046] The evaluation unit 158 ​​may evaluate the play based on the actions of the user U by comparing the positions of the user U's feet with the positions of the judging object at the time when the moving object reaches the judging object.

[0047] [Operation of Pointing Object Placement Processing] Next, a description will be given of the operation of instruction object placement processing for generating a virtual space and placing an instruction object in the dance game processing executed by the CPU 15 of the game device 10. Fig. 5 is a flowchart showing an example of the instruction object placement processing according to this embodiment.

[0048] First, the CPU 15 acquires a captured image of the real space captured by the imaging unit 11 (step S101). For example, before starting to play the dance game, the CPU 15 displays an instruction to the user U to look in a predetermined direction (for example, an instruction to look up, down, left, right, and around) on the display unit 12, and acquires a captured image of the surroundings of the user U in the real space.

[0049] Next, the CPU 15 generates a virtual space corresponding to the real space from the captured image acquired in step S101 (step S103). For example, the CPU 15 detects the positions of objects (floors, walls, etc.) existing in the real space from the captured image. The CPU 15 generates virtual space data of a three-dimensional coordinate space including position information of at least a part of the detected objects (floors, walls, etc.) in a virtual space (three-dimensional coordinate space) having a coordinate origin at a reference position K1 corresponding to the user U. Then, the CPU 15 stores the generated virtual space data in the storage unit 14.

[0050] Next, at or before the start of play of the dance game, CPU 15 places a judgment object (see judgment objects HF, HB, HR, and HL in FIG. 1) at a judgment position based on reference position K1 in the virtual space corresponding to the position of the floor (step S105). When placing the judgment object, CPU 15 adds position information of the placed judgment object to the virtual space data stored in storage unit 14.

[0051] Furthermore, when play of the dance game is started, the CPU 15 determines whether or not there is an appearance trigger for a moving object (step S107). The appearance trigger occurs at a timing set in advance in accordance with the music. If the CPU 15 determines in step S107 that there is an appearance trigger (YES), the process proceeds to step S109.

[0052] In step S109, the CPU 15 places a moving object (one or more of the moving objects NF, NB, NR, and NL in FIG. 1) at an appearance position based on the reference position K1 in the virtual space, and starts moving the moving object toward the judgment position (the position of the judgment object corresponding to each moving object). When placing the moving object, the CPU 15 adds position information of the placed moving object to the virtual space data stored in the storage unit 14. Furthermore, when moving the placed moving object, the CPU 15 updates the position information of the moving object added to the virtual space data stored in the storage unit 14. Then, the process proceeds to step S111. On the other hand, if the CPU 15 determines in step S107 that there is no appearance trigger (NO), the process proceeds to step S111 without performing the process of step S109.

[0053] In step S111, the CPU 15 determines whether the moving object has reached the determination position. The CPU 15 erases the moving object that has been determined to have reached the determination position (YES) in step S111 from the virtual space (step S113). When erasing the moving object from the virtual space, the CPU 15 deletes the position information of the moving object to be erased from the virtual space data stored in the storage unit 14.

[0054] On the other hand, if the CPU 15 determines in step S111 that the moving object has not reached the determination position (NO), the CPU 15 continues to move the moving object gradually toward the determination position (step S115). When moving the placed moving object, the CPU 15 updates the position information of the moving object to be moved in the virtual space data stored in the storage unit 14.

[0055] Next, the CPU 15 determines whether the dance game has ended (step S117). For example, the CPU 15 determines that the dance game has ended when the music being played has ended. If the CPU 15 determines that the dance game has not ended (NO), the process returns to step S107. On the other hand, if the CPU 15 determines that the dance game has ended (YES), the CPU 15 ends the instruction object placement process.

[0056] The order in which the judgment object and the first appearing moving object are placed may be simultaneous, the judgment object may be placed first, or conversely, the judgment object may be placed later (until the first appearing moving object reaches the judgment position).

[0057] [Operation of the instruction object display process] Next, a description will be given of the operation of an instruction object display process for displaying an instruction object placed in the virtual space in the dance game process executed by the CPU 15 of the game device 10. Fig. 6 is a flowchart showing an example of the instruction object display process according to this embodiment.

[0058] The CPU 15 detects the line of sight of the user U (the direction of the game device 10) based on the detection signal output from the sensor 13 (step S201).

[0059] CPU 15 refers to the virtual space data stored in storage unit 14 and causes display unit 12 to display a virtual space corresponding to the range of the field of view (Fov) (range of real space) based on the line of sight direction detected in step S201. For example, CPU 15 causes display unit 12 to display instruction objects (determination object and moving object) arranged in the range of the virtual space corresponding to the range of the field of view (Fov) based on the line of sight direction (step S203). As a result, the moving objects are displayed on display unit 12 at a preset timing in sync with the music.

[0060] Next, the CPU 15 determines whether the dance game has ended (step S205). For example, the CPU 15 determines that the dance game has ended when the music being played has ended. If the CPU 15 determines that the dance game has not ended (NO), the process returns to step S201. On the other hand, if the CPU 15 determines that the dance game has ended (YES), the CPU 15 ends the instruction object display process.

[0061] [Play Evaluation Processing Operation] Next, a description will be given of the operation of the play evaluation process for evaluating a play based on the movement of at least a part of the body of the user U in the dance game processing executed by the CPU 15 of the game device 10. Fig. 7 is a flowchart showing an example of the play evaluation process according to this embodiment.

[0062] The CPU 15 acquires a captured image of the real space captured by the imaging unit 11 (step S301). Next, the CPU 15 detects a movement of at least a part of the body of the user U from the captured image acquired in step S301 (step S303). For example, the CPU 15 detects the movement of the feet of the user U playing a dance game.

[0063] Then, CPU 15 evaluates the movement of at least a part of the body of user U (for example, a foot) detected in step S303 based on the timing and position based on the instruction object arranged in the virtual space (step S305). For example, CPU 15 compares the timing and position at which the moving object reaches the judging object with the timing and position of user U's foot movement (movement of stepping on the judging object), and evaluates the play based on user U's foot movement.

[0064] Furthermore, CPU 15 updates the game score based on the evaluation result in step S305 (step S307). For example, CPU 15 adds a score if it can determine that the timing and position at which the moving object reaches the judging object coincides with the timing and position of the user U's foot action (action of stepping on the judging object), and does not add a score if it can determine that they do not coincide.

[0065] Next, the CPU 15 determines whether the dance game has ended (step S309). For example, the CPU 15 determines that the dance game has ended when the music being played has ended. If the CPU 15 determines that the dance game has not ended (NO), the process returns to step S301. On the other hand, if the CPU 15 determines that the dance game has ended (YES), the play evaluation process ends.

[0066] Summary of the first embodiment As described above, the game device 10 according to the present embodiment is worn on the head of the user U, and outputs images visible to the user U. The game device 10 (an example of a video output device) also executes the processing of a game that can be played using the game device 10, which is an example of a video output device and allows the user U to view real space. For example, the game device 10 acquires captured video of real space and generates a virtual space corresponding to the real space from the acquired captured video. The game device 10 then places an instruction object that instructs the user U to perform a movement in the virtual space at a position based on a reference position K1 corresponding to the user U so that the instruction object is visible to the user, and displays the virtual space in which at least the instruction object is placed in association with the real space. The game device 10 also detects a movement of at least a part of the user U's body from the acquired captured video and evaluates the detected movement based on the timing and position of the instruction object placed in the virtual space.

[0067] As a result, in the game processing in which the user U's actions are evaluated based on the timing and position of the instruction object that instructs the user U's actions, the game device 10 is worn on the head, and the instruction object is associated with real space so that it can be seen by the user U, so that the user can be guided in what actions to take with a simple configuration, allowing for more intuitive play.

[0068] For example, the reference position K1 is a first reference position in the virtual space corresponding to the position of the user U wearing the game device 10 (an example of a video output device), and is based on the position of the see-through HMD in the virtual space. For example, the reference position K1 is a position in the virtual space corresponding to the position of the user U in the real space (the position of the see-through HMD), and is defined as the coordinate origin of the virtual space (three-dimensional coordinate space).

[0069] This allows the game device 10 to display instruction objects in correspondence with the real space based on the position of the user U playing the game, thereby making the instructions to the user U feel more real and enabling more intuitive gameplay.

[0070] Furthermore, the game device 10 moves an instruction object (for example, a moving object) placed at a predetermined position (appearance position) in the virtual space toward a predetermined determination position (for example, the position of a determination object). Then, the game device 10 evaluates the movement of at least a part of the body (for example, a foot) of the user U detected from the captured video, based on the timing at which the instruction object (for example, the moving object) moving in the virtual space reaches the determination position and the determination position.

[0071] This allows the game device 10 to use the captured video to evaluate whether or not the user U has performed the action as instructed.

[0072] Note that, in the game device 10, the user U can only view instruction objects within a range of the field of view based on the user's line of sight, and therefore cannot simultaneously view instruction objects in the front, back, left, and right directions (360° around the user U). Therefore, the game device 10 may limit the positions at which instruction objects are placed to a portion of the virtual space, depending on the orientation of the user U wearing the game device 10 (an example of a video output device). For example, the game device 10 may place only instruction objects in the front, right, and left directions, and not place any instruction objects behind, based on the orientation of the user U (reference position K1) at the time of initialization.

[0073] As a result, the game device 10 does not issue instructions to perform actions outside the range of the user U's field of view (for example, behind), so the user U can play without worrying about what is outside the range of the field of view (for example, behind) during play. Therefore, the game device 10 can prevent the difficulty level of play from becoming too high.

[0074] Furthermore, when the game device 10 restricts the positions at which instruction objects are placed to a portion of the virtual space depending on the orientation of the user U, the game device 10 may change the restricted direction during gameplay depending on the orientation of the user U. For example, when the user U is facing forward, the game device 10 may only place instruction objects in front, right, and left of the user U (reference position K1), and may not place any instruction objects behind the user U. When the user U faces right, the game device 10 may only place instruction objects in front, right, and left of the user U (reference position K1) after facing right (to the right, front, and back before facing right), and may not place any instruction objects behind the user U (to the left before facing right). Similarly, when the user U faces left or backward, the game device 10 may not place any instruction objects in the opposite direction (to the right or front before changing orientation).

[0075] As a result, the game device 10 follows the change in the orientation of the user U and does not always issue instructions to perform actions outside the range of the user U's field of view, thereby making it possible to reduce the difficulty of playing the game.

[0076] [Second embodiment] Next, a second embodiment of the present invention will be described. In the example described in the first embodiment, the instruction objects that the user U can actually see are limited to those that are located within the user U's field of view based on the user U's line of sight among the instruction objects located in the virtual space. Therefore, for example, if an instruction object is located behind the user U, it may be difficult for the user U to recognize it. While this difficulty can be utilized to enhance the game's appeal, it may be difficult for beginners and others to play. In the first embodiment, it was described that the difficulty may be reduced by limiting the location of the instruction object to a portion of the virtual space depending on the user U's orientation. However, this configuration reduces the variety of actions that can be instructed to the user U during play. Furthermore, when the user U actually plays the game, they must play while looking at the user U's feet and the instruction object, which are located below the user U. This may have a negative impact on the user U's physical movements, making it difficult for the user U to dance. Therefore, in this embodiment, the above-mentioned concerns are addressed by using mirrors.

[0077] 8 is a diagram showing an overview of game processing by the game device according to this embodiment. This diagram shows an overview of a play situation in which a user U plays a dance game using the game device 10A according to this embodiment. Like FIG. 1, this diagram shows in one diagram the correspondence between the real space in which the user U is included and the virtual space in which the instruction object is included, and is different from the play screen that the user U can see while playing.

[0078] In the illustrated example, the user U is playing a dance game in a position facing a mirror MR. As in FIG. 1, instruction objects (determination objects and moving objects) are arranged around the user U in the virtual space. Furthermore, the user U is reflected in the mirror MR opposite the user U. Here, the virtual image of the user U reflected in the mirror MR is referred to as a "user image UK." The game device 10A detects a user image UK corresponding to the user U from a captured image captured in the direction of the mirror MR, and also places instruction objects around the detected user image UK, as if the instruction objects arranged around the user U were reflected in the mirror MR.

[0079] FIG. 9 is a diagram illustrating the definition of spatial coordinates in the virtual space and the position of the user image UK according to this embodiment. This diagram adds the position of the user image UK detected from the captured video to the definition of spatial coordinates in the virtual space shown in FIG. 2. A reference position K2 (an example of a second reference position) corresponding to the position of the user image UK in the virtual space is detected at a position at the end (rear) of the mirror MR in the X-axis direction (line of sight direction) relative to a reference position K1 (e.g., the coordinate origin) corresponding to the position of the user U. For example, if the position where the mirror surface of the mirror MR intersects with the X-axis is defined as mirror surface position M1, then reference position K2 is detected at a position in the X-axis direction where the distance from reference position K1 to mirror surface position M1 is the same as the distance from mirror surface position M1 to reference position K2. Note that reference position K2 may be a position corresponding to the center of the head of the user image UK or a position corresponding to the center of gravity of the user image UK, and can be defined at any position.

[0080] Returning to FIG. 8, the game device 10A detects the image area (outline) and distance of the user image UK from the captured image, and detects a reference position K2 corresponding to the position of the user image UK in the virtual space, separately from a reference position K1 corresponding to the position of the user U. The game device 10A then places instruction objects around the reference positions K1 and K2, respectively, based on the reference positions K1 and K2. At this time, because the user image UK is a virtual image of the user U reflected in the mirror MR, its front-to-back orientation relative to the user U is reversed. Therefore, the game device 10A places the instruction objects to be placed around the reference position K2 (the position of the user image UK) with the front-to-back orientation (front-to-back positional relationship in spatial coordinates) reversed from that of the instruction objects to be placed around the reference position K1 (the position of the user U).

[0081] For example, if the direction on the X axis from reference position K1 to reference position K2 is defined as the positive direction, then determination object HF and moving object NF that are positioned forward relative to reference position K1 (the position of the user U) are positioned in the positive direction of the X axis relative to reference position K1. In contrast, determination object HF' and moving object NF' that are positioned forward relative to reference position K2 (the position of the user image UK) are positioned in the negative direction of the X axis relative to reference position K2. Furthermore, determination object HB and moving object NB that are positioned behind reference position K1 (the position of the user U) are positioned in the negative direction of the X axis relative to reference position K1. In contrast, determination object HB' and moving object NB' that are positioned behind reference position K2 (the position of the user image UK) are positioned in the positive direction of the X axis relative to reference position K2.

[0082] On the other hand, the determination object HR and moving object NR arranged to the right of the reference position K1 (the position of the user U), and the determination object HR' and moving object NR' arranged to the right of the reference position K2 (the position of the user image UK), are arranged in the same direction (e.g., the positive direction) of the Y axis with respect to their respective reference positions. Furthermore, the determination object HL and moving object NL arranged to the left of the reference position K1 (the position of the user U), and the determination object HL' and moving object NL' arranged to the left of the reference position K2 (the position of the user image UK), are arranged in the same direction (e.g., the negative direction) of the Y axis with respect to their respective reference positions. Furthermore, the upward and downward positional relationships between the instruction object arranged to the reference position K1 and the instruction object arranged to the reference position K2 are also the same.

[0083] [Configuration of game device 10A] The game device 10A according to this embodiment may be a device including an optically transparent HMD, or a device including a video see-through HMD, similar to the game device 10 described in the first embodiment. Here, similar to the first embodiment, the game device 10A will be described as an optically transparent HMD. The hardware configuration of the game device 10A is similar to the configuration example shown in FIG. 3, and therefore description thereof will be omitted.

[0084] FIG. 10 is a block diagram showing an example of the functional configuration of a game device 10A according to this embodiment. The illustrated game device 10A includes a control unit 150A as a functional configuration realized by the CPU 15 executing a program stored in the storage unit 14. The control unit 150A includes an image acquisition unit 151, a virtual space generation unit 152, a user image detection unit 153A, an object placement unit 154A, a gaze direction detection unit 155, a display control unit 156, a motion detection unit 157, and an evaluation unit 158. In this figure, components corresponding to those in FIG. 4 are denoted by the same reference numerals, and their description will be omitted as appropriate. The functional configuration of the game device 10A differs from the functional configuration of the game device 10 shown in FIG. 4 mainly in that a user image detection unit 153A for detecting a reference position corresponding to a user image UK reflected in the mirror MR is added.

[0085] The user image detection unit 153A detects a user image UK (an example of an image) corresponding to the user U from the captured video acquired by the video acquisition unit 151. For example, the user image UK is detected as a virtual image of the user U reflected in a mirror MR located opposite the user U. This detection requires recognizing that the user image UK is a virtual image of the user U playing a dance game. As a recognition method, for example, an identifiable marker (such as a mark or label) may be attached to the body of the user U or the game device 10A (HMD) worn on the head of the user U, and the user image detection unit 153A may detect this marker from the captured video to recognize that it is a virtual image of the user U. Alternatively, by instructing the user U to perform a specific action (for example, raising and lowering the right hand), the user image detection unit 153A may detect a person performing the action in accordance with the instruction from the captured video to recognize that it is a virtual image of the user U.

[0086] The virtual space generation unit 152 generates, as virtual space data, data of a three-dimensional coordinate space including position information of the user image UK in addition to position information of at least a portion of an object (such as a floor or wall) detected from the captured video. For example, the virtual space generation unit 152 detects the position of an object (such as a floor or wall) existing in real space from the captured video. In addition, the virtual space generation unit 152 detects the position of the user image UK (reference position K2) detected by the user image detection unit 153A. The method for detecting the position of the user image UK may be a detection method that utilizes the parallax of the camera (imaging unit), similar to the method for detecting the position of an object (such as a floor or wall) existing in real space described above. Then, the virtual space generation unit 152 generates, as virtual space data, data of a three-dimensional coordinate space including position information of at least a portion of the detected object (such as a floor or wall) and position information of the reference position K2. Note that, as an example, the coordinate origin of the virtual space (three-dimensional coordinate space) is set to the reference position K1 corresponding to the user U, as in the first embodiment. The virtual space generation unit 152 stores the generated virtual space data in the storage unit 14.

[0087] The object placement unit 154A places an instruction object in the virtual space at a position based on a reference position K1 corresponding to the user U, and also places an instruction object in a position based on a reference position K2 corresponding to the user image UK (see FIGS. 8 and 9). When placing the instruction object in the virtual space at a position based on the reference position K2, the object placement unit 154A reverses the front-to-back orientation of the instruction object with respect to the reference position K2.

[0088] Here, the object placement unit 154A may determine whether the detected user image UK is an image reflected in the mirror MR by instructing the user to perform the above-mentioned specific action (for example, raising and lowering the right hand) and detecting a person performing the action from the captured video. Alternatively, the object placement unit 154A may determine that the detected user image UK is an image reflected in the mirror MR by selecting a pre-selectable mirror mode (a mode in which the user plays while watching their image reflected in the mirror MR).

[0089] For example, when the line of sight of the user U is in the direction of the mirror MR, the display control unit 156 causes the display unit 12 to display an instruction object that is located in a range of the virtual space corresponding to the range of the field of view in the direction of the mirror MR. That is, the display control unit 156 can display an instruction object that is located at a position based on the reference position K2 that corresponds to the user image UK reflected in the mirror MR so that the user U can view it from above.

[0090] The movement detection unit 157 detects a movement of at least a part of the body of the user U by detecting a movement of at least a part of the body of the user image UK reflected in the mirror MR from the captured video.

[0091] The evaluation unit 158 ​​evaluates the movement of at least a part of the body of the user image UK (user image UK reflected in the mirror MR) detected by the movement detection unit 157, using an instruction object placed at a position based on the reference position K2 corresponding to the user image UK. Specifically, the evaluation unit 158 ​​evaluates the movement of at least a part of the body of the user image UK (user image UK reflected in the mirror MR) based on the timing and position based on the instruction object placed at a position based on the user image UK reflected in the mirror MR. In other words, the user U can play while looking in the direction of the mirror MR, without having to look at the user U's feet and instruction object located below.

[0092] 8, instruction objects are placed at both a position based on the reference position K1 corresponding to the user U and a position based on the reference position K2 corresponding to the user image UK, but this is not limiting. For example, when an instruction object is placed at a position based on the reference position K2 corresponding to the user image UK, the object placement unit 154A does not need to place an instruction object at a position based on the reference position K1 corresponding to the user U. In other words, when an instruction object is displayed at a position based on the reference position K2, the instruction object at the position based on the reference position K1 may be hidden. This prevents instruction objects displayed around the user U from obscuring the instruction object displayed in the mirror MR, thereby improving the visibility of the instruction object.

[0093] When the instruction object is placed at a position based on the reference position K2, the object placement unit 154A may display the instruction object at a position based on the reference position K1 in an inconspicuous manner with reduced visibility, for example, by making the object semitransparent or reducing its size. Note that the process of changing the display mode of the instruction object may be performed by the display control unit 156.

[0094] Furthermore, the object placement unit 154A (or the display control unit 156) may make the instruction objects around the user U invisible or semi-transparent only when the mirror MR is within the field of view of the user U, and may display the instruction objects around the user U as usual when the mirror MR is out of the field of view. This allows the instruction objects to be visible even when the mirror MR is out of the field of view (for example, when the user U faces in the opposite direction to the mirror MR).

[0095] [Operation of Pointing Object Placement Processing] Next, the operation of the instruction object placement process for generating a virtual space and placing an instruction object in the dance game process executed by the CPU 15 of the game device 10A will be described. Here, the instruction object placement process for the user U (reference position K1) is the same as the process shown in Fig. 5, so its description will be omitted. Instead, the instruction object placement process for placing an instruction object corresponding to the user image UK (reference position K2) reflected in the mirror MR will be described. Fig. 11 is a flowchart showing an example of the instruction object placement process according to this embodiment.

[0096] First, the CPU 15 acquires a captured image of the real space captured by the imaging unit 11 (step S401). For example, the CPU 15 acquires a captured image including a user image UK (see FIG. 8) reflected in a mirror MR in the line of sight of a user U playing a dance game.

[0097] Next, the CPU 15 detects a virtual image (user image UK) of the user U playing the dance game from the captured video acquired in step S401 (step S403).

[0098] Further, the CPU 15 generates a virtual space corresponding to the real space from the captured image acquired in step S401 (step S405). For example, the CPU 15 detects the positions of objects (floor, wall, etc.) existing in the real space from the captured image and the position (reference position K2) of the user image UK detected in step S403, and generates data of a three-dimensional coordinate space including position information of at least a part of the detected objects (floor, wall, etc.) and position information of the reference position K2 as virtual space data. As an example, the CPU 15 generates virtual space data including position information of at least a part of the detected objects (floor, wall, etc.) and position information of the reference position K2 in a virtual space (three-dimensional coordinate space) with the reference position K1 corresponding to the user U as the coordinate origin. Then, the CPU 15 stores the generated virtual space data in the storage unit 14.

[0099] Next, at or before the start of play of the dance game, CPU 15 places a judgment object (see judgment objects HF', HB', HR', and HL' in FIG. 8) at a judgment position based on reference position K2 in the virtual space corresponding to the position of the floor (step S407). When placing the judgment object, CPU 15 adds position information of the placed judgment object to the virtual space data stored in storage unit 14.

[0100] Furthermore, when play of the dance game is started, the CPU 15 determines whether or not there is an appearance trigger for a moving object (step S409). The appearance trigger occurs at a timing set in advance in accordance with the music. If the CPU 15 determines in step S409 that there is an appearance trigger (YES), the process proceeds to step S411.

[0101] In step S411, the CPU 15 places a moving object (one or more of the moving objects NF', NB', NR', and NL' in FIG. 8) at an appearance position based on the reference position K2 in the virtual space, and starts moving the moving object toward the judgment position (the position of the judgment object corresponding to each moving object). When placing the moving object, the CPU 15 adds position information of the placed moving object to the virtual space data stored in the storage unit 14. Furthermore, when moving the placed moving object, the CPU 15 updates the position information of the moving object added to the virtual space data stored in the storage unit 14. Then, the process proceeds to step S413. On the other hand, if the CPU 15 determines in step S409 that there is no appearance trigger (NO), the process proceeds to step S413 without performing the process of step S411.

[0102] In step S413, the CPU 15 determines whether the moving object has reached the determination position. The CPU 15 erases the moving object that has been determined to have reached the determination position (YES) from the virtual space (step S415). When erasing the moving object from the virtual space, the CPU 15 deletes the position information of the moving object to be erased from the virtual space data stored in the storage unit 14.

[0103] On the other hand, if the CPU 15 determines that the moving object has not reached the determination position (NO), the CPU 15 continues to move the moving object gradually toward the determination position (step S417). When moving the moving object, the CPU 15 updates the position information of the moving object to be moved in the virtual space data stored in the storage unit 14.

[0104] Next, the CPU 15 determines whether the dance game has ended (step S419). For example, the CPU 15 determines that the dance game has ended when the music being played has ended. If the CPU 15 determines that the dance game has not ended (NO), the process returns to step S409. On the other hand, if the CPU 15 determines that the dance game has ended (YES), the CPU 15 ends the instruction object placement process.

[0105] The order in which the judgment object and the first appearing moving object are placed may be simultaneous, the judgment object may be placed first, or conversely, the judgment object may be placed later (until the first appearing moving object reaches the judgment position).

[0106] Summary of the second embodiment As described above, the game device 10A according to this embodiment further detects a user image UK, which is a virtual image (one example of an image) corresponding to the user U, from a captured image of the real space. Then, the game device 10A places an instruction object, which instructs the user U to perform an action, in a position based on the reference position K2 of the user image UK corresponding to the user U in the virtual space so that the instruction object is visible to the user.

[0107] Thus, when worn on the head, the game device 10A can display instruction objects instructing the user U to perform actions around the virtual image (user image UK) of the user U reflected in the mirror MR, for example. Therefore, with a simple configuration, the game device 10A can guide the user to perform actions more intuitively, enabling play. For example, the game device 10A allows the user U to simultaneously view instruction objects displayed around the user image UK (for example, front, back, left, and right) from a bird's-eye view without restricting the placement of the instruction objects to a portion of the virtual space. This can diversify the types of actions to instruct the user U during play. Furthermore, the game device 10A allows the user U to play while looking in the direction of the mirror MR facing the user U, without having to look at his or her own feet and instruction objects located below, thereby making it possible to ensure that dancing is not difficult. Furthermore, the game device 10A can display instruction objects around the user U playing the game and around the virtual image (user image UK) of the user reflected in the mirror MR, for example, allowing the user to play by arbitrarily selecting which of the instruction objects is easier to play.

[0108] The mirror MR may be something other than a mirror as long as it has the effect of a mirror (specular reflection). For example, when the user U plays the game in a brightly lit room at night (when it is dark outside) and faces a window, the window may be used as the mirror MR, and a virtual image of the user U reflected on the window may be used.

[0109] Furthermore, when placing the instruction object at a position (around the user image UK) based on the reference position K2 in the virtual space, the game device 10A inverts the front-to-back orientation with respect to the reference position K2. This allows the game device 10A to display the instruction object in accordance with the orientation of the user image UK reflected in the mirror MR, thereby providing guidance on the actions the user should take to enable more intuitive gameplay.

[0110] When placing an instruction object at a position based on the reference position K2 in the virtual space (around the user image UK), the game device 10A may reduce the visibility of the instruction object to be placed at a position based on the reference position K1 (around the user U). For example, the game device 10A may make the instruction object to be placed at a position based on the reference position K1 translucent or reduced in size, thereby reducing its visibility and making it less noticeable. When placing an instruction object at a position based on the reference position K2 in the virtual space (around the user image UK), the game device 10A may not necessarily place the instruction object at a position based on the reference position K1 (around the user U).

[0111] This allows the game device 10A to prevent the instruction object displayed in the mirror MR from being hidden by the instruction objects displayed around the user U, thereby improving the visibility of the instruction object.

[0112] In the present embodiment, the user U places an instruction object in the virtual space in association with the user image UK (the user's own virtual image) reflected in the mirror MR. However, instead of the mirror MR, the instruction object may be placed in the virtual space in association with an image of the user U displayed on a monitor (display device). For example, the game device 10A further includes a camera (imaging device) that captures an image of the user U in real space and a monitor (display device) that displays the captured image in real time, on the side opposite the user U, and displays the image of the user U captured by the camera on the monitor. The game device 10A may then detect an image of the user U from the image displayed on the monitor instead of the user image UK (the user's own virtual image) reflected in the mirror MR, and place an instruction object in the virtual space in association with the detected image of the user U. In this case, the position of the image of the user U displayed on the monitor serves as the reference position. The image of the user U displayed on the monitor is oriented in the opposite direction to the user image UK of the user U reflected in the mirror MR. Therefore, the game device 10A reverses the orientation of the instruction object placed in association with the image of the user U displayed on the monitor, not only forward and backward, but also left and right, relative to the instruction object placed in association with the user U.

[0113] Furthermore, the game mode in this embodiment in which a command object is placed using a mirror MR, the game mode in which a command object is placed using the monitor, and the game mode described in the first embodiment in which a command object is placed without using either a mirror or a monitor (modes of game processing by the game device 10) each have different display modes for the command object (such as the reference position when placing the command object and whether or not the command object is flipped front to back or left to right). Therefore, when two or more of these game modes are available (for example, when the configuration of the game device 10 is combined with the configuration of the game device 10A), the user may be allowed to select which mode to use before starting the dance game. This allows for smooth detection of the user image UK reflected in the mirror MR and the image of the user U displayed on the monitor, and also reduces misrecognition.

[0114] [Third embodiment] Next, a third embodiment of the present invention will be described. In the above first and second embodiments, an example was described in which the game device 10 (10A) is configured as a complete device as a see-through HMD, but it may also be configured as a separate device connected to the see-through HMD via a wired or wireless connection.

[0115] FIG. 12 is a block diagram showing an example of the hardware configuration of a game system including a game device 10C according to this embodiment. The game device 10C does not include a video output device. The illustrated game system 1C includes the game device 10C and an HMD 20C as a video output device. For example, the HMD 20C is a see-through HMD.

[0116] The HMD 20C includes an imaging unit 21C, a display unit 22C, a sensor 23C, a storage unit 24C, a CPU 25C, a communication unit 26C, and a sound output unit 27C. The imaging unit 21C, the display unit 22C, the sensor 23C, and the sound output unit 27C correspond to the imaging unit 11, the display unit 12, the sensor 13, and the sound output unit 17 shown in FIG. 3, respectively. The storage unit 24C temporarily stores data of the captured image captured by the imaging unit 21C, display data acquired from the game device 10C, and the like. The storage unit 24C also stores programs and the like necessary for controlling the HMD 20C. The CPU 25C functions as a control center that controls each unit included in the HMD 20C. The communication unit 26C communicates with the game device 10C via wired or wireless communication. The HMD 20C transmits the captured image captured by the imaging unit 21C, the detection signal of the sensor 23C, and the like to the game device 10C via the communication unit 26C. The HMD 20C also acquires display data, sound data, and the like of the dance game from the game device 10C via the communication unit 26C.

[0117] The game device 10C includes a storage unit 14C, a CPU 15C, and a communication unit 16C. The storage unit 14C stores the dance game program and data, data on the generated virtual space, and the like. The CPU 15C functions as a control center that controls each component of the game device 10C. For example, the CPU 15C executes the game program stored in the storage unit 14C to perform game processing, such as generating a virtual space corresponding to real space from captured video, placing a command object in the generated virtual space, and detecting the user's movements and evaluating them based on the timing and position of the command object. The communication unit 16C communicates with the HMD 20C via wired or wireless communication. The game device 10C acquires captured video captured by the imaging unit 21C of the HMD 20C and detection signals from the sensor 23C via the communication unit 16C. The game device 10C also transmits display data, sound data, and the like of the dance game to the HMD 20C via the communication unit 16C.

[0118] 13 is a block diagram showing an example of the functional configuration of a game device 10C according to this embodiment. The illustrated game device 10C includes a control unit 150C as a functional configuration realized by a CPU 15C executing a program stored in a storage unit 14C. The control unit 150C has the same configuration as the control unit 150 shown in FIG. 4 or the control unit 150A shown in FIG. 10, except that the control unit 150C exchanges data with each unit (such as the imaging unit 21C, the display unit 22C, the sensor 23C, and the sound output unit 27) included in the HMD 20C via a communication unit 16C.

[0119] In this way, the game device 10C may be configured as a separate device that communicates with the external device, the HMD 20. Note that the game device 10C may be, for example, a smartphone, a PC (Personal Computer), a home game console, or the like.

[0120] [Fourth embodiment] Next, a fourth embodiment of the present invention will be described. In the above first to third embodiments, an HMD that is worn on the head is used. However, in this embodiment, an HMD is not used.

[0121] FIG. 14 is a diagram illustrating an overview of game processing by a game device according to this embodiment. This diagram shows an overview of a play situation in which a user U plays a dance game using a game device 10D. The illustrated game device 10D is an example in which a smartphone is used. In this embodiment, instruction objects arranged in a virtual space are displayed on the display unit 12D or monitor 30D of the game device 10D in association with an image of the user U captured by the front camera 11DA of the game device 10D, allowing the user to play intuitively. The monitor 30D is an external display unit (display device) that can be connected to the game device 10D via a wired or wireless connection. For example, the monitor 30D may have a larger screen than the display unit 12D provided on the game device 10D.

[0122] The game device 10D recognizes the image area of ​​the user U from the captured image of the user U. The game device 10D then defines a reference position K3 corresponding to the position of the user U in the virtual space, generates virtual space (XYZ three-dimensional space) data in which an instruction object is placed at a position based on the reference position K3, and displays the virtual space data superimposed on the captured image. Note that the reference position K3 may be a position corresponding to the center of the user U's head or the center of gravity of the user U, and can be defined at any position.

[0123] In the illustrated example, the instruction object is also displayed on the display unit 12D of the game device 10D, but by also displaying it on the monitor 30D, which has a larger screen than the game device 10D, the visibility of the instruction object for the user U can be improved. The following description will be given with reference to the display screen of the monitor 30D. The user image UV shows an image of the user U included in the captured image. Around the user image UV, an image is displayed in which the instruction object, which is placed in a position based on the reference position K3 of the user U in the virtual space, is superimposed on the captured image.

[0124] For example, the image captured by front camera 11DA can be an image that is flipped left and right like a mirror. On the right side of the user image UV as viewed from the screen of monitor 30D, a determination object HR and a moving object NR that instruct the user U to move to the right are displayed, and on the left side of the user image UV, a determination object HL and a moving object NL that instruct the user U to move to the left are displayed. Furthermore, on the front side of the user image UV as viewed from the screen of monitor 30D, a determination object HF and a moving object NF that instruct the user U to move forward are displayed, and on the back side of the user image UV, a determination object HB and a moving object NB that instruct the user U to move backward are displayed.

[0125] In this way, in this embodiment, instruction objects placed in the virtual space can be displayed in correspondence with the image of the user U, similar to the case where instruction objects are displayed around the user image UK reflected in the mirror MR shown in Figure 8, making it possible to guide the user on the actions they should take so that they can play intuitively.

[0126] The instruction object may be displayed on either the game device 10D or the monitor 30D.

[0127] [Hardware Configuration of Game Device 10D] The hardware configuration of the game device 10D will be described with reference to FIG. 15 is a block diagram showing an example of the hardware configuration of a game device 10D according to this embodiment. The game device 10D includes two imaging units, a front camera 11DA and a rear camera 11DB, a display unit 12D, a sensor 13D, a storage unit 14D, a CPU 15D, a communication unit 16D, a sound output unit 17D, and a video output unit 18D.

[0128] Front camera 11DA is provided on the surface (front side) of game device 10D on which display unit 12D is provided, and captures an image in the direction facing display unit 12D. Back camera 11DB is provided on the surface (back side) of game device 10D opposite to the surface on which display unit 12D is provided, and captures an image in the direction facing the back side.

[0129] Display unit 12D is configured to include a liquid crystal display, an organic EL display, etc. For example, display unit 12D may be configured as a touch panel that detects a touch operation on the display screen.

[0130] The sensor 13D is a sensor that outputs a detection signal related to the direction of the game device 10D. For example, the sensor 13D may include one or more of a gyro sensor, an acceleration sensor, a tilt sensor, a geomagnetic sensor, and the like.

[0131] The storage unit 14D includes, for example, an EEPROM, a ROM, a Flash ROM, a RAM, etc., and stores the program and data for this dance game, data for the generated virtual space, etc.

[0132] CPU 15D functions as a control center that controls each unit of game device 10D. For example, CPU 15D executes a game program stored in storage unit 14D to perform game processing, and as described with reference to Fig. 14, executes processing such as superimposing and displaying an instruction object placed in a virtual space on a captured image captured by user U.

[0133] The communication unit 16D includes a communication device that performs wireless communication such as Bluetooth (registered trademark) or Wi-Fi (registered trademark).

[0134] The sound output unit 17D outputs the sounds of the music played in the dance game, the sound effects of the game, etc. For example, the sound output unit 17 is configured to include a speaker, a phone terminal to which earphones, headphones, etc. are connected, etc.

[0135] Video output unit 18D is configured to include a video output terminal that outputs the video to be displayed on display unit 12D to an external display device (for example, monitor 30D shown in FIG. 14). The video output terminal may be a multipurpose terminal that also includes outputs other than video output, or may be a terminal dedicated to video output.

[0136] [Functional Configuration of Game Device 10D] Next, the functional configuration of the game device 10D will be described with reference to FIG. 16 is a block diagram showing an example of the functional configuration of a game device 10D according to this embodiment. The illustrated game device 10D includes a control unit 150D as a functional configuration realized by a CPU 15D executing a program stored in a storage unit 14D. The control unit 150D includes an image acquisition unit 151D, a virtual space generation unit 152D, a user detection unit 153D, an object placement unit 154D, a display control unit 156D, a motion detection unit 157D, and an evaluation unit 158D.

[0137] Video acquisition unit 151D (an example of an acquisition unit) acquires a captured video of the real space captured by front camera 11DA. For example, as shown in FIG. 14, video acquisition unit 151D acquires a captured video including user U playing a dance game.

[0138] The virtual space generation unit 152D (an example of a generation unit) generates a virtual space corresponding to the real space from the captured video acquired by the video acquisition unit 151D. For example, the virtual space generation unit 152D detects the positions of objects (floors, walls, etc.) present in the real space from the captured video, and generates three-dimensional coordinate space data including position information of at least some of the detected objects (floors, walls, etc.) as virtual space data. As an example, when the virtual space generation unit 152D is initialized at the start of play of this dance game, it defines a reference position K3 corresponding to the user U detected from the captured video by the user detection unit 153D as the coordinate origin of the virtual space (three-dimensional coordinate space of XYZ), and generates virtual space data. During play, the reference position K3 (coordinate origin) and the X, Y, and Z axes are fixed. The virtual space generation unit 152D stores the generated virtual space data in the storage unit 14D.

[0139] The user detection unit 153D detects an image of the user U from the captured image acquired by the image acquisition unit 151D. This detection requires recognizing that the image of a person detected from the captured image is an image of the user U playing the dance game. As a method of recognizing that the image is of the user U, for example, an identifiable marker (such as a mark or a label) may be attached to the body of the user U, and the user detection unit 153D may recognize that the image is of the user U by detecting this marker from the captured image. Alternatively, the user detection unit 153D may recognize that the image is of the user U by instructing the user U to perform a specific action (for example, raising and lowering the right hand), and by detecting from the captured image a person making an action in accordance with the instruction.

[0140] The object placement unit 154D (an example of a placement unit) places an instruction object in the virtual space at a position based on the reference position K3 corresponding to the user U so that the instruction object is visible to the user U. Specifically, the object placement unit 154D places a determination object (see determination objects HF, HB, HR, and HL in FIG. 14) at a determination position in the virtual space that corresponds to the position of the floor. The object placement unit 154D also places a moving object (see moving objects NF, NB, NR, and NL in FIG. 14) at an appearance position in the virtual space at a timing set in advance in accordance with the music, and moves the moving object toward the determination object (changes the placement position). When placing the instruction object (determination object and moving object), the object placement unit 154D updates the virtual space data stored in the storage unit 14D based on coordinate information of the placement position in the virtual space.

[0141] The display control unit 156D generates a composite video by combining the captured video acquired by the video acquisition unit 151D with the video of the instruction object placed in the virtual space by the object placement unit 154D. Then, the display control unit 156D causes the generated composite video to be displayed on the display unit 12D. The display control unit 156D also outputs the generated composite video from the video output unit 18D. For example, the display control unit 156D causes the generated composite video to be displayed on the display unit 12D after being flipped horizontally. Similarly, the display control unit 156D also flips the generated composite video horizontally and outputs it from the video output unit 18D.

[0142] The movement detection unit 157D (an example of a detection unit) detects the movement of at least a part of the body of the user U from the captured video acquired by the video acquisition unit 151D. For example, the movement detection unit 157D detects the movement of the feet of the user U playing a dance game. The movement detection unit 157D detects the movement of the feet by extracting and tracking the image area of ​​the feet from each frame of the captured video.

[0143] The evaluation unit 158D evaluates the movement of at least a part of the body of the user U detected by the movement detection unit 157D based on the timing and position based on the instruction object arranged in the virtual space. For example, the evaluation unit 158D compares the timing and position at which the moving object reaches the judging object with the timing and position of the user U's foot movement (movement of stepping on the judging object), and evaluates the play based on the movement of the user U. If the evaluation unit 158D can determine that the timing and position of both match based on the comparison result, it adds a score, and if it can determine that they do not match, it does not add a score.

[0144] The evaluation unit 158D may evaluate the play based on the actions of the user U by comparing the positions of the user U's feet with the positions of the judging object at the time when the moving object reaches the judging object.

[0145] [Operation of Pointing Object Placement Processing] Next, an operation of the instruction object placement process for generating a virtual space and placing an instruction object in the dance game process executed by the CPU 15D of the game device 10D will be described. Fig. 17 is a flowchart showing an example of the instruction object placement process according to this embodiment.

[0146] First, the CPU 15D acquires a captured image of the real space captured by the front camera 11DA (step S501). For example, the CPU 15 acquires a captured image including a user U playing a dance game, as shown in FIG.

[0147] Next, the CPU 15D detects an image of the user U playing the dance game from the captured image acquired in step S501 (step S503).

[0148] Next, the CPU 15D generates a virtual space corresponding to the real space from the captured video acquired in step S501 (step S505). For example, the CPU 15D detects the positions of objects (floors, walls, etc.) present in the real space from the captured video, and generates three-dimensional coordinate space data including position information of at least a portion of the detected objects (floors, walls, etc.) as virtual space data. As an example, the CPU 15D generates virtual space data including position information of at least a portion of the detected objects (floors, walls, etc.) in a virtual space (three-dimensional coordinate space) whose coordinate origin is a reference position K3 corresponding to the user U detected from the captured video by the user detection unit 153D. Then, the CPU 15D stores the generated virtual space data in the storage unit 14D.

[0149] Next, at or before the start of play of the dance game, the CPU 15D places a judgment object (see judgment objects HF, HB, HR, and HL in FIG. 14) at a judgment position based on a reference position K3 in the virtual space corresponding to the position of the floor (step S507). When placing the judgment object, the CPU 15D adds position information of the placed judgment object to the virtual space data stored in the storage unit 14D.

[0150] Furthermore, when play of the dance game is started, the CPU 15D determines whether or not there is an appearance trigger for a moving object (step S509). The appearance trigger occurs at a timing set in advance in accordance with the music. If the CPU 15D determines in step S509 that there is an appearance trigger (YES), the process proceeds to step S511.

[0151] In step S511, the CPU 15D places a moving object (one or more of the moving objects NF, NB, NR, and NL in FIG. 14) at an appearance position based on the reference position K3 in the virtual space, and starts moving the moving object toward the judgment position (the position of the judgment object corresponding to each moving object). When placing the moving object, the CPU 15D adds position information of the placed moving object to the virtual space data stored in the storage unit 14D. Furthermore, when moving the placed moving object, the CPU 15D updates the position information of the moving object added to the virtual space data stored in the storage unit 14D. Then, the process proceeds to step S513. On the other hand, if the CPU 15D determines in step S509 that there is no appearance trigger (NO), the process proceeds to step S513 without performing the process of step S511.

[0152] In step S513, the CPU 15D determines whether the moving object has reached the determination position. The CPU 15D erases the moving object that has been determined to have reached the determination position (YES) from the virtual space (step S515). When erasing the moving object from the virtual space, the CPU 15D deletes the position information of the moving object to be erased from the virtual space data stored in the storage unit 14D.

[0153] On the other hand, if the CPU 15D determines that the moving object has not reached the determination position (NO), the CPU 15D continues to move the moving object gradually toward the determination position (step S517). When moving the moving object, the CPU 15D updates the position information of the moving object to be moved in the virtual space data stored in the storage unit 14D.

[0154] Next, the CPU 15D determines whether the dance game has ended (step S519). For example, the CPU 15D determines that the dance game has ended when the music being played has ended. If the CPU 15D determines that the dance game has not ended (NO), the process returns to step S509. On the other hand, if the CPU 15D determines that the dance game has ended (YES), the CPU 15D ends the instruction object placement process.

[0155] The order in which the judgment object and the first appearing moving object are placed may be simultaneous, the judgment object may be placed first, or conversely, the judgment object may be placed later (until the first appearing moving object reaches the judgment position).

[0156] [Operation of the instruction object display process] Next, an operation of an instruction object display process for displaying an instruction object placed in a virtual space in the dance game process executed by the CPU 15D of the game device 10D will be described. In this embodiment, the instruction object is displayed as a composite image in which the instruction object is superimposed on a captured image of the user U. FIG. 18 is a flowchart showing an example of the instruction object display process according to this embodiment.

[0157] The CPU 15D acquires a captured image of the real space captured by the front camera 11DA, and also acquires virtual space data from the storage unit 14D (step S601).

[0158] Then, CPU 15D generates a composite image by combining the acquired captured image with the instruction object included in the virtual space data, and displays the generated composite image on display unit 12D (step S603). CPU 15D also outputs the generated composite image to image output unit 18D, and displays it on monitor 30D connected to video output unit 18D (step S603). As a result, a composite image in which the instruction object is superimposed on the captured image of user U is displayed on display unit 12D and monitor 30D in real time. Note that CPU 15D may display the composite image on either display unit 12D or monitor 30D.

[0159] Next, the CPU 15D determines whether the dance game has ended (step S605). For example, the CPU 15D determines that the dance game has ended when the music being played has ended. If the CPU 15D determines that the dance game has not ended (NO), the process returns to step S601. On the other hand, if the CPU 15D determines that the dance game has ended (YES), the CPU 15D ends the instruction object display process.

[0160] [Play Evaluation Processing Operation] Next, a description will be given of the operation of the play evaluation process for evaluating a play based on the movement of at least a part of the body of the user U in the dance game process executed by the CPU 15D of the game device 10D. Fig. 19 is a flowchart showing an example of the play evaluation process according to this embodiment.

[0161] The CPU 15D acquires a captured image of the real space captured by the front camera 11DA (step S701). Next, the CPU 15D detects a movement of at least a part of the body of the user U from the captured image acquired in step S701 (step S703). For example, the CPU 15D detects the movement of the feet of the user U playing a dance game.

[0162] Then, the CPU 15D evaluates the movement of at least a part of the body of the user U (for example, a foot) detected in step S703 based on the timing and position based on the instruction object arranged in the virtual space (step S705). For example, the CPU 15D compares the timing and position at which the moving object reaches the judging object with the timing and position of the foot movement of the user U (the movement of stepping on the judging object) to evaluate the play based on the foot movement of the user U.

[0163] Furthermore, CPU 15D updates the game score based on the evaluation result in step S705 (step S707). For example, CPU 15D adds a score if it can determine that the timing and position at which the moving object reaches the judgment object matches the timing and position of the user U's foot action (action of stepping on the judgment object), and does not add a score if it can determine that they do not match.

[0164] Next, the CPU 15D determines whether the dance game has ended (step S709). For example, the CPU 15D determines that the dance game has ended when the music being played has ended. If the CPU 15D determines that the dance game has not ended (NO), the process returns to step S701. On the other hand, if the CPU 15D determines that the dance game has ended (YES), the play evaluation process ends.

[0165] [Summary of the Fourth Embodiment] As described above, the game device 10D according to this embodiment acquires captured video of a real space and generates a virtual space corresponding to the real space from the acquired captured video. The game device 10D then places an instruction object that instructs the user U to perform a movement in the generated virtual space at a position based on a reference position K3 corresponding to the user so that the instruction object is visible to the user U, and displays a composite image obtained by combining the captured video with an image of the instruction object placed in the virtual space on the display unit 12D (an example of a display unit). The game device 10D may also display the composite image on the monitor 30D (an example of a display unit). The game device 10D also detects a movement of at least a part of the user U's body from the acquired captured video and evaluates the detected movement based on the timing and position of the instruction object placed in the virtual space.

[0166] As a result, in the game process in which the action of the user U is evaluated based on the timing and position of the instruction object that instructs the action of the user U, the game device 10D displays a composite image in which the instruction object is combined with the image captured by the user U in a visible manner on the game device 10D (e.g., a smartphone) or an externally connected monitor 30D (e.g., a home television), thereby guiding the user on the actions that should be taken, allowing for more intuitive play with a simple configuration.

[0167] For example, the game device 10D displays the composite image on the display unit 12D or the monitor 30D after flipping it horizontally.

[0168] This allows the user U to play the game device 10D while looking at the display unit 12D or the monitor 30D in the same way as if looking in a mirror.

[0169] Furthermore, the game device 10 moves an instruction object (for example, a moving object) placed at a predetermined position (appearance position) in the virtual space toward a predetermined determination position (for example, the position of a determination object). Then, the game device 10 evaluates the movement of at least a part of the body (for example, a foot) of the user U detected from the captured video, based on the timing at which the instruction object (for example, the moving object) moving in the virtual space reaches the determination position and the determination position.

[0170] This allows the game device 10D to use the captured video to evaluate whether or not the user U was able to perform the action as instructed.

[0171] [Variations] Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configurations are not limited to the above-described embodiments, and the present invention also includes designs that do not deviate from the gist of the present invention. For example, the configurations described in the above-described embodiments can be combined in any manner.

[0172] Note that the instruction objects described in the above embodiments are merely examples, and various modes are possible as long as they instruct the user U to perform an action. For example, the content of the action instructed to the user U varies depending on the type (mode) of the instruction object. For example, changing the thickness (width in the Z-axis direction) of the moving object changes the time from when the bottom of the moving object reaches the judging object until when the top of the moving object reaches the judging object. Therefore, the thickness of the moving object may be used to indicate the time to continue stepping on the judging object with a foot. The moving object does not necessarily appear in the vertical direction of the destination judging object, but may appear from a position deviating from the vertical direction. Furthermore, the moving direction of the moving object and the position of the judging object can be set arbitrarily.

[0173] Furthermore, the determination object does not have to be displayed at the determination position. For example, if the position of the floor is the determination position, the timing and position at which the moving object reaches the floor become the instruction content for instructing the user U's action. For example, if a moving object having a certain thickness (e.g., a length approximately equal to the height of the user U) is moved vertically toward the floor (determination position) in a diagonal direction (e.g., a direction tilted 45° from the vertical direction) rather than vertically, the position at which the moving object reaches the floor changes over time from the position on the XY plane when the bottom of the moving object reaches the floor to the position on the XY plane when the top of the moving object reaches the floor. Therefore, an instruction to move the stepping position may be given using a moving object having a certain thickness in a diagonal direction.

[0174] Furthermore, the determination position is not limited to the floor surface, and can be set at any position between the floor surface and the ceiling, for example. The height of the determination position may be detected and set according to the height of the user U. Furthermore, without setting a determination position, the displayed moving object itself may indicate the movement of the user U. For example, the position of the moving object when it appears or the position and timing of the moving object when it is moving may indicate the movement of the user U. For example, the trajectory of the movement of the moving object may indicate the trajectory of the movement of the user U (for example, the trajectory of the hand movement).

[0175] In the first and second embodiments, the game apparatus 10 and the game apparatus 10A are configured as see-through HMDs and are provided with the imaging unit 11. However, the imaging unit 11 may be installed in a separate location, separate from the game apparatus 10 and the game apparatus 10A, from which the user U playing the dance game can be imaged. In this case, the device including the imaging unit 11 installed in the separate location is communicatively connected to the game apparatus 10 and the game apparatus 10A via wired or wireless communication. In the game system 1C including the game apparatus 10C and the HMD 20C described in the third embodiment, the imaging unit 21C corresponding to the imaging unit 11 is installed in the HMD 20C. However, the imaging unit 21C may also be installed in a separate location, separate from the HMD 20C, from which the user U playing the dance game can be imaged. In this case, the device including the imaging unit 21C installed in the separate location is communicatively connected to the HMD 20C or the game apparatus 10C via wired or wireless communication. The imaging unit 21C may also be installed in the game apparatus 10C.

[0176] Furthermore, in the fourth embodiment, game device 10D is configured to capture an image of user U playing a dance game using front camera 11DA provided as an imaging unit, but a device including an imaging unit installed at a different location from game device 10D may be used to capture an image of user U. In this case, the device including the imaging unit installed at a different location is connected to game device 10D for wired or wireless communication.

[0177] Alternatively, a program for implementing the functions of the control unit 150 (150A, 150C, 150D) may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform processing as the control unit 150 (150A, 150C, 150D). Here, "loading a program recorded on a recording medium into a computer system and executing it" includes installing the program into a computer system. The term "computer system" here includes hardware such as an OS and peripheral devices. The term "computer system" may also include multiple computer devices connected via a network, including the Internet, a WAN, a LAN, a dedicated line, or other communication lines. The term "computer-readable recording medium" refers to portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built into a computer system. The recording medium storing the program may be a non-transitory recording medium such as a CD-ROM. The term "recording medium" also includes internal or external recording media accessible from a distribution server for distributing the program. The program code stored on the recording medium of the distribution server may be different from the program code in a format executable by the terminal device. In other words, the format in which the program is stored on the distribution server is not important as long as it can be downloaded from the distribution server and installed in a format executable by the terminal device. The program may be divided into multiple parts, each downloaded at a different time and then combined on the terminal device, or each part may be distributed by a different distribution server. Furthermore, the term "computer-readable recording medium" also includes a storage medium that stores a program for a certain period of time, such as volatile memory (RAM) within a computer system that serves as a server or client when a program is transmitted over a network. The program may also be for implementing part of the functions described above.Furthermore, the above-mentioned functions may be realized in combination with a program already recorded in the computer system, that is, a so-called differential file (differential program).

[0178] Furthermore, some or all of the functions of the control unit 150 (150A, 150C, 150D) described above may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each of the above functions may be individually implemented as a processor, or some or all of the functions may be integrated into a processor. Furthermore, the integrated circuit method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0179] In the above embodiment, at least a portion of the data stored in the storage unit 14 (14C, 14D) of the game device 10 (10A, 10C, 10D) may be stored in an externally connected storage device. The externally connected storage device is a storage device connected to the game device 10 (10A, 10C, 10D) via a wired or wireless connection. For example, the externally connected storage device may be a storage device connected via a Universal Serial Bus (USB), a wireless Local Area Network (LAN), a wired LAN, or the like, or may be a storage device (data server) connected via the Internet or the like. This storage device (data server) connected via the Internet or the like may be used using cloud computing.

[0180] Furthermore, a configuration corresponding to at least a part of each unit included in the control unit 150 (150A, 150C, 150D) may be included in a server connected via the Internet, etc. For example, the above embodiment can be applied to a so-called cloud game in which the processing of a game such as a dance game is executed on a server.

[0181] In the above embodiment, a dance game is used as an example of a music game, but the present invention is not limited to dance games. For example, the present invention can be applied to all music games in which players operate objects that appear in time with music. In addition to music games, the present invention can also be applied to games in which players perform operations such as punching, kicking, knocking down, or hitting objects that appear at predetermined times with a weapon.

[0182] [Note] From the above description, the present invention can be understood, for example, as follows: It should be noted that, to facilitate understanding of the present invention, reference numerals in the accompanying drawings are conveniently placed in parentheses, but this does not mean that the present invention is limited to the illustrated embodiments.

[0183] (Appendix A1) A game program according to one aspect of the present invention is a game program that is played by a computer using a video output device (10, 10A, 20C) that is worn on the head of a user (U) to output a video that is visible to the user and that allows the user to view a real space. The game program includes steps of acquiring a captured video of the real space (S101, S301, S401), generating a virtual space corresponding to the real space from the captured video (S103, S405), and generating a virtual space corresponding to the real space based on a reference position (K1, K2) corresponding to the user in the virtual space. The method executes steps of placing an instruction object that instructs the user to perform an action at a position visible to the user (S105, S109, S407, S411), displaying the virtual space in which at least the instruction object is placed in correspondence with the real space (S203), detecting a movement of at least a part of the user's body from the captured image (S303), and evaluating the detected movement based on the timing and position of the instruction object placed in the virtual space (S305).

[0184] According to the configuration of Appendix A1, in a game process in which a user's actions are evaluated based on the timing and position of an instruction object that instructs the user's actions, by wearing a video output device such as an HMD on the head, the instruction object is associated with real space and made visible to the user, so that the game program can guide the user on what actions to take with a simple configuration, allowing for more intuitive play.

[0185] (Appendix A2) Another aspect of the present invention is a game program described in Appendix A1, wherein the reference position includes a first reference position (K1) in the virtual space corresponding to the position of the user (U) wearing the video output device (10, 10A, 20C), and the first reference position is based on the position of the video output device in the virtual space.

[0186] According to the configuration of Appendix A2, the game program can display instruction objects in correspondence with real space based on the position of the user playing the game, thereby making instructions to the user feel more real and enabling more intuitive gameplay.

[0187] (Appendix A3) Also, one aspect of the present invention is a game program as described in Appendix A2, wherein in the placing steps (S105, S109, S407, S411), the position at which the instruction object is placed is limited to a part of the virtual space depending on the orientation of the user (U) wearing the video output device (10, 10A, 20C).

[0188] According to the configuration of Appendix A3, the game program does not issue instructions to perform actions outside the user's field of view (e.g., behind), so the user can play without worrying about what is outside the field of view (e.g., behind) while playing, and the difficulty of the game does not become too high.

[0189] (Appendix A4) Another aspect of the present invention is a game program as described in Appendix A1, further comprising causing the computer to execute a step (S403) of detecting an image (UK) corresponding to the user (U) from the captured image, wherein the reference position includes a second reference position (K2) in the virtual space of the detected image corresponding to the user.

[0190] According to the configuration of Appendix A4, the game program can display instruction objects instructing the user's movements around a virtual image of the user (user image UK) reflected in a mirror, for example, by wearing a video output device such as an HMD on the head. This simple configuration can guide the user to the movements they should make, allowing for more intuitive play. For example, the game program allows the instruction objects displayed around the user image UK (e.g., front, back, left, and right) to be simultaneously viewed from above without limiting the placement of the instruction objects to a portion of the virtual space, thereby diversifying the types of movements instructed to the user during play. Furthermore, the game program can evaluate the user's movements without the user having to look at their own feet and instruction objects below, making it possible to ensure that dancing is not difficult.

[0191] (Appendix A5) Another aspect of the present invention is a game program as described in Appendix A2 or Appendix A3, further comprising causing the computer to execute a step (S403) of detecting (UK) corresponding to the user (U) from the captured image, wherein the reference position includes a second reference position (K2) in the virtual space of the image corresponding to the detected user.

[0192] According to the configuration of Appendix A5, the game program can display instruction objects instructing the user's actions around a virtual image (user image UK) of the user, for example, reflected in a mirror, by wearing a video output device such as an HMD on the head. This simple configuration can guide the user to the actions they should take, allowing for more intuitive gameplay. For example, the game program allows the user to simultaneously view instruction objects displayed around the user image UK (e.g., front, back, left, and right) from a bird's-eye view without limiting the placement of the instruction objects to a portion of the virtual space, thereby diversifying the types of actions instructed to the user during play. Furthermore, the game program allows the user to evaluate the user's actions without having to look at their own feet and instruction objects below, making it possible to ensure that dancing is not difficult. Furthermore, the game program can display instruction objects around the user playing the game and around the virtual image of the user reflected in a mirror, for example, allowing the user to freely select which of the instruction objects is easier to play.

[0193] (Appendix A6) Another aspect of the present invention is a game program as described in Appendix A5, wherein in the placing steps (S105, S109, S407, S411), when placing the instruction object at a position based on the second reference position (K2) in the virtual space, the visibility of the instruction object to be placed at a position based on the first reference position (K1) is reduced, or the instruction object is not placed at a position based on the first reference position (K1).

[0194] According to the configuration of Appendix A6, the game program can prevent an instruction object displayed at a position (around the user U) based on a first reference position (e.g., reference position K1) from obscuring an instruction object displayed at a position (around the virtual image of the user reflected in the mirror MR) based on a second reference position (e.g., reference position K2), thereby improving the visibility of the instruction object.

[0195] (Appendix A7) Another aspect of the present invention is a game program described in any one of Appendices A4 to A6, wherein the image (UK) corresponding to the detected user (U) is an image of the user reflected in a mirror (MR) located opposite, and in the placing steps (S105, S109, S407, S411), when placing the instruction object at a position based on the second reference position (K2) in the virtual space, the front-to-back orientation relative to the second reference position is reversed.

[0196] According to the configuration of Appendix A7, the game program can display instruction objects corresponding to the orientation of the virtual image of the user (user image UK) reflected in the mirror, thereby guiding the user on what actions to take so that the user can play intuitively while looking in the mirror.

[0197] (Appendix A8) Another aspect of the present invention is a game program as set forth in any one of Appendices A1 to A7, wherein in the placing step (S105, S109, S407, S411), the instruction object placed at a predetermined position in the virtual space is moved toward a predetermined judgment position, and in the evaluating step (S305), the detected action is evaluated based on the timing at which the instruction object moving in the virtual space reaches the judgment position and the judgment position.

[0198] According to the configuration of Supplementary Note A8, the game program can use the captured video to evaluate whether or not the user has performed the action as instructed.

[0199] (Appendix A9) Furthermore, one aspect of the present invention is the game program according to any one of Appendices A1 to A8, wherein the content of the action instructed to the user (U) varies depending on the type of the instruction object.

[0200] According to the configuration of appendix A9, the game program can diversify the actions that the user takes during play, making it possible to provide a highly entertaining game.

[0201] (Appendix A10) A game processing method according to one aspect of the present invention is a game processing method executed by a computer that executes processing of a game that can be played using a video output device (10, 10A, 20C) that is worn on the head of a user (U) and outputs a video that is visible to the user and allows the user to view a real space, the game processing method including steps of acquiring a captured video of the real space (S101, S301, S401), generating a virtual space corresponding to the real space from the captured video (S103, S405), and determining a reference position corresponding to the user in the virtual space. The method includes steps (S105, S109, S407, S411) of arranging an instruction object that instructs the user to perform an action at a position based on the position (K1, K2) of the instruction object so that the instruction object is visible to the user; a step (S203) of displaying the virtual space in which at least the instruction object is arranged in correspondence with the real space; a step (S303) of detecting a movement of at least a part of the user's body from the captured image; and a step (S305) of evaluating the detected movement based on the timing and position based on the instruction object arranged in the virtual space.

[0202] According to the configuration of Appendix A10, the game processing method is a game processing method in which a user's actions are evaluated based on the timing and position of an instruction object that instructs the user's actions, and by wearing a video output device such as an HMD on the head, the instruction object is associated with real space and made visible to the user, so that the user can be guided as to what actions to take with a simple configuration, allowing for more intuitive play.

[0203] (Appendix A11) A game device (10, 10A, 10C) according to one aspect of the present invention is a game device that executes processing of a game that can be played by wearing it on the head of a user (U), outputting an image visible to the user, and using an image output device (10, 10A, 20C) that allows the user to view a real space, and includes an acquisition unit (151, S101, S301, S401) that acquires a captured image of the real space, a generation unit (152, S103, S405) that generates a virtual space corresponding to the real space from the captured image acquired by the acquisition unit, and a reference position (K1) corresponding to the user in the virtual space generated by the generation unit. , K2) so that an instruction object instructing the user to perform an action is visible to the user; a display control unit (156, S203) that displays the virtual space in which at least the instruction object is placed in correspondence with the real space; a detection unit (157, S303) that detects a movement of at least a part of the user's body from the captured image acquired by the acquisition unit; and an evaluation unit (158, S305) that evaluates the movement detected by the detection unit based on the timing and position based on the instruction object placed in the virtual space.

[0204] According to the configuration of Appendix A11, in a game process in which a user's actions are evaluated based on the timing and position of an instruction object that instructs the user's actions, by wearing a video output device such as an HMD on the head, the instruction object is associated with real space and made visible to the user, so that the game device can guide the user on the actions they should take with a simple configuration, allowing for more intuitive play.

[0205] (Appendix B1) Furthermore, a game program according to one embodiment of the present invention causes a computer to execute the steps of acquiring captured video of a real space (S501, S701), generating a virtual space corresponding to the real space from the captured video (S505), placing an instruction object instructing the user (U) to move in a position within the virtual space based on a reference position (K3) corresponding to the user (U) so that the instruction object is visible to the user (U) (S507, S511), displaying on a display unit (12D, 30D) a composite image obtained by combining the captured video with an image of the instruction object placed within the virtual space (S603), detecting a movement of at least a part of the user's body from the captured video (S703), and evaluating the detected movement based on the timing and position of the instruction object placed within the virtual space (S705).

[0206] According to the configuration of Appendix B1, in a game process in which a user's actions are evaluated based on the timing and position of an instruction object that instructs the user's actions, the game program causes a composite image in which the instruction object is combined with an image captured by the user to be displayed on a display unit such as a smartphone or home television in a visible manner, thereby enabling the user to be guided through the actions they should take with a simple configuration, allowing for more intuitive play.

[0207] (Appendix B2) Another aspect of the present invention is a game program as described in Appendix B1, wherein in the displaying step (S603), the composite image is displayed on the display unit (12D, 30D) with the image flipped left and right.

[0208] According to the configuration of Appendix B2, the game program can be played while the user is looking at the display unit (monitor) with the same sensation as if looking in a mirror.

[0209] (Appendix B3) Another aspect of the present invention is a game program as described in Appendix B1 or Appendix B2, wherein in the placing step (S507, S511), the instruction object placed at a predetermined position in the virtual space is moved toward a predetermined judgment position, and in the evaluating step (S705), the detected action is evaluated based on the timing at which the instruction object moving in the virtual space reaches the judgment position and the judgment position.

[0210] According to the configuration of Appendix B3, the game program can use the captured video to evaluate whether or not the user has performed the action as instructed.

[0211] (Appendix B4) Furthermore, one aspect of the present invention is the game program according to any one of Appendices B1 to B3, wherein the content of the action instructed to the user (U) varies depending on the type of the instruction object.

[0212] According to the configuration of Appendix B4, the game program can diversify the actions that the user takes during play, making it possible to provide a highly entertaining game.

[0213] (Appendix B5) Furthermore, a game processing method according to one aspect of the present invention is a game processing method executed by a computer, and includes the steps of acquiring captured image of a real space (S501, S701), generating a virtual space corresponding to the real space from the captured image (S505), arranging an instruction object instructing a user (U) to perform an action in the virtual space at a position based on a reference position (K3) corresponding to the user (U) so that the instruction object is visible to the user (U) (S507, S511), displaying a composite image obtained by combining the captured image with an image of the instruction object arranged in the virtual space on a display unit (12D, 30D) (S603), detecting a movement of at least a part of the user's body from the captured image (S703), and evaluating the detected movement based on a timing and a position based on the instruction object arranged in the virtual space (S705).

[0214] According to the configuration of Appendix B5, the game processing method is a game process in which a user's actions are evaluated based on the timing and position of an instruction object that instructs the user's actions, and a composite image in which the instruction object is combined with an image captured by the user is displayed on a display unit of a smartphone, home television, etc. in a visible manner, so that the user can be guided to the actions they should take with a simple configuration, allowing for more intuitive play.

[0215] (Appendix B6) A game device (10D) according to one aspect of the present invention includes an acquisition unit (151D, S501, S701) that acquires a captured image of a real space, a generation unit (152D) that generates a virtual space corresponding to the real space from the captured image acquired by the acquisition unit, and a placement unit (154D, S507, S511) that places an instruction object that instructs a user (U) to perform an action in a position that is based on a reference position (K3) corresponding to the user (U) in the virtual space generated by the generation unit so that the instruction object is visible to the user. a display control unit (156D, S603) that causes a display unit (12D, 30D) to display a composite image obtained by combining the captured image with an image of the instruction object placed in the virtual space; a detection unit (157D, S703) that detects a movement of at least a part of the user's body from the captured image acquired by the acquisition unit; and an evaluation unit (158D, S705) that evaluates the movement detected by the detection unit based on the timing and position based on the instruction object placed in the virtual space.

[0216] According to the configuration of Appendix B6, in a game process in which a user's actions are evaluated based on the timing and position of an instruction object that instructs the user's actions, the game device displays a composite image in which the instruction object is combined with an image captured by the user on a display unit of a smartphone, home television, etc. in a visible manner, thereby guiding the user to the actions they should take with a simple configuration to enable more intuitive play. [Explanation of symbols]

[0217] 1C game system, 10, 10A, 10C, 10D game device, 11 imaging unit, 11DA front camera, 11DB rear camera, 12, 12D display unit, 13, 13D sensor, 14, 14C, 14D memory unit, 15, 15C, 15D CPU, 16, 16C, 16D communication unit, 17, 17D sound output unit, 18D video output unit, 20C HMD, 21C imaging unit, 22C display unit, 23C sensor, 24C memory unit, 25C CPU, 26C communication unit, 27C sound output unit, 150, 150A, 150C, 150D control unit, 151, 151D video acquisition unit, 152, 152D virtual space generation unit, 153A user image detection unit, 153D User detection unit, 154, 154A, 154D object placement unit, 155 gaze direction detection unit, 156, 156D display control unit, 157, 157D action detection unit, 158, 158D evaluation unit

Claims

1. a computer that executes processing of a game that can be played using a video output device that is worn on a user's head and outputs a video that can be seen by the user and allows the user to see a real space; acquiring a captured image of the real space; generating a virtual space corresponding to the real space from the captured video; detecting an image corresponding to the user from the captured video; placing an instruction object that instructs the user to perform an action at a position within the virtual space that is based on a reference position corresponding to the user so that the instruction object is visible to the user while playing the game; displaying the virtual space, in which at least the instruction object is placed, in association with the real space so that the instruction object is superimposed on the real space and visible; detecting a foot movement of the user from the captured image; detecting a floor present in the real space from the captured image; the instruction objects arranged in the virtual space include a determination object arranged at a position corresponding to the detected floor surface and a moving object gradually moving toward the determination object, and a step of evaluating the play of the user based on timing based on a positional relationship between the determination object and the moving object and timing of a detected predetermined movement of the user's foot; Execute the reference position includes a second reference position in the virtual space of an image corresponding to the detected user; Game program.

2. In the evaluating step, evaluating the play of the user based on a timing based on a positional relationship between the judging object and the moving object, a detected timing of a predetermined foot movement of the user, and a positional relationship between a position of the judging object and a position where the predetermined foot movement of the user is made; The game program according to claim 1 .

3. 1. A game processing method executed by a computer that executes processing of a game that can be played using a video output device that is worn on a user's head and outputs a video that can be seen by the user and allows the user to see a real space, comprising: acquiring a captured image of the real space; generating a virtual space corresponding to the real space from the captured video; detecting an image corresponding to the user from the captured video; placing an instruction object that instructs the user to perform an action at a position within the virtual space that is based on a reference position corresponding to the user so that the instruction object is visible to the user while playing the game; displaying the virtual space, in which at least the instruction object is placed, in association with the real space so that the instruction object is superimposed on the real space and visible; detecting a foot movement of the user from the captured image; detecting a floor present in the real space from the captured image; the instruction objects arranged in the virtual space include a determination object arranged at a position corresponding to the detected floor surface and a moving object gradually moving toward the determination object, and a step of evaluating the play of the user based on timing based on a positional relationship between the determination object and the moving object and timing of a detected predetermined movement of the user's foot; Including, the reference position includes a second reference position in the virtual space of an image corresponding to the detected user; Game processing method.

4. A game device that is worn on a user's head and that outputs video images that can be viewed by the user and executes processing for a game that can be played using a video output device that enables the user to view a real space, an acquisition unit that acquires a captured image of the real space; a generation unit that generates a virtual space corresponding to the real space from the captured video acquired by the acquisition unit; a detection unit that detects an image corresponding to the user from the captured video acquired by the acquisition unit; a placement unit that places an instruction object that instructs the user to perform an action at a position based on a reference position corresponding to the user within the virtual space generated by the generation unit so that the instruction object is visible to the user while playing the game; a display control unit that displays the virtual space, in which at least the instruction object is placed, in association with the real space so that the instruction object is superimposed on the real space and visible; a detection unit that detects the user's foot movement and a floor that exists in the real space from the captured image acquired by the acquisition unit; the instruction objects arranged in the virtual space include a determination object arranged at a position corresponding to the detected floor surface, and a moving object gradually moving toward the determination object, and an evaluation unit that evaluates the play of the user based on timing based on a positional relationship between the determination object and the moving object and timing of a detected predetermined movement of the user's foot; Equipped with the reference position includes a second reference position in the virtual space of an image corresponding to the detected user; Game device.

Images