Program and display control device

Abstract

To provide a program and a display control device for contributing to an improvement in the amusement of play in a VR space.SOLUTION: In an HMD (head-mounted type display device) system X, a program is executed by a game device 1, an HMD 2, and a controller 3, and that displays a virtual space image on the HMD 2. The program causes the game device to function as: identification means for, in a real space, identifying a state of a hand of a user U wearing the HMD or a third person and / or a positional relationship between a hand and a real object 4 existing in a real space; and display control means for controlling display of a real object image, which is an image superimposed on the virtual space image and corresponding to the real object 4, on the basis of the identified state and / or the positional relationship.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a program for controlling display on a head mounted display and a display control device. [Background technology]

[0002] In recent years, game devices using a head mounted display (HMD) have been developed, and it has become possible to play games in a virtual reality (VR) space provided by the HMD. In such game devices, the posture or movement of the HMD is tracked to control the viewing area in the VR space visually recognized by the user. This allows the user to play the game while immersed in the VR space (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6756769 Summary of the Invention [Problem to be solved by the invention]

[0004] A user who wears an HMD and plays a game in a VR space cannot see the real space while wearing the HMD, and therefore it is expected that it will be difficult to operate objects in the real space (hereinafter referred to as "real objects"). For example, the user may not be able to answer a call on his / her smartphone while wearing the HMD. In this way, the user who wears the HMD may be distracted by the presence and operation of real objects in the real space, and as a result, may not be able to fully enjoy playing in the VR space.

[0005] The present disclosure provides a program and a display control device that contribute to improving the enjoyment of play in a VR space. [Means for solving the problem]

[0006] The first aspect is: A program for displaying a virtual space image on a head-mounted display device, An identification means for identifying, in a real space, a state of a hand of a user or a third party wearing the head-mounted display device, and / or a positional relationship between the hand and a real object present in the real space; a display control means for controlling display of a real object image, which is an image corresponding to the real object and is superimposed on the virtual space image, based on the specified state and / or the positional relationship; It is a program characterized by causing a computer to function as follows.

[0007] In addition, in a first aspect, the specified state includes a direction of the hand as seen by the user, The display control means can display the real object image superimposed on the virtual space image based on the orientation and / or the positional relationship.

[0008] Also, in the first aspect, the identification means can identify the orientation and / or the positional relationship based on an image of the real space captured by a camera provided in the head-mounted display device.

[0009] In addition, in a first aspect, the computer is further made to function as a progress control means for controlling the progress of content associated with the virtual space image, and the progress control means can stop the progress of the content based on the identified state and / or the positional relationship.

[0010] Also, in the first aspect, the progress control means can determine whether or not to stop the progress of the content based on the type of the physical object.

[0011] Also, in a first aspect, the display control means can superimpose and display the real object image on the virtual space image even if a condition is not satisfied, and can change the display mode of the real object image when the condition is satisfied.

[0012] In the first aspect, the device can further function as a receiving means for receiving the type of the physical object by designation from a user.

[0013] In addition, in a first aspect, the computer can further function as a change means that pre-associates parameters associated with content related to the virtual space image with the real object in response to instructions from the user, and changes the parameters in response to an operation performed by the user on the real object.

[0014] The second aspect is A display control device that displays a virtual space image on a head-mounted display device, An identification means for identifying, in a real space, a state of a hand of a user or a third party wearing the head-mounted display device, and / or a positional relationship between the hand and a real object present in the real space; a display control means for controlling display of a real object image, which is an image corresponding to the real object and is superimposed on the virtual space image, based on the specified state and / or the positional relationship; The display control device is characterized by comprising: Effect of the Invention

[0015] According to the present disclosure, it is possible to provide a program and a display control device that contribute to improving the enjoyment of play in a VR space. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing an example of an overview of an HMD system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an HMD system according to the present embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an HMD system according to the present embodiment. [Figure 4] 11A and 11B are diagrams showing an example of display of a physical object image in a normal mode according to the embodiment; [Diagram 5] 11A and 11B are diagrams showing an example of display of a physical object image in a search mode according to the present embodiment. [Figure 6] 11A and 11B are diagrams showing an example of display of a physical object image in an operation mode according to the present embodiment. [Figure 7] 1A to 1C are diagrams illustrating an example of a display of a virtual space image for selecting a real object according to the present embodiment. [Figure 8] 11A and 11B are diagrams showing an example of a display of a virtual space image for selecting a real object for parameter change according to the present embodiment. [Figure 9] FIG. 4 is a diagram illustrating an example of the operation of the HMD system according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] An embodiment of the present disclosure will be described with reference to the drawings.

[0018] [HMD system overview] Fig. 1 is a diagram showing an example of an overview of an HMD system according to this embodiment. As shown in Fig. 1, the HMD system X includes a game device 1, an HMD 2 worn on the head of a user U, and a controller 3 used by the user U to play content in a VR space.

[0019] The game device 1 is a home game console such as PlayStation (registered trademark), a portable game console such as Nintendo Switch (registered trademark), or an electronic device such as a personal computer, a smartphone, or a tablet. The game device 1 controls the display of an image corresponding to a VR space (hereinafter referred to as a "virtual space image") on the HMD 2. In the following, the game device 1 is assumed to be a device that controls the display of the virtual space image and is capable of progressing through content associated with the virtual space image, but is not limited to this. The game device 1 may be any display control device that is capable of controlling the display of the virtual space image.

[0020] The content associated with the virtual space image may be, for example, a VR game in which an object in a VR space is operated in accordance with the movement of the user U, the operation of the controller 3, the movement, etc. The object may be a character, and the content may be a game in which the character is made to act in the VR space or to fight against an enemy character.

[0021] The HMD2 is a head-mounted display device.

[0022] As shown in Fig. 1, a user U wearing an HMD 2 can feel immersed in a VR space formed by a virtual space image displayed on the HMD 2, but may be distracted by a real object 4 in the real space, and as a result, may not be able to fully enjoy playing in the VR space. For example, a smartphone exists as a real object 4 in the real space shown in Fig. 1, and even if the smartphone receives a call, while the user U is wearing the HMD 2, it is difficult for the user U to grasp the smartphone, and therefore the user U may not be able to concentrate on playing in the VR space.

[0023] Therefore, the game device 1 identifies the state of the hand of the user U or a third party (hereinafter referred to as the "hand state") and / or the positional relationship between the hand of the user U or a third party and a real object 4 existing in real space. Based on the identified hand state and / or positional relationship, the game device 1 controls the display of an image corresponding to the real object 4 (hereinafter referred to as the "real object image") on the HMD 2. Specifically, the game device 1 controls a display in which the real object image is superimposed on the virtual space image (hereinafter referred to as the "superimposed display"). Here, "superimposition" refers to drawing the real object image in a predetermined area in the virtual space image displayed on the HMD 2.

[0024] In this way, by superimposing the real object image on the virtual space image displayed on the HMD 2, the user U can visually recognize the existence of the real object 4 even while wearing the HMD 2, and can continue playing in the VR space while paying attention to the real object 4. This can therefore improve the interest of playing in the VR space.

[0025] Hereinafter, an example of controlling the superimposition of a real object image on a virtual space image based on the hand state of a user wearing the HMD 2 and / or the positional relationship between the user's hand and a real object will be described, but the present invention is not limited to this. Of course, the superimposition of a real object image on a virtual space image may be controlled based on the hand state of a third party different from the user wearing the HMD 2 and / or the positional relationship between the third party's hand and a real object.

[0026] [HMD system configuration] <Hardware configuration> The hardware configuration of the HMD system X of this embodiment will be described. Fig. 2 is a diagram showing an example of the hardware configuration of the HMD system according to this embodiment. Here, the main hardware configuration of the game device 1 and the HMD 2 will be described, but it goes without saying that they may also be provided with hardware configurations not shown. In addition, it goes without saying that the controller 3 includes a communication unit, an input receiving unit (e.g., buttons, etc.) not shown.

[0027] The game device 1 includes a control unit 10, a storage unit 11, a communication unit 12, an image processing unit 13, and a sound processing unit 14. The storage unit 11, the communication unit 12, the image processing unit 13, and the sound processing unit 14 are each connected to the control unit 10 via a dedicated bus, an interface, or the like.

[0028] The control unit 10 is a CPU (Central Processing Unit) or the like that controls the operation of the game device 1. The storage unit 11 is mainly composed of an HDD (Hard Disk Drive), a RAM (Random Access Memory), and a ROM (Read Only Memory). The communication unit 12 communicates with the HMD 2 and the controller 3 wirelessly or via a wired connection. The communication unit 12 may also communicate with a server (not shown) or the like.

[0029] The image processing unit 13 renders an image to be displayed on the display unit 25 of the HMD 2 according to an instruction from the control unit 10. Specifically, the image processing unit 13 renders a virtual space image corresponding to the VR space. The VR space is formed, for example, on a celestial sphere with a predetermined position as the center. The image processing unit 13 renders an image corresponding to the field of view of the user U in the VR space (hereinafter referred to as a "field of view image") as a virtual space image based on configuration information of the VR space (hereinafter referred to as "VR space configuration information") held in the storage unit 11. The field of view image changes following the posture or movement of the HMD 2 and the movement of at least a part of the body of the user U (for example, the head, eyes, hands, etc.). In addition, the image processing unit 13 renders a real object image to be superimposed on the virtual space image on the display unit 25 of the HMD 2.

[0030] The audio processor 14 reproduces and synthesizes audio data in response to instructions from the controller 10, converts it into digital to analog (D / A) data, and outputs it as audio. The audio is output from a speaker or an earphone output terminal of the audio input / output unit 26 of the HMD 2. Furthermore, the audio processor 14 can also convert an audio signal input from a microphone of the audio input / output unit 26 of the HMD 2 into analog to digital (A / D) data and input it.

[0031] The HMD 2 includes a control unit 20, a storage unit 21, a communication unit 22, a sensor unit 23, a display unit 25, and an audio input / output unit 26. The storage unit 21, the communication unit 22, the sensor unit 23, the display unit 25, and the audio input / output unit 26 are each connected to the control unit 20 via a dedicated bus, interface, or the like.

[0032] The control unit 20 controls the operation of the HMD 2. The storage unit 21 is mainly composed of a RAM and a ROM. The communication unit 22 communicates with the game device 1 wirelessly or via a wired connection.

[0033] The sensor unit 23 detects the posture or movement of the HMD 2, the movement of at least a part of the body of the user U (e.g., the head, eyes, hands, etc.), etc. The sensor unit 23 may be at least one of a laser sensor, an infrared sensor, a magnetic sensor, an angular velocity sensor, an acceleration sensor, an eye tracking device (ETD), etc. The sensor unit 23 may be built into the HMD 2, or may be provided in a device (e.g., a controller) separate from the HMD 2.

[0034] The imaging unit 24 captures an image of a predetermined area in real space. The imaging unit 24 may be configured with, for example, one or more cameras and may be capable of capturing images in different directions (for example, upward, downward, left, or right directions, etc.).

[0035] The display unit 25 is configured, for example, with a non-transmissive display. The display unit 25 displays the field of view image as a virtual space image. The display unit 25 may include a right-eye display unit that provides a right-eye image and a left-eye display unit that provides a left-eye image. By viewing the right-eye image and the left-eye image with the right eye and the left eye, respectively, the user U can view a three-dimensional VR space. Note that the number of display units may be one as long as the right-eye image and the left-eye image can be provided. The display unit 25 may also display a real object image superimposed on the virtual space image.

[0036] The audio input / output unit 26 is a speaker for headphones, an earphone, a microphone, an exciter, etc. The audio input / output unit 26 is capable of reproducing audio.

[0037] The above-mentioned game device 1, HMD 2, and controller 3 are hardware resources that execute a program 110 (FIG. 3) of this embodiment.

[0038] <Functional configuration> Next, a description will be given of the functional configuration of the control unit 10 and the storage unit 11 of the game device 1 which realizes the HMD system X. Note that the following describes the main functions of the game device 1, and it goes without saying that the game device 1 may also include functions not shown.

[0039] The game device 1 stores in the storage unit 11 a program 110 for realizing the HMD system X of the present invention, VR space configuration information 111, and parameters 112. The program 110 is a program for operating the game device 1 as the above-mentioned functional means. Based on the program 110, the game device 1 progresses the content associated with the virtual space image.

[0040] The VR space configuration information 111 is, for example, information constituting a background image rendered on a celestial sphere forming the VR space, and information constituting various objects existing in the VR space. The information constituting various objects may be, for example, coordinate positions of various objects in the VR space calculated from the position, inclination, etc. of the controller 3 in the real space.

[0041] The parameter 112 is a parameter associated with the content, and may be, for example, a parameter related to an object in a VR space. The parameter related to the object may be, for example, a hit point (HP) of the object.

[0042] The control unit 10 executes a program 110 stored in the storage unit 11 to function as a specification unit 100, a progress control unit 101, a reception unit 102, a change unit 103, and a display control unit 104.

[0043] -Specific means 100- The determination means 100 determines the state of the hand of a user U wearing the HMD 2 and / or the positional relationship between the hand of the user U and a real object 4 in real space.

[0044] The identification means 100 may identify the hand state (e.g., the orientation of the hand) of the user U wearing the HMD 2 based on an image captured in real space by the imaging unit 24 of the HMD 2. Here, the hand state of the user U may include the orientation (e.g., palm or back of the hand) and posture of the hand as seen by the user U wearing the HMD 2. For example, the identification means 100 may detect the hand by image recognition processing of the captured image, and identify whether the orientation of the hand is the palm or the back of the hand as seen by the user U.

[0045] Furthermore, the determination means 100 may determine the positional relationship between the hand of the user U wearing the HMD 2 and a real object based on an image of the real space captured by the imaging unit 24 of the HMD 2. Here, the positional relationship between the hand of the user U and the real object may be, for example, the distance between the hand and the real object, or the direction in which the real object exists relative to the hand.

[0046] For example, the identification means 100 may detect the hand and a real object by image recognition processing of the captured image, and identify the distance between the hand of the user U and the real object and / or the direction in which the real object exists relative to the hand. The distance between the hand and the real object may be, for example, the distance between a predetermined position of the hand recognized from the captured image and a predetermined position of the real object. Furthermore, the direction in which the real object exists relative to the hand may be a direction relative to the surface of the hand recognized from the captured image (for example, a normal direction, etc.), or a direction above, below, left, right, or depth relative to the predetermined position of the hand, etc.

[0047] -Progress control means 101- The progress control means 101 controls the progress of the content associated with the virtual space image. Specifically, the progress control means 101 stops the progress of the content based on the hand state of the user U identified by the identification means 100 and / or the positional relationship between the hand of the user U and the real object 4. Note that stopping the content may mean temporarily stopping the content in a state in which it can be resumed.

[0048] More specifically, the progress control means 101 may switch a mode related to the progress of the content based on the hand state and / or positional relationship identified by the identification means 100, and control the progress of the content based on the switched mode. The mode is a mode related to the progress of the content, and may include, for example, a mode in which the content progresses (hereinafter referred to as a "normal mode"), a mode in which the user U wearing the HMD 2 searches for a physical object 4 (hereinafter referred to as a "search mode"), a mode in which the user U operates the physical object 4 (hereinafter referred to as an "operation mode"), etc.

[0049] For example, the progress control means 101 may switch the normal mode to the search mode when a motion of grasping the physical object 4 is detected from the hand state and / or positional relationship. The progress control means 101 may also switch the search mode to the operation mode when a motion of grasping the physical object 4 is detected from the hand state and / or positional relationship. The progress control means 101 may also switch the operation mode to the normal mode when it is detected from the hand state and / or positional relationship that the operation of the physical object 4 has ended. The progress control means 101 may also switch the modes based on a signal from the controller 3 (input from the user U).

[0050] Furthermore, the progress control means 101 may continue the progress of the content in normal mode and stop the progress of the content in search mode or operation mode. Furthermore, the progress control means 101 may continue the game progress in search mode and stop the game progress when the mode is changed from the search mode to the operation mode.

[0051] Furthermore, the progress control means 101 may determine whether or not to stop the progress of the above content based on the type of the physical object 4 (e.g., a smartphone, a drink, a snack, etc.). For example, the progress control means 101 may stop the progress of the above content when the type of the physical object 4 is a specific type (e.g., a smartphone), and may continue the progress of the above content when the type of the physical object 4 is another type (e.g., a drink or a snack).

[0052] -Method of reception 102- The reception means 102 receives a designation from the user U of the type of physical object 4 to be used during play by the user U wearing the HMD 2. Specifically, the reception means 102 may select the type of physical object 4 based on a designation from the user U before play begins or during play.

[0053] Furthermore, the reception means 102 may receive a selection of a physical object 4, the parameter 112 of which is updated by use during play, by a designation from the user U. The reception means 102 may associate the selected physical object 4 with the parameter 112 to be updated.

[0054] -Change Method 103- The change means 103 changes the parameter 112 in response to an operation performed by the user U on the physical object 4. The parameter 112 is associated with the physical object 4 in response to an instruction from the user U and stored in the storage unit 11. The instruction from the user U may be input using the controller 3, for example.

[0055] -Display control means 104- The display control means 104 causes the image processing unit 13 to draw a field of view image corresponding to the field of view area in the VR space as a virtual space image. The drawn virtual space image is displayed on the display unit 25 of the HMD 2.

[0056] Furthermore, the display control means 104 controls the superimposition display of the real object image on the virtual space image based on the hand state and / or the positional relationship between the hand and the real object 4 identified by the identification means 100. The display control means 104 may detect a predetermined action on the real object 4 using the hand of the user U based on the hand state and / or the positional relationship between the hand and the real object 4, and control the superimposition display of the real object image based on the detection result. The predetermined action may be, for example, an action of grasping the real object 4 using the hand of the user U, an action of attempting to grasp, an action of manipulating, or an action of attempting to manipulate, etc.

[0057] For example, the display control means 104 may superimpose and display a real object image at a specific position in the virtual space image when a predetermined action using the hand of the user U on the real object 4 is detected, and may not superimpose and display the real object image when the predetermined action is not detected. In this way, the display control means 104 may control whether or not to superimpose and display a real object image based on the detection result of the predetermined action.

[0058] Alternatively, the display control means 104 may superimpose and display a real object image at a specific position in the virtual space image, regardless of whether a predetermined action using the hand of the user U on the real object 4 is detected. In this case, the display control means 104 may change the display mode (e.g., position, size, etc.) of the real object image according to the detection result of the predetermined action. That is, the display control means 104 may superimpose and display the real object image on the virtual space image even if a condition (e.g., detection of an action to grasp the real object 4) is not satisfied, and may change the display mode of the real object image when the condition is satisfied.

[0059] Furthermore, the display control means 104 may control the superimposition of a real object image on a virtual space image based on the mode switched by the progress control means 101 .

[0060] Fig. 4 is a diagram showing an example of display control of a physical object image in a normal mode according to the present embodiment. In Fig. 4, the physical object 4 corresponding to the physical object image is a smartphone, but is not limited to this. The physical object 4 and the physical object image may be designated in advance by the user U. Also, in Fig. 4, the predetermined action performed on the physical object 4 using the hand of the user U is an action of trying to grab the physical object 4 using the hand of the user U, but is not limited to this.

[0061] As shown in FIG. 4, in normal mode, the display control means 104 may cause the display unit 25 of the HMD 2 to display an object image corresponding to the movement of an object (here, opening and closing of a virtual hand) in response to a specific operation of the controller 3 by the user U (e.g., pressing or not pressing a button).

[0062] 4, when a motion of the user U trying to grasp the real object 4 with his / her hand is not detected, the display control means 104 may superimpose and display the real object image at a specific position in the virtual space image. The specific position may be, for example, a position in the VR space corresponding to the positional relationship between the user U and the real object 4 in the real space, or may be an inconspicuous position in the virtual space image. Alternatively, when a motion of the user U trying to grasp the real object 4 with his / her hand is not detected, the display control means 104 may not display the real object image in the virtual space image.

[0063] Fig. 5 is a diagram showing an example of display control of a physical object image in a search mode according to this embodiment. In Fig. 5, the same points as in Fig. 4 will not be described repeatedly. As shown in Fig. 5, in the search mode, the progress control means 101 may stop the game progress or may continue the game progress.

[0064] In the search mode, the display control means 104 detects an action of the user U trying to grasp the real object 4 with the hand based on the orientation of the hand of the user U identified by the identification means 100. For example, as shown in FIG. 5, when the back of the user U's hand faces the user U, the display control means 104 may detect an action of the user U trying to grasp the real object 4 with the hand and superimpose the real object image on the virtual space image. At this time, the display control means 104 may make the presence of the real object 4 in the VR space more noticeable than in the normal mode shown in FIG. 4 by controlling the display mode (e.g., size, position, etc.) of the real object image in the virtual space image. For example, the display control means 104 may display the real object image, which has been displayed in a reduced size in the normal mode, at full size in the search mode.

[0065] 5, in the search mode, the display control means 104 may superimpose an image corresponding to a virtual hand (hereinafter, referred to as a "virtual hand image") on the virtual space image. The display control means 104 may superimpose the virtual hand image and the real object image on the virtual space image in correspondence with the positional relationship between the hand of the user U and the real object 4 in the real space.

[0066] Furthermore, the display control means 104 detects an action of the user U trying to grasp the real object 4 with the hand based on the positional relationship between the hand and the real object 4 in addition to the orientation of the hand of the user U identified by the identification means 100. For example, when the orientation of the hand of the user U satisfies a condition and the positional relationship between the hand and the real object 4 has a condition, the display control means 104 may detect an action of grasping the real object 4 and superimpose and display the real object image on the virtual space image as described above. The condition for the positional relationship between the hand and the real object 4 may be at least one of, for example, whether the distance between the hand and the real object 4 in a predetermined direction (e.g., up, down, left, right, and depth direction) satisfies a predetermined condition and whether the real object exists in a predetermined direction (e.g., normal direction) with respect to the surface of the palm of the hand.

[0067] As shown in Figure 5, in search mode, when a predetermined action (here, an action of attempting to grasp a physical object 4) is detected based on the hand state of user U and / or the positional relationship between the hand and physical object 4 identified by identification means 100, the display manner of the physical object image is changed to be more noticeable compared to normal mode, making it easier for user U to grasp the position of physical object 4 in real space.

[0068] Fig. 6 is a diagram showing an example of display control of a physical object image in the operation mode according to this embodiment. In Fig. 6, the same points as in Fig. 4 or Fig. 5 will not be described repeatedly. As shown in Fig. 6, in the operation mode, the progress control means 101 stops the progress of the game.

[0069] In the operation mode, the display control means 104 detects an action of manipulating the physical object 4 using the hand of the user U, based on the orientation of the hand of the user U identified by the identification means 100 and the positional relationship between the hand and the physical object 4. For example, as shown in Fig. 6, when the palm of the user U faces the user U side and the distance between the hand and the physical object 4 is within a predetermined range, the display control means 104 may detect an action of manipulating the physical object 4 using the hand of the user U, and may superimpose and display the physical object image on the virtual space image.

[0070] In addition, the action of manipulating the real object 4 is not limited to the case where the palm of the user U is facing toward the user U, but may be detected based on the orientation of at least a part of the user U's body (e.g., arm, thumb, etc.) and the distance between at least a part of the body and the real object 4.

[0071] In addition, in the operation mode, the real object image superimposed on the virtual space image may be a captured image of a specific part (e.g., a smartphone display) of the real object 4. The captured image may be, for example, an image captured in real time by the imaging unit 24 of the HMD 2.

[0072] In this way, by displaying a captured image in which a specific portion of the real object 4 is focused within the virtual space image as a real object image, the user U can operate the real object 4 in the real space while still wearing the HMD 2.

[0073] 7 is a diagram showing an example of display control when selecting a real object according to the present embodiment. For example, as shown in FIG. 7, real objects 4a and 4b that are candidates for use during play may be photographed by the imaging unit 24 of the HMD 2 and recognized by image recognition processing of the photographed image.

[0074] As shown in Fig. 7, the display control means 104 may display real object images corresponding to the real objects 4a and 4b on a virtual space image in a selection screen before the start of play. The real object images may be displayed in a superimposed manner at positions in the VR space corresponding to the real objects 4a and 4b in the real space. In Fig. 7, the display control means 104 displays a virtual hand image in a superimposed manner at a position in the VR space corresponding to the hand of the user U in the real space.

[0075] As shown in the selection screen before the start of play in Fig. 7, the reception unit 102 may receive the selection of the real object 4a or 4b based on the positional relationship between the virtual hand and the real object 4a or 4b projected in the VR space. For example, in Fig. 7, since the virtual hand in the VR space is about to grab the real object 4a, the reception unit 102 may receive the selection of the real object 4a.

[0076] Furthermore, the display control means 104 may display a selection screen for the real object 4 on the display unit 25 of the HMD 2 during game play. For example, a selection box for the real object 4 is displayed on the selection screen during game play shown in FIG. 7. The reception means 102 may select a type of the real object 4 (here, a smartphone, a drink, or a snack) based on an instruction of the user U input to the controller 3. For example, when a drink is selected, the display control means 104 may superimpose a real object image corresponding to the real object 4b on the virtual space image instead of the real object image corresponding to the real object 4a.

[0077] Fig. 8 is a diagram showing an example of display control relating to the association of a real object with a parameter according to this embodiment. In Fig. 8, the same points as in Fig. 7 will not be described repeatedly. In Fig. 8, the display control means 104 displays a screen for associating the real object 4a or 4b with the parameter 112 before the start of play. In addition, in the operation mode, the display control means 104 displays an image of the real object 4b captured by the imaging unit 24 of the HMD 2 as a real object image, superimposed on the virtual space image.

[0078] For example, on the association screen before play starts in FIG. 8, the reception means 102 may receive a selection of a physical object 4b designated by the user U, and associate the physical object 4b with an in-game effect (for example, the parameter 112 to be updated).

[0079] Furthermore, on the operation mode screen during play in Fig. 8, when a predetermined action on a physical object 4b using the hand of the user U is detected, the change means 103 may update the parameter 112 associated with the physical object 4b. For example, in Fig. 8, the predetermined action is the user U drinking coffee in a cup which is the physical object 4b. The change means 103 may determine that the user U has drunk the coffee in the cup which is the physical object 4b based on a captured image of the physical object 4b, and may recover the HP associated with the physical object 4b.

[0080] [HMD system operation] FIG. 9 is a diagram showing an example of the operation of the HMD system according to this embodiment. FIG. 9 shows the operation of the game device 1 when a user U wearing the HMD 2 plays content associated with a virtual space image. Some steps in FIG. 9 may be omitted, or some of the steps may be changed in order. For example, step st103 may be omitted, and the game device 1 may transition from the normal mode to the operation mode without the search mode. The operation in FIG. 9 ends when the game device 1 detects a termination operation from the user U in any of steps st102, st103, or st104.

[0081] In step st101, the game device 1 performs pre-play settings for the content associated with the virtual space image. For example, as shown in Fig. 7, the game device 1 may receive, before the start of play, a designation from the user U of the type of real object 4 to be used during play. Also, as shown in Fig. 8, the game device 1 may receive, before the start of play, a selection of a real object 4 whose parameter 112 is updated when used during play. Note that the type of real object 4 to be used during play may be changed during play.

[0082] In step st102, the game device 1 operates in normal mode and continues the progress of the content. As shown in Fig. 4, in the normal mode, the game device 1 may control the movement of a content object in the VR space (opening and closing of a virtual hand in Fig. 4) in response to detection of a specific operation of the controller 3 by the user U, and may display a virtual space image including the content object on the display unit 25 of the HMD 2.

[0083] Also, in the normal mode, the game device 1 superimposes a real object image corresponding to the real object 4 selected in step st101 on the virtual space image so as to be less noticeable compared to the search mode or the operation mode, as shown in Fig. 4. Note that in the normal mode, the game device 1 does not need to superimpose the real object image.

[0084] In step st103, when a predetermined action using the hand of the user U is detected, the game device 1 transitions from the normal mode to the search mode. The predetermined action may be, for example, an action of trying to grab the real object 4 using the hand of the user U. In the search mode, the game device 1 may stop or continue the progress of the content.

[0085] 5, in the search mode, the game device 1 may superimpose a real object image corresponding to the real object 4 selected in step st101 on the virtual space image so as to be more noticeable than in the normal mode. Furthermore, the game device 1 may superimpose the real object image and the virtual hand image in correspondence with the positional relationship between the hand of the user U and the real object 4 in the real space. Since the user U can estimate the positional relationship between the real object 4 and his / her own hand from the positional relationship between the real object image and the virtual hand image, it is possible to easily grasp the real object 4 in the real space even when the user U is wearing the HMD 2 and cannot see the real space.

[0086] In step st104, when a predetermined action using the hand of the user U is detected, the game device 1 transitions from the search mode to the operation mode. The predetermined action may be, for example, an action of operating or attempting to operate the real object 4 using the hand of the user U. In the operation mode, the game device 1 may stop the progress of the content.

[0087] 6, in the operation mode, the game device 1 may superimpose a captured image including the real object 4 selected in step st101 on a virtual space image. Since the user U can check a specific part of the real object 4 (for example, the smartphone screen in FIG. 6) in real time in the captured image, the user U can operate the real object 4 even when wearing the HMD 2 and unable to see the real space.

[0088] In addition, in the operation mode, when the game device 1 detects a predetermined action of the user U's hand on a physical object 4 (for example, in FIG. 8, the action of drinking coffee using the physical object 4), as shown in FIG. 8, the game device 1 may change the parameters 112 associated with the physical object 4.

[0089] In step st105, the game device 1 judges whether or not a predetermined action by the user U is detected. The predetermined action may be, for example, an action of ending the operation of the physical object 4, or an action of releasing the physical object 4 from the hand of the user U. When the predetermined action is detected (step st105; YES), the game device 1 transitions from the operation mode to the normal mode and resumes the progress of the content. On the other hand, when the predetermined action is not detected (step st105; NO), the game device 1 continues the operation in the operation mode.

[0090] To summarize the above, the program of this embodiment is a program that displays a virtual space image on an HMD2 (head-mounted display device), and causes a computer to function as an identification means 100 that identifies, in real space, the state of the hand of a user U or a third party wearing the HMD2, and / or the positional relationship between the hand and a real object 4 existing in real space, and a display control means 104 that controls the display of a real object image superimposed on the virtual space image based on the identified state and / or positional relationship. [effect] According to the HMD system X of this embodiment, a real object image is superimposed on a virtual space image on the display unit 25 of the HMD 2, so that the user U can visually recognize the presence of the real object 4 even while wearing the HMD 2. Therefore, the user U can continue playing in the VR space while paying attention to the real object 4, and the interest of playing in the VR space can be improved.

[0091] [Other embodiments] The various control means and processing procedures described in the above embodiment are merely examples, and are not intended to limit the scope of the present invention, its applications, or its uses. The various control means and processing procedures can be appropriately modified in design without departing from the spirit of the present invention.

[0092] In the above embodiment, the game device 1 may identify the user U based on the state of the user U's hand. Specifically, it may identify whether the hand is the user U's own hand or a third party's hand, and change the display mode of the virtual space image on the display unit 25 of the HMD 2 based on the identification result. For example, when a third party other than the user U covers the HMD 2 saying "Who's that?", if the identification means 100 of the game device 1 identifies the hand state of the third party, the display control means 104 may darken the virtual space image to be displayed on the display unit 25 of the HMD 2. Note that whether the hand is the user U's own hand or a third party's hand may be identified based on biometric information or the like.

[0093] Furthermore, the identification of the hand state by the identification means 100 is not limited to an image captured by the imaging unit 24 of the HMD 2, and may be based on an image captured by a camera installed separately from the HMD 2 (for example, an external camera of the HMD, or a camera of a smartphone used as the real object 4). The camera separate from the HMD 2 may be paired with the HMD 2 in advance.

[0094] In addition, the identification means 100 may identify the hand condition based on the back or palm of the hand, the front or back of the wrist, a wristband (for example, a wristband with light-emitting elements that recognizes the front or back of the hand based on the orientation of the light-emitting elements), ease of obtaining biometric information (veins are easier to obtain on the back of the hand, etc.), etc.

[0095] Furthermore, when the identification means 100 identifies the hand state of a third party, the camera capturing the real object 4 may be a camera installed on a predetermined object around the user U (e.g., a security camera installed on a staircase outside the user's home). The camera may identify a predetermined object around the user U (e.g., a staircase outside the user's home) as a real object, and identify the hand state of the third party relative to the real object or the positional relationship between the third party's hand state and the real object. In this case, the display control means 104 may superimpose a warning image that warns of a third party's intrusion on the virtual space image. This may prevent a third party from knowing how the user U is playing using the HMD 2.

[0096] Alternatively, when the identification means 100 identifies the hand state of a third party, the camera capturing the real object 4 may be a camera installed on a predetermined object around the user U (e.g., a security camera installed on an intercom at the entrance). The camera may identify the hand state of the third party relative to a predetermined object around the user U (e.g., an intercom at the entrance) as a real object, or the positional relationship between the third party's hand state and the real object. In this case, the display control means 104 may superimpose a notification image of the third party's visit on the virtual space image.

[0097] In addition, in the above embodiment, the reception means 102 receives a specification from the user U of the type of real object 4 to be used during play by the user U wearing the HMD 2, but the reception may be limited to a type and may also include a specification of the real object 4.

[0098] In the above embodiment, the game device 1 functions independently as a program, but the present invention is not limited to this. At least some of the means of the program may be realized in cooperation with a server to which the game device 1 is connected via a network.

[0099] Even when these other embodiments are adopted, the effects of the present embodiment are exhibited. In addition, the present embodiment can be appropriately combined with other embodiments, and other embodiments can be appropriately combined with each other. [Explanation of symbols]

[0100] X HMD System 1. Game device 2 HMD 3. Controller 4 Real Objects 10 Control section 11 Storage section 12 Communications Department 13 Image processing section 14 Audio processing unit 20 Control section 21 Memory section 22 Communications Department 23 Sensor section 24 Imaging unit 25 Display section 26 Audio input / output section 100 Specific means 101 Progress Control Means 102 Reception methods 103 Means of change 104 Display control means 110 Programs 111 VR space configuration information 112 Parameters U User

Claims

1. A program for displaying a virtual space image on a head-mounted display device, An identification means for identifying a positional relationship between a hand of a user or a third party wearing the head-mounted display device and a real object present in the real space, the real object being different from the head-mounted display device; a display control means for controlling display of a real object image, which is an image corresponding to the real object and is superimposed on the virtual space image, based on the specified positional relationship; a progress control means for controlling the progress of a content associated with the virtual space image; when the physical object is a first type of object, the progress control means stops the progress of the content based on the positional relationship specified by the specifying means in a state in which the display control means displays the physical object image superimposed on the virtual space image; When the real object is a second type of object different from the first type, the display control means displays an image of the real object superimposed on the virtual space image while the progress control means is progressing the content.

2. the identifying means identifies the positional relationship based on an image of the real space captured by a camera provided in the head-mounted display device.

2. The program according to claim 1 .

3. the display control means displays the real object image superimposed on the virtual space image even if the condition is not satisfied, and changes a display mode of the real object image when the condition is satisfied.

3. The program according to claim 1 or 2.

4. A receiving means for receiving at least one of the first type of object and the second type of object by designation from a user, 4. The program according to claim 1, further causing the computer to function.

5. A display control device that displays a virtual space image on a head-mounted display device, An identification means for identifying a positional relationship between a hand of a user or a third party wearing the head-mounted display device and a real object present in the real space, the real object being different from the head-mounted display device; a display control means for controlling display of a real object image, which is an image corresponding to the real object and is superimposed on the virtual space image, based on the specified positional relationship; a progress control means for controlling the progress of a content associated with the virtual space image; Equipped with when the physical object is a first type of object, the progress control means stops the progress of the content based on the positional relationship specified by the specifying means in a state in which the display control means displays the physical object image superimposed on the virtual space image, A display control device characterized in that, when the real object is a second type of object different from the first type, the display control means displays the real object image superimposed on the virtual space image while the progress control means is progressing the content.

Images