Program and display control device

Abstract

To contribute to improving the amusement of play in a VR space.SOLUTION: A program causes a head-mounted display device to display a virtual space image, the program causing a computer to function as: identification means for, in a real space, identifying a state of a hand of a user wearing the head-mounted display device or a third person and / or a positional relation between the hand and a real object existing in the real space; and display control means for controlling display of a real object image, which is an image superimposed on the virtual space image and corresponding to the real object, on the basis of the identified state and / or the positional relation.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a program for controlling display for a head-mounted display and a display control device.

Background Art

[0002] In recent years, game devices using a head-mounted display (HMD) have been developed, and games can be played in a virtual reality (VR) space provided by the HMD. In such a game device, the posture or movement of the HMD is tracked to control the viewing area in the VR space visible to the user. As a result, the user can immerse themselves in the VR space and play the game (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] A user who plays a game in the VR space while wearing an HMD cannot view the real space while wearing the HMD, so it is assumed that it becomes difficult to operate an object in the real space (hereinafter referred to as "real object"). For example, while wearing the HMD, the user may not be able to respond to an incoming call on their smartphone. Thus, a user wearing an HMD may be distracted by the presence and operation of real objects in the real space. ​​​​​​​​​​ As a result, there is a risk that the excitement of playing in a VR space will not be fully realized. .

[0005] This disclosure relates to a program and a display system that contribute to improving the enjoyment of play in a VR space. Provide control equipment. [Means for solving the problem]

[0006] The first aspect is: A program for displaying a virtual space image on a head-mounted display device, A state of the hand of a user or a third party wearing the head-mounted display device in real space; And / or, determining a positional relationship between the hand and a real object existing in the real space. Means, Based on the specified state and / or the positional relationship, a display control means for controlling display of a real object image, which is an image corresponding to the real object, and, It is a program characterized by causing a computer to function as follows.

[0007] In the first aspect, the specified state is a direction of the hand as seen by the user. Including, The display control means displays the real object based on the orientation and / or the positional relationship. An image can be displayed superimposed on the virtual space image.

[0008] In addition, in the first aspect, the identification means is a camera provided in the head mounted display device. The orientation and / or the positional relationship is determined based on a captured image of the real space by It is possible.

[0009] Also, in the first aspect, the computer is further caused to function as progress control means for controlling the progress of the content associated with the virtual space image, and the progress control means can stop the progress of the content based on the specified state and / or the positional relationship.

[0010] Also, in the first aspect, the progress control means can determine whether to stop the progress of the content based on the type of the real object.

[0011] Also, in the first aspect, the display control means can display the real object image superimposed on the virtual space image even if the condition is not satisfied, and can change the display mode of the real object image when the condition is satisfied.

[0012] Also, in the first aspect, the computer can be further caused to function as reception means for receiving the type of the real object by designation from the user.

[0013] Also, in the first aspect, in response to an instruction from the user, a parameter associated with the content associated with the virtual space image is pre-associated with the real object, and the computer can be further caused to function as change means for changing the parameter in response to an operation on the real object by the user.

[0014] The second aspect is a display control device that displays a virtual space image on a head-mounted display device, and specifying means for specifying the state of the hand of the user or a third party wearing the head-mounted display device in the real space and / or the positional relationship between the hand and a real object existing in the real space. ​​​​​​​​​​​​ Based on the specified state and / or the positional relationship, control the display of a real object image, which is an image corresponding to the real object and is superimposed on the virtual space image display control means and A display control device characterized by comprising the same

Effects of the Invention

[0015] According to the present disclosure, a program and a display control device that contribute to improving the interest of play in the VR space can be provided

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0017] Embodiments of the present disclosure will be described with reference to the drawings.

[0018] [Overview of the HMD System] FIG. 1 is a diagram showing an example of the overview of the HMD system according to this embodiment. As shown in FIG. 1, the HMD system X includes a game device 1, an HMD 2 worn on the head of the user U, and a controller 3 used for the user U to play content in the VR space, and is configured thereby.

[0019] The game device 1 is a home game console such as a PlayStation (registered trademark), a portable game console such as a Nintendo Switch (registered trademark), or an electronic device such as a personal computer, a smartphone, or a tablet. The game device 1 controls the display of an image corresponding to the VR space (hereinafter referred to as a "virtual space image") on the HMD 2. Hereinafter, it is assumed that the game device 1 controls the display of the virtual space image and is a device capable of advancing the content associated with the virtual space image, but it is not limited thereto. The game device 1 may be any display control device capable of controlling the display of the virtual space image. The content associated with the virtual space image may be, for example, a VR game in which an object in the VR space is operated following the movement of the user U, the operation of the controller 3, the movement, etc. The object is a character, and the content may be a game in which the character is made to act in the VR space or made to fight against an enemy character in the VR space. The HMD 2 is a head-mounted display device.

[0020]

[0021]

[0022] ​​​​​​​As shown in FIG. 1, a user U wearing an HMD 2 can see a virtual space image displayed on the HMD 2. While the user can feel immersed in the VR space created by the image, the user cannot see the real objects in the real world. As a result of being distracted by the body 4, the enjoyment of playing in the VR space is not fully achieved. For example, in the real space shown in Figure 1, a smartphone exists as a real object 4. However, even if the smartphone receives a call, the user U is wearing the HMD2. In this state, it becomes difficult to hold the smartphone, so There is a risk that you will not be able to concentrate on (i).

[0023] Therefore, the game device 1 detects the state of the hand of the user U or a third party (hereinafter referred to as the "hand state"). and / or the position of the hand of the user U or a third party and a real object 4 existing in the real space. The game device 1 determines a positional relationship based on the determined hand state and / or positional relationship. Then, an image corresponding to the real object 4 (hereinafter referred to as a "real object image") is displayed on the HMD 2. Specifically, the game device 1 controls the display of a real object image on a virtual space image. Here, "superimposition" refers to the display of the HMD2. This refers to drawing a real object image in a specified area within a virtual space image displayed on the screen.

[0024] In this way, the real object image is superimposed on the virtual space image displayed on the HMD2. As a result, the user U can visually recognize the presence of the real object 4 even while wearing the HMD 2. Therefore, you can continue playing in the VR space while paying attention to the real object 4. This can improve the interest of playing the game.

[0025] In the following, we will consider the state of the user's hands when wearing the HMD 2 and / or the relationship between the user's hands and the real object. A method for controlling the superimposition of real object images on virtual space images based on the positional relationship with the human body. However, the present invention is not limited to this example. and / or the positional relationship between the third person's hand and a real object, Of course, it is also possible to control the superimposition of a real object image on the object.

[0026] [HMD system configuration] <Hardware configuration> The hardware configuration of the HMD system X of this embodiment will be described. FIG. 1 is a diagram showing an example of a hardware configuration of an HMD system according to the embodiment of the present invention. The main hardware configuration of the device 1 and the HMD 2 will be described. In addition, the controller 3 may include a communication unit and an input receiving unit (not shown). Of course, it is also possible to provide a control panel (e.g., a button, etc.).

[0027] The game device 1 includes a control unit 10, a storage unit 11, a communication unit 12, an image processing unit 13, and a sound processing unit. The storage unit 11, the communication unit 12, the image processing unit 13, and the audio processing unit 14 are Each of them is connected to the control unit 10 via a dedicated bus, interface, etc.

[0028] The control unit 10 is a CPU (Central Processor) that controls the operation of the game device 1. The storage unit 11 mainly includes a HDD (Hard Disk Drive). kDrive), RAM (Random Access Memory), and ROM (Re The communication unit 12 is configured to communicate with the HMD wirelessly or via a wired connection. 2. Communicate with the controller 3. Further, the communication unit 12 may communicate with a server or the like (not shown). It is also acceptable.

[0029] The image processing unit 13 draws (renders) an image to be displayed on the display unit 25 of the HMD 2 according to an instruction from the control unit 10. Specifically, the image processing unit 13 draws an imaginary space image corresponding to the VR space. The VR space is formed, for example, on the celestial sphere centered at a predetermined position. Based on the configuration information of the VR space (hereinafter referred to as "VR space configuration information") held in the storage unit 11, the image processing unit 13 draws an image (hereinafter referred to as "field of view image") corresponding to the field of view area of the user U in the VR space as an imaginary space image. The field of view image changes following the posture or movement of the HMD 2 and the movement of at least a part of the user U's body (for example, head, eyes, hands, etc.). Also, the image processing unit 13 draws a real object image to be superimposed on the imaginary space image on the display unit 25 of the HMD 2. The VR space is formed, for example, on the celestial sphere centered at a predetermined position. The image processing unit 13 draws an image corresponding to the field of view area of the user U in the VR space (hereinafter referred to as "field of view image") as an imaginary space image based on the configuration information of the VR space (hereinafter referred to as "VR space configuration information") held in the storage unit 11. The field of view image changes following the posture or movement of the HMD 2 and the movement of at least a part of the user U's body (for example, head, eyes, hands, etc.). Also, the image processing unit 13 draws a real object image to be superimposed on the imaginary space image on the display unit 25 of the HMD 2. The field of view image changes following the posture or movement of the HMD 2 and the movement of at least a part of the user U's body (for example, head, eyes, hands, etc.). Also, the image processing unit 13 draws a real object image to be superimposed on the imaginary space image on the display unit 25 of the HMD 2. The field of view image changes following the posture or movement of the HMD 2 and the movement of at least a part of the user U's body (for example, head, eyes, hands, etc.). Also, the image processing unit 13 draws a real object image to be superimposed on the imaginary space image on the display unit 25 of the HMD 2. The field of view image changes following the posture or movement of the HMD 2 and the movement of at least a part of the user U's body (for example, head, eyes, hands, etc.). Also, the image processing unit 13 draws a real object image to be superimposed on the imaginary space image on the display unit 25 of the HMD 2. The field of view image changes following the posture or movement of the HMD 2 and the movement of at least a part of the user U's body (for example, head, eyes, hands, etc.). Also, the image processing unit 13 draws a real object image to be superimposed on the imaginary space image on the display unit 25 of the HMD 2.

[0030] The audio processing unit 14 reproduces and synthesizes audio data corresponding to an instruction from the control unit 10, performs digital-to-analog (D / A) conversion, and outputs it as audio. The audio is output from the speaker or earphone output terminal of the audio input / output unit 26 of the HMD 2. Further, the audio processing unit 14 may also input the audio signal input from the microphone of the audio input / output unit 26 of the HMD 2 after performing analog-to-digital (A / D) conversion. The audio is output from the speaker or earphone output terminal of the audio input / output unit 26 of the HMD 2. Further, the audio processing unit 14 may also input the audio signal input from the microphone of the audio input / output unit 26 of the HMD 2 after performing analog-to-digital (A / D) conversion. The audio is output from the speaker or earphone output terminal of the audio input / output unit 26 of the HMD 2. Further, the audio processing unit 14 may also input the audio signal input from the microphone of the audio input / output unit 26 of the HMD 2 after performing analog-to-digital (A / D) conversion. The audio processing unit 14 reproduces and synthesizes audio data corresponding to an instruction from the control unit 10, performs digital-to-analog (D / A) conversion, and outputs it as audio. The audio is output from the speaker or earphone output terminal of the audio input / output unit 26 of the HMD 2. Further, the audio processing unit 14 may also input the audio signal input from the microphone of the audio input / output unit 26 of the HMD 2 after performing analog-to-digital (A / D) conversion. The audio processing unit 14 reproduces and synthesizes audio data corresponding to an instruction from the control unit 10, performs digital-to-analog (D / A) conversion, and outputs it as audio. The audio is output from the speaker or earphone output terminal of the audio input / output unit 26 of the HMD 2. Further, the audio processing unit 14 may also input the audio signal input from the microphone of the audio input / output unit 26 of the HMD 2 after performing analog-to-digital (A / D) conversion. It is also possible.

[0031] The HMD 2 is provided with a control unit 20, a storage unit 21, a communication unit 22, a sensor unit 23, a display unit 25, and an audio input / output unit 26. The storage unit 21, the communication unit 22, the sensor unit 23, the display unit 25, and the audio input / output unit 26 are also provided. 25. The voice input / output unit 26 is connected to the control unit 20 via a dedicated bus, interface, etc. respectively.

[0032] The control unit 20 controls the operation of the HMD2. The storage unit 21 is mainly composed of a RAM and a ROM. The communication unit 22 communicates with the game device 1 wirelessly or by wire.

[0033] The sensor unit 23 detects the posture or movement of the HMD2, the movement of at least a part of the body of the user U (e.g., the head, eyes, hands, etc.). The sensor unit 23 may be, for example, at least one of a laser sensor, an infrared sensor, a magnetic sensor, an angular velocity sensor, an acceleration sensor, and an eye tracking device (Eye Tracking Device: ETD), etc. The sensor unit 23 may be built into the HMD2, or may be provided in a device separate from the HMD2 (e.g., a controller ).

[0034] The imaging unit 24 images a predetermined area in the real space. The imaging unit 24 is composed of, for example, one or more cameras and may be able to image different directions (e.g., up, down, left, or right directions, etc.).

[0035] The display unit 25 is composed of, for example, a non-transmissive display. The display unit 25 displays the above virtual space image as a virtual space image. The display unit 25 may include a display unit for the right eye that provides a right eye image and a display unit for the left eye that provides a left eye image. By visually recognizing the right eye image and the left eye image with the right eye and the left eye respectively, the user U can visually recognize a three-dimensional VR space. Note that if the right eye image and the left eye image can be provided, the display unit may be one. Yes. Further, the display unit 25 may superimpose and display a real object image on the virtual space image.

[0036] The audio input / output unit 26 is a speaker for headphones, earphones, a microphone, an ex Ita, etc. The audio input / output unit 26 can reproduce sound.

[0037] The above-described game device 1, HMD 2, and controller 3 are hardware resources for executing the program 11 0 (FIG. 3).

[0038] <Functional configuration> Next, the functional configurations of the control unit 10 and the storage unit 11 of the game device 1 that realizes the HMD system X will be described. Note that the following describes the main functions of the game device 1 Of course, the game device 1 may have functions not shown in the drawings.

[0039] The game device 1 stores, in the storage unit 11, a program 110, VR space configuration information 111, and parameters 112 for realizing the HMD system X of the present invention. The program 1 10 is a program for operating the game device 1 as the above-described functional means. The game Device 1 advances the content associated with the virtual space image based on the program 110.

[0040] The VR space configuration information 111 is, for example, information constituting a background image rendered on a celestial sphere forming the VR space, or information constituting various objects existing in the VR space. The information constituting various objects may be, for example, the coordinate positions of various objects in the VR space calculated from the position,

[0041] tilt, etc. of the controller 3 in the real space.

[0041] Parameter 112 is a parameter associated with the above content. For example, V It may be a parameter regarding an object in the R space. Regarding the parameter of the object It may be, for example, the hit point (HP) of the object, etc. and so on.

[0042] By executing the program 110 stored in the storage unit 11, the control unit 10 functions as a specific means 100, a progress control means 101, a reception means 102, a change means 103, and a display control means 10 4.

[0043] - Specific means 100 - The specific means 100 specifies the hand state of the user U wearing the HMD2 in the real space and / or the positional relationship between the hand of the user U and the real object 4.

[0044] The specific means 100 may specify the hand state (for example, the direction of the hand) of the user U wearing the HMD2 based on the captured image of the real space by the imaging unit 24 of the HMD2. Here, the hand state of the user U may include the direction of the hand (for example, the palm or the back of the hand) as seen from the user U wearing the HMD2, the posture, etc. For example, the specific means 100 detects the hand by performing image recognition processing on the captured image, and determines whether the direction of the hand is the palm or the back of the hand as seen from the user U. The hand state of the user U may include the direction of the hand (for example, the palm or the back of the hand), the posture, etc. as seen from the user U wearing the HMD2. For example, the specific means 100 may detect the hand by performing image recognition processing on the captured image, and determine whether the direction of the hand is the palm or the back of the hand as seen from the user U. The hand state of the user U may include the direction of the hand (for example, the palm or the back of the hand), the posture, etc. as seen from the user U wearing the HMD2. For example, the specific means 100 may detect the hand by performing image recognition processing on the captured image, and determine whether the direction of the hand is the palm or the back of the hand as seen from the user U. or the back of the hand by the image recognition processing of the captured image, and may specify which one it is. and so on.

[0045] Also, the specific means 100 may specify the positional relationship between the hand of the user U wearing the HMD2 and the real object based on the captured image of the real space by the imaging unit 24 of the HMD2. Here the positional relationship between the hand of the user U and the real object may be, for example, the distance between the hand and the real object, the direction in which the real object exists with respect to the hand, etc. Here, the positional relationship between the hand of the user U and the real object may be, for example, the distance between the hand and the real object, the direction in which the real object exists with respect to the hand, etc. and so on.

[0046] For example, the specifying means 100 detects a hand and a real object by performing image recognition processing on the captured image. Then, it may specify the distance between the hand of the user U and the real object and / or the direction in which the real object exists with respect to the hand. The distance between the hand and the real object may be, for example, the distance between a predetermined position of the hand recognized from the captured image and a predetermined position of the real object. Also, the direction in which the real object exists with respect to the hand may be the direction with respect to the plane of the hand recognized from the captured image (e.g., the normal direction, etc.), the up, down, left, right, or depth direction with respect to a predetermined position of the hand, etc. .

[0047] - Progression control means 101 - The progression control means 101 controls the progression of the content associated with the virtual space image. Specifically, the progression control means 101 stops the progression of the content based on the hand state of the user U specified by the specifying means 100 and / or the positional relationship between the hand of the user U and the real object 4. Note that the stop of the content may mean temporarily stopping the content in a state where it can be resumed.

[0048] More specifically, the progression control means 101 switches the mode regarding the progression of the content based on the hand state and / or the positional relationship specified by the specifying means 100, and may control the progression of the content based on the switched mode. The mode is a mode regarding the progression of the content, for example, a mode in which the content progresses (hereinafter, referred to as the "normal mode"), a mode in which the user U wearing the HMD 2 searches for the real object 4 (hereinafter, referred to as the "search mode"), a mode in which the user U operates the real object 4 (hereinafter ​​​​​​​​​​​​It may include, for example, "operation mode", etc.

[0049] For example, when the movement control means 101 detects an operation to grasp the real object 4 from the hand state and / or positional relationship, it may switch the normal mode to the search mode. Also, when the movement control means 101 detects an operation to grasp the real object 4 from the hand state and / or positional relationship, it may switch the search mode to the operation mode. Further, when the movement control means 101 detects that the operation of the real object 4 has ended from the hand state and / or positional relationship, it may switch the operation mode to the normal mode. Note that the movement control means 101 may switch the above mode based on a signal from the controller 3 (input from the user U). Also, when the movement control means 101 detects an operation to grasp the real object 4 from the hand state and / or positional relationship, it may switch the search mode to the operation mode. Further, when the movement control means 101 detects that the operation of the real object 4 has ended from the hand state and / or positional relationship, it may switch the operation mode to the normal mode. Note that the movement control means 101 may switch the above mode based on a signal from the controller 3 (input from the user U). Also, when the movement control means 101 detects an operation to grasp the real object 4 from the hand state and / or positional relationship, it may switch the search mode to the operation mode. Further, when the movement control means 101 detects that the operation of the real object 4 has ended from the hand state and / or positional relationship, it may switch the operation mode to the normal mode. Note that the movement control means 101 may switch the above mode based on a signal from the controller 3 (input from the user U). Also, when the movement control means 101 detects that the operation of the real object 4 has ended from the hand state and / or positional relationship, it may switch the operation mode to the normal mode. Note that the movement control means 101 may switch the above mode based on a signal from the controller 3 (input from the user U). Also, when the movement control means 101 detects that the operation of the real object 4 has ended from the hand state and / or positional relationship, it may switch the operation mode to the normal mode. Note that the movement control means 101 may switch the above mode based on a signal from the controller 3 (input from the user U). Also, when the movement control means 101 detects that the operation of the real object 4 has ended from the hand state and / or positional relationship, it may switch the operation mode to the normal mode. Note that the movement control means 101 may switch the above mode based on a signal from the controller 3 (input from the user U). Also, when the movement control means 101 detects that the operation of the real object 4 has ended from the hand state and / or positional relationship, it may switch the operation mode to the normal mode. Note that the movement control means 101 may switch the above mode based on a signal from the controller 3 (input from the user U).

[0050] In addition, the movement control means 101 may continue the progress of the content in the normal mode and stop the progress of the content in the search mode or the operation mode. Also, in the search mode, the movement control means 101 may continue the game progress and stop the game progress when transitioning from the search mode to the operation mode. In addition, the movement control means 101 may continue the progress of the content in the normal mode and stop the progress of the content in the search mode or the operation mode. Also, in the search mode, the movement control means 101 may continue the game progress and stop the game progress when transitioning from the search mode to the operation mode. In addition, the movement control means 101 may continue the progress of the content in the normal mode and stop the progress of the content in the search mode or the operation mode. Also, in the search mode, the movement control means 101 may continue the game progress and stop the game progress when transitioning from the search mode to the operation mode. In addition, the movement control means 101 may continue the game progress and stop the game progress when transitioning from the search mode to the operation mode.

[0051] In addition, the movement control means 101 may determine whether to stop the progress of the content based on the type of the real object 4 (for example, smartphone, drink, snack, etc.). For example, when the type of the real object 4 is a specific type (for example, smartphone), the movement control means 101 may stop the progress of the content, and when the type of the real object 4 is another type (for example, drink or snack), the movement control means 101 may continue the progress of the content. In addition, the movement control means 101 may determine whether to stop the progress of the content based on the type of the real object 4 (for example, smartphone, drink, snack, etc.). For example, when the type of the real object 4 is a specific type (for example, smartphone), the movement control means 101 may stop the progress of the content, and when the type of the real object 4 is another type (for example, drink or snack), the movement control means 101 may continue the progress of the content. In addition, the movement control means 101 may determine whether to stop the progress of the content based on the type of the real object 4 (for example, smartphone, drink, snack, etc.). For example, when the type of the real object 4 is a specific type (for example, smartphone), the movement control means 101 may stop the progress of the content, and when the type of the real object 4 is another type (for example, drink or snack), the movement control means 101 may continue the progress of the content. In addition, the movement control means 101 may determine whether to stop the progress of the content based on the type of the real object 4 (for example, smartphone, drink, snack, etc.). For example, when the type of the real object 4 is a specific type (for example, smartphone), the movement control means 101 may stop the progress of the content, and when the type of the real object 4 is another type (for example, drink or snack), the movement control means 101 may continue the progress of the content. In addition, the movement control means 101 may determine whether to stop the progress of the content based on the type of the real object 4 (for example, smartphone, drink, snack, etc.). For example, when the type of the real object 4 is a specific type (for example, smartphone), the movement control means 101 may stop the progress of the content, and when the type of the real object 4 is another type (for example, drink or snack), the movement control means 101 may continue the progress of the content.

[0052] - Reception means 102 - The reception means 102 receives the type of the real object 4 used by the user U wearing the HMD2 during play according to the designation from the user U. Specifically, the reception means 102 may select the type of the real object 4 based on the designation from the user U before the start of play or during play. It may also be good.

[0053] In addition, the reception means 102 may receive the selection of the real object 4 whose parameter 112 is updated by use during play according to the designation from the user U. The reception means 102 may associate the selected real object 4 with the parameter 112 to be updated.

[0054] -Change means 103- The change means 103 changes the parameter 112 according to the operation on the real object 4 by the user U. The parameter 112 is associated with the real object 4 according to the instruction of the user U and stored in the storage unit 11. The instruction of the user U may be input, for example, using the controller 3.

[0055] -Display control means 104- The display control means 104 causes the image processing unit 13 to draw a view image corresponding to the view area in the VR space as a virtual space image. The drawn virtual space image is displayed on the display unit 25 of the HMD2.

[0056] In addition, the display control means 104 controls the superimposed display of the real object image on the virtual space image based on the hand state specified by the specifying means 100 and / or the positional relationship between the hand and the real object 4. The display control means 104 detects a predetermined operation on the real object 4 using the hand of the user U based on the hand state and / or the positional relationship between the hand and the real object 4, and the detection result ​​​​​​​​​​​​The superimposed display of the real object image may be controlled based on the result. The predetermined operation may be, for example, an operation of grasping the real object 4 using the hand of the user U, an operation of attempting to grasp, an operation of operating, or an operation of attempting to operate and the like.

[0057] For example, when a predetermined operation on the real object 4 using the hand of the user U is detected, the real object image is superimposed and displayed at a specific position in the virtual space image, and when the predetermined operation is not detected, it is not necessary to superimpose and display the real object image. In this way, the display control means 104 may control whether to superimpose and display the real object image based on the detection result of the predetermined operation.

[0058] Alternatively, the display control means 104 may superimpose and display the real object image at a specific position in the virtual space image regardless of whether a predetermined operation on the real object 4 using the hand of the user U is detected. In this case, the display control means 104 may change the display mode (for example, position, size, etc.) of the real object image according to the detection result of the predetermined operation. That is, the display control means 104 may superimpose and display the real object image on the virtual space image even if a condition (for example, detecting an operation of attempting to grasp the real object 4) is not satisfied, and may change the display mode of the real object image when the condition is satisfied.

[0059] Further, the display control means 104 may control the superimposed display of the real object image on the virtual space image based on the mode switched by the progress control means 101.

[0060] FIG. 4 is a diagram showing an example of display control of a real object image in the normal mode according to the present embodiment. That is. In FIG. 4, it is assumed that the real object 4 corresponding to the real object image is a smartphone, but it is not limited to this. The real object 4 and the real object image may be specified in advance by the user U. Also, in FIG. 4, the predetermined operation on the real object 4 using the hand of the user U is assumed to be an operation of trying to grasp the real object 4 using the hand of the user U, but it is not limited to this. As shown in FIG. 4, in the normal mode, the display control means 104 may cause the display unit 25 of the HMD 2 to display an object image corresponding to the movement of the object (here, the opening and closing of the virtual hand) according to a specific operation (for example, pressing or not pressing a button) of the controller 3 by the user U. As shown in FIG. 4, if the operation of trying to grasp the real object 4 using the hand of the user U is not detected, the display control means 104 may superimpose and display the real object image at a specific position in the virtual space image. The specific position may be, for example, a position in the VR space corresponding to the positional relationship between the user U and the real object 4 in the real space, or a non - conspicuous position in the virtual space image. Alternatively, if the operation of trying to grasp the real object 4 using the hand of the user U is not detected, the display control means 104 may not display the real object image in the virtual space image. FIG. 5 is a diagram showing an example of the display control of the real object image in the search mode according to the present embodiment. In FIG. 5, the description of the same points as in FIG. 4 will not be repeated. As shown in FIG. 5, in the search mode, the progress control means 101 may stop the game progress or may continue the game progress.

[0061] As shown in FIG. 4, in the normal mode, the display control means 104 may cause the display unit 25 of the HMD 2 to display an object image corresponding to the movement of the object (here, the opening and closing of the virtual hand) according to a specific operation (for example, pressing or not pressing a button) of the controller 3 by the user U. As shown in FIG. 4, if the operation of trying to grasp the real object 4 using the hand of the user U is not detected, the display control means 104 may superimpose and display the real object image at a specific position in the virtual space image. The specific position may be, for example, a position in the VR space corresponding to the positional relationship between the user U and the real object 4 in the real space, or a non - conspicuous position in the virtual space image. Alternatively, if the operation of trying to grasp the real object 4 using the hand of the user U is not detected, the display control means 104 may not display the real object image in the virtual space image. (Here, the opening and closing of the virtual hand) to the display unit 25 of the HMD 2. It may be shown.

[0062] As shown in FIG. 4, if the operation of trying to grasp the real object 4 using the hand of the user U is not detected, the display control means 104 may superimpose and display the real object image at a specific position in the virtual space image. The specific position may be, for example, a position in the VR space corresponding to the positional relationship between the user U and the real object 4 in the real space, or a non - conspicuous position in the virtual space image. Or, the display control means 104 may not display the real object image in the virtual space image. The specific position may be, for example, a position in the VR space corresponding to the positional relationship between the user U and the real object 4 in the real space, or a non - conspicuous position in the virtual space image. Or, the display control means 104 may not display the real object image in the virtual space image. If the operation of trying to grasp the real object 4 using the hand of the user U is not detected, the display control means 104 may not display the real object image in the virtual space image. It doesn't have to be.

[0063] FIG. 5 is a diagram showing an example of the display control of the real object image in the search mode according to the present embodiment. In FIG. 5, the description of the same points as in FIG. 4 will not be repeated. As shown in FIG. 5, in the search mode, the progress control means 101 may stop the game progress or may continue the game progress. It may be.​

[0064] In the search mode, the display control means 104 displays the user identified by the identification means 100. Based on the orientation of the user U's hand, the motion of the user U trying to grasp the real object 4 with the hand is detected. For example, as shown in FIG. 5, the display control unit 104 displays the back of the user U's hand as When the user U is facing the U side, the user U tries to grasp the real object 4 with his / her hand. In this case, the display control means 1 may display the real object image superimposed on the virtual space image. 04 is the display mode (e.g., size, position, etc.) of the real object image in the virtual space image By controlling the presence of the real object 4 in the VR space, the presence of the real object 4 in the VR space is reduced compared to the normal mode shown in Figure 4. For example, the display control unit 104 may make the current image displayed in a reduced size in the normal mode stand out. The real object image may be displayed at full size in the search mode.

[0065] As shown in FIG. 5, in the search mode, the display control means 104 displays a virtual space image. An image corresponding to a virtual hand (hereinafter referred to as a "virtual hand image") may be superimposed on the image. The display control means 104 controls the display according to the positional relationship between the hand of the user U and the real object 4 in the real space. The virtual hand image and the real object image may be displayed superimposed on the virtual space image.

[0066] In addition, the display control means 104 determines the orientation of the hand of the user U identified by the identification means 100. In addition, the user U moves the real object 4 using the hand based on the positional relationship between the hand and the real object 4. For example, the display control unit 104 detects the user U's hand orientation according to the condition. When the condition is satisfied and the positional relationship between the hand and the real object 4 satisfies the condition, the real object 4 is grasped. Detect the intended operation and, as described above, superimpose and display the real object image within the virtual space image This may be done. The conditions for the positional relationship between the hand and the real object 4 are, for example, that the distance in a predetermined direction (e.g., up, down, left, right, and depth direction, etc.) between the hand and the real object 4 satisfies a predetermined condition and / or at least one of whether the real object exists in a predetermined direction (e.g., the normal direction) with respect to the plane of the hand may be sufficient.

[0067] As shown in FIG. 5, in the search mode, when a predetermined operation (here, an operation to grasp the real object 4) is detected based on the hand state of the user U identified by the identification means 100 and / or the positional relationship between the hand and the real object 4 the display mode of the real object image is changed to be more prominent compared to the normal mode, so that the user U can more easily grasp the position of the real object 4 in the real space

[0068] FIG. 6 is a diagram showing an example of the display control of the real object image in the operation mode according to the present embodiment In FIG. 6, descriptions of the same points as in FIG. 4 or FIG. 5 are not repeated. As shown in FIG. 6 in the operation mode, the progress control means 101 stops the progress of the game

[0069] In the operation mode, the display control means 104 detects an operation of operating the real object 4 using the hand of the user U based on the orientation of the hand of the user U identified by the identification means 100 and the positional relationship between the hand and the real object 4 For example, as shown in FIG. 6, when the palm of the hand of the user U faces the user U side and the distance between the hand and the real object 4 is within a predetermined range the display control means 104 may detect an operation of operating the real object 4 using the hand of the user U and superimpose and display the real object image within the virtual space image ​

[0070] Note that the operation of operating the real object 4 is when the palm of the user U faces the user U side is not limited to this, and may be detected based on the orientation of at least a part of the user U's body (for example, an arm, a thumb, etc.) and the distance between at least a part of the body and the real object 4.

[0071] Also, in the operation mode, the real object image superimposed on the virtual space image may be a captured image that captures a specific part of the real object 4 (for example, the display of a smartphone). The captured image may be, for example, a real-time captured image by the imaging unit 24 of the HMD2 .

[0072] In this way, by displaying the captured image in which the specific part of the real object 4 is focused as the real object image within the virtual space image, the user U can operate the real object 4 in the real space while wearing the HMD2 .

[0073] FIG. 7 is a diagram showing an example of display control when selecting a real object according to the present embodiment. For example , as shown in FIG. 7, the candidate real objects 4a and 4b to be used during play may be photographed by the imaging unit 24 of the HMD2 and recognized by image recognition processing of the captured image.

[0074] As shown in FIG. 7, the display control means 104 may superimpose and display real object images corresponding to the real objects 4a and 4b on the virtual space image on the selection screen before the start of play . The real object images may be superimposed and displayed at positions in the VR space corresponding to the real objects 4a and 4b in the real space, respectively. Also, in FIG. 7, the display control means 104 is real space ​A virtual hand image is superimposed and displayed at a position in the VR space corresponding to the hand of user U in the space.

[0075] As shown in the selection screen before the start of play in FIG. 7, the reception means 102 projects onto the VR space Based on the positional relationship between the real object 4a or 4b projected and the virtual hand, the real object 4a or 4b Selection may be accepted. For example, in FIG. 7, since the virtual hand in the VR space is trying to grasp the real object 4a The reception means 102 may accept the selection of the real object 4a.

[0076] Also, the display control means 104 may display the selection screen of the real object 4 on the display unit 25 of the HMD2 During the play of the game. For example, in the selection screen during play shown in FIG. 7, the real object The selection box of 4 is displayed. The reception means 102 is based on the instruction of user U input to the controller 3 Etc., the type of the real object 4 (here, a smartphone, a drink, or a snack) may be Selected. For example, when a drink is selected, the display control means 104 replaces the real object image corresponding to the real object 4 a with a real object image corresponding to the real object 4b and superimposes and displays it on the virtual space image May be.

[0077] FIG. 8 is a diagram showing an example of display control related to the association between a real object and a parameter according to the present embodiment. In FIG. 8, the description of the same points as in FIG. 7 will not be repeated. In FIG. 8 , The display control means 104 displays an association screen between the real object 4a or 4b and the parameter 112 Before the start of play. Also, the display control means 104, in the operation mode, uses the captured image of the real object 4b by the imaging unit 24 of the HMD2 as the real object image and superimposes and displays it on the virtual space Image. Is superimposed and displayed on the image.

[0078] For example, on the association screen before starting the play in FIG. 8, the reception means 102 receives the selection of the real object 4b according to the designation of the user U and may associate the real object 4b with an in-game effect (for example, the parameter 112 to be updated).

[0079] Also, on the operation mode screen during the play in FIG. 8, when the change means 103 detects a predetermined operation on the real object 4b using the hand of the user U it may update the parameter 112 associated with the real object 4b. For example, in FIG. 8, the predetermined operation is that the user U drinks the coffee in the cup which is the real object 4b The change means 103 may determine that the user U has drunk the coffee in the cup which is the real object 4b based on the captured image of the real object 4b and may recover the HP associated with the real object 4b

[0080] [Operation of the HMD System] FIG. 9 is a diagram showing an example of the operation of the HMD system according to the present embodiment. In FIG. 9, the operation of the game device 1 in the case where the user U wearing the HMD2 plays the content associated with the virtual space image is shown. Note that some steps in FIG. 9 may be omitted or some orders may be changed. For example, if step st103 is omitted the game device 1 may transition from the normal mode to the operation mode without the search mode Also, the operation in FIG. 9 ends when the game device 1 detects an end operation from the user U in any of steps st102, st103 or st104

[0081] In step st101, the game device 1 acquires the content associated with the virtual space image ​​​​​​Perform the settings before playing the game. For example, as shown in FIG. 7, before starting the game, the game device 1 may accept the type of the real object 4 to be used during the game from the designation by the user U. Also, as shown in FIG. 8, before starting the game, the game device 1 may accept the selection of the real object 4 for which the parameter 112 is updated during the game. Note that the type of the real object 4 to be used during the game may be changed during the game. In step st102, the game device 1 operates in the normal mode and continues the progress of the content. As shown in FIG. 4, in the normal mode, the game device 1 controls the operation of the content object (the opening and closing of the virtual hand in FIG. 4) in the VR space in response to the detection of a specific operation of the controller 3 by the user U, and may display the virtual space image including the content object on the display unit 25 of the HMD2. Further, as shown in FIG. 4, in the normal mode, the game device 1 superimposes and displays the real object image corresponding to the real object 4 selected in step st101 in the virtual space image so as not to be conspicuous compared to the search mode or the operation mode. Note that in the normal mode, the game device 1 may not superimpose and display the real object image. In step st103, when a predetermined operation using the hand of the user U is detected, the game device 1 transitions from the normal mode to the search mode. The predetermined operation may be, for example, an operation of trying to grasp the real object 4 using the hand of the user U. In the search mode, the game device 1 may stop or continue the progress of the content.

[0082]

[0083]

[0084]

[0085] ​​​​​​​​​​​​​ As shown in FIG. 5, in the search mode, the game device 1 superimposes and displays, more prominently than in the normal mode, a real object image corresponding to the real object 4 selected in step st101 within the virtual space image. Also, the game device 1 superimposes and displays the real object image and the virtual hand image in correspondence with the positional relationship between the hand of the user U in the real space and the real object 4. Since the user U can estimate the positional relationship between the real object 4 and his own hand from the positional relationship between the real object image and the virtual hand image, even in a state where the real space cannot be seen while wearing the HMD2, it is possible to make it easier to grasp the real object 4 in the real space. In step st104, when the game device 1 detects a predetermined operation using the hand of the user U, it transitions from the search mode to the operation mode. The predetermined operation may be, for example, an operation or an attempt to operate the real object 4 using the hand of the user U. In the operation mode, the game device 1 may stop the progress of the content. As shown in FIG. 6, in the operation mode, the game device 1 may superimpose and display a capture image including the real object 4 selected in step st101 within the virtual space image. Since the user U can confirm in real time a specific part of the real object 4 (for example, the screen of the smartphone in FIG. 6) in the capture image, even in a state where the real space cannot be seen while wearing the HMD2, the real object 4 can be operated. Also, in the operation mode, as shown in FIG. 8, the game device 1 performs a predetermined operation on the real object 4 using the hand of the user U (for example, in FIG. 8, drinking coffee using the real object 4).

[0086]

[0087]

[0088] ​ When detecting the (action), the parameter 112 associated with the real object 4 may be changed. Yes.

[0089] In step st105, the game device 1 determines whether a predetermined action by the user U has been detected. The predetermined action may be, for example, an action to end the operation of the real object 4, an action in which the real object 4 leaves the user U's hand, or the like. When the game device 1 detects the predetermined action (step st105; YES), it transitions from the operation mode to the normal mode and resumes the progress of the content. On the other hand, when the game device 1 does not detect the predetermined action (step st105; NO), it continues the operation in the operation mode.

[0090] To summarize the above, the program according to this embodiment is a program for causing the HMD2 (head-mounted display device) to display a virtual space image. In the real space, the user U wearing the HMD2 or the state of the hand of a third party, and / or the position relationship between the hand and the real object 4 existing in the real space are specified by the specifying means 100, and based on the specified state and / or position relationship, the display control means 104 for controlling the display of the real object image superimposed on the virtual space image functions the computer. [Effect] According to the HMD system X according to this embodiment, since the real object image is superimposed and displayed in the virtual space image on the display unit 25 of the HMD2, the user U can visually recognize the existence of the real object 4 even while wearing the HMD2. Therefore, it is possible to continue playing in the VR space while paying attention to the real object 4, and the interestingness of playing in the VR space can be improved.

[0091] [Other Embodiments] The various control means and processing procedures described in the above embodiments are examples, and are not intended to limit the present invention, its applicable objects, or the scope of its uses. The various control means and processing procedures can be appropriately modified in design without changing the gist of the present invention.

[0092] In the above embodiment, the game device 1 may identify the user U based on the hand state of the user U. Specifically, it may identify whether it is the hand of the user U himself or the hand of a third party, and based on the identification result, change the display mode of the virtual space image on the display unit 25 of the HMD2. For example, when a third party other than the user U covers the HMD2 saying "Who is it!", if the hand state of the third party is specified by the specifying means 100 of the game device 1, the display control means 104 may dim the virtual space image to be displayed on the display unit 25 of the HMD2. Note that whether it is the hand of the user U himself or the hand of a third party may be identified based on biometric information or the like.

[0093] Also, the specification of the hand state by the specifying means 100 is not limited to the captured image by the imaging unit 24 of the HMD2, and may be based on the captured image by a camera installed separately from the HMD2 (for example, an external camera of the HMD, a camera of a smartphone used as the real object 4). The camera separate from the HMD2 may be paired with the HMD2 in advance.

[0094] Also, the specification of the hand state by the specifying means 100 may be based on the back of the hand or the palm, the front or back of the wrist, a wristband (for example, a wristband with a light-emitting element, and the front and back of the hand are recognized by the direction of the light-emitting element),

[0095] the ease of obtaining biometric information (for example, veins are easier to obtain on the back side of the hand), etc.

[0095] Also, when the third - party's hand state is specified by the specifying means 100, image the real object 4. The camera may be a camera installed on a predetermined object around the user U (for example, a security camera installed on the stairs outside the house). With this camera, a predetermined object around the user U (for example, the stairs outside the house) is used as a real object, and the hand state of the third - party with respect to the real object or the positional relationship between the hand state of the third - party and the real object may be specified. In this case, the display control means 104 may superimpose and display a warning image warning of the intrusion of a third - party on the virtual - space image. Thereby, it may be possible to prevent the third - party from knowing the posture of the user U playing using the HMD2.

[0096] Alternatively, when the third - party's hand state is specified by the specifying means 100, the camera for imaging the real object 4 may be a camera installed on a predetermined object around the user U (for example, a security camera installed on the intercom at the entrance). With this camera, a predetermined object around the user U (for example, the intercom at the entrance) is used as a real object, and the hand state of the third - party with respect to the real object or the positional relationship between the hand state of the third - party and the real object may be specified. In this case, the display control means 104 may superimpose and display a notification image of the visit of a third - party on the virtual - space image.

[0097] Also, in the above - described embodiment, the reception means 102 was described as receiving the type of the real object 4 used by the user U wearing the HMD2 during play from the specification by the user U. However, it is not limited to the type, and the specification of the real object 4 may be received.

[0098] In the above - described embodiment, the case where the game device 1 functions a program alone was illustrated. However, ​​​It is not limited to this. At least some of the means of the program may be realized in conjunction with a server connected through a network by the game device 1.

[0099] Even when these other embodiments are adopted, the effects of this embodiment are exhibited. Also, it is possible to appropriately combine this embodiment with other embodiments and between other embodiments.

Explanation of Reference Numerals

[0100] X HMD system 1 Game device 2 HMD 3 Controller 4 Real object 10 Control unit 11 Storage unit 12 Communication unit 13 Image processing unit 14 Audio processing unit 20 Control unit 21 Storage unit 22 Communication unit 23 Sensor unit 24 Imaging unit 25 Display unit 26 Audio input / output unit 100 Specific means 101 Progression control means 102 Reception means 103 Modification means 104 Display control means 110 Program 111 VR space configuration information 112 Parameter U User

Claims

【Claim 1】 A program for causing a head-mounted display device to display a virtual space image, in the real space, the state of the hand of the user or a third party wearing the head-mounted display device, and / or a specifying means for specifying the positional relationship between the hand and a real object existing in the real space, and display control means for controlling the display of a real object image, which is an image corresponding to the real object and is superimposed on the virtual space image, based on the specified state and / or the positional relationship. A program characterized by causing a computer to function. ​ ​

Images