program

Abstract

To improve a user's virtual experience.SOLUTION: A program causes a computer to perform: setting a virtual viewpoint, an area to which a specific action is associated, and an operation object operated by a user, in a virtual space; changing at least any one of a position or attitude of the virtual viewpoint and an attitude of the operation object, according to movement of the user; and changing the area according to a change in at least one of the position or attitude of the virtual viewpoint and the operation object.

Description

Technical Field

[0001] The present disclosure relates to a program.

Background Art

[0002] Conventionally, games are known in which a user operates an avatar placed in a virtual space and enjoys being immersed in the virtual space. Patent Document 1 discloses an invention that performs control to avoid the risk of impairing safety due to the user moving too fast in the real space as a result of being too immersed in the virtual space when operating an operation target object such as a "sword" in the virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] However, the invention described in Patent Document 1 does not disclose the concept of how an avatar equips an equipment item such as a "sword" in the virtual space. For this reason, the technique as disclosed in Patent Document 1 leaves room for improvement in the virtual experience that the user virtually undergoes.

[0005] An object of the present disclosure is to provide a program capable of improving a user's virtual experience.

Means for Solving the Problems

[0006] According to one aspect of the present disclosure, the program causes the processor to perform the steps of: defining a virtual space; arranging a virtual viewpoint, a collision region having a defined relative position to the virtual viewpoint and associated with a collision action, and an operation object in the virtual space; changing at least one of the position and orientation of the virtual viewpoint in accordance with the movement of the head of a user to which a head-mounted device is associated; moving the collision region to the relative position after a first time has elapsed when at least one of the position and orientation of the virtual viewpoint has changed; moving the operation object in the virtual space in accordance with the movement of a first part of the user's body other than the user's head; generating the collision action in accordance with the fact that the operation object and the collision region have achieved a first positional relationship; and displaying a field of view image corresponding to the field of view from the virtual viewpoint on the head-mounted device. [Effects of the Invention]

[0007] According to this disclosure, it is possible to provide a program that can improve the user's virtual experience. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating the schematic configuration of an HMD system according to one embodiment. [Figure 2] This is a block diagram showing an example of a computer hardware configuration according to a certain embodiment. [Figure 3] This diagram conceptually represents the uvw field of view coordinate system set in an HMD according to a certain embodiment. [Figure 4] This is a diagram conceptually representing one aspect of a virtual space according to a certain embodiment. [Figure 5] This is a top-down view of the head of a user wearing an HMD according to one embodiment. [Figure 6] This diagram shows a YZ cross-section of the field of view in a virtual space, viewed from the X direction. [Figure 7] This diagram shows the XZ cross-section of the field of view in a virtual space, viewed from the Y direction. [Figure 8(A)] This diagram shows a schematic configuration of a controller according to one embodiment. [Figure 8(B)] This figure shows an example of the yaw, roll, and pitch directions defined for the user's right hand according to one embodiment. [Figure 9] This is a block diagram showing an example of a server hardware configuration according to a certain embodiment. [Figure 10] This is a block diagram representing a computer according to a certain embodiment as a modular configuration. [Figure 11] This is a sequence chart representing a portion of the processing performed in an HMD set according to a certain embodiment. [Figure 12(A)] This is a schematic diagram illustrating the situation in a network where each HMD provides a virtual space to the user. [Figure 12(B)] This figure shows the field of view of user 5A in Figure 12(A). [Figure 13] A sequence diagram showing the process to be performed in an HMD system according to a certain embodiment. [Figure 14] This is a block diagram showing the detailed configuration of a computer module according to a certain embodiment. [Figure 15] This figure shows an example of a field of view image corresponding to the view from a virtual camera according to one embodiment. [Figure 16] This figure shows the virtual space corresponding to the field of view image in Figure 15. [Figure 17] This is a left side view of an avatar object according to a certain embodiment, with its right hand visible in the foreground. [Figure 18] This is a rear view showing the back of an avatar object according to a certain embodiment. [Figure 19] This figure shows a state in which a collision occurs between a hand object and the right shoulder region of a collision area in a virtual space according to a certain embodiment. [Figure 20]This is a diagram showing a state in which a collision action occurs due to the occurrence of the collision in FIG. 19, and the equipment is exchanged. [Figure 21] This is a diagram showing a state in which a collision occurs between a hand object and the right waist region of a collision region in a virtual space according to an embodiment. [Figure 22] This is a plan view of an avatar object viewed from above in a virtual space according to an embodiment. [Figure 23] This is a diagram showing a state after the face of the avatar object moves according to an embodiment, and the collision region has not yet moved. [Figure 24] This is a flowchart showing processing of a processor according to an embodiment. [Figure 25] This is an enlarged diagram showing an avatar object according to an embodiment. [Figure 26] This is an enlarged diagram showing an avatar object according to an embodiment. [Figure 27] This is a diagram showing the avatar object when the user tilts the head according to an embodiment. [Figure 28] This is a flowchart showing processing of a processor according to an embodiment. [Figure 29] This is a diagram showing the avatar object when the user lowers the head according to an embodiment. [Figure 30] This is a flowchart showing processing of a processor according to an embodiment. [Figure 31] This is a diagram showing a state in which a collision region is disposed behind an avatar object arranged in a virtual space according to an embodiment, at a distance from the avatar object. [Figure 32] This is a flowchart showing processing of a processor according to an embodiment. [Modes for Carrying Out the Invention]

[0009] Embodiments of this technical concept will be described in detail below with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions of them will not be repeated. In one or more embodiments shown in this disclosure, the elements included in each embodiment can be combined with each other, and the resulting combination will also form part of the embodiments shown in this disclosure.

[0010] [HMD System Configuration] The configuration of the HMD (Head-Mounted Device) system 100 will be described with reference to Figure 1. Figure 1 is a schematic diagram showing the configuration of the HMD system 100 according to this embodiment. The HMD system 100 is offered as a home system or a commercial system.

[0011] The HMD system 100 includes a server 600, HMD sets 110A, 110B, 110C, and 110D, external equipment 700, and a network 2. Each of the HMD sets 110A, 110B, 110C, and 110D is configured to communicate with the server 600 and external equipment 700 via the network 2. Hereinafter, the HMD sets 110A, 110B, 110C, and 110D will be collectively referred to as HMD set 110. The number of HMD sets 110 constituting the HMD system 100 is not limited to four; it may be three or fewer, or five or more. Each HMD set 110 comprises an HMD 120, a computer 200, an HMD sensor 410, a display 430, and a controller 300. The HMD120 includes a monitor 130, a gaze sensor 140, a first camera 150, a second camera 160, a microphone 170, and a speaker 180. The controller 300 may include a motion sensor 420.

[0012] In one scenario, the computer 200 is connectable to the Internet or other network 2 and can communicate with a server 600 or other computer connected to network 2. Other computers include, for example, the computer of another HMD set 110 or external equipment 700. In another scenario, the HMD 120 may include a sensor 190 instead of the HMD sensor 410.

[0013] The HMD120 is worn on the head of user 5 and can provide user 5 with a virtual space during operation. More specifically, the HMD120 displays an image for the right eye and an image for the left eye on the monitor 130, respectively. When each eye of user 5 views the respective image, user 5 can perceive the image as a three-dimensional image based on the parallax between the two eyes. The HMD120 may include either a so-called head-mounted display equipped with a monitor, or a head-mounted device to which a smartphone or other terminal with a monitor can be attached.

[0014] The monitor 130 can be implemented, for example, as a non-transparent display device. In one scenario, the monitor 130 is positioned on the HMD 120 so as to be in front of both of the user 5's eyes. Therefore, when user 5 views the three-dimensional image displayed on the monitor 130, they can become immersed in the virtual space. In one scenario, the virtual space may include, for example, a background, objects that user 5 can interact with, and images of menus that user 5 can select. In another scenario, the monitor 130 can be implemented as a liquid crystal display (LCD) monitor or an OLED (Electro-Luminescence) monitor, such as those found in smartphones or other information display terminals.

[0015] In another scenario, the monitor 130 can be implemented as a transparent display device. In this case, the HMD 120 may be an open type, such as glasses, rather than a closed type that covers the user's eyes, as shown in Figure 1. The transparent monitor 130 may also be temporarily configured as an opaque display device by adjusting its transmittance. The monitor 130 may include a configuration that simultaneously displays a portion of the image constituting the virtual space and the real space. For example, the monitor 130 may display an image of the real space captured by a camera mounted on the HMD 120, or it may make the real space visible by setting the transmittance of a portion to a high level.

[0016] In one scenario, the monitor 130 may include a sub-monitor for displaying the image for the right eye and a sub-monitor for displaying the image for the left eye. In another scenario, the monitor 130 may be configured to display the image for the right eye and the image for the left eye together. In this case, the monitor 130 includes a high-speed shutter. The high-speed shutter operates to alternately display the image for the right eye and the image for the left eye so that the image is perceived by only one eye at a time.

[0017] In certain situations, the HMD120 includes multiple light sources (not shown). Each light source is implemented, for example, by an LED (Light Emitting Diode) that emits infrared light. The HMD sensor 410 has a position tracking function for detecting the movement of the HMD120. More specifically, the HMD sensor 410 reads multiple infrared rays emitted by the HMD120 and detects the position and tilt of the HMD120 in real space.

[0018] In another scenario, the HMD sensor 410 may be implemented by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the HMD 120 by performing image analysis processing using the image information of the HMD 120 output from the camera.

[0019] In another scenario, the HMD120 may be equipped with a sensor 190 as a position detector, either in place of or in addition to the HMD sensor 410. The HMD120 can use the sensor 190 to detect its own position and tilt. For example, if the sensor 190 is an angular velocity sensor, a geomagnetic sensor, or an accelerometer, the HMD120 can use any of these sensors instead of the HMD sensor 410 to detect its own position and tilt. As an example, if the sensor 190 is an angular velocity sensor, the angular velocity sensor detects the angular velocity of the HMD120 around its three axes in real space over time. Based on each angular velocity, the HMD120 calculates the temporal change in the angle around its three axes, and further calculates the tilt of the HMD120 based on the temporal change in the angle.

[0020] The gaze sensor 140 detects the direction in which the gaze of the user 5's right and left eyes is directed. In other words, the gaze sensor 140 detects the gaze of the user 5. Detection of the direction of gaze is achieved, for example, by a known eye-tracking function. The gaze sensor 140 is achieved by a sensor having such an eye-tracking function. In some cases, it is preferable that the gaze sensor 140 includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be a sensor that detects the rotation angle of each eyeball by irradiating the user 5's right and left eyes with infrared light and receiving reflected light from the cornea and iris in response to the irradiated light. Based on the detected rotation angles, the gaze sensor 140 can detect the gaze of the user 5.

[0021] The first camera 150 captures the lower part of user 5's face. More specifically, the first camera 150 captures user 5's nose and mouth, etc. The second camera 160 captures user 5's eyes and eyebrows, etc. The part of the HMD 120 that faces user 5 is defined as the inside of the HMD 120, and the part of the HMD 120 that faces away from user 5 is defined as the outside of the HMD 120. In one scenario, the first camera 150 may be positioned on the outside of the HMD 120, and the second camera 160 may be positioned on the inside of the HMD 120. The images generated by the first camera 150 and the second camera 160 are input to the computer 200. In another scenario, the first camera 150 and the second camera 160 may be implemented as a single camera, and this single camera may capture user 5's face.

[0022] The microphone 170 converts the user 5's speech into an audio signal (electrical signal) and outputs it to the computer 200. The speaker 180 converts the audio signal back into speech and outputs it to the user 5. In another context, the HMD 120 may include earphones instead of the speaker 180.

[0023] The controller 300 is connected to the computer 200 by wire or wireless. The controller 300 receives commands from the user 5 to the computer 200. In one aspect, the controller 300 is configured to be graspable by the user 5. In another aspect, the controller 300 is configured to be attached to the user 5's body or part of their clothing. In yet another aspect, the controller 300 may be configured to output at least one of vibration, sound, or light based on signals transmitted from the computer 200. In yet another aspect, the controller 300 receives operations from the user 5 to control the position and movement of objects placed in the virtual space.

[0024] In one scenario, the controller 300 includes multiple light sources. Each light source is, for example, an LED that emits infrared light. The HMD sensor 410 has a position tracking function. In this case, the HMD sensor 410 reads the multiple infrared light emitted by the controller 300 and detects the position and tilt of the controller 300 in real space. In another scenario, the HMD sensor 410 may be implemented by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the controller 300 by performing image analysis processing using the image information of the controller 300 output from the camera.

[0025] The motion sensor 420 is attached to the user 5's hand in one scenario to detect the user 5's hand movements. For example, the motion sensor 420 detects the hand's rotation speed, rotation rate, etc. The detected signal is sent to the computer 200. The motion sensor 420 is, for example, provided on the controller 300. In one scenario, the motion sensor 420 is provided on the controller 300, which is configured to be graspable by the user 5. In another scenario, for safety in the real world, the controller 300 is attached to something that will not easily fly away when worn on the user 5's hand, such as a glove. In yet another scenario, a sensor not attached to the user 5 may detect the user 5's hand movements. For example, the signal from a camera that photographs the user 5 may be input to the computer 200 as a signal representing the user 5's actions. The motion sensor 420 and the computer 200 are connected to each other wirelessly, as an example. In the case of wireless communication, the communication method is not particularly limited, and for example, Bluetooth® or other known communication methods can be used.

[0026] The display 430 displays an image similar to the one shown on the monitor 130. This allows users other than user 5, who is wearing the HMD 120, to view the same image as user 5. The image displayed on the display 430 does not need to be a three-dimensional image; it may be an image for the right eye or an image for the left eye. Examples of displays 430 include liquid crystal displays and organic EL monitors.

[0027] The server 600 may transmit programs to the computer 200. In another scenario, the server 600 may communicate with other computers 200 to provide virtual reality to HMDs 120 used by other users. For example, in an amusement facility where multiple users play an interactive game, each computer 200 communicates signals based on each user's actions to other computers 200 via the server 600, enabling multiple users to enjoy the same game in the same virtual space. Each computer 200 may communicate signals based on each user's actions to other computers 200 without going through the server 600.

[0028] External device 700 can be any device that can communicate with computer 200. External device 700 may be, for example, a device that can communicate with computer 200 via network 2, or a device that can communicate directly with computer 200 via short-range wireless communication or wired connection. Examples of external device 700 include, but are not limited to, smart devices, PCs (Personal Computers), and peripherals for computer 200.

[0029] [Computer hardware configuration] Referring to Figure 2, the computer 200 according to this embodiment will be described. Figure 2 is a block diagram showing an example of the hardware configuration of the computer 200 according to this embodiment. The computer 200 comprises, as its main components, a processor 210, memory 220, storage 230, input / output interface 240, and communication interface 250. Each component is connected to a bus 260.

[0030] The processor 210 executes a series of instructions contained in a program stored in memory 220 or storage 230 based on signals given to the computer 200, or based on predetermined conditions being met. In some cases, the processor 210 is implemented as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), MPU (Micro Processor Unit), FPGA (Field-Programmable Gate Array), or other device.

[0031] Memory 220 temporarily stores programs and data. Programs are loaded, for example, from storage 230. Data includes data input to computer 200 and data generated by processor 210. In some cases, memory 220 is implemented as RAM (Random Access Memory) or other volatile memory.

[0032] Storage 230 permanently holds programs and data. Storage 230 can be implemented as, for example, ROM (Read-Only Memory), a hard disk drive, flash memory, or other non-volatile storage device. Programs stored in storage 230 include programs for providing a virtual space in the HMD system 100, simulation programs, game programs, user authentication programs, and programs for communicating with other computers 200. Data stored in storage 230 includes data and objects for defining the virtual space.

[0033] In another scenario, the storage 230 may be implemented as a removable storage device, such as a memory card. In yet another scenario, a configuration may be used that uses programs and data stored in an external storage device instead of the storage 230 built into the computer 200. Such a configuration makes it possible to update programs and data in a batch, for example, in a setting where multiple HMD systems 100 are used, such as in an amusement facility.

[0034] The input / output interface 240 communicates signals with the HMD 120, HMD sensor 410, motion sensor 420, and display 430. The monitor 130, gaze sensor 140, first camera 150, second camera 160, microphone 170, and speaker 180 included in the HMD 120 can communicate with the computer 200 via the input / output interface 240 of the HMD 120. In some cases, the input / output interface 240 is implemented using USB (Universal Serial Bus), DVI (Digital Visual Interface), HDMI (Registered Trademark) (High-Definition Multimedia Interface), or other terminals. The input / output interface 240 is not limited to those described above.

[0035] In one scenario, the input / output interface 240 can also communicate with the controller 300. For example, the input / output interface 240 receives signals output from the controller 300 and the motion sensor 420. In another scenario, the input / output interface 240 sends a command output from the processor 210 to the controller 300. This command instructs the controller 300 to perform actions such as vibration, sound output, or light emission. Upon receiving the command, the controller 300 performs one of the following actions: vibration, sound output, or light emission.

[0036] The communication interface 250 is connected to network 2 and communicates with other computers (e.g., server 600) connected to network 2. In some cases, the communication interface 250 may be implemented as, for example, a LAN (Local Area Network) or other wired communication interface, or as a wireless communication interface such as WiFi (Wireless Fidelity), Bluetooth®, NFC (Near Field Communication), etc. The communication interface 250 is not limited to those described above.

[0037] In a given scenario, the processor 210 accesses the storage 230, loads one or more programs stored in the storage 230 into the memory 220, and executes a series of instructions contained in those programs. These one or more programs may include the operating system of the computer 200, application programs for providing a virtual space, game software executable in the virtual space, etc. The processor 210 sends a signal for providing the virtual space to the HMD 120 via the input / output interface 240. The HMD 120 displays an image on the monitor 130 based on that signal.

[0038] In the example shown in Figure 2, the computer 200 is configured to be located outside the HMD 120, but in other aspects, the computer 200 may be built into the HMD 120. For example, a portable information and communication terminal (e.g., a smartphone) including a monitor 130 may function as the computer 200.

[0039] The computer 200 may be configured to be used in common by multiple HMDs 120. With such a configuration, for example, it is possible to provide the same virtual space to multiple users, so that each user can enjoy the same applications as other users in the same virtual space.

[0040] In one embodiment, the HMD system 100 has a pre-set real coordinate system, which is a coordinate system in real space. The real coordinate system has three reference directions (axes) parallel to the vertical direction in real space, the horizontal direction perpendicular to the vertical direction, and the front-to-back direction perpendicular to both the vertical and horizontal directions. The horizontal direction, vertical direction (up and down direction), and front-to-back direction in the real coordinate system are defined as the x-axis, y-axis, and z-axis, respectively. More specifically, in the real coordinate system, the x-axis is parallel to the horizontal direction in real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-to-back direction in real space.

[0041] In a given scenario, the HMD sensor 410 includes an infrared sensor. When the infrared sensor detects infrared radiation emitted from each light source of the HMD 120, it detects the presence of the HMD 120. The HMD sensor 410 further detects the position and tilt (orientation) of the HMD 120 in real space, corresponding to the movement of the user 5 wearing the HMD 120, based on the values ​​of each point (each coordinate value in the real coordinate system). More specifically, the HMD sensor 410 can detect the temporal changes in the position and tilt of the HMD 120 using the values ​​detected over time.

[0042] Each tilt of the HMD 120 detected by the HMD sensor 410 corresponds to each tilt of the HMD 120 around its three axes in the real coordinate system. Based on the tilt of the HMD 120 in the real coordinate system, the HMD sensor 410 sets the uvw field of view coordinate system for the HMD 120. The uvw field of view coordinate system set for the HMD 120 corresponds to the viewpoint coordinate system when the user 5 wearing the HMD 120 views objects in the virtual space.

[0043] [uvw field of view coordinate system] The uvw field of view coordinate system will be described with reference to Figure 3. Figure 3 is a conceptual diagram of the uvw field of view coordinate system set on an HMD 120 according to one embodiment. The HMD sensor 410 detects the position and inclination of the HMD 120 in the real coordinate system when the HMD 120 is started up. The processor 210 sets the uvw field of view coordinate system on the HMD 120 based on the detected values.

[0044] As shown in Figure 3, the HMD120 sets up a three-dimensional uvw field of view coordinate system centered (origin) on the head of user 5 wearing the HMD120. More specifically, the HMD120 sets three new directions obtained by tilting the horizontal, vertical, and front-to-back directions (x, y, and z axes) that define the real coordinate system by the same amount as the inclination of the HMD120 around each axis within the real coordinate system, as the pitch axis (u axis), yaw axis (v axis), and roll axis (w axis) of the uvw field of view coordinate system in the HMD120.

[0045] In a given scenario, if user 5 wearing the HMD120 is standing upright and looking straight ahead, the processor 210 sets the HMD120 to a uvw field of view coordinate system parallel to the real coordinate system. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and forward / backward direction (z-axis) in the real coordinate system coincide with the pitch axis (u-axis), yaw axis (v-axis), and roll axis (w-axis) of the uvw field of view coordinate system in the HMD120.

[0046] After the uvw field of view coordinate system is set on the HMD120, the HMD sensor 410 can detect the tilt of the HMD120 in the set uvw field of view coordinate system based on the movement of the HMD120. In this case, the HMD sensor 410 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD120 in the uvw field of view coordinate system as the tilt of the HMD120. The pitch angle (θu) represents the tilt angle of the HMD120 around the pitch axis in the uvw field of view coordinate system. The yaw angle (θv) represents the tilt angle of the HMD120 around the yaw axis in the uvw field of view coordinate system. The roll angle (θw) represents the tilt angle of the HMD120 around the roll axis in the uvw field of view coordinate system.

[0047] The HMD sensor 410 sets the uvw field of view coordinate system for the HMD 120 after it has moved, based on the detected tilt of the HMD 120. The relationship between the HMD 120 and its uvw field of view coordinate system is always constant, regardless of the position and tilt of the HMD 120. When the position and tilt of the HMD 120 change, the position and tilt of the uvw field of view coordinate system of the HMD 120 in the real coordinate system change in conjunction with the change in position and tilt.

[0048] In a given scenario, the HMD sensor 410 may determine the position of the HMD 120 in real space as its relative position to the HMD sensor 410, based on the infrared light intensity obtained from the output of the infrared sensor and the relative positional relationship between multiple points (e.g., the distance between each point). The processor 210 may determine the origin of the uvw field of view coordinate system of the HMD 120 in real space (real coordinate system) based on the determined relative position.

[0049] [Virtual Space] The virtual space will be further explained with reference to Figure 4. Figure 4 is a conceptual diagram representing one aspect of a virtual space 11 according to a certain embodiment. The virtual space 11 has a spherical structure that covers the entire 360-degree direction of the center 12. In Figure 4, the upper half of the celestial sphere of the virtual space 11 is shown as an example for simplicity of explanation. Each mesh is defined in the virtual space 11. The position of each mesh is predetermined as a coordinate value in the XYZ coordinate system, which is the global coordinate system defined in the virtual space 11. The computer 200 associates each partial image that constitutes a panoramic image 13 (still image, video, etc.) that can be displayed in the virtual space 11 with the corresponding mesh in the virtual space 11.

[0050] In a given scenario, a virtual space 11 defines an XYZ coordinate system with its origin at the center 12. The XYZ coordinate system is, for example, parallel to the real coordinate system. The horizontal, vertical (up and down), and front-to-back directions in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis (horizontal) of the XYZ coordinate system is parallel to the x-axis of the real coordinate system, the Y-axis (vertical) of the XYZ coordinate system is parallel to the y-axis of the real coordinate system, and the Z-axis (front-to-back direction) of the XYZ coordinate system is parallel to the z-axis of the real coordinate system.

[0051] When the HMD120 is started, i.e., in its initial state, the virtual camera 14 is positioned at the center 12 of the virtual space 11. At a certain point, the processor 210 displays the image captured by the virtual camera 14 on the monitor 130 of the HMD120. The virtual camera 14 moves in the virtual space 11 in conjunction with the movement of the HMD120 in the real world. This allows changes in the position and tilt of the HMD120 in the real world to be similarly reproduced in the virtual space 11.

[0052] The virtual camera 14 is defined with a uvw field of view coordinate system, similar to that of the HMD 120. The uvw field of view coordinate system of the virtual camera 14 in the virtual space 11 is defined to be linked to the uvw field of view coordinate system of the HMD 120 in the real space (real coordinate system). Therefore, when the tilt of the HMD 120 changes, the tilt of the virtual camera 14 also changes accordingly. The virtual camera 14 can also move in the virtual space 11 in conjunction with the movement of the user 5 wearing the HMD 120 in the real space.

[0053] The processor 210 of the computer 200 defines the field of view 15 in the virtual space 11 based on the position and tilt (reference line of sight 16) of the virtual camera 14. The field of view 15 corresponds to the area of ​​the virtual space 11 that is visible to the user 5 wearing the HMD 120. In other words, the position of the virtual camera 14 can be said to be the viewpoint of the user 5 in the virtual space 11.

[0054] The gaze direction of user 5 detected by the gaze sensor 140 is the direction in the viewpoint coordinate system when user 5 views an object. The uvw field of view coordinate system of the HMD 120 is equal to the viewpoint coordinate system when user 5 views the monitor 130. The uvw field of view coordinate system of the virtual camera 14 is linked to the uvw field of view coordinate system of the HMD 120. Therefore, the HMD system 100, following a certain phase, can consider the gaze direction of user 5 detected by the gaze sensor 140 as the gaze direction of user 5 in the uvw field of view coordinate system of the virtual camera 14.

[0055] [User's perspective] The determination of user 5's gaze will be explained with reference to Figure 5. Figure 5 is a top view of the head of user 5 wearing an HMD120 according to one embodiment.

[0056] In one scenario, the gaze sensor 140 detects the lines of sight of the user 5's right and left eyes. In one scenario, when the user 5 is looking at something close up, the gaze sensor 140 detects lines of sight R1 and L1. In another scenario, when the user 5 is looking at something far away, the gaze sensor 140 detects lines of sight R2 and L2. In this case, the angle formed by lines of sight R2 and L2 with respect to the roll axis w is smaller than the angle formed by lines of sight R1 and L1 with respect to the roll axis w. The gaze sensor 140 transmits the detection results to the computer 200.

[0057] When the computer 200 receives detected values ​​for lines of sight R1 and L1 from the gaze sensor 140 as a result of line of sight detection, it identifies the gaze point N1, which is the intersection of lines of sight R1 and L1, based on these detected values. On the other hand, when the computer 200 receives detected values ​​for lines of sight R2 and L2 from the gaze sensor 140, it identifies the intersection of lines of sight R2 and L2 as the gaze point. Based on the position of the identified gaze point N1, the computer 200 identifies the user 5's line of sight N0. For example, the computer 200 detects the direction in which a line extends from the midpoint of the line connecting the user 5's right eye R and left eye L, and through the gaze point N1, as the line of sight N0. The line of sight N0 is the direction in which the user 5 is actually looking with both eyes. The line of sight N0 corresponds to the direction in which the user 5 is actually looking with respect to the field of view 15.

[0058] In another configuration, the HMD system 100 may include a television broadcast receiving tuner. With such a configuration, the HMD system 100 can display television programs in the virtual space 11.

[0059] In another aspect, the HMD system 100 may also be equipped with a communication circuit for connecting to the internet, or a call function for connecting to a telephone line.

[0060] [Visibility area] The field of view region 15 will be described with reference to Figures 6 and 7. Figure 6 is a diagram showing a YZ cross-section of the field of view region 15 as viewed from the X direction in the virtual space 11. Figure 7 is a diagram showing an XZ cross-section of the field of view region 15 as viewed from the Y direction in the virtual space 11.

[0061] As shown in Figure 6, the field of view region 15 in the YZ cross-section includes region 18. Region 18 is defined by the position of the virtual camera 14, the reference line of sight 16, and the YZ cross-section of the virtual space 11. The processor 210 defines region 18 as the range centered on the reference line of sight 16 in the virtual space and including the polar angle α.

[0062] As shown in Figure 7, the field of view region 15 in the XZ cross-section includes region 19. Region 19 is defined by the position of the virtual camera 14, the reference line of sight 16, and the XZ cross-section of the virtual space 11. The processor 210 defines region 19 as the range including the azimuth angle β centered on the reference line of sight 16 in the virtual space 11. The polar angles α and β are determined according to the position of the virtual camera 14 and the inclination (orientation) of the virtual camera 14.

[0063] In a given scenario, the HMD system 100 provides the user 5 with a view of the virtual space 11 by displaying a field of view image 17 on the monitor 130 based on a signal from the computer 200. The field of view image 17 is an image corresponding to the portion of the panoramic image 13 that corresponds to the field of view area 15. When the user 5 moves the HMD 120 worn on their head, the virtual camera 14 also moves in conjunction with that movement. As a result, the position of the field of view area 15 in the virtual space 11 changes. Consequently, the field of view image 17 displayed on the monitor 130 is updated with an image from the panoramic image 13 that superimposes on the field of view area 15 in the direction that the user 5 is facing in the virtual space 11. The user 5 can then see the desired direction in the virtual space 11.

[0064] Thus, the tilt of the virtual camera 14 corresponds to the user 5's line of sight (reference line of sight 16) in the virtual space 11, and the position where the virtual camera 14 is placed corresponds to the user 5's viewpoint in the virtual space 11. Therefore, by changing the position or tilt of the virtual camera 14, the image displayed on the monitor 130 is updated, and the user 5's field of view is shifted.

[0065] While wearing the HMD 120, user 5 can only see the panoramic image 13 displayed in the virtual space 11, without seeing the real world. Therefore, the HMD system 100 can give user 5 a high degree of immersion in the virtual space 11.

[0066] In a certain scenario, the processor 210 may move the virtual camera 14 in the virtual space 11 in conjunction with the movement of the user 5 wearing the HMD 120 in the real space. In this case, the processor 210 determines the image area (field of view 15) to be projected onto the monitor 130 of the HMD 120 based on the position and tilt of the virtual camera 14 in the virtual space 11.

[0067] In one scenario, the virtual camera 14 may include two virtual cameras, namely a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. Appropriate parallax is set for the two virtual cameras so that the user 5 can perceive the three-dimensional virtual space 11. In another scenario, the virtual camera 14 may be implemented with a single virtual camera. In this case, the image for the right eye and the image for the left eye may be generated from the image obtained by the single virtual camera. In this embodiment, the technical concept of the present disclosure is illustrated by assuming that the virtual camera 14 includes two virtual cameras, and that the roll axis (w) generated by combining the roll axes of the two virtual cameras is configured to match the roll axis (w) of the HMD 120.

[0068] [controller] An example of the controller 300 will be described with reference to Figure 8. Figure 8 is a diagram showing the schematic configuration of the controller 300 according to one embodiment.

[0069] As shown in Figure 8, in a given scenario, the controller 300 may include a right controller 300R and a left controller (not shown). The right controller 300R is operated by the user 5's right hand. The left controller is operated by the user 5's left hand. In a given scenario, the right controller 300R and the left controller are configured symmetrically as separate devices. Therefore, the user 5 can freely move both the right hand holding the right controller 300R and the left hand holding the left controller. In another scenario, the controller 300 may be an integrated controller that accepts operation from both hands. The right controller 300R will be described below.

[0070] The right controller 300R comprises a grip 310, a frame 320, and a top surface 330. The grip 310 is configured to be grasped by the user 5's right hand. For example, the grip 310 can be held by the palm and three fingers (middle finger, ring finger, and little finger) of the user 5's right hand.

[0071] The grip 310 includes buttons 340 and 350 and a motion sensor 420. Button 340 is located on the side of the grip 310 and accepts operation by the middle finger of the right hand. Button 350 is located on the front of the grip 310 and accepts operation by the index finger of the right hand. In some cases, buttons 340 and 350 are configured as trigger-type buttons. The motion sensor 420 is built into the housing of the grip 310. If the movements of user 5 can be detected from around user 5 by a camera or other device, the grip 310 does not need to include the motion sensor 420.

[0072] Frame 320 includes a plurality of infrared LEDs 360 arranged along its circumference. The infrared LEDs 360 emit infrared light in accordance with the progress of a program that uses the controller 300 during its execution. The infrared light emitted from the infrared LEDs 360 can be used to detect the positions and orientations (tilts, orientations) of the right controller 300R and the left controller. In the example shown in Figure 8, two rows of infrared LEDs 360 are shown, but the number of arrangements is not limited to that shown in Figure 8. One row or three or more rows may be used.

[0073] The top surface 330 includes buttons 370 and 380 and an analog stick 390. Buttons 370 and 380 are configured as push-buttons. Buttons 370 and 380 accept operation by the user's right thumb. The analog stick 390 accepts operation in any direction of 360 degrees from its initial position (neutral position) in a given situation. Such operation includes, for example, operation to move an object placed in the virtual space 11.

[0074] In one scenario, the right controller 300R and the left controller include batteries to power the infrared LED 360 and other components. The batteries include, but are not limited to, rechargeable, button-type, or dry-cell type batteries. In another scenario, the right controller 300R and the left controller may be connected, for example, to the USB interface of a computer 200. In this case, the right controller 300R and the left controller do not require batteries.

[0075] As shown in states (A) and (B) of Figure 8, for example, the yaw, roll, and pitch directions are defined for the right hand of user 5. When user 5 extends their thumb and index finger, the direction in which the thumb extends is defined as the yaw direction, the direction in which the index finger extends is defined as the roll direction, and the direction perpendicular to the plane defined by the axis of the yaw direction and the axis of the roll direction is defined as the pitch direction.

[0076] [Server hardware configuration] Referring to Figure 9, the server 600 according to this embodiment will be described. Figure 9 is a block diagram showing an example of the hardware configuration of a server 600 according to a certain embodiment. The server 600 comprises, as its main components, a processor 610, memory 620, storage 630, input / output interface 640, and communication interface 650. Each component is connected to a bus 660.

[0077] The processor 610 executes a series of instructions contained in a program stored in memory 620 or storage 630 based on signals given to the server 600, or based on the fulfillment of predetermined conditions. In some cases, the processor 610 is implemented as a CPU, GPU, MPU, FPGA, or other device.

[0078] Memory 620 temporarily stores programs and data. Programs are loaded, for example, from storage 630. Data includes data input to server 600 and data generated by processor 610. In some cases, memory 620 is implemented as RAM or other volatile memory.

[0079] Storage 630 permanently holds programs and data. Storage 630 can be implemented as, for example, ROM, a hard disk drive, flash memory, or other non-volatile storage device. Programs stored in storage 630 may include programs for providing a virtual space in the HMD system 100, simulation programs, game programs, user authentication programs, and programs for communicating with the computer 200. Data stored in storage 630 may include data and objects for defining the virtual space.

[0080] In another scenario, the storage 630 may be implemented as a removable storage device, such as a memory card. In yet another scenario, a configuration may be used that uses programs and data stored in an external storage device instead of the storage 630 built into the server 600. Such a configuration makes it possible to update programs and data in a batch, for example, in a setting where multiple HMD systems 100 are used, such as in an amusement facility.

[0081] The input / output interface 640 communicates signals with input / output devices. In some cases, the input / output interface 640 is implemented using USB, DVI, HDMI®, or other terminals. The input / output interface 640 is not limited to those described above.

[0082] The communication interface 650 is connected to network 2 and communicates with computer 200 connected to network 2. In some cases, the communication interface 650 may be implemented as, for example, a LAN or other wired communication interface, or as WiFi, Bluetooth®, NFC or other wireless communication interface. The communication interface 650 is not limited to those described above.

[0083] In a given scenario, the processor 610 accesses the storage 630, loads one or more programs stored in the storage 630 into memory 620, and executes a series of instructions contained in those programs. These one or more programs may include the operating system of the server 600, application programs for providing a virtual space, game software executable in the virtual space, and the like. The processor 610 may also send signals to the computer 200 via the input / output interface 640 for providing the virtual space.

[0084] [HMD control device] The control device of the HMD120 will be described with reference to Figure 10. In one embodiment, the control device is implemented by a computer 200 having a well-known configuration. Figure 10 is a block diagram showing the computer 200 according to one embodiment as a modular configuration.

[0085] As shown in Figure 10, the computer 200 comprises a control module 510, a rendering module 520, a memory module 530, and a communication control module 540. In one scenario, the control module 510 and the rendering module 520 are implemented by a processor 210. In another scenario, multiple processors 210 may operate as the control module 510 and the rendering module 520. The memory module 530 is implemented by a memory 220 or storage 230. The communication control module 540 is implemented by a communication interface 250.

[0086] The control module 510 controls the virtual space 11 provided to the user 5. The control module 510 defines the virtual space 11 in the HMD system 100 using virtual space data that represents the virtual space 11. The virtual space data is stored, for example, in the memory module 530. The control module 510 may also generate the virtual space data or obtain it from a server 600 or the like.

[0087] The control module 510 places objects in the virtual space 11 using object data that represents the objects. The object data is stored, for example, in the memory module 530. The control module 510 may also generate object data or retrieve object data from the server 600 or the like. The objects may include, for example, an avatar object that represents the user 5, a character object, an action object such as a virtual hand operated by the controller 300, and landscapes including forests, mountains, etc., cityscapes, animals, etc., which are placed according to the progression of the game's story.

[0088] The control module 510 places the avatar object of user 5 of another computer 200 connected via network 2 into the virtual space 11. In one scenario, the control module 510 places the avatar object of user 5 into the virtual space 11. In another scenario, the control module 510 places an avatar object modeled after user 5 into the virtual space 11 based on an image that includes user 5. In yet another scenario, the control module 510 places an avatar object selected by user 5 from among several types of avatar objects (for example, objects modeled after animals or objects of stylized people) into the virtual space 11.

[0089] The control module 510 determines the tilt of the HMD 120 based on the output of the HMD sensor 410. In another case, the control module 510 determines the tilt of the HMD 120 based on the output of the sensor 190, which functions as a motion sensor. The control module 510 detects the facial features (e.g., mouth, eyes, eyebrows) of user 5 from the images of user 5's face generated by the first camera 150 and the second camera 160. The control module 510 detects the movement (shape) of each detected feature.

[0090] The control module 510 detects the user 5's line of sight in the virtual space 11 based on the signal from the gaze sensor 140. The control module 510 detects the viewpoint position (coordinate values ​​in the XYZ coordinate system) where the detected line of sight of the user 5 intersects with the celestial sphere of the virtual space 11. More specifically, the control module 510 detects the viewpoint position based on the line of sight of the user 5 defined in the uvw coordinate system and the position and tilt of the virtual camera 14. The control module 510 transmits the detected viewpoint position to the server 600. In another scenario, the control module 510 may be configured to transmit line of sight information representing the user 5's line of sight to the server 600. In this case, the server 600 can calculate the viewpoint position based on the received line of sight information.

[0091] The control module 510 reflects the movement of the HMD 120 detected by the HMD sensor 410 onto the avatar object. For example, if the control module 510 detects that the HMD 120 has tilted, it tilts and positions the avatar object accordingly. The control module 510 reflects the detected facial movements onto the face of the avatar object placed in the virtual space 11. The control module 510 receives gaze information from other users 5 from the server 600 and reflects it onto the gaze of the other user 5's avatar object. In a certain situation, the control module 510 reflects the movement of the controller 300 onto the avatar object and the manipulated object. In this case, the controller 300 is equipped with a motion sensor, an acceleration sensor, or multiple light-emitting elements (e.g., infrared LEDs) to detect the movement of the controller 300.

[0092] The control module 510 places an operation object in the virtual space 11 to accept input from user 5. User 5 manipulates objects placed in the virtual space 11 by manipulating the operation object. In some cases, the operation object may include, for example, a hand object which is a virtual hand corresponding to user 5's hand. In some cases, the control module 510 moves the hand object in the virtual space 11 in conjunction with the movement of user 5's hand in the real space, based on the output of the motion sensor 420. In some cases, the operation object may correspond to the hand portion of an avatar object.

[0093] The control module 510 detects collisions when each object placed in the virtual space 11 collides with another object. For example, the control module 510 can detect the timing when the collision area of ​​one object touches the collision area of ​​another object, and when this detection occurs, it performs a predetermined process. The control module 510 can detect the timing when two objects separate from each other, and when this detection occurs, it performs a predetermined process. The control module 510 can detect when two objects are in contact. For example, when an object being manipulated touches another object, the control module 510 detects that these two objects have touched and performs a predetermined process.

[0094] In a given scenario, the control module 510 controls the image display on the monitor 130 of the HMD 120. For example, the control module 510 places a virtual camera 14 in the virtual space 11. The control module 510 controls the position and tilt (orientation) of the virtual camera 14 in the virtual space 11. The control module 510 defines the field of view 15 according to the tilt of the head of the user 5 wearing the HMD 120 and the position of the virtual camera 14. The rendering module 520 generates a field of view image 17 to be displayed on the monitor 130 based on the determined field of view 15. The field of view image 17 generated by the rendering module 520 is output to the HMD 120 by the communication control module 540.

[0095] When the control module 510 detects speech from user 5 using microphone 170 via HMD 120, it identifies the computer 200 to which the corresponding voice data will be sent. The voice data is sent to the computer 200 identified by the control module 510. When the control module 510 receives voice data from another user's computer 200 via network 2, it outputs the corresponding voice (speech) from speaker 180.

[0096] The memory module 530 holds data used by the computer 200 to provide the virtual space 11 to user 5. In one instance, the memory module 530 holds spatial information, object information, and user information.

[0097] The spatial information holds one or more templates defined to provide the virtual space 11.

[0098] The object information includes multiple panoramic images 13 that constitute the virtual space 11, and object data for placing objects in the virtual space 11. The panoramic images 13 may include still images and moving images. The panoramic images 13 may include images of a non-real space and images of a real space. Examples of images of a non-real space include images generated by computer graphics.

[0099] User information holds a user ID that identifies user 5. The user ID may be, for example, an IP (Internet Protocol) address or MAC (Media Access Control) address set on the computer 200 used by the user. In another context, the user ID may be set by the user. User information includes programs, etc., that cause the computer 200 to function as a control device for the HMD system 100.

[0100] The data and programs stored in the memory module 530 are input by the user 5 of the HMD 120. Alternatively, the processor 210 downloads programs or data from a computer operated by the content provider (e.g., server 600) and stores the downloaded programs or data in the memory module 530.

[0101] The communication control module 540 can communicate with the server 600 and other information and communication devices via the network 2.

[0102] In one scenario, the control module 510 and the rendering module 520 can be implemented, for example, using Unity® provided by Unity Technologies, Inc. In another scenario, the control module 510 and the rendering module 520 can also be implemented as a combination of circuit elements that perform each process.

[0103] Processing in computer 200 is realized by hardware and software executed by processor 210. Such software may be pre-stored in a hard disk or other memory module 530. The software may also be stored on a CD-ROM or other computer-readable non-volatile data recording medium and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other network. Such software is read from the data recording medium by an optical disc drive or other data reading device, or downloaded from a server 600 or other computer via a communication control module 540, and then temporarily stored in a storage module. The software is read from the storage module by processor 210 and stored in RAM in the form of an executable program. Processor 210 then executes the program.

[0104] [Control structure of HMD system] The control structure of the HMD set 110 will be described with reference to Figure 11. Figure 11 is a sequence chart showing some of the processes performed in the HMD set 110 according to one embodiment.

[0105] As shown in Figure 11, in step S1110, the processor 210 of the computer 200, as the control module 510, identifies the virtual space data and defines the virtual space 11.

[0106] In step S1120, the processor 210 initializes the virtual camera 14. For example, the processor 210 places the virtual camera 14 in the memory work area at a predetermined center 12 in the virtual space 11 and directs the line of sight of the virtual camera 14 in the direction that the user 5 is facing.

[0107] In step S1130, the processor 210, acting as the rendering module 520, generates field-of-view image data for displaying the initial field-of-view image. The generated field-of-view image data is output to the HMD 120 by the communication control module 540.

[0108] In step S1132, the monitor 130 of the HMD 120 displays a field of view image based on the field of view image data received from the computer 200. User 5 wearing the HMD 120 can perceive the virtual space 11 by viewing the field of view image.

[0109] In step S1134, the HMD sensor 410 detects the position and tilt of the HMD 120 based on multiple infrared beams emitted from the HMD 120. The detection results are output to the computer 200 as motion detection data.

[0110] In step S1140, the processor 210 determines the field of view of the user 5 wearing the HMD 120 based on the position and tilt included in the motion detection data of the HMD 120.

[0111] In step S1150, the processor 210 executes the application program and places objects in the virtual space 11 based on the instructions contained in the application program.

[0112] In step S1160, the controller 300 detects user 5's operation based on the signal output from the motion sensor 420 and outputs detection data representing the detected operation to the computer 200. In another scenario, user 5's operation of the controller 300 may be detected based on images from cameras positioned around user 5.

[0113] In step S1170, the processor 210 detects an operation of the controller 300 by the user 5 based on the detection data obtained from the controller 300.

[0114] In step S1180, the processor 210 generates field-of-view image data based on the user 5's operation of the controller 300. The generated field-of-view image data is output to the HMD 120 by the communication control module 540.

[0115] In step S1190, the HMD120 updates the field of view image based on the received field of view image data and displays the updated field of view image on the monitor 130.

[0116] [Avatar Object] Referring to Figures 12(A) and (B), an avatar object according to this embodiment will be described. The following figures illustrate the avatar objects of each user 5 in HMD sets 110A and 110B. Hereinafter, a user of HMD set 110A will be referred to as user 5A, a user of HMD set 110B as user 5B, a user of HMD set 110C as user 5C, and a user of HMD set 110D as user 5D. Each component of HMD set 110A is denoted by the reference numeral A, each component of HMD set 110B is denoted by the reference numeral B, each component of HMD set 110C is denoted by the reference numeral C, and each component of HMD set 110D is denoted by the reference numeral D. For example, HMD 120A is included in HMD set 110A.

[0117] Figure 12(A) is a schematic diagram illustrating the situation in network 2 where each HMD 120 provides a virtual space 11 to user 5. Computers 200A to 200D provide virtual spaces 11A to 11D to users 5A to 5D via HMDs 120A to 120D. In the example shown in Figure 12(A), virtual spaces 11A and 11B are composed of the same data. In other words, computers 200A and 200B share the same virtual space. Virtual spaces 11A and 11B contain avatar object 6A for user 5A and avatar object 6B for user 5B. Although avatar object 6A in virtual space 11A and avatar object 6B in virtual space 11B are shown wearing HMDs 120, this is for the sake of clarity, and in reality, these objects are not wearing HMDs 120.

[0118] In a certain scenario, the processor 210A may position a virtual camera 14A, which captures the user 5A's field of view image 17A, at the position of the avatar object 6A's eyes.

[0119] Figure 12(B) shows the view image 17A of user 5A in Figure 12(A). The view image 17A is the image displayed on the monitor 130A of the HMD 120A. This view image 17A is an image generated by the virtual camera 14A. The view image 17A displays the avatar object 6B of user 5B. Although not specifically shown, the view image of user 5B also displays the avatar object 6A of user 5A.

[0120] In the state shown in Figure 12(B), user 5A can communicate with user 5B through dialogue via the virtual space 11A. More specifically, user 5A's voice, acquired by microphone 170A, is transmitted to user 5B's HMD 120B via server 600 and output from speaker 180B installed on HMD 120B. User 5B's voice is transmitted to user 5A's HMD 120A via server 600 and output from speaker 180A installed on HMD 120A.

[0121] The actions of user 5B (the actions of HMD120B and controller 300B) are reflected by processor 210A in avatar object 6B placed in virtual space 11A. This allows user 5A to perceive user 5B's actions through avatar object 6B.

[0122] Figure 13 is a sequence chart showing some of the processes performed in the HMD system 100 according to this embodiment. Although HMD set 110D is not shown in Figure 13, HMD set 110D operates in the same way as HMD sets 110A, 110B, and 110C. In the following description, each component of HMD set 110A will be denoted by the reference numeral A, each component of HMD set 110B will be denoted by the reference numeral B, each component of HMD set 110C will be denoted by the reference numeral C, and each component of HMD set 110D will be denoted by the reference numeral D.

[0123] In step S1310A, the processor 210A in the HMD set 110A acquires avatar information to determine the behavior of the avatar object 6A in the virtual space 11A. This avatar information includes, for example, information about the avatar such as motion information, face tracking data, and audio data. Motion information includes information showing the temporal changes in the position and tilt of the HMD 120A, and information showing the hand movements of user 5A detected by the motion sensor 420A, etc. Face tracking data includes data that identifies the position and size of each part of user 5A's face. Face tracking data also includes data showing the movement of each organ that makes up user 5A's face and gaze data. Audio data includes data showing the voice of user 5A acquired by the microphone 170A of the HMD 120A. The avatar information may also include information that identifies the avatar object 6A, or user 5A associated with the avatar object 6A, and information that identifies the virtual space 11A in which the avatar object 6A exists. User ID is an example of information that identifies the avatar object 6A or user 5A. The room ID is one piece of information that identifies the virtual space 11A where the avatar object 6A exists. The processor 210A transmits the avatar information obtained as described above to the server 600 via network 2.

[0124] In step S1310B, the processor 210B in the HMD set 110B acquires avatar information to determine the behavior of the avatar object 6B in the virtual space 11B, similar to the processing in step S1310A, and transmits it to the server 600. Similarly, in step S1310C, the processor 210C in the HMD set 110C acquires avatar information to determine the behavior of the avatar object 6C in the virtual space 11C and transmits it to the server 600.

[0125] In step S1320, the server 600 temporarily stores the player information received from HMD set 110A, HMD set 110B, and HMD set 110C, respectively. Based on the user ID and room ID included in each avatar information, the server 600 integrates the avatar information of all users (in this example, users 5A to 5C) associated with the common virtual space 11. Then, at a predetermined timing, the server 600 sends the integrated avatar information to all users associated with the virtual space 11. This executes the synchronization process. Through this synchronization process, HMD set 110A, HMD set 110B, and HMD 110C can share each other's avatar information at almost the same time.

[0126] Next, based on the avatar information transmitted from server 600 to each HMD set 110A-110C, each HMD set 110A-110C executes the processing in steps S1330A-S1330C. The processing in step S1330A corresponds to the processing in step S1180 in Figure 11.

[0127] In step S1330A, the processor 210A in the HMD set 110A updates the information of the avatar objects 6B and 6C of other users 5B and 5C in the virtual space 11A. Specifically, the processor 210A updates the position and orientation of the avatar object 6B in the virtual space 11 based on the motion information contained in the avatar information transmitted from the HMD set 110B. For example, the processor 210A updates the information (position and orientation, etc.) of the avatar object 6B contained in the object information stored in the memory module 530. Similarly, the processor 210A updates the information (position and orientation, etc.) of the avatar object 6C in the virtual space 11 based on the motion information contained in the avatar information transmitted from the HMD set 110C.

[0128] In step S1330B, the processor 210B in the HMD set 110B updates the information of the avatar objects 6A and 6C of users 5A and 5C in the virtual space 11B, similar to the processing in step S1330A. Similarly, in step S1330C, the processor 210C in the HMD set 110C updates the information of the avatar objects 6A and 6B of users 5A and 5B in the virtual space 11C.

[0129] [Detailed module configuration] Refer to Figure 14 to describe the details of the module configuration of computer 200. Figure 14 is a block diagram showing the detailed configuration of the modules of computer 200 according to one embodiment.

[0130] As shown in Figure 14, the control module 510 includes a virtual camera control module 1421, a field of view determination module 1422, a reference gaze identification module 1423, a facial organ detection module 1424, a motion detection module 1425, a virtual space definition module 1426, a virtual object generation module 1427, an operation object control module 1428, and an avatar control module 1429. The rendering module 520 includes a field of view image generation module 1438. The memory module 530 holds spatial information 1431, object information 1432, user information 1433, and face information 1434.

[0131] The virtual camera control module 1421 places the virtual camera 14 in the virtual space 11. The virtual camera control module 1421 controls the placement position and orientation (tilt) of the virtual camera 14 in the virtual space 11. The field of view determination module 1422 defines the field of view 15 according to the orientation of the user's head wearing the HMD 120 and the placement position of the virtual camera 14. The field of view image generation module 1438 generates a field of view image 17 to be displayed on the monitor 130 based on the determined field of view 15.

[0132] The reference gaze identification module 1423 identifies the gaze of user 5 based on the signal from the gaze sensor 140. The facial organ detection module 1424 detects the facial organs (e.g., mouth, eyes, eyebrows) that make up user 5's face from the images of user 5's face generated by the first camera 150 and the second camera 160. The motion detection module 1425 detects the movement (shape) of each organ detected by the facial organ detection module 1424.

[0133] The virtual space definition module 1426 defines the virtual space 11 in the HMD system 100 by generating virtual space data that represents the virtual space 11.

[0134] The virtual object generation module 1427 generates objects to be placed in the virtual space 11. These objects may include, for example, landscapes such as forests and mountains, animals, etc., which are placed according to the progression of the game's story.

[0135] The operation object control module 1428 places operation objects in the virtual space 11 to accept user input. The user manipulates objects placed in the virtual space 11 by manipulating the operation objects. In some cases, the operation objects may include, for example, a hand object corresponding to the hand of a user wearing the HMD 120. In other cases, the operation objects may correspond to the hand portion of an avatar object, which will be described later.

[0136] The avatar control module 1429 generates data for placing avatar objects of users of other computers 200 connected via network 2 into the virtual space 11. In one scenario, the avatar control module 1429 generates data for placing user 5's avatar object into the virtual space 11. In another scenario, the avatar control module 1429 generates an avatar object modeled after user 5 based on an image including user 5. In yet another scenario, the avatar control module 1429 generates data for placing an avatar object selected by user 5 from among several types of avatar objects (for example, objects modeled after animals or objects of stylized people) into the virtual space 11.

[0137] The avatar control module 1429 reflects the movement of the HMD 120 detected by the HMD sensor 410 onto the avatar object. For example, the avatar control module 1429 detects that the HMD 120 has tilted and generates data to tilt and position the avatar object. In a certain situation, the avatar control module 1429 reflects the movement of the controller 300 onto the avatar object. In this case, the controller 300 is equipped with a motion sensor, an accelerometer, or multiple light-emitting elements (e.g., infrared LEDs) to detect the movement of the controller 300. The avatar control module 1429 reflects the movement of the facial organs detected by the motion detection module 1425 onto the face of the avatar object placed in the virtual space 11. In other words, the avatar control module 1429 reflects the movement of user 5A's face onto the avatar object.

[0138] The control module 510 detects collisions when each object placed in the virtual space 11 collides with another object. For example, the control module 510 can detect the timing when one object touches another object, and when this is detected, it performs a predetermined process. The control module 510 can detect the timing when two objects separate from each other, and when this is detected, it performs a predetermined process. The control module 510 can detect when two objects are touching each other. Specifically, the operation object control module 1428 detects when an operation object touches another object, and performs a predetermined process.

[0139] The memory module 530 holds data used by the computer 200 to provide the virtual space 11 to user 5. In one instance, the memory module 530 holds spatial information 1431, object information 1432, user information 1433, and face information 1434.

[0140] Spatial information 1431 holds one or more templates defined for providing the virtual space 11.

[0141] Object information 1432 holds content to be played in the virtual space 11, objects used in said content, and information for placing the objects in the virtual space 11 (for example, location information). The content may include, for example, games, content representing landscapes similar to those in the real world, etc.

[0142] User information 1433 holds programs for making the computer 200 function as a control device for the HMD system 100, application programs that use each of the contents held in object information 1432, and so on.

[0143] The facial information 1434 holds templates pre-stored by the facial organ detection module 1424 for detecting the facial organs of user 5. In a given scenario, the facial information 1434 holds a mouth template 1435, an eye template 1436, and an eyebrow template 1437. Each template may be an image corresponding to an organ that makes up a face. For example, the mouth template 1435 may be an image of a mouth. Each template may contain multiple images.

[0144] [Method for updating equipment status] Next, a method for updating the equipment state according to a certain embodiment will be described. Figure 15 shows a view image 1517, which is an example of a view image corresponding to the view from the virtual camera 14. Figure 16 shows a virtual space 11 corresponding to the view image 1517 in Figure 15. Figures 15 and 16 show a scene from a game that provides the user 5 with a virtual experience of battling enemy characters such as monsters in the virtual space 11. The enemy characters are placed as enemy character objects 1642 in the virtual space 11.

[0145] The virtual camera 14 is a virtual viewpoint 1643 located in the virtual space 11. The virtual viewpoint 1643 is a first-person viewpoint associated with user 5. The processor 210 of the computer 200 determines the field of view from the virtual viewpoint 1643 according to the head movements of user 5 wearing the HMD 120 (the head movements of user 5 associated with the head-mounted device), and generates field of view image data corresponding to the field of view from this virtual viewpoint 1643 by rendering. The field of view image data generated by the processor 210 is output to the HMD 120 by the communication control module 540. The monitor 130 of the HMD 120 displays a field of view image 1517 based on the field of view image data received from the computer 200. User 5 wearing the HMD 120 can recognize the virtual space 11 by viewing the field of view image 1517.

[0146] The field of view image 1517 in Figure 15 is a field of view image of a situation in which an avatar object 1644 corresponding to user 5 is confronting an enemy character object 1642 in the virtual space 11. The field of view image 1517 includes an enemy character image 1542 corresponding to the enemy character object 1642 placed in the virtual space 11. The field of view image 1517 includes a hand image 1539 corresponding to a hand object 1639, which is an example of an action object placed in the virtual space 11. The field of view image 1517 includes a sword image 1541 corresponding to a sword object 1641, which is an example of an equipment object placed in the virtual space 11. In the virtual space 11 shown in Figure 16, the avatar object 1644 is equipped with the sword object 1641. The processor 210 moves the hand object 1639, which is an action object in the virtual space 11, in accordance with the movement of the controller 300, that is, the movement of the first part (hand), which is a part of user 5's body other than the head. The first part may be any part of the body other than the hand. In this case, the object placed in the virtual space corresponding to the first part moves according to the movement of that first part. Equipment objects are weapon objects or armor objects. Weapon objects include sword objects and gun objects. Armor objects include shield objects.

[0147] The control module 510 of the processor 210 can detect collisions between objects placed in the virtual space 11 and other objects, and when such collisions are detected, it performs predetermined processing. For example, if the control module 510 detects a collision between a sword object 1641 and an enemy character object 1642, it can perform processing to place an injured enemy character object 1642 in the virtual space 11 instead of an uninjured enemy character object 1642.

[0148] Figures 17 and 18 are enlarged views of the avatar object 1644 shown in Figure 16. Figure 17 is a left side view showing the right hand of the avatar object 1644 in the foreground. Figure 18 is a rear view showing the back of the avatar object 1644. In this embodiment, a collision area 1745 is placed behind the avatar object 1644 in the virtual space 11. The collision area 1745 is placed at a predetermined relative position from the virtual viewpoint 1643. The collision area 1745 has a defined relative position to the virtual viewpoint 1643 and is associated with collision action. Object data representing the collision area 1745 is stored in the memory module 530. The control module 510 uses the object data of the collision area 1745 stored in the memory module 530 to place the collision area 1745 in the virtual space 11.

[0149] The collision region 1745 is an area that is not visible in the view image, i.e., a colorless and transparent area, even when it is included in the view from the virtual viewpoint 1643. In another embodiment, the collision region 1745 may be an area that is visible when it is included in the view from the virtual viewpoint 1643. The collision region 1745 is a single plate shape with thickness on the rear side of the avatar object 1644. The collision region 1745 has an area located on the right shoulder of the avatar object 1644 (hereinafter referred to as the "right shoulder region"), an area located on the left shoulder of the avatar object 1644 (hereinafter referred to as the "left shoulder region"), an area located on the right hip of the avatar object 1644 (hereinafter referred to as the "right hip region"), and an area located on the left hip of the avatar object 1644 (hereinafter referred to as the "left hip region").

[0150] When the control module 510 detects a collision between the collision area 1745 and the hand object 1639, it performs a predetermined process. The execution of this predetermined process is the occurrence of a collision action. For example, as an example of collision occurrence, when a collision between the collision area 1745 and the hand object 1639 is detected, as an example of collision action occurrence, equipment is newly equipped from a state where no equipment is equipped. Another example of collision occurrence is when a collision between the collision area 1745 and the hand object 1639 is detected, as an example of collision action occurrence, the equipped equipment is replaced. Equipping equipment means that the processor 210 places the equipment object at a predetermined relative position from the hand object 1639 in the virtual space 11. Equipment includes weapons such as swords and guns, and armor such as shields. Another example of collision action occurrence is placing an item in association with an operation object. This item is, for example, an equipment object. This item is, for example, an armor object or a weapon object. This item may be used directly in the game. This item may not be used directly in the game. Also, as an example of collision behavior, a second item may be placed in association with the manipulated object instead of the first item which is placed in association with the manipulated object.

[0151] In a given scenario, the processor 210 places a menu object in the virtual space 11 for the user 5 to operate a menu that can be selected. The user 5 can use the hand object 1639 to manipulate this menu object and select the correspondence between a position within the collision area 1745 and the type of collision action. Based on this selection by the user 5, the processor 210 sets the collision action. In this embodiment, the right shoulder area of ​​the collision area 1745 is associated with a gun as a collision action, and the right hip area of ​​the collision area 1745 is associated with a sword as a collision action.

[0152] Figure 19 shows the state in which a collision occurs between the hand object 1639 and the right shoulder region of the collision area 1745 in the virtual space 11. Figure 20 shows the state in which a collision effect occurs due to the collision in Figure 19 and the equipment is replaced. The occurrence of a collision is not limited to the collision between the hand object 1639 and the collision area 1745. The occurrence of a collision may also be defined as the positional relationship between the hand object 1639 (first part) and the collision area 1745 becoming a predetermined positional relationship (first positional relationship). The processor 210 moves the collision area 1745 in the virtual space 11 in response to the movement of the user 5's head, so that the collision area 1745 moves in accordance with the movement of the avatar object 1644.

[0153] When the processor 210 moves either or both of the hand object 1639 and the collision area 1745 in the virtual space 11, a collision occurs when the positional relationship between the hand object 1639 and the collision area 1745 becomes a first positional relationship (e.g., a collision). In Figure 19, a collision occurs between the hand object 1639 and the right shoulder area of ​​the collision area 1745, and the gun set in the menu is equipped (see Figure 20). That is, as a collision action, the processor 210 places the gun object 2041 in the virtual space 11 at a predetermined relative position from the hand object 1639, instead of the sword object 1641.

[0154] Figure 21 shows the state in virtual space 11 where a collision occurs between the hand object 1639 and the right hip region of the collision area 1745. After the state shown in Figure 20, when a collision occurs between the hand object 1639 and the right hip region of the collision area 1745 as shown in Figure 21, the sword set in the menu is equipped. That is, as a collision action, the processor 210 places the sword object 1641 at a predetermined relative position from the hand object 1639 in virtual space 11, instead of the gun object 2041.

[0155] Incidentally, the collision area 1745 moves in accordance with the movement of the avatar object 1644, and may move to a position where collision is less likely to occur. This point will be explained with reference to Figure 22. Figure 22 is a top view of the avatar object 1644 in the virtual space 11. When user 5 turns their face 60 degrees to the right, as shown in Figure 22, the face of the avatar object 1644 also turns 60 degrees to the right, and the collision area 1745 also tilts 60 degrees to the right in conjunction with this, moving away from the right shoulder of the avatar object 1644. In this case, it becomes less likely for a collision to occur between the hand object 1639 and the right shoulder area of ​​the collision area 1745.

[0156] Therefore, in this embodiment, processing is performed to delay the movement of the collision region 1745. Figure 23 shows the state in this embodiment where the face of the avatar object 1644 has moved, but the collision region 1745 has not yet moved. Figure 23 is a top view of the avatar object 1644 in the virtual space 11. Figure 24 is a flowchart showing the processing of the processor 210 in this embodiment.

[0157] In step S2446 of Figure 24, the avatar object 1644 is moved based on the detection result of the HMD sensor 410. Then, in step S2447, a predetermined time (first time) is waited for to elapse. At this time, as shown in Figure 23, the face of the avatar object 1644 is turned 60 degrees to the right, but the collision region 1745 has not moved, and the collision region 1745 remains on the right shoulder of the avatar object 1644, maintaining a state where collision between the hand object 1639 and the right shoulder region of the avatar object 1644 is likely to occur. Then, after the predetermined time has elapsed in step S2447, the collision region 1745 is moved based on the detection result of the HMD sensor 410. As a result, the collision region 1745 can follow the movement of the avatar object 1644. In this embodiment, if at least one of the position and orientation of the virtual viewpoint 1643 changes, the collision region 1745 is moved after the first time has elapsed so that it is in a predetermined relative position with respect to the virtual viewpoint 1643.

[0158] Although Figures 22 and 23 describe the case when user 5 is facing to the right, in one embodiment, the process shown in Figure 24 can be applied regardless of the action taken by user 5.

[0159] Next, the control of a collision area according to a certain embodiment will be described using Figures 25 to 30. Figures 25 and 26 are enlarged views of the avatar object 1644 shown in Figure 16. Figure 25 is a rear view showing the back of the avatar object 1644. Figure 26 is a left side view showing the right hand of the avatar object 1644 in the foreground. In this embodiment, in the virtual space 11, divided collision areas 2545A, 2545B, 2545C, and 2545D are placed behind the avatar object 1644 instead of the collision area 1745 shown in Figure 17. The divided collision areas 2545A, 2545B, 2545C, and 2545D are placed at predetermined relative positions from the virtual viewpoint 1643. The memory module 530 stores object data representing the divided collision areas 2545A, 2545B, 2545C, and 2545D. The control module 510 uses the object data of the partitioned collision areas 2545A, 2545B, 2545C, and 2545D stored in the memory module 530 to place the partitioned collision areas 2545A, 2545B, 2545C, and 2545D in the virtual space 11.

[0160] The divided collision regions 2545A, 2545B, 2545C, and 2545D are areas that are not visible in the field of view image, i.e., colorless and transparent areas, even when they are included in the field of view from the virtual viewpoint 1643. In another embodiment, the divided collision regions 2545A, 2545B, 2545C, and 2545D may be areas that are visible when included in the field of view from the virtual viewpoint 1643. Each of the divided collision regions 2545A, 2545B, 2545C, and 2545D is a single plate shape with thickness on the rear side of the avatar object 1644. The divided collision region 2545A corresponds to the left shoulder area of ​​the collision region 1745. The divided collision region 2545B corresponds to the right shoulder area of ​​the collision region 1745. The divided collision region 2545C corresponds to the left hip area of ​​the collision region 1745. The partitioned collision region 2545D corresponds to the right hip region of the collision region 1745. The processor 210 can move each of the partitioned collision regions 2545A, 2545B, 2545C, and 2545D individually in the virtual space 11.

[0161] As shown in Figure 25, when viewing the back of avatar object 1644, the divided collision area 2545B is located to the right of the divided collision area 2545A. As shown in Figure 25, when viewing the back of avatar object 1644, the divided collision area 2545D is located below the divided collision area 2545B. As shown in Figure 25, when viewing the back of avatar object 1644, the divided collision area 2545C is located below the divided collision area 2545A. As shown in Figure 25, when viewing the back of avatar object 1644, the divided collision area 2545D is located to the right of the divided collision area 2545C. The divided collision areas 2545A and 2545B are the upper collision areas, and the divided collision areas 2545C and 2545D are the lower collision areas. The divided collision regions 2545A and 2545C are the left-side collision regions, while the divided collision regions 2545B and 2545D are the right-side collision regions.

[0162] When the control module 510 detects a collision between each of the divided collision regions 2545A, 2545B, 2545C, and 2545D and the hand object 1639, it performs a predetermined process. The execution of this predetermined process is the occurrence of a collision action. The occurrence of the collision action is the same as in the example described above for the collision region 1745, so a detailed explanation is omitted. In one embodiment, the process shown in Figure 24 may be applied to the divided collision regions 2545A, 2545B, 2545C, and 2545D in the same way as for the collision region 1745, or the process shown in Figure 24 may not be applied.

[0163] Figure 27 shows the avatar object 1644 when user 5 tilts their head in one embodiment. When the HMD sensor 410 detects a rotation of the pitch angle as described with reference to Figure 3, the processor 210 moves the avatar object 1644 in the virtual space 11 according to this detection result. At this time, if the collision area is a single plate shape as the collision area 1745 described above, the collision area will also tilt in accordance with the avatar object 1644, separating from the avatar object 1644 on the lower body side, making it difficult to generate collisions between the hand object 1639 and the right and left hip areas of the collision area.

[0164] Therefore, in this embodiment, the condition in which collisions with the hand object 1639 are less likely to occur is eliminated by moving each of the multiple divided collision regions individually. Figure 27 shows the state in this embodiment where, when the HMD sensor 410 detects rotation of the pitch angle, the amount of rotation of the lower of the multiple divided collision regions is reduced. Figure 28 is a flowchart of the processing of the processor 210 in this embodiment.

[0165] In step S2851 of Figure 28, it is determined whether rotation of the pitch angle has been detected based on the detection result of the HMD sensor 410. If rotation of the pitch angle is not detected, the process ends. If rotation of the pitch angle is detected, the process proceeds to step S2852. In step S2852, the pitch angle of the upper collision region, divided collision regions 2545A and 2545B, is rotated according to the detection result of the HMD sensor 410. In the example in Figure 27, the divided collision regions 2545A and 2545B are rotated by an angle θ1. Also in step S2852, the amount of rotation of the lower collision region (second collision region), divided collision regions 2545C and 2545D (second collision region), is reduced compared to the upper collision region (first collision region), divided collision regions 2545A and 2545B. In the example in Figure 27, the divided collision regions 2545C and 2545D are rotated by an angle θ2. Angle θ2 is smaller than angle θ1. This prevents the lower collision area from moving too far away from the avatar object 1644, making it easier to generate collisions between the hand object 1639 and the right and left hip regions of the collision area.

[0166] Figure 29 shows the avatar object 1644 when user 5 lowers their head in one embodiment. The left side of Figure 29 shows the avatar object 1644 in the virtual space 11 when user 5 is not lowering their head. The right side of Figure 29 shows the avatar object 1644 in the virtual space 11 when user 5 is lowering their head. When the HMD sensor 410 detects downward movement in the yaw axis (v axis) direction (-v direction movement) as explained with reference to Figure 3, the processor 210 moves the avatar object 1644 in the virtual space 11 according to this detection result. At this time, if the collision area is a single plate shape as the collision area 1745 described above, the collision area will also move downward following the avatar object 1644, and the lower side of the collision area will be positioned below the ground, making it difficult to generate collisions between the hand object 1639 and the right and left hip areas of the collision area.

[0167] Therefore, in this embodiment, the state in which collisions with the hand object 1639 are less likely to occur is resolved by moving each of the multiple divided collision regions individually. The avatar object 1644 on the right side of Figure 29 shows the state in this embodiment where, when the HMD sensor 410 detects movement in the -v direction, the amount of movement of the lower collision region among the multiple divided collision regions is reduced. Figure 30 is a flowchart of the processing of the processor 210 in this embodiment.

[0168] In step S3053 of Figure 30, it is determined whether movement in the -v direction has been detected based on the detection result of the HMD sensor 410. If no movement in the -v direction is detected, the process terminates. If movement in the -v direction is detected, the process proceeds to step S3054. In step S3054, the upper collision regions, divided collision regions 2545A and 2545B, are moved in the -v direction according to the detection result of the HMD sensor 410. In the example in Figure 29, the divided collision regions 2545A and 2545B are moved downward by a distance L1. Also in step S3054, the lower collision regions (first collision regions), divided collision regions 2545C and 2545D, are moved by a smaller amount than the upper collision regions (second collision regions), divided collision regions 2545A and 2545B. In the example in Figure 29, the divided collision regions 2545C and 2545D are moved downward by a distance L2. Distance L2 is smaller than distance L1. This prevents the lower collision area from being positioned below ground level, making it easier to generate collisions between the hand object 1639 and the right and left hip areas of the collision area.

[0169] Furthermore, when the HMD sensor 410 detects a yaw angle rotation as explained with reference to Figure 3, the processor 210 moves the avatar object 1644 in the virtual space 11 according to this detection result. At this time, if the collision region is a single plate shape as the collision region 1745 described above, the collision region will also tilt in accordance with the avatar object 1644, and for example, if the user 5 turns to the right, the right half of the user's body will move away from the avatar object 1644, making it difficult to generate a collision between the hand object 1639 and the right shoulder region and the right hip region of the collision region. Therefore, in one embodiment, when the HMD sensor 410 detects a yaw angle rotation, if the user 5 turns to the right, the amount of rotation of the right-side collision region (divided collision regions 2545B and 2545D) among the multiple divided collision regions is reduced. This prevents the right-side collision region from moving too far away from the avatar object 1644, making it easier to generate a collision between the hand object 1639 and the right shoulder region and the right hip region of the collision region. Furthermore, when user 5 is facing left, the rotation amount of the leftmost collision region (divided collision regions 2545A and 2545C) among the multiple divided collision regions is reduced. This prevents the leftmost collision region from moving too far away from the avatar object 1644, making it easier to generate collisions between the hand object 1639 and the left shoulder region and left hip region of the collision region.

[0170] Furthermore, when the HMD sensor 410 detects a rotation of the roll angle as explained with reference to Figure 3, the processor 210 moves the avatar object 1644 in the virtual space 11 according to this detection result. At this time, if the collision region is a single plate shape as the collision region 1745 described above, the collision region will also tilt in accordance with the avatar object 1644, moving away from the avatar object 1644 on the lower body side, making it difficult to generate collisions between the hand object 1639 and the right and left hip regions of the collision region. Therefore, in one embodiment, when the HMD sensor 410 detects a rotation of the roll angle, the amount of rotation of the lower collision region (divided collision regions 2545C and 2545D) among the multiple divided collision regions is reduced. This prevents the lower collision region from moving too far away from the avatar object 1644, making it easier to generate collisions between the hand object 1639 and the right and left hip regions of the collision region.

[0171] Next, we will describe a case in which user 5 can choose whether or not to generate a collision effect in a certain embodiment. For example, if the collision area is placed on the back of the avatar object 1644, it is difficult to reach the back of the body to change equipment. On the other hand, if the collision area is moved to the front of the avatar object 1644, a collision will occur with even a slight movement, and the equipment will be changed by actions that are not for the purpose of changing equipment, such as attack actions (the equipment will be changed against the user's will). Therefore, in this embodiment, user 5 is made able to choose whether or not to generate a collision effect.

[0172] Figure 31 shows a state in one embodiment where a collision region 3145 is placed behind an avatar object 1644 located in the virtual space 11, at a distance from the avatar object 1644. In the state shown in Figure 31, even if the hand object 1639 is moved backward, no collision occurs between the hand object 1639 and the collision region 3145, and therefore, no collision effect occurs.

[0173] Figure 32 is a flowchart showing the processing of the processor 210 in this embodiment. In step 3255, it is determined whether a button has been pressed. The button to be determined in step 3255 is one of the buttons 340, 350, 370, and 380 provided on the controller 300 shown in Figure 8. If a button is pressed in step 3255, the process ends and the state shown in Figure 31 is maintained. On the other hand, if a button is not pressed in step 3255, in step S3256, the collision area 3145 is moved forward. That is, by moving the collision area 3145 to the position of collision area 1745 shown in Figure 19, for example, the hand object 1639 is moved backward, causing a collision between the hand object 1639 and the collision area 3145, and thus a collision effect occurs. In this example, the fact that a button is not pressed is the movement of the hand object 1639 that establishes the first positional relationship between the hand object 1639 and the collision area 3145. In this embodiment, a process is performed to detect this movement, and in response to the detection of this movement, a process is performed to move the collision region 3145 closer to the avatar object 1644.

[0174] In one embodiment, the position where the collision region 3145 is moved in step S3256 may be any position as long as a collision with the hand object 1639 can occur.

[0175] In one embodiment, if the button is pressed in step 3255 of Figure 32, the collision area 3145 does not need to be placed in the virtual space 11. In this case, if the button is not pressed in step 3255 of Figure 32, the collision area 3145 may be placed closer to the avatar object 1644.

[0176] In one embodiment, if the button is not pressed in step 3255 of Figure 32, the collision action may be suppressed, and if the button is pressed, the collision action may be suppressed. Also, in one embodiment, the user 5's choice of whether or not to generate a collision action may be an action other than pressing a button. The user 5's choice of whether or not to generate a collision action may also be the movement of the manipulated object.

[0177] In the above embodiment, a virtual space (VR space) into which the user is immersed by an HMD was used as an example for explanation, but a transparent HMD may also be used as the HMD. In this case, by outputting a field of view image which is a composite of a part of the image constituting the virtual space with the real space that the user sees through the transparent HMD, a virtual experience in an augmented reality (AR) space or a mixed reality (MR) space may be provided to the user. In this case, instead of an actionable object, an action may be performed on a target object in the virtual space based on the user's hand movements. Specifically, the processor may identify the coordinate information of the user's hand position in the real space and define the position of the target object in the virtual space in relation to the coordinate information in the real space. As a result, the processor can grasp the positional relationship between the user's hand in the real space and the target object in the virtual space, and execute processing corresponding to the collision control described above between the user's hand and the target object. As a result, it becomes possible to perform an action on the target object based on the user's hand movements.

[0178] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0179] (composition) The technical features disclosed above can be summarized as follows:

[0180] (Composition 1) According to one embodiment, a program is provided that is executed by a processor 210. The program provides the processor 210 with the steps of: defining a virtual space 11 (S1110); placing a virtual viewpoint 1643, collision regions 1745, 2545A, 2545B, 2545C, 2545D whose relative positions to the virtual viewpoint 1643 are defined and to which collision action is associated (S1120, S1150); changing at least one of the position and orientation of the virtual viewpoint 1643 in accordance with the movement of the head of a user 5 to which a head-mounted device (HMD 120) is associated (S1170); and, when at least one of the position and orientation of the virtual viewpoint 1643 changes, after a first time has elapsed, the relative position The following steps are performed: moving collision regions 1745, 2545A, 2545B, 2545C, and 2545D so that they are in the correct position; moving an operation object (hand object 1639) in the virtual space 11 in accordance with the movement of a first part (hand) which is a part of the user 5's body other than the user 5's head (S1170); generating the collision action in accordance with the fact that the operation object (hand object 1639) and the collision regions 17545, 2545A, 2545B, 2545C, and 2545D have reached a first positional relationship (e.g., collision) (S1170); and displaying a field of view image 1517 corresponding to the field of view from the virtual viewpoint 1643 on the head-mounted device (HMD 120) (S1180).

[0181] (Configuration 2) In configuration 1, collision regions 2545A, 2545B, 2545C, and 2545D are a plurality of divided collision regions, and the program causes the processor 210 to individually move each of the plurality of divided collision regions 2545A, 2545B, 2545C, and 2545D to the relative positions with respect to the virtual viewpoint 1643 (S1170), and to generate the collision action in response to the operation object (hand object 1639) and any of the plurality of divided collision regions 2545A, 2545B, 2545C, and 2545D being in the first positional relationship (S1170).

[0182] (Composition 3) In configuration (2), the multiple divided collision regions 2545A, 2545B, 2545C, and 2545D include a first collision region (2545A and 2545B), and the multiple divided collision regions 2545A, 2545B, 2545C, and 2545D include a second collision region (2545C and 2545D) located below the first collision region, and when the virtual viewpoint 1643 is tilted, the amount of rotation of the second collision region corresponding to the movement of the virtual viewpoint 1643 is smaller than the amount of rotation of the first collision region corresponding to the movement of the virtual viewpoint 1643 (S2852, S1170).

[0183] (Composition 4) In configuration 2, the multiple divided collision regions 2545A, 2545B, 2545C, and 2545D include a first collision region (2545A and 2545B), and the multiple divided collision regions 2545A, 2545B, 2545C, and 2545D include a second collision region (2545C and 2545D) located below the first collision region. When the position of the virtual viewpoint 1643 moves downward, the amount of movement of the second collision region corresponding to the movement of the virtual viewpoint 1643 is smaller than the amount of movement of the first collision region corresponding to the movement of the virtual viewpoint 1643 (S3054, S1170).

[0184] (Composition 5) In any one of configurations 1 to 4, the program causes the processor 210 to further execute the following steps: (S1150) position an avatar (avatar object 1644) that moves in conjunction with the movements of the user 5 in the virtual space 11; (S1150) position a collision area 3145 behind the avatar (avatar object 1644); (S1170) detect the movement of an operation object (hand object 1639) to establish the first positional relationship with the collision area 3145; and (S1170) move the collision area 3145 closer to the avatar (avatar object 1644) in response to the detection of the movement of the operation object (hand object 1639) to establish the first positional relationship.

[0185] (Composition 6) In any one of the configurations from (Configuration 1) to (Configuration 5), collision areas 1745, 2545A, 2545B, 2545C, 2545D, and 3145 are positioned outside the field of view from the virtual viewpoint 1643.

[0186] (Composition 7) In any one of the items from (Configuration 1) to (Configuration 6), the collision action places an item in association with the operation object (hand object 1639).

[0187] (Composition 8) In any one of the items from (Configuration 1) to (Configuration 6), the collision action replaces the first item that is positioned in association with the operation object (hand object 1639) with a second item positioned in association with the operation object (hand object 1639).

[0188] (Composition 9) In configuration 7 or configuration 8, the item is either an armor object or a weapon object (sword object 1641, gun object 2041).

[0189] (Composition 10) In any one of the items from (Configuration 1) to (Configuration 9), the virtual viewpoint 1643 is a first-person viewpoint associated with user 5.

[0190] (Composition 11) According to one embodiment, a method is provided that is performed by a processor. The method is performed by a processor and includes the steps of: defining a virtual space; arranging a virtual viewpoint, a collision region having a defined relative position to the virtual viewpoint and associated with a collision action, and an operation object in the virtual space; changing at least one of the position and orientation of the virtual viewpoint in response to the movement of the head of a user to which a head-mounted device is associated; moving the collision region to the relative position after a first time has elapsed, when at least one of the position and orientation of the virtual viewpoint has changed; moving the operation object in the virtual space in response to the movement of a first part which is a part of the user's body other than the user's head; generating the collision action in response to the operation object and the collision region being in a first positional relationship; and displaying a field of view image corresponding to the field of view from the virtual viewpoint on the head-mounted device.

[0191] (Composition 12) According to one embodiment, an information processing device is provided. The information processing device comprises a processor 210 and a memory (storage 230) that stores a program. The program causes the processor 210 to perform the following steps: define a virtual space; place a virtual viewpoint, a collision area whose relative position to the virtual viewpoint is defined and to which a collision action is associated, and an operation object in the virtual space; change at least one of the position and orientation of the virtual viewpoint in accordance with the movement of the head of a user to which a head-mounted device is associated; when at least one of the position and orientation of the virtual viewpoint changes, move the collision area after a first time has elapsed so that it becomes the relative position; move the operation object in the virtual space in accordance with the movement of a first part which is a part of the user's body other than the user's head; generate the collision action in accordance with the fact that the operation object and the collision area have achieved a first positional relationship; and display a field of view image corresponding to the field of view from the virtual viewpoint on the head-mounted device.

[0192] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims rather than by the foregoing description, and all modifications within the meaning and scope equivalent to the claims are intended to be included. [Explanation of Symbols]

[0193] 2…Network, 5…User, 6…Avatar Object, 11…Virtual Space, 12…Center, 14…Virtual Camera, 15…Field of View, 100…HMD System, 110…HMD Set, 130…Monitor, 170…Microphone, 180…Speaker, 190…Sensor, 200…Computer, 210…Processor, 220…Memory, 230…Storage, 240…Input / Output Interface, 250…Communication Interface, 300…Controller, 310…Grip, 320…Frame, 340, 350, 370, 380…Buttons, 390…Analog Stick, 410…HMD Sensor, 420…Motion Sensor, 430…Display, 510…Control Module, 520…Rendering Module, 530…Memory Module 540...Communication control module, 600...Server, 610...Processor, 620...Memory, 630...Storage, 640...Input / Output interface, 650...Communication interface, 1421...Virtual camera control module, 1422...Field of view determination module, 1423...Reference gaze identification module, 1424...Motion detection module, 1424...Facial organ detection module, 1425...Motion detection module, 1426...Virtual space definition module, 1427...Virtual object generation module, 1428...Operation object control module, 1429...Avatar control module, 1431...Spatial information, 1432...Object information, 1433...User information, 1434...Face information, 1435...Mouth template, 1438...Field of view image generation module.

Claims

[Claim 1] On the computer, In the virtual space, a virtual viewpoint, an area to which collision behavior is associated, and a hand object that is a virtual hand corresponding to the user's hand are set up. The collision action described above does not include the movement of the hand object. In response to the user's movements, the position and orientation of the virtual viewpoint are changed, If at least one of the position and orientation of the virtual viewpoint changes, move the region after a first time has elapsed. program.

Images