Program, information processing device, and method

Abstract

To provide a program, an information processing device, and a method for making the amount to be paid by a user for a virtual experience in a virtual space to be appropriate further.SOLUTION: In an HMD (Head-Mounted Device) set including an HMD, a computer, an HMD sensor, a display, and a controller, a processor of the computer causes an avatar object 6A associated with a user A to enter a second virtual space 2711A, causes an avatar object 6B placed in the second virtual space 2711A to execute performance in accordance with motion of a user B, and executes processing related to first charging for a first charging amount in accordance with the second virtual space 2711A and processing related to second charging for a second charging amount in accordance with the performance.SELECTED DRAWING: Figure 30

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing device, and a method. [Background technology]

[0002] Patent Documents 1 to 3 disclose examples of techniques that allow users to view content in a virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2017-176728 [Patent Document 2] Patent Publication No. 2018-007828 [Patent Document 3] Patent Publication No. 2016-025633 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technology has room to improve the amount of money users pay for virtual experiences in virtual spaces.

[0005] One aspect of the present disclosure aims to make the amount that users pay for a virtual experience in a virtual space more appropriate. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided a program executed by a computer having a processor for providing a virtual experience to a first user. The program causes the processor to execute the following steps: define a virtual space including a first area for providing the virtual experience to the first user; have a first avatar associated with the first user enter the first area; place a second avatar associated with a second user in the first area; have the second avatar perform a performance in accordance with the movement of the second user; execute a process related to a first charge for a first amount corresponding to the first area; and execute a process related to a second charge for a second amount corresponding to the performance. [Effects of the Invention]

[0007] According to one aspect of the present disclosure, the amount that a user pays for a virtual experience in a virtual space can be made more appropriate. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an outline of the configuration of an HMD system according to an embodiment. [Figure 2] FIG. 1 is a block diagram illustrating an example of a hardware configuration of a computer according to an embodiment. [Figure 3] FIG. 1 is a diagram conceptually illustrating a uvw field of view coordinate system set in an HMD according to an embodiment. [Figure 4] FIG. 1 is a diagram conceptually illustrating one mode of representing a virtual space according to an embodiment. [Figure 5] 1 is a top view of a user's head wearing an HMD according to an embodiment. FIG. [Figure 6] 10 is a diagram showing a YZ cross section of a field of view in a virtual space as viewed from the X direction. [Figure 7] 10 is a diagram showing an XZ cross section of a field of view in a virtual space as viewed from the Y direction. [Figure 8(A)] FIG. 2 is a diagram illustrating a schematic configuration of a controller according to an embodiment. [Figure 8(B)] FIG. 10 illustrates an example of yaw, roll, and pitch directions defined relative to a user's right hand according to one embodiment. [Figure 9] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a server according to an embodiment. [Figure 10] FIG. 1 is a block diagram illustrating a modular configuration of a computer according to an embodiment. [Figure 11] 10 is a sequence chart illustrating a part of a process executed in an HMD set according to an embodiment. [Figure 12(A)] FIG. 1 is a schematic diagram showing a situation in which each HMD provides a virtual space to a user in a network. [Figure 12(B)] FIG. 13 is a diagram showing a field of view image of a user 5A in FIG. 12(A). [Figure 13] FIG. 10 is a sequence diagram illustrating processing executed in an HMD system according to an embodiment. [Figure 14] FIG. 2 is a block diagram showing a detailed configuration of modules in a computer according to an embodiment. [Figure 15] 1 illustrates a virtual space and view image according to one embodiment. FIG. [Figure 16] 1 illustrates a virtual space and view image according to one embodiment. FIG. [Figure 17] FIG. 10 is a sequence diagram illustrating an example of processing executed in an HMD system according to an embodiment. [Figure 18] 10A and 10B are diagrams for explaining a method of acquiring dimension data according to an embodiment. [Figure 19] FIG. 2 illustrates an example of a data structure of location information according to an embodiment. [Figure 20] FIG. 2 is a diagram illustrating an example of a data structure of dimension data according to an embodiment. [Figure 21] 1 is a flowchart illustrating a process for obtaining dimensional data according to one embodiment. [Figure 22] FIG. 10 illustrates an example of a data structure for rotation directions according to an embodiment. [Figure 23] 10 is a sequence chart illustrating a part of a process executed in an HMD set according to an embodiment. [Figure 24] FIG. 2 illustrates a first virtual space and field of view image according to one embodiment. [Figure 25] FIG. 2 illustrates a first virtual space and field of view image according to one embodiment. [Figure 26] 10 is a sequence chart illustrating a part of a process executed in an HMD set according to an embodiment. [Figure 27] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 28] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 29] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 30] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 31] FIG. 1 is a diagram illustrating an example of a user's posture according to an embodiment. [Figure 32] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 33] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 34] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 35] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 36] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 37] FIG. 2 illustrates a first virtual space and field of view image according to one embodiment. [Figure 38] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 39] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 40] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 41] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 42] FIG. 10 illustrates a second virtual space and live video according to an embodiment. [Figure 43] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 44] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 45] 10 is a sequence chart illustrating a part of a process executed in an HMD set according to an embodiment. [Figure 46] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 47] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 48] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 49] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 50] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 51] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 52] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 53] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 54] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 55] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 56] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 57] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 58] 10 is a sequence chart illustrating a part of a process executed in an HMD set according to an embodiment. [Figure 59] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 60] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 61] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 62] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 63] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 64] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 65] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 66] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 67] 10 is a sequence chart illustrating a part of a process executed in an HMD set according to an embodiment. [Figure 68] FIG. 10 illustrates a third virtual space and field of view image according to one embodiment. [Figure 69] FIG. 10 illustrates a third virtual space and field of view image according to one embodiment. [Figure 70] 10 is a sequence chart illustrating a part of a process executed in an HMD set according to an embodiment. [Figure 71] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 72] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 73] FIG. 1 is a diagram illustrating an example of a user's posture according to an embodiment. [Figure 74] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 75] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 76] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 77] FIG. 10 illustrates a third virtual space and field of view image according to one embodiment. [Figure 78] 10 is a sequence chart showing an example of processing executed in the HMD system. [Figure 79] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 80] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 81] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 82] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 83] FIG. 1 is a diagram illustrating an example of a user's posture according to an embodiment. [Figure 84] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 85] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 86] 10 is a sequence chart showing an example of processing executed in the HMD system. [Figure 87] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 88] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 89] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 90] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 91]FIG. 10 is a diagram illustrating an example of data that the processor refers to in order to notify the user of at least one second performance. [Figure 92] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 93] 10 is a sequence chart showing an example of processing executed in the HMD system. [Figure 94] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 95] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 96] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 97] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 98] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 99] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 100] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 101] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 102] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 103] 2A and 2B are diagrams illustrating a first virtual space (first region) and a field of view image according to an embodiment. [Figure 104] FIG. 2 illustrates a first virtual space and field of view image according to one embodiment. [Figure 105] 10 is a sequence chart illustrating a part of a process executed in an HMD set according to an embodiment. [Figure 106] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 107] FIG. 2 illustrates a first virtual space and field of view image according to one embodiment. [Figure 108] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 109] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 110] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 111] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 112] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 113] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 114] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. [Figure 115] FIG. 10 illustrates a second virtual space and field of view image according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] [Embodiment 1] Hereinafter, embodiments of this technical idea will be described in detail with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. In one or more embodiments shown in this disclosure, elements included in each embodiment can be combined with each other, and the combined result also forms part of the embodiment shown in this disclosure.

[0010] [Configuration of HMD System] The configuration of an HMD (Head-Mounted Device) system 100 will be described with reference to Fig. 1. Fig. 1 is a diagram showing an outline of the configuration of HMD system 100 according to the present embodiment. HMD system 100 is provided as a system for home use or a system for business use.

[0011] The HMD system 100 includes a server 600, HMD sets 110A, 110B, 110C, and 110D, an external device 700, and a network 2. Each of the HMD sets 110A, 110B, 110C, and 110D is configured to be able to communicate with the server 600 and the external device 700 via the network 2. Hereinafter, the HMD sets 110A, 110B, 110C, and 110D will be collectively referred to as the HMD set 110. The number of HMD sets 110 constituting the HMD system 100 is not limited to four, and may be three or less, or five or more. The HMD set 110 includes an HMD 120, a computer 200, an HMD sensor 410, a display 430, and a controller 300. The HMD 120 includes a monitor 130, a gaze sensor 140, a first camera 150, a second camera 160, a microphone 170, and a speaker 180. The controller 300 may include a motion sensor 420.

[0012] In one aspect, the computer 200 can be connected to the Internet or other network 2, and can communicate with a server 600 or other computers connected to the network 2. Examples of other computers include computers of other HMD sets 110 and external devices 700. In another aspect, the HMD 120 can include a sensor 190 instead of the HMD sensor 410.

[0013] The HMD 120 is worn on the head of the user 5 and can provide a virtual space to the user 5 during operation. More specifically, the HMD 120 displays an image for the right eye and an image for the left eye on the monitor 130. When each eye of the user 5 views the respective image, the user 5 can recognize the image as a three-dimensional image based on the parallax between the two eyes. The HMD 120 can include both a so-called head-mounted display equipped with a monitor and a head-mounted device to which a smartphone or other terminal equipped with a monitor can be attached.

[0014] Monitor 130 is realized, for example, as a non-transmissive display device. In one aspect, monitor 130 is disposed on the main body of HMD 120 so as to be positioned in front of both eyes of user 5. Therefore, when user 5 views the three-dimensional image displayed on monitor 130, user 5 can be immersed in the virtual space. In one aspect, the virtual space includes, for example, a background, objects that user 5 can operate, and images of menus that user 5 can select. In another aspect, monitor 130 can be realized as a liquid crystal monitor or organic EL (Electro Luminescence) monitor provided in a so-called smartphone or other information display terminal.

[0015] In another aspect, the monitor 130 may be realized as a transmissive display device. In this case, the HMD 120 may be an open type such as a pair of glasses, rather than a closed type that covers the eyes of the user 5 as shown in FIG. 1 . The transmissive monitor 130 may be temporarily configured as a non-transmissive display device by adjusting its transmittance. The monitor 130 may include a configuration that simultaneously displays a portion of an image that constitutes a virtual space and the real space. For example, the monitor 130 may display an image of the real space captured by a camera mounted on the HMD 120, or may make the real space visible by setting the transmittance of a portion of the monitor 130 high.

[0016] In one aspect, monitor 130 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, monitor 130 may be configured to display an image for the right eye and an image for the left eye as a single image. In this case, monitor 130 includes a high-speed shutter. The high-speed shutter operates to alternately display an image for the right eye and an image for the left eye so that the image is recognized by only one of the eyes.

[0017] In one aspect, the HMD 120 includes a plurality of light sources (not shown). Each light source is realized, for example, by an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 410 has a position tracking function for detecting the movement of the HMD 120. More specifically, the HMD sensor 410 reads a plurality of infrared rays emitted by the HMD 120 and detects the position and tilt of the HMD 120 in real space.

[0018] In another aspect, the HMD sensor 410 may be realized by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the HMD 120 by performing image analysis processing using image information of the HMD 120 output from the camera.

[0019] In another aspect, the HMD 120 may include the sensor 190 as a position detector instead of or in addition to the HMD sensor 410. The HMD 120 may use the sensor 190 to detect the position and tilt of the HMD 120 itself. For example, if the sensor 190 is an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor, the HMD 120 may use any of these sensors instead of the HMD sensor 410 to detect the position and tilt of the HMD 120 itself. As an example, if the sensor 190 is an angular velocity sensor, the angular velocity sensor detects the angular velocity of the HMD 120 around three axes in real space over time. The HMD 120 calculates changes over time in the angles of the HMD 120 around the three axes based on the angular velocities, and further calculates the tilt of the HMD 120 based on the changes over time in the angles.

[0020] The gaze sensor 140 detects the direction in which the right and left eyes of the user 5 are looking. In other words, the gaze sensor 140 detects the gaze of the user 5. The detection of the gaze direction is achieved, for example, by a known eye tracking function. The gaze sensor 140 is achieved by a sensor having the eye tracking function. In certain aspects, the gaze sensor 140 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be, for example, a sensor that irradiates the right and left eyes of the user 5 with infrared light and detects the rotation angle of each eyeball by receiving light reflected from the cornea and iris of the irradiated light. The gaze sensor 140 can detect the gaze of the user 5 based on the detected rotation angles.

[0021] The first camera 150 captures the lower part of the face of the user 5. More specifically, the first camera 150 captures the nose, mouth, and the like of the user 5. The second camera 160 captures the eyes, eyebrows, and the like of the user 5. The housing of the HMD 120 on the user 5 side is defined as the inside of the HMD 120, and the housing of the HMD 120 on the opposite side from the user 5 is defined as the outside of the HMD 120. In one aspect, the first camera 150 may be disposed outside the HMD 120, and the second camera 160 may be disposed inside the HMD 120. Images generated by the first camera 150 and the second camera 160 may be input to the computer 200. In another aspect, the first camera 150 and the second camera 160 may be implemented as a single camera, and the face of the user 5 may be captured by this single camera.

[0022] The microphone 170 converts speech of the user 5 into an audio signal (electrical signal) and outputs it to the computer 200. The speaker 180 converts the audio signal into sound and outputs it to the user 5. In another aspect, the HMD 120 may include earphones instead of the speaker 180.

[0023] Controller 300 is connected to computer 200 via a wire or wirelessly. Controller 300 accepts input of commands from user 5 to computer 200. In one aspect, controller 300 is configured to be able to be held by user 5. In another aspect, controller 300 is configured to be able to be attached to the body or a part of clothing of user 5. In yet another aspect, controller 300 may be configured to output at least one of vibration, sound, and light based on a signal transmitted from computer 200. In yet another aspect, controller 300 accepts operations from user 5 to control the position and movement of an object placed in a virtual space.

[0024] In one aspect, the controller 300 includes multiple light sources. Each light source is realized, for example, by an LED that emits infrared light. The HMD sensor 410 has a position tracking function. In this case, the HMD sensor 410 reads multiple infrared rays emitted by the controller 300 and detects the position and tilt of the controller 300 in real space. In another aspect, the HMD sensor 410 may be realized by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the controller 300 by performing image analysis processing using image information of the controller 300 output from the camera.

[0025] In one aspect, the motion sensor 420 is attached to the hand of the user 5 and detects the movement of the hand of the user 5. For example, the motion sensor 420 detects the rotation speed, number of rotations, etc. of the hand. The detected signal is sent to the computer 200. The motion sensor 420 is provided, for example, in the controller 300. In one aspect, the motion sensor 420 is provided, for example, in the controller 300 configured to be able to be held by the user 5. In another aspect, for safety in the real world, the controller 300 is attached to something that will not easily fly away when worn on the hand of the user 5, such as a glove. In yet another aspect, the controller 300 is not attached to the user 5. Alternatively, a sensor may detect the hand movement of user 5. For example, a signal from a camera capturing an image of user 5 may be input to computer 200 as a signal representing the movement of user 5. As an example, motion sensor 420 and computer 200 are connected to each other wirelessly. In the case of wireless communication, there are no particular limitations on the communication format, and for example, Bluetooth (registered trademark) or other known communication methods may be used.

[0026] The display 430 displays an image similar to the image displayed on the monitor 130. This allows users other than the user 5 wearing the HMD 120 to view the same image as the user 5. The image displayed on the display 430 does not need to be a three-dimensional image, and may be an image for the right eye or an image for the left eye. Examples of the display 430 include a liquid crystal display and an organic EL monitor.

[0027] The server 600 may transmit a program to the computer 200. In another aspect, the server 600 may communicate with other computers 200 to provide virtual reality to the HMDs 120 used by other users. For example, in an amusement facility, when multiple users play a participatory game, each computer 200 communicates signals based on the actions of each user with the other computers 200 via the server 600, allowing multiple users to enjoy a common game in the same virtual space. Each computer 200 may also communicate signals based on the actions of each user with the other computers 200 without going through the server 600.

[0028] The external device 700 may be any device that can communicate with the computer 200. The external device 700 may be, for example, a device that can communicate with the computer 200 via the network 2, or a device that can directly communicate with the computer 200 via short-range wireless communication or a wired connection. Examples of the external device 700 include, but are not limited to, smart devices, PCs (Personal Computers), and peripheral devices of the computer 200.

[0029] [Hardware Configuration of Computer] A computer 200 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of computer 200 according to this embodiment. Computer 200 includes, as main components, a processor 210, a memory 220, a storage 230, an input / output interface 240, and a communication interface 250. Each component is connected to a bus 260.

[0030] Processor 210 executes a series of instructions included in a program stored in memory 220 or storage 230 based on a signal provided to computer 200 or based on the establishment of a predetermined condition. In one aspect, processor 210 is realized as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), a field-programmable gate array (FPGA), or other device.

[0031] Memory 220 temporarily stores programs and data. Programs are loaded from storage 230, for example. Data includes data input to computer 200 and data generated by processor 210. In one aspect, memory 220 is realized as RAM (Random Access Memory) or other volatile memory.

[0032] The storage 230 permanently stores programs and data. The storage 230 is realized, for example, as a ROM (Read-Only Memory), a hard disk drive, a flash memory, or other non-volatile storage device. The programs stored in the storage 230 include a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with other computers 200. The data stored in the storage 230 includes data and objects for defining the virtual space.

[0033] In another aspect, storage 230 may be realized as a removable storage device such as a memory card. In yet another aspect, a configuration may be used in which programs and data stored in an external storage device are used instead of storage 230 built into computer 200. With such a configuration, for example, in a situation where multiple HMD systems 100 are used, such as an amusement facility, it becomes possible to collectively update programs and data.

[0034] The input / output interface 240 communicates signals between the HMD 120, the HMD sensor 410, the motion sensor 420, and the display 430. The monitor 130, the gaze sensor 140, the first camera 150, the second camera 160, the microphone 170, and the speaker 180 included in the HMD 120 can communicate with the computer 200 via the input / output interface 240 of the HMD 120. In one aspect, the input / output interface 240 is realized using a terminal such as a Universal Serial Bus (USB), a Digital Visual Interface (DVI), or a High-Definition Multimedia Interface (HDMI®). The input / output interface 240 is not limited to those described above.

[0035] In one aspect, the input / output interface 240 may further communicate with the controller 300. For example, the input / output interface 240 receives input of signals output from the controller 300 and the motion sensor 420. In another aspect, the input / output interface 240 sends instructions output from the processor 210 to the controller 300. The instructions instruct the controller 300 to vibrate, output sound, emit light, etc. Upon receiving the instructions, the controller 300 executes one of the following in response to the instructions: vibration, sound output, or light emission.

[0036] Communication interface 250 is connected to network 2 and communicates with other computers (e.g., server 600) connected to network 2. In one aspect, communication interface 250 is realized as, for example, a wired communication interface such as a local area network (LAN), or a wireless communication interface such as Wi-Fi (Wireless Fidelity), Bluetooth (registered trademark), or NFC (Near Field Communication). Communication interface 250 is not limited to the above.

[0037] In one aspect, the processor 210 accesses the storage 230, loads one or more programs stored in the storage 230 into the memory 220, and executes a series of instructions included in the programs. The one or more programs may include an operating system for the computer 200, an application program for providing a virtual space, game software executable in the virtual space, etc. The processor 210 sends a signal for providing the virtual space to the HMD 120 via the input / output interface 240. The HMD 120 displays an image on the monitor 130 based on the signal.

[0038] 2 shows a configuration in which the computer 200 is provided outside the HMD 120, but in another aspect, the computer 200 may be built into the HMD 120. As an example, a portable information communication terminal (e.g., a smartphone) including the monitor 130 may function as the computer 200.

[0039] The computer 200 may be configured to be shared by multiple HMDs 120. With such a configuration, for example, the same virtual space can be provided to multiple users, allowing each user to enjoy the same application as other users in the same virtual space.

[0040] In one embodiment, a real coordinate system, which is a coordinate system in real space, is set in advance in the HMD system 100. The real coordinate system has three reference directions (axes) parallel to the vertical direction in real space, the horizontal direction perpendicular to the vertical direction, and the front-to-back direction perpendicular to both the vertical and horizontal directions. The horizontal direction, vertical direction (up-down direction), and front-to-back direction in the real coordinate system are defined as the x-axis, y-axis, and z-axis, respectively. More specifically, in the real coordinate system, the x-axis is parallel to the horizontal direction in real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-to-back direction in real space.

[0041] In one aspect, the HMD sensor 410 includes an infrared sensor. When the infrared sensor detects infrared rays emitted from each light source of the HMD 120, it detects the presence of the HMD 120. The HMD sensor 410 further detects the position and tilt (orientation) of the HMD 120 in real space in accordance with the movement of the user 5 wearing the HMD 120, based on the values ​​of each point (each coordinate value in the real coordinate system). More specifically, the HMD sensor 410 can detect changes over time in the position and tilt of the HMD 120 using each value detected over time.

[0042] Each tilt of the HMD 120 detected by the HMD sensor 410 corresponds to each tilt around the three axes of the HMD 120 in the real coordinate system. The HMD sensor 410 sets a uvw field of view coordinate system for the HMD 120 based on the tilt of the HMD 120 in the real coordinate system. The uvw field of view coordinate system set for the HMD 120 corresponds to a viewpoint coordinate system when the user 5 wearing the HMD 120 views an object in a virtual space.

[0043] [UVW Field of View Coordinate System] The uvw field of view coordinate system will be described with reference to Figure 3. Figure 3 is a conceptual diagram showing the uvw field of view coordinate system set in the HMD 120 according to an embodiment. When the HMD 120 is started up, the HMD sensor 410 detects the position and tilt of the HMD 120 in the real coordinate system. The processor 210 sets the uvw field of view coordinate system in the HMD 120 based on the detected values.

[0044] 3, the HMD 120 sets a three-dimensional uvw field of view coordinate system with the head of the user 5 wearing the HMD 120 as its center (origin). More specifically, the HMD 120 tilts the horizontal, vertical, and front-to-back directions (x-axis, y-axis, z-axis) that define the real coordinate system around each axis by the tilt of the HMD 120 around each axis in the real coordinate system, and sets these three directions as the pitch axis (u-axis), yaw axis (v-axis), and roll axis (w-axis) of the uvw field of view coordinate system in the HMD 120.

[0045] In a certain situation, when the user 5 wearing the HMD 120 stands upright and looks straight ahead, the processor 210 sets a uvw field of view coordinate system parallel to the real coordinate system in the HMD 120. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-to-back direction (z-axis) in the real coordinate system coincide with the pitch axis (u-axis), yaw axis (v-axis), and roll axis (w-axis) of the uvw field of view coordinate system in the HMD 120.

[0046] After the uvw field of view coordinate system is set in the HMD 120, the HMD sensor 410 can detect the tilt of the HMD 120 in the set uvw field of view coordinate system based on the movement of the HMD 120. In this case, the HMD sensor 410 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 120 in the uvw field of view coordinate system as the tilt of the HMD 120. The pitch angle (θu) represents the tilt angle of the HMD 120 around the pitch axis in the uvw field of view coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 120 around the yaw axis in the uvw field of view coordinate system. The roll angle (θw) represents the tilt angle of the HMD 120 around the roll axis in the uvw field of view coordinate system.

[0047] The HMD sensor 410 sets the uvw field of view coordinate system of the HMD 120 after the HMD 120 has moved to the HMD 120 based on the detected tilt of the HMD 120. The relationship between the HMD 120 and the uvw field of view coordinate system of the HMD 120 remains the same regardless of the position and tilt of the HMD 120. When the position and tilt of the HMD 120 change, the position and tilt of the uvw field of view coordinate system of the HMD 120 in the real coordinate system change in conjunction with the change in the position and tilt.

[0048] In one aspect, the HMD sensor 410 may identify the position of the HMD 120 in real space as a relative position with respect to the HMD sensor 410 based on the light intensity of the infrared light acquired based on the output from the infrared sensor and the relative positional relationship between multiple points (e.g., the distance between each point, etc.). The processor 210 may determine the origin of the uvw field of view coordinate system of the HMD 120 in real space (actual coordinate system) based on the identified relative position.

[0049] [Virtual Space] The virtual space will be further described with reference to FIG. 4. FIG. 4 is a conceptual diagram illustrating one mode of expressing virtual space 11 according to an embodiment. Virtual space 11 has a spherical structure that covers the entire 360-degree area from center 12. To avoid complicating the explanation, FIG. 4 illustrates only the upper half of the celestial sphere in virtual space 11. Meshes are defined in virtual space 11. The position of each mesh is defined in advance as a coordinate value in an XYZ coordinate system, which is a global coordinate system defined in virtual space 11. Computer 200 associates each partial image that constitutes panoramic image 13 (still image, video, etc.) that can be deployed in virtual space 11 with a corresponding mesh in virtual space 11.

[0050] In a certain aspect, an XYZ coordinate system is defined in virtual space 11 with center 12 as the origin. The XYZ coordinate system is, for example, parallel to the real coordinate system. The horizontal direction, vertical direction (up-down direction), and front-to-back direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the real coordinate system, and the Z-axis (front-to-back direction) of the XYZ coordinate system is parallel to the z-axis of the real coordinate system.

[0051] When the HMD 120 is started up, i.e., in the initial state of the HMD 120, the virtual camera 14 is placed at the center 12 of the virtual space 11. In a certain stage, the processor 210 displays an image captured by the virtual camera 14 on the monitor 130 of the HMD 120. The virtual camera 14 moves in the virtual space 11 in conjunction with the movement of the HMD 120 in the real space. This allows changes in the position and tilt of the HMD 120 in the real space to be reproduced in the virtual space 11 in the same manner.

[0052] A uvw field of view coordinate system is defined for the virtual camera 14, as in the case of the HMD 120. The uvw field of view coordinate system of the virtual camera 14 in the virtual space 11 is defined so as to be linked to the uvw field of view coordinate system of the HMD 120 in the real space (actual coordinate system). Therefore, when the tilt of the HMD 120 changes, the tilt of the virtual camera 14 also changes accordingly. The virtual camera 14 can also move in the virtual space 11 in conjunction with the movement in the real space of the user 5 wearing the HMD 120.

[0053] Processor 210 of computer 200 defines field of view 15 in virtual space 11 based on the position and tilt (reference line of sight 16) of virtual camera 14. Field of view 15 corresponds to the area of ​​virtual space 11 that is visible to user 5 wearing HMD 120. In other words, the position of virtual camera 14 can be said to be the viewpoint of user 5 in virtual space 11.

[0054] The line of sight of the user 5 detected by the gaze sensor 140 is the direction in the viewpoint coordinate system when the user 5 views an object. The uvw field of view coordinate system of the HMD 120 is equal to the viewpoint coordinate system when the user 5 views the monitor 130. The uvw field of view coordinate system of the virtual camera 14 is linked to the uvw field of view coordinate system of the HMD 120. Therefore, in a certain aspect, the HMD system 100 can regard the line of sight of the user 5 detected by the gaze sensor 140 as the line of sight of the user 5 in the uvw field of view coordinate system of the virtual camera 14.

[0055] [User's Line of Sight] Determining the line of sight of user 5 will be described with reference to Fig. 5. Fig. 5 is a diagram showing the head of user 5 wearing HMD 120 according to an embodiment as viewed from above.

[0056] In one aspect, the gaze sensor 140 detects the gaze of each of the right and left eyes of the user 5. In one aspect, when the user 5 is looking at something close, the gaze sensor 140 detects gazes R1 and L1. In another aspect, when the user 5 is looking at something far away, the gaze sensor 140 detects gazes R2 and L2. In this case, the angle formed by the gazes R2 and L2 with respect to the roll axis w is smaller than the angle formed by the gazes R1 and L1 with respect to the roll axis w. The gaze sensor 140 transmits the detection result to the computer 200.

[0057] When the computer 200 receives the detection values ​​of the lines of sight R1 and L1 from the gaze sensor 140 as the gaze detection result, it identifies the gaze point N1, which is the intersection of the lines of sight R1 and L1, based on the detection values. On the other hand, when the computer 200 receives the detection values ​​of the lines of sight R2 and L2 from the gaze sensor 140, it identifies the intersection of the lines of sight R2 and L2 as the gaze point. The computer 200 identifies the gaze N0 of the user 5 based on the position of the identified gaze point N1. For example, the computer 200 detects the direction of the line passing through the midpoint of the line connecting the right eye R and left eye L of the user 5 and the gaze point N1 as the gaze N0. The gaze N0 is the direction in which the user 5 actually looks with both eyes. The gaze N0 corresponds to the direction in which the user 5 actually looks with respect to the field of view 15.

[0058] In another aspect, the HMD system 100 may include a television broadcast receiving tuner. With this configuration, the HMD system 100 can display television programs in the virtual space 11.

[0059] In yet another aspect, the HMD system 100 may be provided with a communication circuit for connecting to the Internet or a telephone function for connecting to a telephone line.

[0060] [Viewing Area] The viewing area 15 will be described with reference to Figures 6 and 7. Figure 6 is a diagram showing a YZ cross section of the viewing area 15 in the virtual space 11 as viewed from the X direction. Figure 7 is a diagram showing an XZ cross section of the viewing area 15 in the virtual space 11 as viewed from the Y direction.

[0061] 6, the field of view 15 in the YZ cross section includes an area 18. The area 18 is defined by the position of the virtual camera 14, the reference line of sight 16, and the YZ cross section of the virtual space 11. The processor 210 defines the range including the polar angle α centered on the reference line of sight 16 in the virtual space as the area 18.

[0062] 7, the field of view 15 in the XZ cross section includes an area 19. The area 19 is defined by the position of the virtual camera 14, the reference line of sight 16, and the XZ cross section of the virtual space 11. The processor 210 defines a range including an azimuth angle β centered on the reference line of sight 16 in the virtual space 11 as the area 19. The polar angles α and β are determined according to the position of the virtual camera 14 and the tilt (orientation) of the virtual camera 14.

[0063] In one aspect, the HMD system 100 provides the user 5 with a field of view in the virtual space 11 by displaying a field of view image 17 on the monitor 130 based on a signal from the computer 200. The field of view image 17 is an image corresponding to a portion of the panoramic image 13 that corresponds to the field of view area 15. When the user 5 moves the HMD 120 worn on his / her head, the virtual camera 14 also moves in conjunction with the movement. As a result, the position of the field of view 15 in the virtual space 11 changes. As a result, the field of view image 17 displayed on the monitor 130 is updated to an image of the panoramic image 13 that is superimposed on the field of view 15 in the direction in which the user 5 is facing in the virtual space 11. The user 5 can view a desired direction in the virtual space 11.

[0064] In this way, the tilt of virtual camera 14 corresponds to the line of sight (reference line of sight 16) of user 5 in virtual space 11, and the position at which virtual camera 14 is placed corresponds to the viewpoint of user 5 in virtual space 11. Therefore, by changing the position or tilt of virtual camera 14, the image displayed on monitor 130 is updated and the field of view of user 5 is moved.

[0065] While wearing the HMD 120, the user 5 can view only the panoramic image 13 unfolded in the virtual space 11 without viewing the real world. Therefore, the HMD system 100 can give the user 5 a highly immersive feeling in the virtual space 11.

[0066] In one aspect, processor 210 may move virtual camera 14 in virtual space 11 in conjunction with movement in real space of user 5 wearing HMD 120. In this case, processor 210 identifies the image area (field of view 15) to be projected onto monitor 130 of HMD 120 based on the position and tilt of virtual camera 14 in virtual space 11.

[0067] In one aspect, virtual camera 14 may include two virtual cameras, i.e., a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. An appropriate parallax is set for the two virtual cameras so that user 5 can recognize three-dimensional virtual space 11. In another aspect, virtual camera 14 may be implemented using a single virtual camera. In this case, an image for the right eye and an image for the left eye may be generated from an image acquired by the single virtual camera. In this embodiment, the technical concept of the present disclosure is illustrated assuming that virtual camera 14 includes two virtual cameras and is configured such that a roll axis (w) generated by combining the roll axes of the two virtual cameras is adapted to the roll axis (w) of HMD 120.

[0068] [Controller] An example of controller 300 will be described with reference to Figure 8. Figure 8 is a diagram showing a schematic configuration of controller 300 according to an embodiment.

[0069] As shown in FIG. 8, in one aspect, the controller 300 may include a right controller 300R and a left controller (not shown). The right controller 300R is operated with the right hand of the user 5. The left controller is operated with the left hand of the user 5. In one aspect, the right controller 300R and the left controller are configured symmetrically as separate devices. Therefore, the user 5 can freely move both the right hand holding the right controller 300R and the left hand holding the left controller. In another aspect, the controller 300 may be an integrated controller that can be operated with both hands. The right controller 300R will be described below.

[0070] The right controller 300R includes a grip 310, a frame 320, and a top surface 330. The grip 310 is configured to be held by the right hand of the user 5. For example, the grip 310 can be held by the palm and three fingers (middle finger, ring finger, and little finger) of the right hand of the user 5.

[0071] Grip 310 includes buttons 340 and 350 and a motion sensor 420. Button 340 is located on the side of grip 310 and is operated by the middle finger of the right hand. Button 350 is located on the front of grip 310 and is operated by the index finger of the right hand. In some aspects, buttons 340 and 350 are configured as trigger-type buttons. Motion sensor 420 is built into the housing of grip 310. If the movements of user 5 can be detected from around user 5 by a camera or other device, grip 310 does not need to include motion sensor 420.

[0072] The frame 320 includes a plurality of infrared LEDs 360 arranged along its circumferential direction. The infrared LEDs 360 emit infrared light in accordance with the progress of a program that uses the controller 300 while the program is being executed. The infrared light emitted from the infrared LEDs 360 can be used to detect the positions and attitudes (tilt, direction) of the right controller 300R and the left controller. In the example shown in FIG. 8, the infrared LEDs 360 are arranged in two rows, but the number of arrangements is not limited to that shown in FIG. 8. Alternatively, an array of three or more rows may be used.

[0073] The top surface 330 includes buttons 370, 380 and an analog stick 390. The buttons 370, 380 are configured as push buttons. The buttons 370, 380 are operated by the thumb of the right hand of the user 5. In a certain situation, the analog stick 390 is operated in any direction within 360 degrees from the initial position (neutral position). Such operations include, for example, operations for moving an object placed in the virtual space 11.

[0074] In one aspect, the right controller 300R and the left controller include batteries for driving the infrared LED 360 and other components. Batteries include, but are not limited to, rechargeable, button-type, and dry cell batteries. In another aspect, the right controller 300R and the left controller can be connected to, for example, a USB interface of the computer 200. In this case, the right controller 300R and the left controller do not require batteries.

[0075] 8, for example, the yaw, roll, and pitch directions are defined for the right hand of user 5. When user 5 extends his thumb and index finger, the direction in which the thumb extends is defined as the yaw direction, the direction in which the index finger extends is defined as the roll direction, and the direction perpendicular to the plane defined by the yaw direction axis and the roll direction axis is defined as the pitch direction.

[0076] [Hardware Configuration of Server] Server 600 according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of the hardware configuration of server 600 according to an embodiment. Server 600 includes, as main components, a processor 610, a memory 620, a storage 630, an input / output interface 640, and a communication interface 650. Each component is connected to a bus 660.

[0077] The processor 610 executes a series of instructions included in a program stored in the memory 620 or the storage 630 based on a signal provided to the server 600 or based on the establishment of a predetermined condition. In one aspect, the processor 610 is implemented as a CPU, a GPU, an MPU, an FPGA, or other device.

[0078] The memory 620 temporarily stores programs and data. The programs are loaded from, for example, the storage 630. The data includes data input to the server 600 and data generated by the processor 610. In one aspect, the memory 620 is implemented as a RAM or other volatile memory.

[0079] The storage 630 permanently stores programs and data. The storage 630 is realized, for example, as a ROM, a hard disk drive, a flash memory, or other non-volatile storage device. The programs stored in the storage 630 may include a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with the computer 200. The data stored in the storage 630 may include data and objects for defining the virtual space.

[0080] In another aspect, storage 630 may be realized as a removable storage device such as a memory card. In yet another aspect, a configuration may be used in which programs and data stored in an external storage device are used instead of storage 630 built into server 600. With such a configuration, for example, in a situation where multiple HMD systems 100 are used, such as an amusement facility, it becomes possible to collectively update programs and data.

[0081] The input / output interface 640 communicates signals with input / output devices. In one aspect, the input / output interface 640 is implemented using a terminal such as a USB, DVI, HDMI (registered trademark), or the like. The input / output interface 640 is not limited to the above.

[0082] The communication interface 650 is connected to the network 2 and communicates with the computer 200 connected to the network 2. In one aspect, the communication interface 650 is realized as, for example, a wired communication interface such as a LAN, or a wireless communication interface such as WiFi, Bluetooth, NFC, etc. The communication interface 650 is not limited to the above.

[0083] In one aspect, the processor 610 accesses the storage 630, loads one or more programs stored in the storage 630 into the memory 620, and executes a series of instructions included in the programs. The one or more programs may include an operating system for the server 600, an application program for providing a virtual space, game software executable in the virtual space, etc. The processor 610 may send a signal for providing the virtual space to the computer 200 via the input / output interface 640.

[0084] [HMD Control Device] The control device of the HMD 120 will be described with reference to Fig. 10. In one embodiment, the control device is realized by a computer 200 having a known configuration. Fig. 10 is a block diagram showing the modular configuration of the computer 200 according to one embodiment.

[0085] 10 , the computer 200 includes a control module 510, a rendering module 520, a memory module 530, and a communication control module 540. In one aspect, the control module 510 and the rendering module 520 are implemented by the processor 210. In another aspect, multiple processors 210 may operate as the control module 510 and the rendering module 520. The memory module 530 is implemented by the memory 220 or the storage 230. The communication control module 540 is implemented by the communication interface 250.

[0086] The control module 510 controls the virtual space 11 provided to the user 5. The control module 510 defines the virtual space 11 in the HMD system 100 using virtual space data representing the virtual space 11. The virtual space data is stored in, for example, the memory module 530. The control module 510 may generate the virtual space data or obtain the virtual space data from the server 600 or the like.

[0087] The control module 510 places the object in the virtual space 11 using object data representing the object. The object data is stored in, for example, the memory module 530. The control module 510 may generate the object data or obtain the object data from, for example, the server 600. The object may include, for example, an avatar object representing the user 5, a character object, an operation object such as a virtual hand operated by the controller 300, landscapes including forests, mountains, and the like, cityscapes, animals, and the like that are placed according to the progress of the game story.

[0088] Control module 510 places an avatar object of user 5 of another computer 200 connected via network 2 in virtual space 11. In one aspect, control module 510 places an avatar object of user 5 in virtual space 11. In another aspect, control module 510 places an avatar object that resembles user 5 in virtual space 11 based on an image including user 5. In another aspect, control module 510 places an avatar object that has been selected by user 5 from multiple types of avatar objects (e.g., objects that resemble animals or deformed human objects) in virtual space 11.

[0089] The control module 510 determines the tilt of the HMD 120 based on the output of the HMD sensor 410. In another aspect, the control module 510 determines the tilt of the HMD 120 based on the output of the sensor 190 functioning as a motion sensor. The control module 510 detects the organs (e.g., mouth, eyes, eyebrows) that make up the face of the user 5 from the images of the face of the user 5 generated by the first camera 150 and the second camera 160. The control module 510 detects the movement (shape) of each detected organ.

[0090] The control module 510 detects the line of sight of the user 5 in the virtual space 11 based on a signal from the gaze sensor 140. The control module 510 detects a viewpoint position (coordinate value in the XYZ coordinate system) where the detected line of sight of the user 5 intersects with the celestial sphere of the virtual space 11. More specifically, the control module 510 detects the viewpoint position based on the line of sight of the user 5 defined in the uvw coordinate system and the position and inclination of the virtual camera 14. The control module 510 transmits the detected viewpoint position to the server 600. In another aspect, the control module 510 may be configured to transmit gaze information representing the line of sight of the user 5 to the server 600. In such a case, the server 600 may calculate the viewpoint position based on the gaze information received.

[0091] The control module 510 reflects the movement of the HMD 120 detected by the HMD sensor 410 in the avatar object. For example, the control module 510 detects that the HMD 120 has been tilted and tilts and positions the avatar object. The control module 510 reflects the detected movement of the facial organs in the face of the avatar object placed in the virtual space 11. The control module 510 receives gaze information of another user 5 from the server 600 and reflects the information in the gaze of the avatar object of the other user 5. In a certain aspect, the control module 510 reflects the movement of the controller 300 in the avatar object or the operation object. In this case, the controller 300 includes a motion sensor, an acceleration sensor, or a plurality of light-emitting elements (e.g., infrared LEDs) for detecting the movement of the controller 300.

[0092] The control module 510 places a control object in the virtual space 11 for receiving operations by the user 5 in the virtual space 11. The user 5 operates the control object to, for example, operate an object placed in the virtual space 11. In one aspect, the control object may include, for example, a hand object that is a virtual hand corresponding to the hand of the user 5. In one aspect, the control module 510 moves the hand object in the virtual space 11 in conjunction with the movement of the hand of the user 5 in the real space based on the output of the motion sensor 420. In one aspect, the control object may correspond to the hand portion of an avatar object.

[0093] The control module 510 detects a collision when each of the objects placed in the virtual space 11 collides with another object. The control module 510 can, for example, detect the timing when the collision area of ​​one object comes into contact with the collision area of ​​another object, and when this detection is made, performs a predetermined process. The control module 510 can detect the timing when two objects are no longer in contact with each other, and when this detection is made, performs a predetermined process. The control module 510 can detect when two objects are in contact with each other. For example, when a control object comes into contact with another object, the control module 510 detects that the control object has come into contact with the other object, and performs a predetermined process. .

[0094] In one aspect, the control module 510 controls image display on the monitor 130 of the HMD 120. For example, the control module 510 places a virtual camera 14 in the virtual space 11. The control module 510 controls the position of the virtual camera 14 in the virtual space 11 and the tilt (orientation) of the virtual camera 14. The control module 510 defines a field of view 15 according to the tilt of the head of the user 5 wearing the HMD 120 and the position of the virtual camera 14. The rendering module 520 generates a field of view image 17 to be displayed on the monitor 130 based on the determined field of view 15. The field of view image 17 generated by the rendering module 520 is output to the HMD 120 by the communication control module 540.

[0095] When the control module 510 detects an utterance made by the user 5 using the microphone 170 from the HMD 120, it identifies the computer 200 to which audio data corresponding to the utterance is to be sent. The audio data is sent to the computer 200 identified by the control module 510. When the control module 510 receives audio data from another user's computer 200 via the network 2, it outputs audio (utterance) corresponding to the audio data from the speaker 180.

[0096] The memory module 530 stores data used by the computer 200 to provide the virtual space 11 to the user 5. In one aspect, the memory module 530 stores space information, object information, and user information.

[0097] The space information includes one or more templates defined to provide the virtual space 11 .

[0098] The object information includes a plurality of panoramic images 13 that constitute the virtual space 11, and object data for placing objects in the virtual space 11. The panoramic images 13 may include still images and moving images. The panoramic images 13 may include images of unreal spaces and images of real spaces. Examples of images of unreal spaces include images generated by computer graphics.

[0099] The user information holds a user ID that identifies the user 5. The user ID may be, for example, an IP (Internet Protocol) address or a MAC (Media Access Control) address set in the computer 200 used by the user. In another aspect, the user ID may be set by the user. The user information includes a program for causing the computer 200 to function as a control device for the HMD system 100.

[0100] The data and programs stored in memory module 530 are input by user 5 of HMD 120. Alternatively, processor 210 downloads programs or data from a computer (e.g., server 600) operated by a business that provides the content, and stores the downloaded programs or data in memory module 530.

[0101] The communication control module 540 can communicate with the server 600 and other information communication devices via the network 2.

[0102] In one aspect, the control module 510 and the rendering module 520 may be realized using, for example, Unity (registered trademark) provided by Unity Technologies, Inc. In another aspect, the control module 510 and the rendering module 520 may be realized as a combination of circuit elements that realize each process.

[0103] Processing in computer 200 is realized by hardware and software executed by processor 210. Such software may be pre-stored on a hard disk or other memory module 530. Software may be stored on a CD-ROM or other computer-readable non-volatile data recording medium and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other network. Such software is read from the data recording medium by an optical disk drive or other data reading device, or downloaded from server 600 or other computers via communication control module 540, and then temporarily stored in a storage module. The software is read from the storage module by processor 210 and stored in RAM in the form of an executable program. Processor 210 executes the program.

[0104] [Control Structure of HMD System] The control structure of the HMD set 110 will be described with reference to Fig. 11. Fig. 11 is a sequence chart showing a part of the processing executed in the HMD set 110 according to an embodiment.

[0105] As shown in FIG. 11, in step S1110, the processor 210 of the computer 200, acting as the control module 510, identifies virtual space data and defines the virtual space 11.

[0106] In step S1120, processor 210 initializes virtual camera 14. For example, processor 210 places virtual camera 14 at a predefined center 12 in virtual space 11 in a work area of ​​memory, and directs the line of sight of virtual camera 14 in the direction in which user 5 is facing.

[0107] In step S1130, the processor 210, functioning as the rendering module 520, generates field of view image data for displaying an initial field of view image. The generated field of view image data is output to the HMD 120 by the communication control module 540.

[0108] In step S1132, the monitor 130 of the HMD 120 displays a field of view image based on the field of view image data received from the computer 200. The user 5 wearing the HMD 120 can recognize the virtual space 11 by viewing the field of view image.

[0109] In step S1134, the HMD sensor 410 detects the position and tilt of the HMD 120 based on the multiple infrared lights emitted from the HMD 120. The detection result is output to the computer 200 as motion detection data.

[0110] In step S1140, the processor 210 identifies the viewing direction of the user 5 wearing the HMD 120 based on the position and tilt included in the motion detection data of the HMD 120.

[0111] In step S1150, processor 210 executes the application program and places objects in virtual space 11 based on instructions included in the application program.

[0112] In step S1160, controller 300 detects an operation by user 5 based on a signal output from motion sensor 420, and outputs detection data representing the detected operation to computer 200. In another aspect, the operation of controller 300 by user 5 may be detected based on images from cameras arranged around user 5.

[0113] In step S1170, processor 210 detects an operation of controller 300 by user 5 based on the detection data acquired from controller 300.

[0114] In step S1180, the processor 210 generates field of view image data based on the operation of the controller 300 by the user 5. The generated field of view image data is output to the HMD 120 by the communication control module 540.

[0115] In step S1190, the HMD 120 updates the field of view image based on the received field of view image data, and displays the updated field of view image on the monitor 130.

[0116] [Avatar Objects] Avatar objects according to this embodiment will be described with reference to Figures 12(A) and (B). Below, figures explaining avatar objects of each user 5 of HMD sets 110A and 110B. Hereinafter, the user of HMD set 110A will be referred to as user 5A, the user of HMD set 110B as user 5B, the user of HMD set 110C as user 5C, and the user of HMD set 110D as user 5D. A is added to the reference symbol of each component related to HMD set 110A, B is added to the reference symbol of each component related to HMD set 110B, C is added to the reference symbol of each component related to HMD set 110C, and D is added to the reference symbol of each component related to HMD set 110D. For example, HMD 120A is included in HMD set 110A.

[0117] FIG. 12(A) is a schematic diagram illustrating a situation in which each HMD 120 provides a virtual space 11 to a user 5 in a network 2. Computers 200A to 200D provide virtual spaces 11A to 11D to users 5A to 5D, respectively, via HMDs 120A to 120D. In the example illustrated in FIG. 12(A), virtual space 11A and virtual space 11B are configured using the same data. In other words, computer 200A and computer 200B share the same virtual space. In virtual space 11A and virtual space 11B, there exist an avatar object 6A of user 5A and an avatar object 6B of user 5B. Although avatar object 6A in virtual space 11A and avatar object 6B in virtual space 11B are shown wearing HMDs 120, this is for ease of explanation; in reality, these objects do not wear HMDs 120.

[0118] In one aspect, the processor 210A may position a virtual camera 14A that captures a field of view image 17A of the user 5A at the eye position of the avatar object 6A.

[0119] 12(B) is a diagram showing a field of view image 17A of user 5A in FIG. 12(A). Field of view image 17A is an image displayed on monitor 130A of HMD 120A. This field of view image 17A is an image generated by virtual camera 14A. An avatar object 6B of user 5B is displayed in field of view image 17A. Although not specifically shown, avatar object 6A of user 5A is also displayed in the field of view image of user 5B.

[0120] 12(B), user 5A can communicate with user 5B through the virtual space 11A. More specifically, the voice of user 5A acquired by microphone 170A is transmitted to HMD 120B of user 5B via server 600 and output from speaker 180B provided in HMD 120B. The voice of user 5B is transmitted to HMD 120A of user 5A via server 600 and output from speaker 180A provided in HMD 120A.

[0121] The actions of user 5B (the actions of HMD 120B and controller 300B) are reflected in avatar object 6B placed in virtual space 11A by processor 210A, allowing user 5A to recognize the actions of user 5B through avatar object 6B.

[0122] Fig. 13 is a sequence chart showing a part of the processing executed in the HMD system 100 according to the present embodiment. Although the HMD set 110D is not shown in Fig. 13, the HMD set 110D operates in the same manner as the HMD sets 110A, 110B, and 110C. In the following description, A will be added to the reference symbol of each component related to the HMD set 110A, B will be added to the reference symbol of each component related to the HMD set 110B, C will be added to the reference symbol of each component related to the HMD set 110C, and D will be added to the reference symbol of each component related to the HMD set 110D.

[0123] In step S1310A, the processor 210A in the HMD set 110A acquires avatar information for determining the movement of the avatar object 6A in the virtual space 11A. The avatar information includes, for example, information about the avatar, such as movement information, face tracking data, and voice data. The movement information is The face tracking data includes information indicating temporal changes in the position and tilt of the HMD 120A, and information indicating hand movements of the user 5A detected by the motion sensor 420A, etc. The face tracking data includes data specifying the position and size of each feature of the user 5A's face. The face tracking data includes data indicating the movement of each organ constituting the user 5A's face, and gaze data. The audio data includes data indicating the audio of the user 5A acquired by the microphone 170A of the HMD 120A. The avatar information may include information specifying the avatar object 6A or the user 5A associated with the avatar object 6A, and information specifying the virtual space 11A in which the avatar object 6A exists. Examples of information specifying the avatar object 6A or the user 5A include a user ID. Examples of information specifying the virtual space 11A in which the avatar object 6A exists include a room ID. The processor 210A transmits the avatar information acquired as described above to the server 600 via the network 2.

[0124] In step S1310B, similar to the processing in step S1310A, processor 210B in HMD set 110B obtains avatar information for determining the movement of avatar object 6B in virtual space 11B and transmits it to server 600. Similarly, in step S1310C, processor 210C in HMD set 110C obtains avatar information for determining the movement of avatar object 6C in virtual space 11C and transmits it to server 600.

[0125] In step S1320, the server 600 temporarily stores the player information received from each of the HMD sets 110A, 110B, and 110C. The server 600 integrates the avatar information of all users (users 5A to 5C in this example) associated with the common virtual space 11 based on the user ID, room ID, and the like included in each piece of avatar information. The server 600 then transmits the integrated avatar information to all users associated with the virtual space 11 at a predetermined timing. This executes a synchronization process. This synchronization process allows the HMD sets 110A, 110B, and 110C to share each other's avatar information at approximately the same time.

[0126] Subsequently, each of the HMD sets 110A to 110C executes the processes of steps S1330A to S1330C based on the avatar information transmitted to each of the HMD sets 110A to 110C from the server 600. The process of step S1330A corresponds to the process of step S1180 in FIG.

[0127] In step S1330A, the processor 210A in the HMD set 110A updates the information on the avatar objects 6B and 6C of the other users 5B and 5C in the virtual space 11A. Specifically, the processor 210A updates the position, orientation, etc. of the avatar object 6B in the virtual space 11A based on the movement information included in the avatar information transmitted from the HMD set 110B. For example, the processor 210A updates the information (position, orientation, etc.) of the avatar object 6B included in the object information stored in the memory module 530. Similarly, the processor 210A updates the information (position, orientation, etc.) of the avatar object 6C in the virtual space 11A based on the movement information included in the avatar information transmitted from the HMD set 110C.

[0128] In step S1330B, processor 210B in HMD set 110B updates information on avatar objects 6A and 6C of users 5A and 5C in virtual space 11B, similar to the processing in step S1330A. Similarly, in step S1330C, processor 210C in HMD set 110C updates information on avatar objects 6A and 6B of users 5A and 5B in virtual space 11C.

[0129] [Detailed Module Configuration] The module configuration of computer 200 will be described in detail with reference to Fig. 14. Fig. 14 is a block diagram showing the detailed configuration of the modules of computer 200 according to an embodiment. As shown in Fig. 14, control module 510 includes a virtual object generation module 1421, a virtual camera control module 1422, a controllable object control module 1423, an avatar object control module 1424, a motion detection module 1425, a collision detection module 1426, a virtual object control module 1427, a billing processing module 1428, and a payment processing module 1429.

[0130] The virtual object generation module 1421 generates various virtual objects in the virtual space 11. In one aspect, the virtual objects may include, for example, landscapes including forests, mountains, and other objects, animals, etc., which are placed according to the progress of the game story. In another aspect, the virtual objects may include avatar objects, operation objects, stage objects, and UI (User Interface) objects.

[0131] The virtual camera control module 1422 controls the behavior of the virtual camera 14 in the virtual space 11. The virtual camera control module 1422 controls, for example, the placement position of the virtual camera 14 in the virtual space 11 and the orientation (tilt) of the virtual camera 14.

[0132] The operation object control module 1423 controls an operation object for receiving an operation by the user 5 in the virtual space 11. The user 5 operates the operation object to operate, for example, a virtual object placed in the virtual space 11. In one aspect, the operation object may include, for example, a hand object (virtual hand) corresponding to the hand of the user 5 wearing the HMD 120. In one aspect, the operation object may correspond to the hand portion of an avatar object described below.

[0133] The avatar object control module 1424 reflects the movement of the HMD 120 detected by the HMD sensor 410 in the avatar object. For example, the avatar object control module 1424 detects tilting of the HMD 120 and generates data for tilting and positioning the avatar object. In one aspect, the avatar object control module 1424 reflects the movement of the controller 300 in the avatar object. In this case, the controller 300 includes a motion sensor, an acceleration sensor, or multiple light-emitting elements (e.g., infrared LEDs) for detecting the movement of the controller 300. The avatar object control module 1424 reflects the movement of the facial organs detected by the movement detection module 1425 in the face of the avatar object placed in the virtual space 11. In other words, the avatar object control module 1424 reflects the facial movement of the user 5 in the avatar object.

[0134] The motion detection module 1425 detects the motion of the user 5. For example, the motion detection module 1425 detects the motion of the hand of the user 5 in response to the output of the controller 300. For example, the motion detection module 1425 detects the motion of the body of the user 5 in response to the output of a motion sensor attached to the body of the user 5. The motion detection module 1425 can also detect the movement of the facial organs of the user 5.

[0135] The collision detection module 1426 detects a collision when each virtual object placed in the virtual space 11 collides with another virtual object. The collision detection module 1426 can detect, for example, the timing when a virtual object comes into contact with another virtual object. The collision detection module 1426 can detect the timing when a virtual object and another virtual object cease to be in contact with each other. The collision detection module 1426 can also detect when a virtual object and another virtual object are in contact with each other. For example, when a control object and another virtual object come into contact with each other, the collision detection module 1426 detects that the control object and the other object have come into contact with each other. The collision detection module 1426 executes predetermined processing based on these detection results.

[0136] The virtual object control module 1427 controls the behavior of virtual objects excluding avatar objects in the virtual space 11. As one example, the virtual object control module 1427 transforms a virtual object. As another example, the virtual object control module 1427 changes the placement position of a virtual object. As another example, the virtual object control module 1427 moves a virtual object.

[0137] The billing processing module 1428 executes processing related to billing.

[0138] The payment processing module 1429 executes processing related to payment.

[0139] [Viewer's Virtual Space] FIG. 15 illustrates a virtual space 11A and a field of view image 1517A according to an embodiment. In FIG. 15(A), at least avatar objects 6A-6D, a virtual camera 14A, and a stage object 1532 are arranged in virtual space 11A for providing a virtual experience to user 5A (first user). User 5A wears HMD 120A on his / her head. User 5A holds a right controller 300RA with his / her right hand (first part) constituting a part of the right side of the user's body, and holds a left controller 300LA with his / her left hand (second part) constituting a part of the left side of the user's body. Avatar object 6A (first avatar) includes a virtual right hand 1531RA and a virtual left hand 1531LA. The virtual right hand 1531RA is a type of control object and can move in virtual space 11A in response to the movement of user 5A's right hand. The virtual left hand 1531LA is a type of operation object, and can move in the virtual space 11A in accordance with the movement of the left hand of the user 5A.

[0140] 15(A) is constructed by playing back live content on computer 200A. In virtual space 11A, avatar object 6B performs as a live performer, while other avatar objects, including avatar object 6A, watch the performance as live viewers. In virtual space 11A, avatar objects 6A to 6D are individually associated with users 5A to 5D, respectively.

[0141] In virtual space 11A, avatar object 6B (second avatar) is placed on stage object 1532. Stage object 1532 has an appearance that imitates a stage at a real live performance venue. Avatar objects 6A, 6C, and 6D are all placed in front of stage object 1532. In virtual space 11A, avatar object 6B performs a live performance by moving in accordance with the movements of user 5B (second user). In virtual space 11A, avatar object 6A watches the performance by avatar object 6B. At the same time, in virtual space 11C provided for user 5C, avatar object 6C watches the performance performed by avatar object 6B. Similarly, in virtual space 11D provided for user 5D, avatar object 6D watches the performance performed by avatar object 6B. Therefore, it can be said that user 5B is the performer, and users 5A, 5C, and 5D are the viewers.

[0142] In FIG. 15A, a virtual camera 14A is placed on the head of an avatar object 6A. The virtual camera 14A defines a field of view 15A according to the position and orientation of the virtual camera 14A. The virtual camera 14A defines a field of view 15A according to the position and orientation of the virtual camera 14A. A field of view image 1517A corresponding to avatar object 6A is generated and displayed on HMD 120A as shown in FIG. 15(B). By viewing field of view image 1517A, user 5A views a part of the virtual space from the viewpoint of avatar object 6A. This allows user 5A to have a virtual experience as if user 5A were avatar object 6A. Field of view image 1517A includes avatar object 6B performing a performance. Therefore, user 5A can watch the performance by avatar object 6B from the viewpoint of avatar object 6A.

[0143] A plurality of different avatar objects 6B may be placed in virtual space 11A. In one aspect, the plurality of avatar objects 6B may be associated with different users 5. In another aspect, the same user 5B may be associated with the plurality of avatar objects 6B.

[0144] [Actor's Virtual Space] Figure 16 shows a virtual space 11B and a field of view image 1617B according to one embodiment. In Figure 16(A), at least avatar objects 6A-6D, a virtual camera 14B, and a stage object 1532 are arranged in virtual space 11B for providing a virtual experience to user 5B. User 5B is wearing HMD 120B on his head. User 5B holds right controller 300RB with his right hand (first part) that constitutes part of the right side of the user 5B's body, and holds left controller 300LB with his left hand (second part) that constitutes part of the left side of the user 5B's body.

[0145] The HMD 120B includes a sensor 190 that functions as a motion sensor. The right controller 300RB and the left controller 300LB include a motion sensor 420. The user 5B also wears motion sensors 1641 to 1643. The motion sensor 1641 is attached to the waist of the user 5B by a belt 1644. The motion sensor 1642 is attached to the top of the right foot of the user 5B. The motion sensor 1643 is attached to the top of the left foot of the user 5B. The motion sensors 1641 to 1643 are connected to the computer 200B by wire or wirelessly.

[0146] In one aspect, a motion sensor worn by user 5B detects the arrival time and angle of a signal (e.g., an infrared laser) emitted from a base station (not shown). Processor 210B of computer 200B (hereinafter simply referred to as processor 210B) detects the position of the motion sensor relative to the base station based on the detection result of the motion sensor. Processor 210B may further normalize the position of the motion sensor relative to the base station using a predetermined point (e.g., the position of sensor 190 worn on the head) as a reference.

[0147] The avatar object 6B includes a virtual right hand 1531RB and a virtual left hand 1531LB. The virtual right hand 1531RB is a type of control object and can move in the virtual space 11B in accordance with the movement of the right hand of the user 5B. The virtual left hand 1531LA is a type of control object and can move in the virtual space 11B in accordance with the movement of the left hand of the user 5B.

[0148] Virtual space 11B shown in FIG. 16(A) is constructed by playing live content on computer 200B. Virtual spaces 11A and 11B are synchronized with each other under the control of server 600. User 5B moves his or her body to make avatar object 6B perform a performance. Computer 200B detects the movement of user 5B based on the output of various motion sensors attached to user 5B. In virtual space 11B, avatar object 6B performs a performance that reflects the movement of user 5B in real space, in accordance with the identified movement of user 5B. In virtual space 11B, avatar objects 6A, 6C, and 6D watch the performance by avatar object 6B. When avatar object 6B performs a performance in accordance with the movement of user 5B in virtual space 11B, avatar object 6B also performs the same performance in virtual spaces 11A, 11C, and 11D in synchronization with the performance. In this way, user 5B plays the role of a broadcaster who broadcasts the live performance of avatar object 6B to users 5A, 5C, and 5D, respectively.

[0149] In FIG. 16(A), virtual camera 14B is positioned at the head of avatar object 6B. Virtual camera 14B defines field of view 15B according to the position and orientation of virtual camera 14B. Virtual camera 14B generates field of view image 1617B corresponding to field of view 15B and displays it on HMD 120B as shown in FIG. 16(B). By viewing field of view image 1617B, user 5B views a portion of the virtual space from the perspective of avatar object 6B. This allows user 5B to have a virtual experience as if user 5B were avatar object 6B. Field of view image 1617B includes avatar objects 6A, 6C, and 6D watching the performance. Therefore, user 5B can grasp the state of avatar objects 6A, 6C, and 6D watching the performance by avatar object 6B from the perspective of avatar object 6B.

[0150] [Live Streaming Flow] Figure 17 is a sequence diagram showing an example of processing executed in HMD system 100. The following describes a series of processing steps for streaming a live performance of avatar object 6B in virtual space 11B from computer 200B to computer 200A. The live performance of avatar object 6B is also streamed to computers 200C and 200D based on a similar series of processing steps.

[0151] In step S1701, the processor 210B detects the positions of the head, waist, both hands, and both feet of the user 5B from motion sensors attached to the user 5B. Hereinafter, the positions of the parts of the user 5B detected by each motion sensor will also be referred to as "position information." In step S1702, the processor 210B calculates the rotation direction of the joints of the user 5B based on the current position information of the user 5B and dimensional data of the user 5B acquired in advance. The dimensional data is data representing the dimensions of the body of the user 5B. The dimensional data and the rotation direction will be described later. Detecting the current position information and calculating the rotation direction are synonymous with detecting the movement of the user 5B.

[0152] In step S1703, the processor 210B moves the avatar object 6B placed in the virtual space 11B based on the current position information and rotation direction. The processor 210B moves the right upper arm of the avatar object 6B, for example, based on the rotation direction of the right shoulder. The processor 210B further moves the position of the avatar object 6B in the virtual space 11B based on the current position information (for example, current position information of the waist). In this way, the processor 210B reflects the movement of the user 5B in the real space in the avatar object 6B in the virtual space. In other words, the processor 210B causes the avatar object 6B to perform a performance according to the movement of the user 5B.

[0153] The process for reflecting the movement of user 5B in avatar object 6B is not limited to the process based on the position information and rotation direction described above. For example, processor 210B may move avatar object 6B in accordance with the movement of user 5B without calculating the rotation direction. For example, processor 210B may control the position of each part object of avatar object 6B corresponding to each part of user 5B so that the position corresponds to the position of each part constituting the body of user 5B.

[0154] In step S1704, processor 210B generates movement information representing the movement of avatar object 6B when performing a performance, and transmits to server 600 avatar information of avatar object 6B including this movement information.

[0155] In step S1710, processor 210A of computer 200A (hereinafter simply referred to as processor 210A) transmits avatar information of avatar object 6A to the server. In step S1720, server 600 executes synchronization processing to synchronize virtual spaces 11A and 11B. Specifically, server 600 transmits the avatar information of avatar object 6B received from computer 200B to computer 200A. Server 600 further transmits the avatar information of avatar object 6A received from computer 200A to computer 200B.

[0156] In step S1705, processor 210B receives avatar information of avatar object 6A transmitted from server 600. In step S1706, processor 210B controls the avatar object in virtual space 11B based on the received avatar information. As a result, the behavior of avatar object 6A in virtual space 11A is reflected in avatar object 6A in virtual space 11B. In other words, the behavior of avatar object 6A is synchronized in virtual spaces 11A and 11B. For example, when avatar object 6A moves in virtual space 11A, avatar object 6A also moves in virtual space 11B in the same manner.

[0157] In step S1711, the processor 210A receives avatar information of the avatar object 6B transmitted from the server 600. In step S1712, the processor 210A moves the avatar object 6B based on the movement information included in the received avatar information. As a result, the processor 210B reflects the movement of the user 5B in the real space in the avatar object 6B placed in the virtual space 11A. In other words, the processor 210A causes the avatar object 6B to perform a performance in the virtual space 11A that corresponds to the movement of the user 5B. As a result, the behavior of the avatar object 6B in the virtual space 11B is reflected in the avatar object 6B in the virtual space 11A. In other words, the behavior of the avatar object 6B is synchronized in the virtual spaces 11A and 11B. For example, if the avatar object 6B performs a first performance in the virtual space 11B, the avatar object 6B also performs the same first performance in the virtual space 11A. In this way, the live performance of avatar object 6B in virtual space 11B is distributed to virtual space 11A.

[0158] Although not shown, the processor 210B records the voice uttered by the user 5B using the microphone 170B. The processor 210B generates voice data representing the voice of the user 5B and transmits it to the server 600. The server 600 transmits the received voice data of the user 5B to the computer 200A by synchronous processing. The processor 210A outputs the voice represented by the received voice data of the user 5B to the speaker 180A. As a result of this series of processes, the user 5A can hear the voice uttered by the user 5B during the live performance in real time.

[0159] [Acquisition of Dimensional Data] Figure 18 is a diagram for explaining a method for acquiring dimensional data. Figure 18(A) shows a state in which user 5B is facing forward, with both arms spread horizontally, and standing upright. Hereinafter, the state shown in Figure 18(A) is also referred to as the first posture. Figure 18(B) shows a state in which user 5B is facing forward, with both arms placed on the sides of the thighs, and standing upright. Hereinafter, the state shown in Figure 18(B) is also referred to as the second posture.

[0160] In one aspect, the processor 210B prompts the user 5B to take the first posture and the second posture. , the character in the first posture and the second posture is displayed on monitor 130B, and a message is displayed instructing the character to assume the same posture. As another example, processor 210B may output a sound from speaker 180B instructing the character to assume the first posture and the second posture.

[0161] The processor 210B acquires position information of the head, waist, both hands, and both feet of the user 5B based on the output of the motion sensors attached to the user 5B in each of the two postures (first posture and second posture). This position information can be acquired as positions in a real coordinate system (x, y, z) as shown in FIG.

[0162] The processor 210B calculates dimensional data of the user 5B from the position information corresponding to the two postures. In one embodiment, the processor 210B calculates the height, shoulder width, arm length, leg length, and height from head to shoulders of the user 5B as dimensional data, as shown in FIG. 20 . The processor 210B may calculate the distance between the hands in the second posture as the shoulder width. The processor 210B may calculate half of the value obtained by subtracting the shoulder width from the distance between the hands in the first posture as the arm length. The processor 210B may calculate the distance from the height of the feet to the height of the head as the height. The processor 210B may calculate the distance from the height of the feet to the height of the waist as the leg length. The processor 210B may calculate the height from the height of the hands to the head in the first posture as the height from the shoulders.

[0163] 21 is a flowchart showing a process for acquiring dimension data. In step S2110, processor 210B places virtual camera 14B in virtual space 11B. Processor 210B further outputs field of view image 17B corresponding to the shooting range of virtual camera 14B to monitor 130B.

[0164] In step S2120, the processor 210B instructs the user 5B to assume the first posture. For example, the processor 210B realizes the processing of step S2120 by placing an object with the instruction written on it in the virtual space 11B. In step S2130, the processor 210B acquires position information corresponding to the first posture.

[0165] In step S2140, the processor 210B instructs the user 5B to assume the second posture. In step S2150, the processor 210B acquires position information corresponding to the second posture.

[0166] In step S2160, processor 210B calculates dimension data of user 5B from the position information corresponding to the first posture and the position information corresponding to the second posture. Processor 210B stores the dimension data in storage 230B.

[0167] As described above, user 5B can easily input his / her own dimensions into computer 200B simply by assuming two postures. In another aspect, user 5B may input his / her own dimensions into computer 200B using an input device such as a keyboard.

[0168] [Joint Rotation Direction] In one embodiment, the processor 210B estimates the rotation direction of the joints of the user 5B based on the outputs (position information) of six motion sensors attached to the user 5B and dimensional data. As an example, the processor 210B estimates the position of the shoulders based on head position information, shoulder width, and height from the head to the shoulders. The processor 210B estimates the position of the elbows from the shoulder positions and hand position information. This estimation can be performed by a known application using inverse kinematics.

[0169] In one embodiment, the processor 210B acquires the tilts (rotation directions) of the joints of the user 5B's neck (head), waist, wrists, and ankles from six motion sensors. In addition, the processor 210B estimates the rotation directions of the joints of the shoulders, elbows, hips (groins), and knees based on inverse kinematics. As shown in FIG. 22, the processor 210B acquires or estimates the rotation direction of each joint in the uvw field of view coordinate system.

[0170] Note that when the rotation direction is calculated based on position information and dimensional data, the processor 210B cannot accurately estimate the position of the shoulders when the user 5B is not facing forward (i.e., when the head and waist are facing in different directions). Therefore, in another embodiment, the computer 200B may estimate the rotation direction of the joints by further taking into consideration the tilt of parts of the user 5B detected by a motion sensor. For example, the computer 200B estimates the shoulder position based on the head position information, the tilt of the head, the tilt of the waist, the shoulder width, and the height from the head to the shoulders. With this configuration, the computer 200B can improve the accuracy of the rotation direction of the joints.

[0171] [Live Venue Selection Process Flow] Figure 23 is a sequence chart illustrating a portion of the process executed by HMD set 110A according to an embodiment. Figure 24 is a diagram illustrating first virtual space 2411A and field of view image 2417A according to an embodiment. In this embodiment, a series of processes for selecting a virtual venue where user 5A will watch a virtual live performance will be described as being executed by HMD set 110A. However, this process may also be executed by other HMD sets 110B, 110C, and 110D, or part or all of this process may be executed by server 600.

[0172] In step S2301, processor 210A defines first virtual space 2411A as shown in FIG. 24(A). This process corresponds to the process of step S1110 in FIG. 11. Specifically, processor 210A identifies virtual space data, thereby defining first virtual space 2411A represented by the virtual space data. First virtual space 2411A is a virtual space in which avatar object 6A is placed before the start of a live performance by avatar object 6B. First virtual space 2411A is also a virtual space that allows user 5A to select a venue.

[0173] In step S2302, processor 210A, functioning as virtual object generation module 1421, generates virtual camera 14A and places it in first virtual space 2411A. In step S2303, processor 210A, functioning as virtual object generation module 1421, generates avatar object 6A including virtual right hand 1531RA and virtual left hand 1531LA and places it in first virtual space 2411A. In step S2304, processor 210A, functioning as virtual object generation module 1421, generates UI panel 2450 and pen object 2453 and places them in first virtual space 2411A.

[0174] UI panel 2450 is a type of UI object and is used by user 5A to cause processor 210A to execute processing for selecting a venue. UI panel 2450 includes options 2451 and 2452 arranged on the front surface of UI panel 2450. Options 2451 and 2452 include information describing the name of the venue that will be selected when the option is selected by user 5A. Option 2451 is an item for selecting a first venue, and option 2452 is an item for selecting a second venue that is different from the first venue. Pen object 2453 is a type of virtual object and is held and used by virtual right hand 1531RA or virtual left hand 1531LA to select option 2451 or 2452 on UI panel 2450.

[0175] In step S2305, processor 210A, functioning as virtual camera control module 1422, determines the position and tilt of virtual camera 14A in first virtual space 2411A in accordance with the movement of HMD 120A. More specifically, processor 210A controls field of view 15A, which is the field of view from virtual camera 14A in first virtual space 2411A, in accordance with the pose of user 5A's head and the position of virtual camera 14A in first virtual space 2411A. This processing corresponds to part of the processing of step S1140 in FIG. 11. Because virtual camera 14A is positioned in the same position as avatar object 6A, the position of virtual camera 14A is synonymous with the position of avatar object 6A. Furthermore, the field of view from virtual camera 14A is synonymous with the field of view from avatar object 6A.

[0176] In step S2306, processor 210A displays field of view image 17A on monitor 130A. Specifically, processor 210A defines field of view image 17A corresponding to field of view area 15A based on the movement of HMD 120A (i.e., the position and tilt of virtual camera 14A) and virtual space data defining first virtual space 2411A. Defining field of view image 17A is synonymous with generating field of view image 17A. Processor 210A further displays field of view image 17A on HMD 120A by outputting field of view image 17A to monitor 130A of HMD 120A. This processing corresponds to the processing of steps S1180 and S1190 in FIG. 11.

[0177] Processor 210A displays, for example, field of view image 2417A corresponding to first virtual space 2411A shown in Fig. 24(A) on monitor 130A as shown in Fig. 24(B). By viewing field of view image 2417A, user 5A recognizes that option 2451 or option 2452 on UI panel 2450 needs to be selected.

[0178] The processes of steps S2305 and S2306 described above (that is, updating of field of view image 17A in accordance with the movement of HMD 120A) are continuously and repeatedly executed while steps S2307 to S2310 described below are being executed.

[0179] In step S2307, the processor 210A, functioning as the movement detection module 1425, detects the movement of the right hand of the user 5A based on the output of the right controller 300RA. In step S2308, the processor 210A, functioning as the manipulation object control module 1423, moves the virtual right hand 1531RA in the first virtual space 2411A in accordance with the detected movement of the right hand of the user 5A. In a certain situation, the processor 210A moves the virtual right hand 1531RA in the first virtual space 2411A in accordance with the movement of the right hand of the user 5A so as to bring the virtual right hand 1531RA closer to the pen object 2453. After the virtual right hand 1531RA has come sufficiently close to the pen object 2453, the processor 210A selects (grabs) the pen object 2453 with the virtual right hand 1531RA based on the movement of the right hand of the user 5A. An example of the movement of the right hand of the user 5A is the movement of pressing any button on the right controller 300RA. As a result, the pen object 2453 is grasped by the imaginary right hand 1531RA.

[0180] FIG. 25 illustrates a first virtual space 2411A and a field of view image 2517A according to an embodiment. As shown in FIG. 25(A), after virtual right hand 1531RA selects pen object 2453, processor 210A moves virtual right hand 1531RA and pen object 2453 in first virtual space 2411A based on the movement of the right hand of user 5A so as to bring the tip of pen object 2453 closer to option 2451. Processor 210A, as collision detection module 1426, detects that the tip of pen object 2453 has collided with option 2451 when the tip of pen object 2453 and option 2451 have a first positional relationship. The first positional relationship means, for example, that the distance between the tip of pen object 2453 and option 2451 is less than a first distance. Alternatively, the collision area defined by the tip of pen object 2453 and the collision area set for option 2451 at least partially collide with each other. The processor 210A detects that the option 2451 has been selected by the pen object 2453 based on the collision between the tip of the pen object 2453 and the option 2451. In step S2309, the processor 210A accepts the selection of the first venue corresponding to the selected option 2451. The first venue is a venue where the user This is a virtual venue corresponding to the second virtual space 2711A, which is the environment in which The 5A watches the live performance.

[0181] Processor 210A displays, for example, field of view image 2517A corresponding to first virtual space 2411A shown in Fig. 25(A) on monitor 130A as shown in Fig. 25(B). By viewing field of view image 2517A, user 5A recognizes that he or she has selected option 2451 on UI panel 2450 (i.e., that he or she has selected the first venue).

[0182] In step S2310, processor 210A, functioning as avatar object control module 1424, causes avatar object 6A to exit first virtual space 2411A. Specifically, processor 210A ends the placement of avatar object 6A in first virtual space 2411A. After this, processor 210A continues defining first virtual space 2411A. Furthermore, even after avatar object 6A is no longer placed, processor 210A leaves other virtual objects placed in first virtual space 2411A. Processor 210A discontinues providing first virtual space 2411A to user 5A. Processor 210A then executes a series of processes for causing avatar object 6A to watch a performance by avatar object 6B, thereby newly providing second virtual space 2711A corresponding to the first venue to user 5A.

[0183] The venue selection method is not limited to the above example. For example, processor 210A places a spherical model that visualizes the contents of the venue in first virtual space 2411A. Even if this spherical model is selected by virtual right hand 1531RA or virtual left hand 1531LA, processor 210A can newly provide user 5A with second virtual space 2711A that corresponds to the venue indicated by the selected spherical model.

[0184] [Live Viewing Processing Flow] Fig. 26 is a sequence chart illustrating a portion of the processing executed by HMD set 110A according to an embodiment. Fig. 27 is a diagram illustrating second virtual space 2711A and field of view image 2717A according to an embodiment. In this embodiment, a series of processes for starting live viewing will be described as being executed by HMD set 110A. However, the processes may be executed by other HMD sets 110B, 110C, and 110D, or some or all of the processes may be executed by server 600.

[0185] In step S2601, processor 210A defines second virtual space 2711A as shown in FIG. 27(A). This process corresponds to the process of step S1110 in FIG. 11. Specifically, processor 210A identifies virtual space data, thereby defining second virtual space 2711A represented by the virtual space data. Second virtual space 2711A is a virtual space different from first virtual space 2411A, and is a virtual space in which avatar object 6A watches a live performance by avatar object 6B. In other words, second virtual space 2711A is a virtual space in which a performance by avatar object 6B is performed.

[0186] In step S2602, processor 210A generates virtual camera 14A and places it in second virtual space 2711A. In step S2603, processor 210A generates stage object 1532 and places it in second virtual space 2711A. In step S2604, processor 210A receives avatar information for avatar objects 6C and 6D from server 600. In step S2605, processor 210A places avatar objects 6C and 6D in second virtual space 2711A based on the received avatar information. Processor 210A also receives avatar information for other avatar objects 6 and places them in second virtual space 2711A. Many other avatar objects 6 are placed in second virtual space 2711A, but for ease of explanation, only one of them is representatively illustrated in FIG. 27. Hereinafter, other avatar objects 6 will not be mentioned unless specifically necessary.

[0187] In step S2606, processor 210A, functioning as avatar object control module 1424, causes avatar object 6A to enter second virtual space 2711A. Specifically, processor 210A places avatar object 6A, which has exited first virtual space 2411A, in second virtual space 2711A. In this manner, causing avatar object 6A to enter second virtual space 2711A is synonymous with placing avatar object 6A in second virtual space 2711A. Although not shown, processor 210A generates avatar information for avatar object 6A at any timing and transmits it to server 600.

[0188] In step S2607, processor 210A, functioning as virtual object generation module 1421, generates tag object 2733A and associates it with avatar object 6A. Specifically, processor 210A places tag object 2733A at virtual left hand 1531LA of avatar object 6A. Tag object 2733A is a type of virtual object. Tag object 2733A is a virtual object for managing billing information that indicates the amount charged to user 5A watching the live performance in second virtual space 2711A.

[0189] The processor 210A sets an upper limit on the amount of money that can be charged to the user 5A during a live performance (hereinafter referred to as the upper limit charge amount). The processor 210A further registers upper limit charge information indicating the set upper limit charge amount in the tag object 2733A. For example, the processor 210A can place the tag object 2733A on the virtual left hand 1531LA, and then accept input of the upper limit charge amount by the user 5A and set the input upper limit charge amount. For example, when the user 5B registers user information in the HMD system 100, the processor 210A can also set the upper limit charge amount in advance based on credit card information or the like included in the user information. For example, the processor 210A can also calculate the upper limit charge amount based on the user's past charges for live performances that the user 5A has viewed in the past. The processor 210A registers upper limit charge information indicating the set upper limit charge amount in the tag object 2733A.

[0190] In FIG. 27(A), avatar object 6B is not positioned in second virtual space 2711A. In other words, avatar object 6B has not yet started live in second virtual space 2711A. In step S2608, processor 210A, as virtual camera control module 1422, determines the position and tilt of virtual camera 14A in second virtual space 2711A in accordance with the movement of HMD 120A. This process is essentially the same as the process in step S2305, and therefore detailed description will not be repeated. In step S2609, processor 210A displays field of view image 17A on monitor 130A. This process is essentially the same as the process in step S2306, and therefore detailed description will not be repeated.

[0191] Processor 210A displays, for example, field of view image 2717A corresponding to second virtual space 2711A shown in Fig. 27(A) on monitor 130A as shown in Fig. 27(B). By viewing field of view image 2717A, user 5A recognizes that avatar object 6A has entered second virtual space 2711A. User 5A further recognizes that avatar object 6B has not yet started performing because avatar object 6B has not been placed on stage object 1532.

[0192] FIG. 28 illustrates a second virtual space 2711A and a field of view image 2817A according to an embodiment. In step S2610, processor 210A, acting as billing processing module 1428, registers billing information representing the admission fee for the first venue corresponding to second virtual space 2711A in tag object 2733A. The admission fee for the first venue is the amount charged to user 5A to watch a live performance held in second virtual space 2711A. The registered billing information can also be considered to be billing information corresponding to the action of avatar object 6A entering second virtual space 2711A. Processor 210A visualizes the billing information registered in tag object 2733A on tag object 2733A. In detail, processor 210A displays the sum of the billing amounts represented by the registered billing information (hereinafter, the total billing amount) on the surface of tag object 2733A. In FIG. 28(A), one piece of billing information is registered in tag object 2733A. In this case, the processor 210A displays the billing amount (500 yen) indicated by the registered billing information as the total billing amount in the tag object 2733A. The processor 210A further displays the upper limit billing amount (30,000 yen) indicated by the upper limit billing information in the tag object 2733A.

[0193] After the billing information is registered, user 5A moves his or her left hand into his or her field of view, as if checking the time on a wristwatch. Processor 210A detects the movement of user 5A's left hand based on the output of left controller 300LA. Processor 210A moves virtual left hand 1531LA into field of view 15A as shown in FIG. 28(A) in accordance with the movement of user 5A's left hand. For example, processor 210A displays field of view image 2817A corresponding to second virtual space 2711A shown in FIG. 28(A) on monitor 130A as shown in FIG. 28(B). By viewing field of view image 2817A, user 5A confirms the upper limit billing amount and total billing amount displayed on tag object 2733A. In the example of FIG. 28(B), the user recognizes that the upper limit billing amount is 30,000 yen and that the current total billing amount is 500 yen. In other words, the user 5A understands that as a result of the avatar object 6A entering the second virtual space 2711A, the user 5A has been charged 500 yen as an admission fee.

[0194] [Live Start] FIG. 29 illustrates a second virtual space 2711B and a field of view image 2917B according to one embodiment. After avatar object 6A enters second virtual space 2711A, processor 210B defines second virtual space 2711B shown in FIG. 29. Second virtual space 2711B is provided to user 5B and is synchronized with second virtual space 2711A. Second virtual space 2711B is also the virtual space where performance by avatar object 6B takes place. Processor 210B places virtual camera 14B, avatar object 6B, and stage object 1532 in second virtual space 2711B. Processor 210B receives avatar information for avatar objects 6A, 6C, and 6D from server 600 and places avatar objects 6A, 6C, and 6D in second virtual space 2711B based on the avatar information.

[0195] For example, the processor 210B displays a field of view image 2917B corresponding to the second virtual space 2711B shown in FIG. 29(A) on the monitor 130B as shown in FIG. 29(B). By viewing the field of view image 2917B, the user 5B recognizes that the avatar objects 6A, 6C, and 6D, who are viewers of the live performance, are in front of the stage object 1532. After the avatar object 6B is placed on the stage object 1532, the live performance by the avatar object 6B begins. After this, the user 5B progresses the live performance of the avatar object 6B in the second virtual space 2711B by moving, emitting sound, and operating the right controller 300RB and the left controller 300LB. The processor 210B transmits avatar information, including movement information of the avatar object 6B, to the server 600 in real time during the live performance. Send to.

[0196] 30 is a diagram showing second virtual space 2711A and field of view image 3017A according to one embodiment. After second virtual space 2711B is defined, in step S2611, processor 210A receives avatar information of avatar object 6B from server 600 in real time. In step S2612, processor 210A places avatar object 6B in second virtual space 2711A, as shown in FIG. 30(A), based on the avatar information of avatar object 6B that was initially received. In particular, processor 210A places avatar object 6B on stage object 1532.

[0197] Processor 210A displays, for example, field of view image 3017A corresponding to second virtual space 2711A shown in Fig. 30(A) on monitor 130A as shown in Fig. 30(B). By viewing field of view image 3017A, user 5A recognizes that avatar object 6B, a performer in the live performance, has appeared on stage object 1532. User 5B further recognizes that a live performance by avatar object 6B is about to begin.

[0198] FIG. 31 is a diagram illustrating an example of a posture of user 5B according to an embodiment. After the start of a live performance, user 5B moves his / her body to assume the posture shown in FIG. 31, for example. The posture shown in FIG. 31 corresponds to a first performance. Processor 210B causes avatar object 6B to perform a first performance in second virtual space 2711B in accordance with the movement of user 5B when assuming the posture shown in FIG. 31. Processor 210B transmits avatar information including movement information of avatar object 6B when avatar object 6B performs the first performance to server 600 in real time.

[0199] 32 illustrates second virtual space 2711A and field of view image 3217A according to an embodiment. After avatar object 6B performs a first performance in second virtual space 2711B, in step S2613, processor 210A receives avatar information of avatar object 6B from server 600 in real time. In step S2614, processor 210A causes avatar object 6B to perform the first performance based on the movement information included in the received avatar information. In this way, processor 210A causes avatar object 6B to perform the first performance in accordance with the movement of user 5B shown in FIG. 31.

[0200] 31, avatar object 6B can move to any position within second virtual space 2711A to perform the first performance, not just on stage object 1532. Avatar object 6A can watch the first performance by avatar object 6B from any position within second virtual space 2711A. Therefore, it can be said that the area within second virtual space 2711A where the first performance takes place (first area) is the entire second virtual space 2711A.

[0201] Processor 210A causes avatar object 6A to view the first performance by avatar object 6B in second virtual space 2711A. Processor 210A displays, for example, field of view image 3217A corresponding to second virtual space 2711A shown in FIG. 32(A) on monitor 130A as shown in FIG. 32(B). By viewing field of view image 3217A, user 5A recognizes that avatar object 6B has performed the first performance on stage object 1532. This allows user 5A to enjoy the performance by avatar object 6B from the perspective of avatar object 6A.

[0202] [Registering Billing Information] FIG. 33 is a diagram illustrating second virtual space 2711A and field of view image 3317A according to one embodiment. Processor 210A notifies user 5A during a live broadcast with a message inviting user 5A to purchase live video footage of avatar object 6B. Processor 210A displays the message on tag object 2733A, for example. The message includes text inviting user 5A to purchase the live video footage and the amount charged to user 5B to purchase the live video footage. In the example of FIG. 33, the amount charged to purchase the live video footage is 1,000 yen.

[0203] In second virtual space 2711A, a dedicated virtual camera (not shown) for capturing live video of items available for purchase by user 5A is placed in second virtual space 2711A. Processor 210A generates live video by capturing a portion of second virtual space 2711A with the dedicated virtual camera. Processor 210A can also capture live video while moving the dedicated virtual camera within second virtual space 2711A.

[0204] After the message is displayed on tag object 2733A, user 5A moves his or her left hand into his or her field of view, as if checking the time on a wristwatch. Processor 210A moves virtual left hand 1531LA into field of view area 15A, as shown in FIG. 33(A), in accordance with the movement of user 5A's left hand. Processor 210A displays, for example, field of view image 3317A corresponding to second virtual space 2711A shown in FIG. 33(A) on monitor 130A, as shown in FIG. 33(B). User 5A confirms the message displayed on tag object 2733A by viewing field of view image 3317A. This allows user 5B to recognize that he or she can purchase live video by paying an additional 1,000 yen.

[0205] 34 is a diagram showing second virtual space 2711A and field of view image 3417A according to an embodiment. After checking the message displayed on tag object 2733A, user 5A performs an operation to purchase the live video. User 5A moves his / her right hand so that the tip of virtual right hand 1531RA approaches tag object 2733A. In response to the movement of the right hand of user 5B, processor 210A moves virtual right hand 1531RA in second virtual space 2711A so that the tip of virtual right hand 1531RA approaches tag object 2733A.

[0206] The processor 210A, as the collision detection module 1426, detects that the tip of the virtual right hand 1531RA has collided with the tag object 2733A when the tip of the virtual right hand 1531RA and the tag object 2733A have a first positional relationship. The first positional relationship means, for example, that the distance between the tip of the virtual right hand 1531RA and the tag object 2733A is shorter than a first distance. Alternatively, the collision area defined by the tip of the virtual right hand 1531RA and the collision area set for the tag object 2733A at least partially collide. The processor 210A detects that the tag object 2733A has been selected by the virtual right hand 1531RA based on the collision between the tip of the virtual right hand 1531RA and the tag object 2733A. The processor 210A detects an operation by the user 5A to purchase the live video in response to the selection of the tag object 2733A on which a message inviting the user to purchase the live video is displayed.

[0207] Processor 210A displays, for example, field of view image 3417A corresponding to second virtual space 2711A shown in Fig. 34(A) on monitor 130A as shown in Fig. 34(B). By viewing field of view image 3417A, user 5A recognizes that tag object 2733A, on which a message is displayed, has been selected with virtual right hand 1531RA. In the example of Fig. 34, the operation by user 5A to purchase live video is performed on tag object 2733A, so that user 5A's immersion in second virtual space 2711A is not impaired.

[0208] FIG. 35 illustrates a second virtual space 2711A and a field of view image 3517A according to an embodiment. In step S2615, processor 210A, functioning as billing processing module 1428, registers billing information corresponding to the action of avatar object 6B in tag object 2733A. The action of avatar object 6A here refers to an action of avatar object 6A that leads to a charge for user 5B. In the example of FIG. 34, the action that leads to a charge for user 5B is avatar object 6A selecting tag object 2733A with virtual right hand 1531RA. The action that leads to a charge may also be an action that leads to a charge in embodiments 2 to 8, which will be described later. The action that leads to a charge may also be an action of avatar object 6A that leads to user 5A purchasing a virtual good (e.g., a virtual penlight) that can be used during a live performance. In the example of FIG. 34, processor 210A newly registers billing information indicating the charge amount (1,000 yen) required to purchase live video in tag object 2733A.

[0209] After the billing information is registered, processor 210A visualizes the billing information again on tag object 2733A. Specifically, processor 210A erases the message displayed on tag object 2733A and displays the upper limit billing amount and the total billing amount on tag object 2733A. Specifically, processor 210A calculates the sum (1,500 yen) of the two billing amounts (500 yen and 1,000 yen) represented by the two pieces of billing information registered in tag object 2733A, and displays this as the total billing amount on tag object 2733A. For example, processor 210A displays field of view image 3517A corresponding to second virtual space 2711A shown in FIG. 35(A) on monitor 130A as shown in FIG. 35(B). By viewing field of view image 3517A, user 5A recognizes that the current total billing amount has increased to 1,500 yen due to the purchase of live video. In this way, user 5A can check the total billing amount with a simple operation.

[0210] Processor 210A can visualize not only the total billing amount but also its breakdown on tag object 2733A. Processor 210A, for example, displays each billing amount represented by multiple pieces of billing information registered in tag object 2733A individually on tag object 2733A. User 5A can confirm each individual billing amount by visually checking tag object 2733A.

[0211] Processor 210A can also visualize the billing information directly on field of view image 3517A without visualizing it on tag object 2733A. Processor 210A can, for example, display a notification UI, such as a pop-up image superimposed on field of view image 3517A, on monitor 130A and display the total billing amount in the notification UI. Processor 210A can also associate the pop-up image with tag object 2733A on field of view image 3517A and display it on monitor 130A.

[0212] After registering the billing information corresponding to the live video purchase, processor 210A registers new billing information corresponding to the action in tag object 2733A each time avatar object 6A takes a new action in second virtual space 2711A that results in a charge. As the amount of billing information registered in tag object 2733A increases, the total amount billed to user 5A further increases. User 5B can check the latest total amount billed by visually checking tag object 2733A at a desired timing.

[0213] [Settlement of Total Charge Amount] Figure 36 shows second virtual space 2711A and field of view image 3617A according to one embodiment. In Figure 36(A), the live performance in second virtual space 2711A has ended. With the end of the live performance, avatar objects 6B and 6C have left second virtual space 2711A. Avatar objects 6A and 6D remain in second virtual space 2711A after the end of the live performance. At the end of the live performance, the total amount charged to user 5B is The total billing amount is assumed to be 12,000 yen. In other words, the total of the billing amounts represented by the billing information registered in tag object 2733A is 12,000 yen.

[0214] Processor 210A displays, for example, field of view image 3617A corresponding to second virtual space 2711A shown in Fig. 36(A) on monitor 130A as shown in Fig. 36(B). User 5A recognizes that the live performance has ended by viewing field of view image 3617A. User 5B further recognizes that the total amount charged to user 5A is 12,000 yen.

[0215] After checking the total amount charged at the end of the live performance, the user 5A performs an operation to cause the avatar object 6A to exit the second virtual space 2711A. This operation is, for example, the user 5A pressing any button on the right controller 300RA. In step S2616, in response to detecting the operation by the user 5A to cause the avatar object 6A to exit, the processor 210A causes the avatar object 6A to exit the second virtual space 2711A. This causes the processor 210A to end provision of the second virtual space 2711A to the user 5A.

[0216] FIG. 37 is a diagram illustrating first virtual space 2411A and field of view image 3717A according to an embodiment. In step S2617, processor 210A causes avatar object 6A, which has exited second virtual space 2711A, to re-enter first virtual space 2411A as shown in FIG. 37. In other words, processor 210A places avatar object 6A in first virtual space 2411A again. At this time, processor 210A disassociates tag object 2733A from avatar object 6A. In other words, processor 210A does not place tag object 2733A at virtual left hand 1531LA in first virtual space 2411A.

[0217] In step S2618, the processor 210A, as the payment processing module 1429, executes processing related to payment according to the billing information registered in the tag object 2733A. The processor 210A, for example, notifies the server 600 of the total of the billing amounts represented by the registered billing information. The server 600 settles the notified total billing amount. The server 600, for example, executes credit card settlement of the total billing amount according to the credit card information of the user 5A registered in the HMD system 100. As a result, the credit card issuer will later invoice the user 5A for the 12,000 yen charged to the user 5A. A server different from the server 600 can also settle the total billing amount charged to the user 5A. The processor 210A can also settle the total billing amount charged to the user 5A.

[0218] For example, the processor 210A displays a field of view image 3717A corresponding to the first virtual space 2411A shown in FIG. 37(A) on the monitor 130A as shown in FIG. 37(B). The server 600 notifies the computer 200A that the payment has been completed. In response to notification from the server 600 that the payment has been completed, the processor 210A notifies the user 5A that the payment has been completed on the field of view image 3717A. For example, the processor 210A displays a notification image 3735 including a message explaining that the payment has been completed, superimposed on the field of view image 3717A on the monitor 130A. The processor 210A can also place a virtual object having the same appearance as the notification image 3735 at any position within the field of view area 15A in the second virtual space 2711A. Alternatively, the processor 210A can include the notification image 3735 at any position within the field of view image 3717A when generating the field of view image 3717A. By viewing field of view image 3717A, user 5B recognizes that avatar object 6A has been placed again in first virtual space 2411A after the live performance has ended. User 5A further recognizes by viewing notification image 3735 that the payment of the fee charged to user 5A during the live performance has been completed. User 5A further recognizes that because tag object 2733A is no longer positioned in virtual left hand 7131LB in first virtual space 2411A, the fee corresponding to the actions of avatar object 6A will no longer be charged to user 5A in first virtual space 2411A.

[0219] The processor 210A can also leave the tag object 2733A placed on the avatar object 6A after the avatar object 6A moves to the first virtual space 2411A. In this case, after the payment is completed, the processor 210A displays a message on the tag object 2733A notifying the user 5A that the payment has been completed. The processor 210A further displays an "Agree" button on the tag object 2733A. When the avatar object 6A presses the "Agree" button, the processor 210A disassociates the tag object 2733A from the avatar object 6A. As a result, the tag object 2733A is no longer placed on the avatar object 6A. Therefore, the user 5A will no longer be able to see the tag object 2733A even if he or she moves the virtual left hand 1531LA into the field of view 15A.

[0220] After the live performance ends, the processor 210B calculates the sales revenue of the live performance by adding up the total amounts charged to each user 5 who is a viewer. The server 600 can also calculate the sales revenue and notify the computer 200B. The processor 210B executes processing to pay a fee to the user 5B according to the sales revenue of the live performance. For example, the processor 210B sends request information to the server 600 after the live performance ends, requesting payment of the fee to the server 600. In response to the received request information, the server 600 pays at least a portion of the sales revenue of the live performance to the user 5B as a fee after completing settlement for each user 5. The server 600 can pay the fee to the user 5B each time the live performance ends. Alternatively, the server 600 can pay the fee to the user 5B in a lump sum after a certain period (e.g., one month) has elapsed, according to the total sales revenue of each live performance held within that period. The processor 210B, rather than the server 600, can also pay the fee to the user 5B according to the sales revenue.

[0221] [Major Advantages of the Present Embodiment] As described above, the processor 210A automatically registers billing information corresponding to the behavior of the avatar object 6A in the second virtual space 2711A in association with the avatar object 6A. In this way, the HMD system 100 can provide the user 5A with a suitable billing method in the second virtual space 2711A. The processor 210A further automatically registers the billing information in the tag object 2733A without superimposing a UI that requests the user 5A to input information related to billing on the field of view image 17A and displaying it on the monitor 130A. This allows the user 5A's immersion in the second virtual space 2711A to be maintained when billing processing is performed.

[0222] The processor 210A further automatically executes a payment-related process when the avatar object 6A exits the second virtual space 2711A. In this way, the HMD system 100 can provide the user 5A with a preferred payment method in the second virtual space 2711A. The processor 210A further automatically executes the payment-related process without superimposing a UI that requests the user 5A to input payment information on the field of view image 17A and displaying it on the monitor 130A. This allows the user 5A's immersion in the first virtual space 2411A to be maintained when the payment-related process is executed.

[0223] The processor 210A further automatically executes the process related to the payment without requiring the user 5A to use a virtual object for payment (virtual currency) that resembles a credit card or cash, thereby preventing the user 5A from being bothered when the process related to the payment is executed.

[0224] [Warning to User 5A] Figure 38 is a diagram showing second virtual space 2711A and field of view image 3817A according to one embodiment. In the example of Figure 38, the total billed amount is 25,000 yen, which exceeds a certain percentage of the upper limit billed amount. The certain percentage is, for example, 70% of the upper limit billed amount (21,000 yen). Processor 210A warns user 5B that the total billed amount exceeds the certain percentage of the upper limit billed amount. Processor 210A warns user 5B by, for example, illuminating tag object 2733A as shown in Figure 38(A).

[0225] Processor 210A displays, for example, field of view image 3817A corresponding to second virtual space 2711A shown in FIG. 38(A) on monitor 130A as shown in FIG. 38(B). By viewing field of view image 3817A, user 5A recognizes that the current total billed amount is 25,000 yen. By viewing the light emitted from tag object 2733A, user 5A recognizes that he or she has been warned that the total billed amount is approaching the upper limit billed amount. This allows user 5A to avoid the total billed amount exceeding the upper limit billed amount without his or her knowledge.

[0226] After warning user 5A, processor 210A can increase the upper limit charge amount in accordance with the operation of user 5A. User 5A inputs a new upper limit charge amount into tag object 2733A, for example, by operating imaginary right hand 1531RA. Processor 210A detects the new input upper limit charge amount. Processor 210A updates the upper limit charge amount indicated by the upper limit charge information registered in tag object 2733A to the detected new upper limit charge amount.

[0227] FIG. 39 is a diagram illustrating second virtual space 2711A and field of view image 3917A according to one embodiment. In the example of FIG. 39, processor 210A controls the appearance of second virtual space 2711A according to the total billing amount. Specifically, processor 210A associates monitor object 3936A, which notifies user 5A of the total billing amount, with avatar object 6A and places it in second virtual space 2711A. Processor 210A also generates total billing information indicating the total billing amount and periodically transmits it to server 600. Hereinafter, this total billing information will be referred to as the total billing information of avatar object 6A.

[0228] In a second virtual space 2711C (not shown), a tag object 2733C is placed at the virtual left hand 1531LC of avatar object 6C. Each piece of charging information indicating the amount charged to user 5C is registered in tag object 2733C. Processor 210C calculates the sum (total charging amount) of the charging amounts indicated by the pieces of charging information registered in tag object 2733C. Processor 210C further generates total charging information indicating the total charging amount and periodically transmits it to server 600. Hereinafter, this total charging information will be referred to as the total charging information of avatar object 6C.

[0229] In a second virtual space 2711D (not shown), a tag object 2733D is placed at the virtual left hand 1531LD of avatar object 6D. Each piece of billing information representing the amount to be charged to user 5D is registered in tag object 2733D. Processor 210D calculates the sum (total billing amount) of the billing amounts represented by the billing information registered in tag object 2733D. Processor 210D further generates total billing information representing the total billing amount and periodically transmits it to server 600. Hereinafter, this total billing information will be referred to as the total billing information of avatar object 6D.

[0230] Processor 210A receives the total billing information for avatar objects 6C and 6D from server 600. Processor 210A controls the appearance of avatar object 6C in second virtual space 2711A in accordance with the total billing amount indicated by the total billing information for avatar object 6C. A monitor object 3936C that visualizes the total amount charged to user 5C is associated with avatar object 6C and placed in second virtual space 2711A.

[0231] Processor 210A controls the appearance of avatar object 6D in second virtual space 2711A according to the total billing amount represented by the billing information. Processor 210A controls the appearance of avatar object 6D in second virtual space 2711A according to the total billing amount represented by the total billing information of avatar object 6D. In detail, processor 210A places monitor object 3936D, which visualizes the total billing amount of user 5D, in second virtual space 2711A in association with avatar object 6D.

[0232] Monitor objects 3936A-3936C are disposed above avatar objects 6A-6C, respectively. Processor 210A controls the appearance of avatar object 6A by visualizing the billing information of avatar object 6A on monitor object 3936A. Specifically, processor 210A displays the total billed amount for user 5A on monitor object 3936A. Processor 210A controls the appearance of avatar object 6C by visualizing the billing information of avatar object 6C on monitor object 3936C. Specifically, processor 210A displays the total billed amount for user 5C on monitor object 3936C. Processor 210A controls the appearance of avatar object 6D by visualizing the billing information of avatar object 6D on monitor object 3936D. Specifically, processor 210D displays the total billed amount for user 5D on monitor object 3936D.

[0233] Processor 210A displays, for example, field of view image 3917A corresponding to second virtual space 2711A shown in Fig. 39(A) on monitor 130A as shown in Fig. 39(B). By viewing field of view image 3917A, user 5A recognizes that user 5C's current total bill amount is 20,000 yen and that user 5D's current total bill amount is 5,000 yen. Because monitor object 3936A is not located within field of view area 15A, user 5A does not view the total bill amount for user 5A displayed on monitor object 3936A.

[0234] The total charge amount for user 5A is displayed on monitor object 3936A, and therefore, the larger the total charge amount, the higher the amount displayed on monitor object 3936A. If the amount displayed on monitor object 3936A is low, that amount will not attract much attention from users 5B to 5D. Therefore, the avatar object 6A associated with monitor object 3936A will not stand out much to the other users 5B to 5D. The higher the amount displayed on monitor object 3936A, the more attention it will attract from users 5B to 5D. Therefore, the avatar object 6A associated with monitor object 3936A will appear more prominent to the other users 5B to 5D. In this way, by displaying the total charge amount for user 5A on monitor object 3936A, processor 210A can make the appearance of avatar object 6A stand out more as the total charge amount increases.

[0235] In field of view image 3917A, the total amount charged for user 5C (20,000 yen) displayed in association with avatar object 6C is greater than the total amount charged for user 5D (5,000 yen) displayed in association with avatar object 6D. Therefore, by visually checking field of view image 3917A, user 5A recognizes that the appearance of avatar object 6C stands out more than the appearance of avatar object 6D.

[0236] FIG. 40 illustrates second virtual space 2711B and field of view image 4017B according to one embodiment. Server 600 transmits the received total billing information for avatar objects 6A, 6C, and 6D to computer 200B through synchronization processing. Processor 210B receives the received total billing information for avatar objects 6A, 6C, and 6D from server 600. Processor 210B controls the appearance of avatar objects 6A, 6C, and 6D in accordance with the received total billing information for avatar objects 6A, 6C, and 6D. The appearance control in this case is similar to the control of the appearance of avatar objects 6A, 6C, and 6D in second virtual space 2711A shown in FIG. 39, and therefore detailed description will not be repeated.

[0237] For example, processor 210B displays field of view image 4017B corresponding to second virtual space 2711B shown in FIG. 40(A) on monitor 130B as shown in FIG. 40(B). By viewing field of view image 4017B, user 5B recognizes that user 5A's current total billed amount is 7,000 yen, user 5C's current total billed amount is 20,000 yen, and user 5D's current total billed amount is 5,000 yen. To user 5B, the appearance of avatar object 6C corresponding to the higher total billed amount appears more distinctive than the appearances of avatar object 6A corresponding to the lower total billed amount and avatar object 6D associated with user 5D. In this way, user 5C can appeal to user 5B that his / her total billed amount is higher than users 5B and 5D through the change in appearance of avatar object 6C.

[0238] Figure 41 shows second virtual space 2711A and field of view image 4117A according to one embodiment. In the example of Figure 41, processor 210A controls the exterior colors of avatar objects 6A, 6C, and 6D in accordance with the billing information for each of avatar objects 6A, 6C, and 6D. In particular, processor 210A changes the exterior colors of avatar objects 6A, 6C, and 6D by applying textures 4137A, 4137C, and 4137D in accordance with the billing information to avatar objects 6A, 6C, and 6D.

[0239] Texture 4137D is a texture that is colored a first color, corresponding to the total billed amount exceeding a first threshold. Texture 4137A is a texture that is colored a second color, corresponding to the total billed amount exceeding a second threshold. Texture 4137C is a texture that is colored a third color, corresponding to the total billed amount exceeding a third threshold. The second color is a color that stands out more than the first color. The third color is a color that stands out more than the second color. The first color is, for example, copper, the second color is, for example, silver, and the third color is, for example, gold.

[0240] Processor 210A displays, for example, field of view image 4117A corresponding to second virtual space 2711A shown in Figure 41(A) on monitor 130A as shown in Figure 41(B). By viewing field of view image 4117A, user 5A recognizes that the appearance of avatar object 6C has changed to the third color and that the appearance of avatar object 6D has changed to the first color. By viewing field of view image 3917A, user 5A recognizes that the appearance of avatar object 6C is more noticeable than the appearance of avatar object 6D.

[0241] The processor 210A can also change the appearance of the avatar object 6A to make it appear wealthier as the total amount charged increases. The processor 210A can also place a bulletin board object (not shown) in the second virtual space 2711A and display a ranking of total charges on the bulletin board object. For example, the processor 210A displays the names of the top five users 5 with the highest total charges on the bulletin board object. This makes the avatar objects 6 associated with users 5 with the highest total charges more noticeable in the second virtual space 2711A.

[0242] The processor 210A can also change a part of the appearance of the avatar object 6A, rather than the entirety of it. The processor 210A can also equip the avatar object 6A with accessories that can make the appearance of the avatar object 6A appear more wealthy. Examples of such accessories include a gold-plated watch or necklace.

[0243] [Watching Live Viewing] FIG. 42 illustrates a second virtual space 4211B and live video 4239 according to an embodiment. Similar to second virtual space 2711B, processor 210B places avatar objects 6A-6D, stage object 1532, and virtual camera 14B in second virtual space 4211B. Processor 210B also places a virtual camera 4238 in second virtual space 4211B. Virtual camera 4238 is a virtual object that generates live video captured of avatar object 6B. Processor 210B places virtual camera 4238, for example, at a certain distance in front of avatar object 6B. Processor 210B also controls the orientation of virtual camera 4238 to place stage object 1532 and avatar object 6B within field of view 4215 of virtual camera 4238.

[0244] Processor 210B, for example, controls virtual camera 4238 to capture an image of a portion of second virtual space 4211B that is included in field of view 4215 with virtual camera 4238. As a result, processor 210B generates live video 4239 corresponding to field of view 4215 as shown in FIG. 42(B). Live video 4239 is a type of field of view image. Live video 4239 shows avatar object 6B performing on stage object 1532. Processor 210B distributes the generated live video 4239 to each of the other computers 200, such as computer 200A, via server 600.

[0245] 43 is a diagram illustrating a second virtual space 4311A and a field of view image 4317A according to one embodiment. Similar to the second virtual space 2711A, the processor 210A places avatar objects 6A, 6C, and 6D in the second virtual space 4311A. However, unlike the second virtual space 2711A, the processor 210A does not place a stage object 1532 and an avatar object 6B in the second virtual space 4311A. Instead, the processor 210A places a screen object 4340 in the second virtual space 4311A.

[0246] The processor 210A receives the live video 4239 distributed from the computer 200B via the server 600. The processor 210A displays the received live video 4239 on the screen object 4340. As a result, the processor 210A causes the avatar object 6A to view the performance by the avatar object 6B displayed in the live video 4239. The processor 210A displays, for example, a field of view image 4317A corresponding to the second virtual space 4311A shown in FIG. 43(A) on the monitor 130A as shown in FIG. 43(B). The user 5A views the live video 4239 displayed on the screen object 4340 by viewing the field of view image 4317A. The live video 4239 displays the figure of the avatar object 6B performing on the stage object 1532. By viewing the live video 4239, the user 5A can have a virtual experience as if he or she were watching a live viewing of the avatar object 6B in the second virtual space 4311A.

[0247] The distribution of live viewing is not limited to the above-mentioned example. For example, in a virtual space for live video shooting in which only the avatar object 6B, the stage object 1532, and the virtual camera 4238 are arranged, a live video of the avatar object 6B is shot, and the live video is distributed to the public. On screen object 4340 placed in second virtual space 4311A, live video can also be distributed.

[0248] Processor 210A can also register, in tag object 2733A, billing information corresponding to the behavior of avatar object 6A in second virtual space 4311A. For example, when avatar object 6A takes an action that leads to user 5A purchasing live video 4239, processor 210A registers, in tag object 2733A, billing information corresponding to that behavior.

[0249] [First Area in Virtual Space] Figure 44 is a diagram showing a second virtual space 4411A and a field of view image 4417A according to one embodiment. Similar to second virtual space 2711A, processor 210A places avatar objects 6A and 6C and stage object 1532 in second virtual space 4411A. Second virtual space 4411A includes a first area 4451. First area 4451 corresponds to a portion of second virtual space 4411A. Processor 210A places a gate object 4452 at a location on the outer edge of first area 4451.

[0250] In FIG. 44 , avatar object 6C and stage object 1532 are placed within the first area 4451. Avatar object 6A is placed outside the first area 4451. The first area 4451 is surrounded by an opaque wall object (not shown). The first area 4451 is a location where a live performance by avatar object 6B is held. When avatar object 6A is within the first area 4451, it can watch the performance by avatar object 6B. When avatar object 6A is outside the first area 4451, it cannot watch the performance by avatar object 6B. Avatar object 6A can enter or exit the first area 4451 by passing through a gate object 4452.

[0251] After defining the second virtual space 4411A, the processor 210A places the avatar object 6A outside the first region 4451 in the second virtual space 4411A. While the avatar object 6A is placed outside the first region 4451, the processor 210A does not place the tag object 2733A in the virtual left hand 1531LA. In response to an operation by the user 5A, the processor 210A moves the avatar object 6A from outside the first region 4451 to inside the first region 4451 through the gate object 4452. As a result, the processor 210A causes the avatar object 6A to enter the first region 4451.

[0252] When avatar object 6A enters first area 4451, processor 210A places tag object 2733A in virtual left hand 1531LA. Processor 210A registers, in tag object 2733A, billing information according to the action of avatar object 6A within first area 4451. Processor 210A moves avatar object 6A from within first area 4451 to outside first area 4451 via gate object 4452 in response to an operation by user 5A. As a result, processor 210A causes avatar object 6A to exit first area 4451.

[0253] Processor 210A executes processing related to payment in accordance with the billing information registered in tag object 2733A when avatar object 6A exits first area 4451. Processor 210A further deletes tag object 2733A from virtual left hand 1531LA.

[0254] [Variation] Processor 210A can associate tag object 2733A with any part of avatar object 6A, not just virtual left hand 1531LA. Processor 210A, for example, associates tag object 2733A with virtual right hand 1531RA. Processor 210A associates tag object 2733A with, for example, the torso or foot of avatar object 6A. Processor 210A preferably places tag object 2733A on virtual right hand 1531RA or virtual left hand 1531LA. This allows user 5A to easily check the total charge amount displayed on tag object 2733A.

[0255] When the avatar object 6A enters the second virtual space 2711A, the processor 210A does not necessarily have to associate the tag object 2733A with the avatar object 6A. The processor 210A can associate billing information with the avatar object 6A and register it in the avatar object 6A. When the avatar object 6A exits the second virtual space 2711A, the avatar object 6A can also execute a payment process in accordance with the billing information registered in the avatar object 6A.

[0256] User 5A can purchase points in advance that can be used to pay the total billing amount. If user 5A has purchased points in advance, server 600 can pay the total billing amount by consuming points equivalent to the total billing amount from all points owned by user 5A.

[0257] When registering the billing information in the tag object 2733A, the processor 210A can store the billing information in the storage 230A. If the avatar object 6A fraudulently leaves the second virtual space 2711A, the processor 210A executes a settlement process using the billing information stored in the storage 230A. For example, the fraudulent departure of the avatar object 6A from the second virtual space 2711A may occur if the user 5A forcibly turns off the power of the computer 200A before the avatar object 6A exits the second virtual space 2711A. After the power of the computer 200A is turned on, the processor 210A transmits the billing information stored in the storage 230A to the server 600. The processor 610 settles the total amount of the billing indicated by the received billing information.

[0258] If the user 5A forcibly turns off the power of the computer 200A before causing the avatar object 6A to exit the second virtual space 2711A, the processor 210A detects that the avatar object 6A has unauthorizedly left the computer 200A the next time the computer 200A is turned on. When the processor 210A detects the occurrence of unauthorized leaving, it notifies the server 600 of this. When the processor 610 is notified of the occurrence of unauthorized leaving by the computer 200A, it settles the billing amount based on the billing information of the user 5A at the time of the unauthorized leaving. This allows the HMD system 100 to prevent the user 5A from evading the payment of the billing amount. If unauthorized leaving of the avatar object 6A occurs frequently or if the payment cannot be completed successfully after the unauthorized leaving, the processor 610 can delete the account of the user 5A.

[0259] When the avatar object 6A enters the second virtual space 2711A, the processor 210A registers billing information corresponding to the admission fee in the tag object 2733A and can immediately execute a process related to the settlement for the admission fee. In other words, the processor 210A can execute a process related to the settlement after the billing information is registered in the tag object 2733A and before the avatar object 6A exits the second virtual space 2711A. Furthermore, when the processor 210A registers billing information according to the behavior of the avatar object 6A in the tag object 2733A, it can also execute a process related to the settlement corresponding to the registered billing information before the avatar object 6A exits.

[0260] Processor 210A can cause avatar object 6A to enter second virtual space 2711A before a live performance of avatar object 6B takes place in second virtual space 2711A. When avatar object 6A enters second virtual space 2711A before the live performance takes place, processor 210A can cause avatar object 6A to purchase a ticket for the live performance from avatar object 6B in second virtual space 2711A.

[0261] User 5A can have avatar object 6A purchase tickets for a live concert before the concert begins. In one aspect, processor 210B detects user 5B's movements for selling tickets to the concert before the concert begins. Processor 210B causes avatar object 6B to perform a first action for selling tickets in response to the detected movements. Processor 210B transmits avatar information of avatar object 6B, including movement information corresponding to the first action, to server 600.

[0262] When avatar object 6A enters second virtual space 2711A before the live performance begins, processor 210A requests server 600 for avatar information of avatar object 6B, which computer 200B has transmitted to server 600 in advance. In response to the request, server 600 transmits the avatar information of avatar object 6B to computer 200A. Based on the received avatar information, processor 210A places avatar object 6B in second virtual space 2711A. Based on movement information included in the avatar information, processor 210A causes avatar object 6B to perform a first action for selling tickets. For example, processor 210A causes avatar object 6B to hand the ticket to avatar object 6A. Processor 210A further causes avatar object 6B to wave virtual left hand 1531LB.

[0263] In another aspect, when processor 210A causes avatar object 6A to enter second virtual space 2711A during a specific time period, processor 210A transmits avatar information of avatar object 6A to server 600. As a result of synchronization processing by server 600, processor 210B receives the avatar information of avatar object 6A from server 600. Processor 210B places avatar object 6A in second virtual space 2711B in accordance with the received avatar information. After avatar object 6A is placed, processor 210B detects a movement of user 5B to sell tickets for a live performance. In accordance with the detected movement of user 5B, processor 210B causes avatar object 6B to perform a second action to sell tickets in second virtual space 2711B. Processor 210B transmits the avatar information of avatar object 6B, including movement information corresponding to the second action, to server 600.

[0264] As a result of the synchronization process of the server 600, the processor 210A receives avatar information of the avatar object 6B from the server 600. In response to the movement information included in the received avatar information, the processor 210A causes the avatar object 6B to perform a second action for selling tickets based on the movement information included in the avatar information. In other words, the processor 210A causes the avatar object 6B to perform the action of selling tickets in response to the real-time movement of the user 5B. This allows the user 5A to interact with the user 5B in real time, thereby making the user 5A more satisfied.

[0265] [Embodiment 2] In this embodiment, the processor 210A executes processing related to charging and settlement for the user 5A by a method using the tag object 2733A described in embodiment 1. However, without being limited to this, the processor 210A can also execute processing related to charging and settlement for the user 5A by any method different from the method according to embodiment 1.

[0266] [Charging Process Flow According to Travel Distance] FIG. 45 shows a flow of charging according to travel distance in an HMD according to an embodiment. FIG. 46 is a sequence chart illustrating a portion of a process executed in set 110A. FIG. 46 illustrates second virtual space 2711A and field of view image 4617A according to an embodiment. In FIG. 46, a performance is being performed by avatar object 6B (second avatar) associated with user 5B (second user) in second virtual space 2711A. Only billing information corresponding to the admission fee (500 yen) is registered in tag object 2733A. Avatar object 6A (first avatar) associated with user 5A (first user) is located at first position 4661 in second virtual space 2711A, which is far away from stage object 1532. User 5A is watching the performance of avatar object 6B, who is located far away from avatar object 6A.

[0267] User 5A wishes to watch the performance from closer to avatar object 6B by moving avatar object 6A to second position 4662, which is closer to stage object 1532. To fulfill this desire, user 5A moves his / her right hand so that the tip of virtual right hand 1531RA points to second position 4662. In step S4501, processor 210A detects the movement of user 5A's right hand based on the output of right controller 300RA. In step S4502, processor 210A moves virtual right hand 1531RA in accordance with the movement of user 5A's right hand. As shown in FIG. 46(A), after virtual right hand 1531RA moves, the tip of virtual right hand 1531RA points to second position 4662 in second virtual space 2711A. In step S4503, processor 210A identifies second position 4662 in second virtual space 2711A pointed to by the tip of imaginary right hand 1531RA.

[0268] For example, the processor 210A displays a field of view image 4617A corresponding to the second virtual space 2711A shown in FIG. 46(A) on the monitor 130A as shown in FIG. 46(B). In step S4504, the processor 210A notifies the user 5A of a mark 4663 indicating a second position 4662 in the second virtual space 2711A on the field of view image 4617A. In detail, the processor 210A displays the mark 4663 on the monitor 130A by superimposing it on a position 4664 in the field of view image 4617A corresponding to the second position 4662. The processor 210A can place the mark 4663 as a virtual object at the second position 4662 in the second virtual space 2711A. Alternatively, the processor 210A can include the mark 4663 at the position 4664 corresponding to the second position 4662 when generating the field of view image 4617A. By viewing the field of view image 4617A, the user 5A confirms that a mark 4663 indicating the second position 4662 in the second virtual space 2711A is displayed on the monitor 130A. By confirming the mark 4663, the user 5A understands that the second position 4662 has been designated as the destination of the avatar object 6A.

[0269] After checking the mark 4663, the user 5A performs a first operation to move the avatar object 6A to the second position 4662. The first operation is, for example, the user pressing any button on the right controller 300RA. In step S4505, the processor 210A detects the first operation of the user 5A to move the avatar object 6A.

[0270] 47A and 47B illustrate a second virtual space 2711A and a field of view image 4717A according to an embodiment. In step S4506, when a first operation by user 5A is detected, processor 210A, functioning as avatar object control module 1424, discretely moves avatar object 6A from first position 4661 to second position 4662 within second virtual space 2711A, as shown in FIG. 47A. Specifically, processor 210A instantaneously moves (warps) avatar object 6A from first position 4661 to second position 4662 without passing through any of the positions between first position 4661 and second position 4662. Processor 210A also discretely moves virtual camera 14A in conjunction with the movement of avatar object 6A.

[0271] In step S4507, the processor 210A registers, in the tag object 2733A, charging information corresponding to the distance 4775 traveled by the avatar object 6A from the first position 4661 to the second position 4662. Specifically, the processor 210A calculates a charging amount corresponding to the distance 4775 traveled by multiplying the distance 4775 by a charging unit price for each fixed distance. The charging unit price is determined in advance by, for example, the operator of the HMD system 100. For example, when defining the second virtual space 2711A, the processor 210A sets, in the computer 200A, a charging unit price represented by the unit price information included in the virtual space data. In FIG. 47, 500 yen is calculated as the charging amount. The processor 210A newly registers, in the tag object 2733A, charging information representing the calculated charging amount. The processor 210A displays, on the surface of the tag object 2733A, the total of the charging amounts represented by the two pieces of registered charging information (1,000 yen).

[0272] After the billing information is registered, the processor 210A moves the virtual left hand 1531LA within the field of view 15A in accordance with the movement of the left hand of the user 5A, as shown in FIG. 47(A). The processor 210A displays, for example, a field of view image 4717A corresponding to the second virtual space 2711A shown in FIG. 47(A) on the monitor 130A, as shown in FIG. 47(B). By viewing the field of view image 4717A, the user 5A recognizes that the avatar object 6A has moved to the second position 4662 near the stage object 1532. The user 5A further recognizes that the avatar object 6B is in front of the avatar object 6A. This allows the user 5A to view the performance near the avatar object 6B, thereby further enhancing the enjoyment of the live performance. The user 5A further recognizes that the current total billed amount is 1,000 yen by viewing the tag object 2733A. In other words, the user 5A understands that, as a result of moving the avatar object 6A to the second position 4662, a new travel fee of 500 yen has been charged to the user 5A.

[0273] FIG. 48 illustrates a second virtual space 2711A and a field of view image 4817A according to an embodiment. The second virtual space 2711A in FIG. 48 is the same as that in FIG. 46. An avatar object 6A is placed at a first position 4661. Only billing information corresponding to the admission fee (500 yen) is registered in a tag object 2733A. A user 5A moves his or her right hand so that the tip of the virtual right hand 1531RA points to a third position 4866 that is farther from the stage object 1532 than the second position 4662. The processor 210A identifies the third position 4866 in the second virtual space 2711A pointed to by the tip of the virtual right hand 1531RA.

[0274] For example, the processor 210A displays a field of view image 4817A corresponding to the second virtual space 2711A shown in FIG. 48(A) on the monitor 130A as shown in FIG. 48(B). The processor 210A notifies the user 5A of a mark 4663 indicating a third position 4866 in the second virtual space 2711A on the field of view image 4817A. In detail, the processor 210A displays the mark 4663 on the monitor 130A by superimposing it on a position 4867 in the field of view image 4817A corresponding to the third position 4866 in the second virtual space 2711A. The processor 210A can place the mark 4663 as a virtual object at the third position 4866 in the second virtual space 2711A. Alternatively, the processor 210A can include the mark 4663 at the position 4867 corresponding to the third position 4866 when generating the field of view image 4817A. By viewing the field of view image 4817A, the user 5A confirms that the mark 4663 indicating the third position 4866 in the second virtual space 2711A is displayed on the monitor 130A. By confirming the mark 4663, the user 5A understands that the third position 4866 has been designated as the destination of the avatar object 6A.

[0275] FIG. 49 illustrates a second virtual space 2711A and a field of view image 4917A according to an embodiment. The processor 210A detects a first operation by the user 5A for moving the avatar object 6A to the third position 4866. When the first operation by the user 5A is detected, the processor 210A discretely moves the avatar object 6A and the virtual camera 14A from the first position 4661 to the third position 4866 within the second virtual space 2711A, as shown in FIG. 49(A). The processor 210A registers, in the tag object 2733A, billing information corresponding to the movement distance 4969 of the avatar object 6A from the first position 4661 to the third position 4866. In particular, the processor 210A calculates the billing amount corresponding to the movement distance 4969 by multiplying the movement distance 4969 by the charging unit price. Since the travel distance 4969 is shorter than the travel distance 4775, the billing amount is calculated to be 250 yen, which is cheaper than 500 yen, in Fig. 49. The processor 210A newly registers billing information indicating the calculated billing amount in the tag object 2733A.

[0276] The processor 210A displays the total of the charges (750 yen) represented by the two registered billing information on the surface of the tag object 2733A. For example, the processor 210A displays a field of view image 4917A corresponding to the second virtual space 2711A shown in FIG. 49(A) on the monitor 130A as shown in FIG. 49(B). By visually checking the field of view image 4917A, the user 5A recognizes that the avatar object 6A has moved to a third position 4866, which is slightly away from the stage object 1532. The user 5A further recognizes that the total current charge is 750 yen. In other words, the user 5A understands that, as a result of moving the avatar object 6A to the third position 4866, an additional 250 yen has been charged to the user 5A as a travel fee. As shown in FIGS. 47 and 49, the processor 210A registers billing information in the tag object 2733A, which indicates that the longer the travel distance, the higher the charge amount.

[0277] [Major Advantage of This Embodiment] Processor 210A registers billing information according to the distance traveled by avatar object 6A in tag object 2733A. In this way, HMD system 100 can provide user 5A with a suitable billing method in second virtual space 2711A.

[0278] The user 5A can watch the live performance from a position of their choice by moving the avatar object 6A to a desired position in the second virtual space 2711A during the live performance. For example, when a performance that the user 5A likes is being performed, the user 5A moves the avatar object 6A to a position close to the avatar object 6B. When the desired viewing position varies depending on the content of the performance performed by the avatar object 6B, the user 5A can watch the live performance by moving the avatar object 6A to a position of their choice for each performance. For example, when the user 5A's favorite performance consists of two parts, the user 5A can watch the first part by moving the avatar object 6A to a position directly in front of the avatar object 6B, and can watch the second part by moving the avatar object 6A to the right side of the stage object 1532 and watching the avatar object 6B from a diagonal direction. When there are multiple avatar objects 6B performing in the second virtual space 2711A and the positions of the avatar objects 6B on the stage object 1532 are interchanged, the user 5A can The user 5A can also watch the live performance by moving the avatar object 6A so that it follows the position of the user's favorite avatar object 6B. As a result, the HMD system 100 can make the live performance more enjoyable according to the preferences of the user 5A.

[0279] [Movement of Avatar Object 6B] Although an example in which the avatar object 6A can move at any timing has been described above, the movement of the avatar object 6A is not limited to this example. Below, a detailed description will be given of an example in which the avatar object 6A can move only at a timing specified by the avatar object 6B.

[0280] FIG. 50 illustrates second virtual space 2711A and field of view image 5017A according to an embodiment. In FIG. 50, a performance by avatar object 6B is being performed in second virtual space 2711A. Only billing information corresponding to the admission fee (500 yen) is registered in tag object 2733A. Avatar object 6B is located at first position 5070 near the right edge of stage object 1532. Avatar object 6A is located at second position 5071 in second virtual space 2711A, near stage object 1532 and opposite first position 5070. In this manner, avatar object 6A faces avatar object 6B and watches the performance by avatar object 6B from a position close to avatar object 6B.

[0281] Processor 210A displays, for example, field of view image 5017A corresponding to second virtual space 2711A shown in Fig. 50(A) on monitor 130A as shown in Fig. 50(B). By viewing field of view image 5017A, user 5A enjoys the performance of avatar object 6B that is near avatar object 6A.

[0282] Figure 51 is a diagram showing second virtual space 2711B and field of view image 5117B according to one embodiment. Second virtual space 2711B shown in Figure 51 is synchronized with second virtual space 2711A shown in Figure 50. Processor 210B displays field of view image 5117B corresponding to second virtual space 2711B shown in Figure 51(A) on monitor 130B, as shown in Figure 51(B). By viewing field of view image 5117B, user 5B recognizes that avatar object 6A is watching the performance near avatar object 6B.

[0283] FIG. 52 illustrates a second virtual space 2711A and a field of view image 5217A according to an embodiment. If a condition enabling movement of the avatar object 6A within the second virtual space 2711A is not met, the processor 210A prohibits the avatar object 6A from moving within the second virtual space 2711A. In FIG. 52, the processor 210A moves the virtual right hand 1531RA in response to the movement of the right hand of the user 5A so that the tip of the virtual right hand 1531RA points to a desired position in the second virtual space 2711A. However, since movement of the avatar object 6A is prohibited, the processor 210A does not identify the position to which the tip of the virtual right hand 1531RA points.

[0284] For example, processor 210A displays field of view image 5217A corresponding to second virtual space 2711A shown in FIG. 52(A) on monitor 130A as shown in FIG. 52(B). Since processor 210A does not identify the position pointed to by the tip of virtual right hand 1531RA, it does not notify user 5A of mark 4663 indicating the position on field of view image 5217A. By viewing field of view image 5217A, user 5A confirms that, even though the tip of virtual right hand 1531RA points to a desired position in second virtual space 2711A, mark 4663 indicating the position is not displayed on monitor 130A. User 5A further understands that, because mark 4663 is not displayed, avatar object 6A cannot be moved.

[0285] 53 is a diagram illustrating a second virtual space 2711B and a field of view image 5317B according to an embodiment. User 5B wishes to change the position at which avatar object 6B performs on stage object 1532 to a third position 5372 near the left edge of stage object 1532. To fulfill this wish, user 5B moves his / her left hand so that the tip of virtual left hand 1531LB points to third position 5372. Processor 210B moves virtual left hand 1531LB in second virtual space 2711B in accordance with the movement of user 5B's left hand. Processor 210B identifies third position 5372 in second virtual space 2711B to which the tip of virtual left hand 1531LB points.

[0286] For example, the processor 210B displays a field of view image 5317B corresponding to the second virtual space 2711B shown in FIG. 53(A) on the monitor 130B as shown in FIG. 53(B). The processor 210B notifies the user 5B of a mark 4663 indicating a third position 5372 in the second virtual space 2711B on the field of view image 5317B. In detail, the processor 210B displays the mark 4663 on the monitor 130B by superimposing it on a position 5373 in the field of view image 5317B corresponding to the third position 5372 in the second virtual space 2711B. The processor 210B can place the mark 4663 as a virtual object at the third position 5372 in the second virtual space 2711B. Alternatively, the processor 210B can include the mark 4663 at the position 5373 corresponding to the third position 5372 when generating the field of view image 5317B. By viewing field of view image 5317B, user 5B confirms that mark 4663 indicating third position 5372 is displayed on monitor 130B. By confirming mark 4663, user 5B understands that third position 5372 has been designated as the destination of avatar object 6B.

[0287] After checking the mark 4663, the user 5B performs a second operation to move the avatar object 6B to the third position 5372. The second operation is, for example, the user 5B pressing any button on the right controller 300RA. The processor 210B detects the second operation of the user 5B to move the avatar object 6B.

[0288] FIG. 54 illustrates a second virtual space 2711B and a field of view image 5417B according to an embodiment. When a second operation by user 5B is detected, processor 210B discretely moves avatar object 6B and virtual camera 14B within second virtual space 2711B from first position 5070 to third position 5372, as shown in FIG. 54(A). For example, processor 210B displays field of view image 5417B corresponding to second virtual space 2711B shown in FIG. 54(A) on monitor 130B, as shown in FIG. 54(B). By viewing field of view image 5417B, user 5B recognizes that avatar object 6B has moved to third position 5372. User 5B further recognizes that, as a result of avatar object 6B moving, avatar object 6A has disappeared from the field of view of avatar object 6B.

[0289] FIG. 55 illustrates a second virtual space 2711A and a field of view image 5517A according to an embodiment. After avatar object 6B moves in second virtual space 2711B, processor 210A receives the latest avatar information for avatar object 6B from server 600. Based on the movement information included in the received avatar information, processor 210A discretely moves avatar object 6B from first position 5070 to third position 5372 in second virtual space 2711A. As a result, avatar object 6B is positioned outside field of view 15A. For example, processor 210A displays field of view image 5517A corresponding to second virtual space 2711A shown in FIG. 55(A) on monitor 130A as shown in FIG. 55(B). By viewing field of view image 5517A, user 5A recognizes that avatar object 6B has moved on stage object 1532 and therefore has disappeared from the field of view of avatar object 6A.

[0290] When avatar object 6B moves within second virtual space 2711A, processor 210A enables avatar object 6A to move within second virtual space 2711A. However, the conditions for enabling avatar object 6A to move within second virtual space 2711A are not limited to these. Processor 210A notifies user 5A, for example, by audio, that avatar object 6A can now be moved. Processor 210A can also notify user 5A that avatar object 6A can now be moved by changing the lighting within second virtual space 2711A.

[0291] 56 is a diagram illustrating a second virtual space 2711A and a field of view image 5617A according to an embodiment. In FIG. 56, the processor 210A allows the avatar object 6A to move within the second virtual space 2711A in response to the movement of the avatar object 6B. By turning to the right, the user 5A recognizes that the avatar object 6B has moved near the right edge of the stage object 1532. The user 5A wishes to continue watching the live broadcast near the avatar object 6B by moving the avatar object 6A to a fourth position 5674 in the second virtual space 2711A, which is closer to the moved avatar object 6B.

[0292] To fulfill this wish, user 5A moves his / her right hand so that the tip of virtual right hand 1531RA points to fourth position 5674. Processor 210A moves virtual right hand 1531RA in accordance with the movement of user 5A's right hand. Since movement of avatar object 6A is permitted, processor 210A identifies fourth position 5674, to which the tip of virtual right hand 1531RA points after the movement.

[0293] For example, processor 210A displays field of view image 5617A corresponding to second virtual space 2711A shown in FIG. 56(A) on monitor 130A as shown in FIG. 56(B). Processor 210A displays mark 4663 indicating fourth position 5674 in second virtual space 2711A on monitor 130A by superimposing it on position 5675 in field of view image 4817A corresponding to fourth position 5674 in second virtual space 2711A. Processor 210A can place mark 4663 as a virtual object at fourth position 5674 in second virtual space 2711A. Alternatively, processor 210A can include mark 4663 at position 5675 corresponding to fourth position 5674 when generating field of view image 5617A. By viewing field of view image 5617A, user 5A recognizes that avatar object 6B has moved close to the right edge of stage object 1532. User 5A further recognizes by checking mark 4663 included in field of view image 5617A that fourth position 5674 has been designated as the destination of avatar object 6A.

[0294] 57A and 57B illustrate a second virtual space 2711A and a field of view image 5717A according to an embodiment. Processor 210A detects a first operation by user 5A for moving avatar object 6A to fourth position 5674. When processor 210A detects the first operation by user 5A, processor 210A discretely moves avatar object 6A and virtual camera 14A from second position 5071 to fourth position 5674 within second virtual space 2711A, as shown in FIG. 57A. Processor 210A registers, in tag object 2733A, billing information corresponding to distance 5776 traveled by avatar object 6A from second position 5071 to fourth position 5674.

[0295] The processor 210A, for example, A field of view image 5717A corresponding to space 2711A is displayed on monitor 130A as shown in FIG. 57(B). By viewing field of view image 5717A, user 5A recognizes that avatar object 6A has moved to fourth position 5674 near avatar object 6B. User 5A further recognizes that avatar object 6B is right in front of avatar object 6A. This allows user 5A to continue enjoying the live performance near avatar object 6B.

[0296] In the modes shown in FIGS. 50 to 57, as a result of avatar object 6B moving within second virtual space 2711A, user 5A is strongly motivated to move avatar object 6A closer to avatar object 6B. In this way, by moving avatar object 6A, user 5B can strongly encourage user 5A to move avatar object 6A. Therefore, it is expected that user 5A will actively move avatar object 6A within second virtual space 2711A, which further increases the possibility of charging user 5A an amount corresponding to the distance moved. As a result, it is possible to increase the possibility of increasing sales of avatar object 6B's live performance.

[0297] [Variation] Processor 210A does not necessarily have to register charging information according to the distance traveled in tag object 2733A. For example, processor 210A can associate charging information according to the distance traveled with avatar object 6A and register it in avatar object 6A. Instead of registering charging information according to the distance traveled in tag object 2733A, processor 210A can execute any process related to charging according to the distance traveled. The process related to charging is, for example, transmitting charging information to server 600 without registering it in tag object 2733A.

[0298] The processor 210A can also execute processing related to charging according to the number of times the avatar object 6A has moved. The processor 210A, for example, registers charging information according to the number of times the avatar object 6A has moved in the tag object 2733A. The processor 210A, for example, calculates a charging amount that increases as the number of times the avatar object 6A has moved, and registers charging information representing the calculated charging amount in the tag object 2733A. In this case, the processor 210A can register optimal charging information according to the number of times the avatar object 6A has moved in the tag object 2733A. The processor 210A can also prevent the avatar object 6A from moving unlimitedly within the second virtual space 2711A and thereby disrupting the live performance of the avatar object 6B.

[0299] The processor 210A can execute a charging process according to at least one of the distance traveled and the number of times traveled. The processor 210A can execute any of a first process related to charging according to only the distance traveled, a second process related to charging according to only the number of times traveled, and a third process related to charging according to both the distance traveled and the number of times traveled.

[0300] The processor 210A can also execute processing related to charging according to the distance and direction of movement of the avatar object 6A. For example, the processor 210A increases the unit price of charging when the avatar object 6A is moving closer to the avatar object 6B. For example, the processor 210A decreases the unit price of charging when the avatar object 6A is moving farther away from the avatar object 6B.

[0301] The processor 210A can also continuously move the avatar object 6A within the second virtual space 2711A. For example, in response to a first operation by the user 5A, the processor 210A moves the avatar object 6A from the first position 4661 to the second position 4662 so as to pass through all of the positions between the first position 4661 and the second position 4662. In this case, the processor 210A also registers, in the tag object 2733A, billing information indicating a billing amount according to the distance traveled 4775.

[0302] Processor 210A can also automatically move avatar object 6A within second virtual space 2711A without detecting the first operation of user 5B. Even when processor 210A automatically moves avatar object 6A, processor 210A registers, in tag object 2733A, billing information indicating a billing amount according to the distance moved.

[0303] In this embodiment, it is assumed, but not limited to, that the processor 210A can move the avatar object 6A only to a position in the second virtual space 2711A where no avatar object 6 other than the avatar object 6A is located. In other words, it is assumed, but not limited to, that the user 5A cannot specify a position where an avatar object 6 other than the avatar object 6A is located as the destination position of the avatar object 6A.

[0304] It is assumed that processor 210A can only move avatar object 6A to positions within a specified range in second virtual space 2711A. For example, processor 210A generally prohibits moving avatar object 6A onto stage object 1532. If permission is granted by user 5B, processor 210A can also move avatar object 6A onto stage object 1532. However, this is not a limitation.

[0305] The user 5B can perform a second operation to permit movement of the avatar object 6A. When the processor 210B detects the second operation of the user 5B, it transmits detection information indicating the detection of the second operation to the computer 200A via the server 600. The processor 210A receives the detection information transmitted from the computer 200B. The processor 210A detects the second operation by the user 5B based on the received detection information. When the processor 210A detects the second operation, it enables the avatar object 6A to be moved within the second virtual space 2711A. For example, the processor 210A enables the movement for a certain period of time after detecting the second operation. The second operation does not necessarily have to be the operation of the user 5B itself, but may be a movement of the user 5B, such as what the user 5B says, or may be information specifying the period during which the movement is permitted. This allows the processor 210A to prevent the avatar object 6A from being moved without the permission of the user 5B during a live broadcast. In other words, the user 5B can specify the timing at which the avatar object 6A can move at his / her own will, which allows the user 5B to control the live performance in a way that suits the user 5B's wishes.

[0306] The processor 210A can detect the occurrence of a first event in the second virtual space 2711A. The first event is, for example, the avatar object 6B taking some kind of action. The action of the avatar object 6B is, for example, the avatar object 6B moving to another position on the stage object 1532, as shown in FIG. 55. The first event may be the avatar object 6B performing a specified performance on the stage object 1532 for a certain period of time or more. When the processor 210A detects the first event, it enables the avatar object 6A to move within the second virtual space 2711A. This makes it possible to prevent the avatar object 6A from moving randomly during a live performance.

[0307] In the HMD system 100, the charging unit price for each fixed distance may vary depending on the size of the second virtual space 2711A. For example, the larger the size of the second virtual space 2711A, the lower the charging unit price. When the defined size of the second virtual space 2711A is a first size, the processor 210A sets a first charging unit price (e.g., 50 yen) in the computer 200A as the charging unit price. When the defined size of the second virtual space 2711A is a second size larger than the first size, the processor 210A sets a second charging unit price (e.g., 20 yen) that is lower than the first charging unit price in the computer 200A as the charging unit price. The processor 210A can calculate an optimal charging amount according to both the size and the movement distance of the second virtual space 2711A by multiplying the charging unit price set in the computer 200A by the movement distance of the avatar object 6A.

[0308] Processor 210A can execute processing related to charging according to at least either the distance traveled or the number of times traveled, even when avatar object 6A moves within second virtual space 4311A in which live video 4239 shown in Fig. 43 is played. Processor 210A can also execute processing related to charging according to at least either the distance traveled or the number of times traveled, even when avatar object 6A moves within a virtual space in which no performance is performed by avatar object 6B and live video 4239 is not played.

[0309] [Embodiment 3] In this embodiment, the processor 210A executes processing related to charging and settlement for the user 5A by a method using the tag object 2733A described in embodiment 1. However, without being limited to this, the processor 210A can also execute processing related to charging and settlement for the user 5A by any method different from the method according to embodiment 1.

[0310] In this embodiment, the processor 210C executes the process related to charging and settlement for the user 5C in the same manner as the method using the tag object 2733A described in embodiment 1. However, without being limited thereto, the processor 210C can also execute the process related to charging and settlement for the user 5C in any manner different from the method according to embodiment 1.

[0311] [Viewpoint Sharing Process Flow] FIG. 58 is a sequence chart illustrating a portion of a process executed in HMD set 110A according to an embodiment. FIG. 59 illustrates second virtual space 2711C and field of view image 5917C according to an embodiment. In FIG. 59, processor 210C causes avatar object 6B (third avatar) associated with user 5B (third user) to perform a performance (first performance) in accordance with the movement of user 5B in second virtual space 2711C provided to user 5C (second user). Processor 210C causes avatar object 6C (second avatar) associated with user 5C to watch the performance near avatar object 6B. Processor 210C causes avatar object 6A (first avatar) associated with user 5A (first user) to watch the performance from a position distant from avatar object 6B.

[0312] In second virtual space 2711A synchronized with second virtual space 2711C, processor 210A causes avatar object 6B to perform in accordance with the movements of user 5B. Processor 210A causes avatar objects 6A and 6C to watch the performance by avatar object 6B in second virtual space 2711A.

[0313] The processor 210C notifies the user 5C during a live broadcast of a message guiding the user 5C to move the avatar object 6C to the stage object 1532. For example, as shown in FIG. 59(A), the processor 210C displays the message on the tag object 2733C. The message includes text guiding the user 5C to move to the stage object 1532 and the amount charged to the user 5C when the avatar object 6C is moved. In the example of FIG. 59, the avatar object 6C is moved onto the stage object 1532. The charge for this is 1,000 yen.

[0314] After the message is displayed on the tag object 2733C, the processor 210C moves the virtual left hand 1531LC into the field of view 15C in accordance with the movement of the left hand of the user 5C, as shown in FIG. 59(A). The processor 210C displays, for example, a field of view image 5917C corresponding to the second virtual space 2711C shown in FIG. 59(A) on the monitor 130C, as shown in FIG. 59(B). The user 5C confirms the message displayed on the tag object 2733C by viewing the field of view image 5917C. This allows the user 5C to recognize that if he or she pays an additional charge of 1,000 yen, he or she can move the avatar object 6C onto the stage object 1532.

[0315] FIG. 60 illustrates a second virtual space 2711C and a field of view image 6017C according to an embodiment. By viewing the field of view image 6017C, the user 5C checks the message displayed in the tag object 2733C. After checking the message, the user 5C wishes to move the avatar object 6C onto the stage object 1532. To fulfill this wish, the user 5C performs an operation to select the tag object 2733C. In response to detecting the operation of the user 5C, the processor 210C registers billing information corresponding to the stage movement in the tag object 2733C. In detail, the processor 210C registers billing information indicating the amount charged for moving the avatar object 6C onto the stage object 1532 in the tag object 2733C. After registering the billing information, the processor 210C moves the avatar object 6C onto the stage object 1532, as shown in FIG. 60(A). Processor 210C places avatar object 6C after the movement at a position in front of avatar object 6B on stage object 1532. Processor 210C transmits avatar information of avatar object 6C to server 600 after moving avatar object 6C.

[0316] For example, processor 210C displays a field of view image 6017C corresponding to second virtual space 2711C shown in FIG. 60(A) on monitor 130C as shown in FIG. 60(B). By viewing field of view image 6017C, user 5C recognizes that avatar object 6C has moved onto stage object 1532. User 5C can further enjoy a performance in front of avatar object 6B and can enjoy interaction with avatar object 6B (in other words, with user 5B).

[0317] FIG. 61 illustrates a second virtual space 2711A and a field of view image 6117A according to an embodiment. In step S5801, after avatar object 6C moves onto stage object 1532 in second virtual space 2711C, processor 210A receives avatar information of avatar object 6C from server 600. In step S5802, processor 210A moves avatar object 6C onto stage object 1532 in second virtual space 2711A based on the received avatar information, as shown in FIG. 61(A). In other words, processor 210A synchronizes second virtual space 2711A with second virtual space 2711C shown in FIG. 60. Processor 210A places avatar object 6C in front of avatar object 6B on stage object 1532.

[0318] After moving avatar object 6C onto stage object 1532, processor 210A receives avatar information for avatar object 6B from server 600. Processor 210A causes avatar object 6B to perform a performance (second performance) for avatar object 6C in accordance with the movement information included in the received avatar information. In this way, in second virtual space 2711A, avatar object 6C is watching the performance by avatar object 6B for avatar object 6C in accordance with the billing information corresponding to the stage movement being registered in tag object 2733C.

[0319] After the avatar object 6C moves onto the stage object 1532, the processor 210A notifies the user 5A during live broadcast of a message informing the user 5A that the avatar object 6C will share the viewpoint of another avatar object 6 different from the avatar object 6A. The processor 210A displays the message on, for example, the tag object 2733A. The message includes text informing the user 5A about the viewpoint sharing and the amount charged to the user 5A for the viewpoint sharing. In the example of FIG. 61, the amount charged to the user 5A for the viewpoint sharing is 200 yen. After the message is displayed on the tag object 2733A, the processor 210A moves the virtual left hand 1531LA into the field of view 15A in accordance with the movement of the left hand of the user 5A, as shown in FIG. 61(A).

[0320] For example, the processor 210A displays a field of view image 6117A corresponding to the second virtual space 2711A shown in FIG. 61(A) on the monitor 130A as shown in FIG. 61(B). By viewing the field of view image 6117A, the user 5A recognizes that the avatar object 6C has moved onto the stage object 1532. The user 5A further recognizes that the avatar object 6C is viewing a performance near the avatar object 6B on the stage object 1532. The user 5A further recognizes, by checking the message displayed on the tag object 2733C, that by paying an additional charge of 200 yen, the viewpoint of any avatar object 6 can be shared with the avatar object 6A.

[0321] [Viewpoint Sharing for a Fee] FIG. 62 illustrates a second virtual space 2711A and a field of view image 3417A according to an embodiment. After checking the message displayed in the tag object 2733A, the user 5A wishes to have the avatar object 6A share the viewpoint of the avatar object 6C. To fulfill this wish, the user 5A performs an operation to select the tag object 2733A. In step S5803, when the processor 210A detects an operation by the user 5A, the processor 210A registers billing information corresponding to viewpoint sharing in the tag object 2733A. Specifically, the processor 210A newly registers billing information indicating the billing amount (200 yen) required for viewpoint sharing in the tag object 2733A. In response to the registration of the billing information, the processor 210A enables the avatar object 6A to share the viewpoint of another avatar object 6 in the second virtual space 2711A. The processor 210A notifies the user 5A, for example, by audio, that viewpoint sharing is now possible. Processor 210A can also notify user 5A that viewpoint sharing is now possible by changing the lighting in second virtual space 2711A.

[0322] In response to the movement of the right hand after tag object 2733A is selected, processor 210A moves virtual right hand 1531RA so that the tip of virtual right hand 1531RA points to avatar object 6C, as shown in Fig. 62(A). In step S5804, processor 210A identifies avatar object 6C pointed to by the tip of virtual right hand 1531RA.

[0323] For example, the processor 210A displays a field of view image 6217A corresponding to the second virtual space 2711A shown in FIG. 62(A) on the monitor 130A as shown in FIG. 62(B). In step 5805, the processor 210A displays a mark 6277 indicating the avatar object 6C on the monitor 130A by superimposing the mark 6277 on the avatar object 6C in the field of view image 6217A. The processor 210A can also place the mark 6277 in the second virtual space 2711A as a virtual object that is in contact with or close to the avatar object 6C. When generating the field of view image 6217A, the processor 210A can also include the mark 6277 at the position of the avatar object 6C in the field of view image 4617A. The user 5A recognizes that the avatar object 6C has been designated by viewing the mark 6277 in the field of view image 6217A. After checking the mark 6277, the user 5A performs a first operation to make the avatar object 6A share the viewpoint of the avatar object 6C. The first operation is, for example, the user 5A pressing any button on the right controller 300RA.

[0324] FIG. 63 illustrates a second virtual space 2711A and a field of view image 6317A according to an embodiment. In step S5806, in response to detecting a first operation, processor 210A selects avatar object 6C. In step S5807, in response to the selection of avatar object 6C, processor 210A causes avatar object 6A to share the viewpoint of avatar object 6C, which is watching the performance for avatar object 6C. Specifically, processor 210A moves virtual camera 14A from the position of avatar object 6A to the position of avatar object 6C. At this time, processor 210A does not move avatar object 6A. Therefore, after virtual camera 14A moves, the positions at which avatar object 6A and virtual camera 14A are placed in second virtual space 2711A are different from each other. Processor 210A controls the orientation of virtual camera 14A after the movement in accordance with the orientation of the head of avatar object 6C. This allows avatar object 6A to share the viewpoint of avatar object 6C.

[0325] When charging information corresponding to viewpoint sharing is registered in the tag object 2733A, the processor 210A can also display a UI for selecting an avatar object 6C on the tag object 2733A. In this case, the user 5A selects the avatar object 6C through input to the UI displayed in the tag object 2733A. The processor 210A causes the avatar object 6A to share the viewpoint of the avatar object 6C selected through the UI. When charging information corresponding to viewpoint sharing is registered in the tag object 2733A, the processor 210A can also cause the avatar object 6A to automatically share the viewpoint of the avatar object 6C without having the user 5A select the avatar object 6C.

[0326] After the viewpoint is shared, in step S5808, processor 210A receives avatar information of avatar object 6B from server 600. In step S5809, processor 210A causes avatar object 6B to perform a performance (second performance) for avatar object 6C, based on the movement information included in the received avatar information. In this way, avatar object 6C is watching the performance by avatar object 6B for avatar object 6C in second virtual space 2711A, in response to the billing information corresponding to the stage movement being registered in tag object 2733C.

[0327] Processor 210A displays, for example, a field of view image 6317A (first field of view image) corresponding to second virtual space 2711A shown in FIG. 63(A) on monitor 130A as shown in FIG. 63(B). By viewing field of view image 6317A, user 5A recognizes that the viewpoint of avatar object 6C is shared with avatar object 6A. In other words, user 5A, like user 5C, recognizes that the viewpoint of avatar object 6C is shared with avatar object 6A in front of avatar object 6B. You can watch the performance against C.

[0328] In FIG. 63(A), the viewpoint of avatar object 6C is shared with avatar object 6A, but avatar object 6A itself is located far away from avatar object 6B. Therefore, user 5A cannot make avatar object 6A directly interact with avatar object 6B. For example, user 5A cannot hold out or wave virtual right hand 1531RA toward avatar object 6B. Processor 210A may prevent audio produced by user 5B from being output to speaker 180A. This may prevent user 5A from hearing audio spoken by user 5B toward avatar object 6C. In this way, user 5A can only visually recognize user 5C interacting with avatar object 6B (user 5B) from the viewpoint of avatar object 6C. Since the fee for viewpoint sharing (200 yen) is cheaper than the fee for moving onto the stage object 1532 (1000 yen), the user 5A understands that after viewpoint sharing, the avatar object 6A cannot interact with the avatar object 6B. Furthermore, the user 5A is motivated to try viewpoint sharing.

[0329] Figure 64 shows a second virtual space 2711B and a field of view image 6417B according to an embodiment. In Figure 64(A), an avatar object 6B and an avatar object 6C are placed on a stage object 1532. Three different avatar objects 6, including an avatar object 6A, share the viewpoint of avatar object 6C. The processor 210B places a monitor object 6478 in the second virtual space 2711B in association with the avatar object 6C to notify the user 5B of the number of avatar objects 6 that share the viewpoint of the avatar object 6C. The monitor object 6478 is placed above the avatar object 6C. The processor 210B displays the number of avatar objects 6 that share the viewpoint of the avatar object 6C on the monitor object 6478.

[0330] For example, the processor 210B displays a field of view image 6417B corresponding to the second virtual space 2711B shown in FIG. 64(A) on the monitor 130B as shown in FIG. 64(B). By viewing the field of view image 6417B, the user 5B recognizes that the avatar object 6C is watching the performance right in front of the avatar object 6B. The user 5B further recognizes that the viewpoint of the avatar object 6C is shared by three other avatar objects 6 by checking the number displayed on the monitor object 6478. In other words, the user 5B can grasp how many other users 5 are receiving the performance directed at the avatar object 6C.

[0331] [Moving Avatar Object 6A for a Fee] Figure 65 shows second virtual space 2711A and field of view image 6517A according to one embodiment. As shown in Figure 65(A), after processor 210A moves avatar object 6C onto stage object 1532, if a specified condition (e.g., the passage of a certain amount of time) is met, processor 210A returns avatar object 6C to its original position in second virtual space 2711A. In step S5810, processor 210A ends viewpoint sharing when avatar object 6C is returned to its original position. Specifically, processor 210A moves virtual camera 14A to the position of avatar object 6A.

[0332] After the viewpoint sharing ends, the processor 210A notifies the user 5A during the live broadcast of a message guiding the user 5A to move the avatar object 6A onto the stage object 1532. The processor 210A displays the message on, for example, the tag object 2733A. The message includes text guiding the user 5A to move to the stage object 1532 and the amount charged to the user 5A when the avatar object 6A is moved onto the stage object 1532. In the example of FIG. 65, the amount charged to move the avatar object 6A onto the stage object 1532 is 1,000 yen.

[0333] After the message is displayed, processor 210A moves virtual left hand 1531LA into field of view 15A in accordance with the movement of user 5A's left hand, as shown in FIG. 65(A). Processor 210A displays, for example, field of view image 6517A corresponding to second virtual space 2711A shown in FIG. 65(A) on monitor 130A, as shown in FIG. 65(B). By viewing field of view image 6517A, user 5A recognizes that the sharing of viewpoints has ended. User 5A further recognizes, by checking the message displayed on tag object 2733A, that avatar object 6A can be moved onto stage object 1532 by paying an additional charge of 1,000 yen.

[0334] FIG. 66 illustrates a second virtual space 2711A and a field of view image 6617A according to an embodiment. After enjoying the experience of having avatar object 6A share the viewpoint of avatar object 6C, user 5A wishes to directly interact with avatar object 6B by moving avatar object 6A onto stage object 1532. To fulfill this desire, user 5A checks the message displayed on tag object 2733A and then performs an operation to select tag object 2733A. In step S5811, when processor 210A detects an operation by user 5A to select tag object 2733A, processor 210A registers billing information corresponding to the stage movement in tag object 2733A. In detail, after the viewpoint sharing ends, processor 210A newly registers billing information indicating the billing amount (1,000 yen) corresponding to the stage movement in tag object 2733A.

[0335] In step S5812, if billing information corresponding to the stage movement is registered in tag object 2733A, processor 210A moves avatar object 6A onto stage object 1532 as shown in FIG. 66(A). Processor 210A places avatar object 6A after the movement in a position in front of avatar object 6B on stage object 1532. In step S5813, processor 210A receives avatar information for avatar object 6B from server 600. In step S5814, avatar object control module 1424 causes avatar object 6B to perform a performance (second performance) for avatar object 6A in accordance with the movement information included in the received avatar information.

[0336] For example, the processor 210A displays a field of view image 6617A corresponding to the second virtual space 2711A shown in Fig. 66(A) on the monitor 130A as shown in Fig. 66(B). By viewing the field of view image 6617A, the user 5A recognizes that the avatar object 6A has moved onto the stage object 1532. The user 5A can further enjoy a performance in front of the avatar object 6B and can enjoy interaction with the avatar object 6B (in other words, with the user 5B).

[0337] In FIG. 66, unlike the case of viewpoint sharing, avatar object 6A is located directly in front of avatar object 6B in second virtual space 2711A. Therefore, user 5A can hold out or wave virtual right hand 1531RA toward avatar object 6B. Processor 210A outputs the sound emitted by user 5B to speaker 180A. Therefore, user 5A can hear the sound of user 5B speaking to avatar object 6A. In this way, unlike the case of viewpoint sharing, user 5A can have avatar object 6A directly interact with avatar object 6B, which can be more satisfying than the case of viewpoint sharing.

[0338] [Major Advantage of the Present Embodiment] The processor 210A can cause the avatar object 6A to share the viewpoint of another avatar object 6 different from the avatar object 6A in the second virtual space 2711A. This allows the user 5A to view the performance of the avatar object 6B not only from the viewpoint of the avatar object 6A but also from the viewpoint of another avatar object 6 located at a different position from the avatar object 6A. For example, even if the user 5A is unable to move the avatar object 6A to a desired position because the avatar object 6C is already there, the user 5A can view the performance of the avatar object 6B from the viewpoint of the avatar object 6C through viewpoint sharing. In this way, the HMD system 100 can further increase the interest of the user 5A in the second virtual space 2711A.

[0339] In this embodiment, avatar object 6C watches the performance by avatar object 6B for avatar object 6C as a result of the charge for stage movement to user 5C. When the charge for viewpoint sharing is charged to user 5A, avatar object 6A watches the performance for avatar object 6C performed as a result of the charge from the viewpoint of avatar object 6C. This allows user 5A to experience the performance for avatar object 6C in advance, albeit in an incomplete form. If user 5B is satisfied with the performance of avatar object 6B during viewpoint sharing, he or she requests that avatar object 6B perform the performance for avatar object 6A. This allows user 5A to accept the charge for stage movement to be charged to user 5A.

[0340] In this way, the HMD system 100 charges user 5C the amount corresponding to the stage movement, then charges user 5A the amount corresponding to the viewpoint sharing, and then, using this charging as a trigger, can also charge user 5A the amount corresponding to the stage movement. In this way, the HMD system 100 can realize multiple charging cycles for user 5A, and can therefore charge a larger amount to user 5A. As a result, user 5B can further increase sales of the live performance.

[0341] [Variation] It is not necessarily necessary to charge the user 5A for viewpoint sharing. The processor 210A can permit the user 5A to share a viewpoint without registering billing information corresponding to viewpoint sharing in the tag object 2733A. In other words, the processor 210A can cause the avatar object 6A to share the viewpoint of another avatar object 6 in response to an operation by the user 5A at any time during a live broadcast. The processor 210A can also cause the avatar object 6A to automatically share the viewpoint of another avatar object 6 without detecting an operation by the user 5A.

[0342] Processor 210A can cause avatar object 6A to share the viewpoint of another avatar object 6 that is not placed on stage object 1532. For example, when avatar object 6D is selected by user 5A, processor 210A causes avatar object 6A to share the viewpoint of avatar object 6D.

[0343] The processor 210B can notify the user 5B of the number of other avatar objects 6 that share the viewpoint with each of the multiple different avatar objects 6. This allows user 5B to understand which viewpoint of avatar object 6 in second virtual space 2711B is more popular with users 5.

[0344] The processor 210C does not necessarily need to register the charging information corresponding to the stage movement in the tag object 2733C. For example, the processor 210C can associate the charging information corresponding to the stage movement with the avatar object 6C and register it in the avatar object 6C. Instead of registering the charging information corresponding to the stage movement in the tag object 2733C, the processor 210C can execute any other process related to the charging corresponding to the stage movement (second charging). For example, the other process is to transmit the charging information corresponding to the stage movement to the server 600 without registering it in the tag object 2733C.

[0345] The processor 210A does not necessarily need to register the charging information corresponding to viewpoint sharing in the tag object 2733A. For example, the processor 210A can associate the charging information corresponding to viewpoint sharing with the avatar object 6A and register it in the avatar object 6A. Instead of registering the charging information corresponding to viewpoint sharing in the tag object 2733A, the processor 210A can execute any other process related to charging corresponding to viewpoint sharing (first charging). For example, the other process is transmitting the charging information corresponding to viewpoint sharing to the server 600 without registering it in the tag object 2733A.

[0346] The processor 210A does not necessarily need to register the charging information corresponding to the stage movement in the tag object 2733A. For example, the processor 210A can associate the charging information corresponding to the stage movement with the avatar object 6A and register it in the avatar object 6A. Instead of registering the charging information corresponding to the stage movement in the tag object 2733A, the processor 210A can execute any other process related to the charging corresponding to the stage movement (third charging). For example, the other process is to transmit the charging information to the server 600 without registering it in the tag object 2733A.

[0347] The user 5B may perform a first operation to permit viewpoint sharing. When the processor 210B detects the first operation of the user 5B, it transmits detection information indicating the detection of the first operation to the computer 200A via the server 600. The processor 210A receives the detection information transmitted from the computer 200B. The processor 210A detects the first operation by the user 5B based on the received detection information. When the first operation is detected, the processor 210A enables the avatar object 6A to share the viewpoint of any avatar object 6 other than the avatar object 6A. For example, after detecting the first operation, the processor 210A enables the avatar object 6A to share the viewpoint of any avatar object 6 other than the avatar object 6A for a certain period of time. The first operation does not necessarily have to be the operation of the user 5B, but may be a behavior of the user 5B, such as the content of an utterance by the user 5B, or may be information specifying a period of time during which viewpoint sharing is permitted. This prevents the viewpoint of another avatar object 6 from being shared with the avatar object 6A without the permission of the user 5B. In other words, the user 5B can specify, at his / her own will, the timing when the avatar object 6A can share the viewpoint of another avatar object 6. This allows the user 5B to control the production of the live performance in a way that is more suited to the user 5B's wishes.

[0348] The processor 210A can cause the avatar object 6A to share the viewpoint of the avatar object 6C without moving the virtual camera 14A to the position of the avatar object 6C. For example, when the processor 210A detects an operation by the user 5A to cause the avatar object 6A to share the viewpoint of the avatar object 6C, the processor 210A transmits request information for requesting viewpoint sharing to the computer 200C via the server 600. When the processor 210C receives the request information, the processor 210C transmits a field of view image 6017C (second field of view image) corresponding to the field of view area 15C from the avatar object 6C, which is generated by the virtual camera 14C, to the computer 200A via the server 600. When the processor 210A receives the field of view image 6017C, the processor 210A temporarily stops controlling the virtual camera 14A. The processor 210A further outputs the received field of view image 6017C to the monitor 130A. By viewing field of view image 6017C, user 5A can view the performance of avatar object 6B from the perspective of avatar object 6C.

[0349] The processor 210A can detect the occurrence of a first event in the second virtual space 2711A. The first event is, for example, the avatar object 6B taking some kind of action. The action of the avatar object 6B is, for example, the avatar object 6B moving to another position on the stage object 1532. The first event may be the avatar object 6B performing a specified performance on the stage object 1532 for a certain period of time or more. Alternatively, the first event may be the avatar object 6C moving onto the stage object 1532. When the processor 210A detects the first event, it enables the avatar object 6A to share the viewpoint of another avatar object 6. This makes it possible to prevent the viewpoint of another avatar object 6 from being shared with the avatar object 6A without permission.

[0350] After sharing the viewpoint, processor 210A can control virtual camera 14A in conjunction with the control of virtual camera 14C. For example, processor 210A moves virtual camera 14A to the position of avatar object 6C, and then receives control information representing control details for the position and orientation of virtual camera 14C from computer 200C via server 600. Processor 210A controls virtual camera 14A in conjunction with the control of virtual camera 14C in accordance with the received control information. This allows user 5A to more clearly realize that avatar object 6A is sharing the viewpoint of avatar object 6C.

[0351] [Embodiment 4] In this embodiment, the processor 210B executes the process related to charging and settlement for the user 5B using the same method as the method using the tag object 2733A described in embodiment 1. However, without being limited to this, the processor 210B can also execute the process related to charging and settlement for the user 5B using any method different from the method according to embodiment 1.

[0352] [Avatar Selection Flow] FIG. 67 is a sequence chart illustrating a portion of the processing executed by HMD set 110B according to an embodiment. FIG. 68 illustrates a third virtual space 6811B and a field of view image 6817B according to an embodiment. In step S6701, processor 210B of computer 200B (hereinafter simply referred to as “processor 210B”) defines third virtual space 6811B as shown in FIG. 68(A). This processing corresponds to the processing of step S1110 in FIG. 11. Specifically, processor 210B identifies virtual space data to define third virtual space 6811B represented by the virtual space data. Third virtual space 6811B is a virtual space in which avatar object 6B is placed before the start of the live performance in second virtual space 2711B. Third virtual space 6811B is also a virtual space that allows user 5B to select an avatar object to be used by user 5B during the live performance.

[0353] In step S6702, processor 210B generates virtual camera 14B and places it in the third virtual space 6811B. In step S6703, processor 210B generates avatar object 6B including virtual right hand 1531RB and virtual left hand 1531LB and places it in the third virtual space 6811B. Avatar object 6B is used by user 5B to select an avatar in the third virtual space 6811B to be used by user 5B during a live broadcast. In this embodiment, avatar object 6B is not used in the second virtual space 2711B. In step S6704, processor 210B generates a UI panel 6850 and a pen object 6853 and places them in the third virtual space 6811B.

[0354] In this embodiment, a plurality of different avatar objects are prepared in advance in HMD system 100. User 5B can select one avatar object to use live in second virtual space 2711B from the plurality of avatar objects.

[0355] UI panel 6850 is a type of UI object and is used by user 5B to cause processor 210B to execute processing for selecting one avatar object from multiple avatar objects. UI panel 6850 includes options 6851 and 6852 arranged on the front surface of UI panel 6850. Options 6851 and 6852 include images showing the appearance of the avatar object that will be selected when the option is selected by user 5B. Option 6851 is an item for selecting a first avatar object from the multiple avatar objects, and option 6852 is an item for selecting a second avatar object from the multiple avatars. Pen object 6853 is a type of virtual object and is held and used by virtual right hand 1531RB or virtual left hand 1531LB to select option 6851 or 6852 on UI panel 6850.

[0356] In step S6705, processor 210B determines the position and tilt of virtual camera 14B in third virtual space 6811B in accordance with the movement of HMD 120B. This process corresponds to part of the process of step S1140 in FIG. 11. In step S6706, processor 210B displays field of view image 17B on monitor 130A. This process corresponds to the processes of steps S1180 and S1190 in FIG. 11.

[0357] Processor 210B displays, for example, field of view image 6817B corresponding to third virtual space 6811B shown in Fig. 68(A) on monitor 130B as shown in Fig. 68(B). By viewing field of view image 6817B, user 5B recognizes that option 6851 or option 6852 on UI panel 6850 needs to be selected.

[0358] The processes of steps S6705 and S6706 described above (that is, updating of field of view image 17B in accordance with the movement of HMD 120B) are continuously and repeatedly executed while steps S6707 to S6709 described below are being executed.

[0359] In step S6707, the processor 210B detects the movement of the right hand of the user 5B based on the output of the right controller 300RB. In step S6708, the processor 210B moves the virtual right hand 1531RB in the third virtual space 6811B in accordance with the detected movement of the right hand of the user 5B. In a certain stage, the processor 210B moves the virtual right hand 1531RB in the third virtual space 6811B in accordance with the movement of the right hand of the user 5B so as to bring the virtual right hand 1531RB closer to the pen object 6853. After the virtual right hand 1531RB has come sufficiently close to the pen object 6853, the processor 210B selects (grabs) the pen object 6853 with the virtual right hand 1531RB based on the movement of the right hand of the user 5B.

[0360] FIG. 69 illustrates a third virtual space 6811B and a field of view image 6917B according to an embodiment. As described above, after virtual right hand 1531RB selects pen object 6853, processor 210B moves virtual right hand 1531RB and pen object 6853 in the third virtual space 6811B based on the movement of the right hand of user 5B so as to bring the tip of pen object 6853 closer to option 6851. Processor 210B detects that option 6851 has been selected by pen object 6853 based on a collision between the tip of pen object 6853 and option 6851. In step S6709, processor 210B selects avatar object 7106B corresponding to the selected option 6851 from among the multiple avatar objects. The appearance and size of avatar object 7106B differ from the appearance and size of avatar object 6B.

[0361] Processor 210B displays, for example, field of view image 6917B corresponding to third virtual space 6811B shown in Fig. 69(A) on monitor 130A as shown in Fig. 69(B). By viewing field of view image 6917B, user 5B recognizes that he or she has selected avatar object 7106B by selecting option 6851 on UI panel 6850.

[0362] After selecting avatar object 7106B, processor 210B finishes placing avatar object 6B in third virtual space 6811B. After this, processor 210B continues defining third virtual space 6811B. Furthermore, even after avatar object 6B is no longer placed, processor 210B leaves other virtual objects placed in third virtual space 6811B. Processor 210B discontinues providing third virtual space 6811B to user 5B. Processor 210B then executes a series of processes to cause avatar object 7106B to perform a performance, thereby newly providing second virtual space 2711B to user 5B.

[0363] The method of selecting an avatar object by user 5B is not limited to the above example. Processor 210B may, for example, place the first avatar object indicated by option 6851 and the second avatar object indicated by option 6852 in third virtual space 6811B. User 5B, for example, brings virtual right hand 1531RB into contact with the first avatar object or the second avatar object placed in third virtual space 6811B. Processor 210B selects the avatar object that has come into contact with virtual right hand 1531RB.

[0364] [Performance Execution Process Flow] FIG. 70 is a sequence chart illustrating a portion of the process executed in HMD set 110B according to an embodiment. FIG. 71 is a diagram illustrating second virtual space 2711B and field of view image 7117A according to an embodiment. In step S7001, processor 210B defines second virtual space 2711B as shown in FIG. 71(A). In step S7002, processor 210B generates virtual camera 14B and places it in second virtual space 2711B. In step S7003, processor 210B generates stage object 1532 and places it in second virtual space 2711B. In step S7004, processor 210B receives avatar information for avatar objects 6A, 6C, and 6D from server 600. In step S7005, processor 210B places avatar objects 6A, 6C, and 6D in second virtual space 2711B based on the received avatar information.

[0365] In step S7006, processor 210B places avatar object 7106B (first avatar) including virtual right hand 7131RB and virtual left hand 7131LB in the second virtual space 2711B in association with user 5B (first user). In detail, processor 210B places avatar object 7106B on stage object 1532. Although not shown, processor 210B generates avatar information for avatar object 6B at any timing and transmits it to server 600.

[0366] In step S7007, the processor 210B generates a tag object 2733B and associates it with the avatar object 7106B. Specifically, the processor 210B places the tag object 2733B at the virtual left hand 7131LB of the avatar object 7106B. The processor 210B registers, in the tag object 2733B, upper limit charging information that indicates the upper limit of the amount that can be charged to the user 5B during the live performance.

[0367] After avatar object 7106B is placed on stage object 1532, a live performance by avatar object 7106B begins in second virtual space 2711B. FIG. 71(A) shows second virtual space 2711B at the start of the live performance. Processor 210B does not register billing information in tag object 2733B at the start of the live performance. In step S7008, processor 210B determines the position and tilt of virtual camera 14B in second virtual space 2711B in accordance with the movement of HMD 120B. This process is essentially the same as the process in step S2305, and therefore detailed description will not be repeated. In step S7009, processor 210B displays field of view image 17B on monitor 130B. This process is essentially the same as the process in step S2306, and therefore detailed description will not be repeated.

[0368] For example, processor 210B displays field of view image 7117B corresponding to second virtual space 2711B shown in FIG. 71(A) on monitor 130B as shown in FIG. 71(B). Processor 210B displays the upper limit charge amount indicated by the upper limit charge information in tag object 2733B. Processor 210B further displays 0 yen as the total charge amount in tag object 2733A in response to the fact that charge information is not registered in tag object 2733A. By viewing field of view image 7117B, user 5B recognizes that avatar objects 6A, 6C, and 6D, who are viewers of the live performance, are in front of stage object 1532. User 5B further recognizes that user 5B has not been charged for the use of avatar object 7106B at the start of the live performance.

[0369] 72A and 72B are diagrams illustrating a second virtual space 2711A and a field of view image 7217A according to one embodiment. After a second virtual space 2711B is defined, the processor 210A receives avatar information of an avatar object 6B from the server 600 in real time. Based on the avatar information of the avatar object 6B that was initially received, the processor 210A places an avatar object 7106B in the second virtual space 2711A, as shown in FIG. 72A. Specifically, the processor 210A places the avatar object 7106B on the stage object 1532.

[0370] Processor 210A displays, for example, field of view image 7217A corresponding to second virtual space 2711A shown in Fig. 72(A) on monitor 130A as shown in Fig. 72(B). By viewing field of view image 7217A, user 5A recognizes that avatar object 7106B, a performer in the live performance, has appeared on stage object 1532. User 5 further recognizes that a live performance by avatar object 7106B is about to begin.

[0371] 73 is a diagram illustrating an example of a posture of user 5B according to an embodiment. After the start of a live performance, user 5B moves his / her body to assume the posture shown in FIG. 73, for example. In step S7010, processor 210B detects the movement of user 5B when assuming the posture shown in FIG. 73 based on the output of each motion sensor.

[0372] Figure 74 shows second virtual space 2711B and field of view image 7417B according to an embodiment. In step S7011, processor 210B causes avatar object 7106B to perform a performance in second virtual space 2711B, as shown in Figure 74(A), in accordance with the movements of user 5B. In detail, processor 210B reflects the body movements of user 5B shown in Figure 73 in avatar object 7106B. Processor 210B transmits avatar information, including movement information of avatar object 7106B when avatar object 7106B performs a performance, to server 600 in real time.

[0373] Processor 210B displays, for example, field of view image 7417B corresponding to second virtual space 2711B shown in Fig. 74(A) on monitor 130B as shown in Fig. 74(B). User 5B views field of view image 7417B to confirm the reactions of avatar objects 6A, 6C, and 6D to the performance.

[0374] FIG. 75 illustrates a second virtual space 2711A and a field of view image 7517A according to an embodiment. After avatar object 7106B performs a performance in second virtual space 2711B, processor 210A receives avatar information for avatar object 7106B from server 600 in real time. Processor 210A causes avatar object 7106B to perform a performance based on the movement information included in the received avatar information. This allows processor 210A to cause avatar object 7106B to perform a performance in accordance with the movement of user 5B shown in FIG. 73. Processor 210A causes avatar object 6A to watch the performance by avatar object 7106B in second virtual space 2711A. For example, processor 210A displays field of view image 7517A corresponding to second virtual space 2711A shown in FIG. 75(A) on monitor 130A as shown in FIG. 75(B). By viewing field of view image 7517A, user 5A recognizes that avatar object 7106B has performed the first performance on stage object 1532. This allows user 5A to enjoy the performance by avatar object 7106B from the perspective of avatar object 6A.

[0375] [Registering Charging Information] FIG. 76 illustrates second virtual space 2711B and field of view image 7617B according to an embodiment. In second virtual space 2711B shown in FIG. 76(A), the live performance of avatar object 7106B has already ended. In step S7012, processor 210B registers, in tag object 2733B, charging information corresponding to the performance of avatar object 7106B in second virtual space 2711B. After the live performance ends, processor 210B registers, in tag object 2733B, charging information indicating a charging amount corresponding to the time avatar object 7106B was used in second virtual space 2711B, for example. Processor 210B calculates the charging amount, which is the usage fee for avatar object 7106B, by multiplying the elapsed time from the start to the end of the live performance by the charging unit price per fixed time period set for avatar object 7106B. In FIG. 76, the calculated charging amount is 1,000 yen. The processor 210B registers the billing information indicating the calculated billing amount in the tag object 2733B. The processor 210B visualizes the registered billing information on the tag object 2733B.

[0376] Processor 210B displays, for example, field of view image 7617B corresponding to second virtual space 2711B shown in Fig. 76(A) on monitor 130B as shown in Fig. 76(B). By viewing field of view image 7617B, user 5B understands that 1,000 yen has been charged to user 5B as a usage fee for avatar object 7106B.

[0377] After checking the total amount charged at the end of the live performance, the user 5B moves the avatar object 7106B to the second virtual space 2 711B. This operation is, for example, the user 5B pressing any button on the right controller 300RB. In step S7013, in response to detecting the operation by the user 5B to cause the avatar object 7106B to exit the second virtual space 2711B, the processor 210B causes the avatar object 7106B to exit the second virtual space 2711B. As a result, the processor 210B ends provision of the second virtual space 2711B to the user 5B.

[0378] FIG. 77 shows third virtual space 6811B and field of view image 7717B according to an embodiment. In step S7014, processor 210B again places avatar object 6B in third virtual space 6811B as shown in FIG. 77(A). At this time, processor 210B does not place tag object 2733B in virtual left hand 1531LB. In step S7015, processor 210B executes processing related to payment in accordance with the billing information registered in tag object 2733B. This processing is basically the same as the processing in step S2618, and therefore detailed description will not be repeated.

[0379] For example, the processor 210B displays a field of view image 7717B corresponding to the third virtual space 6811B shown in FIG. 77(A) on the monitor 130B as shown in FIG. 77(B). The server 600 notifies the computer 200B that the payment has been completed. In response to the notification from the server 600 that the payment has been completed, the processor 210B notifies the user 5B that the payment has been completed via the field of view image 7717B. For example, the processor 210B displays a notification image 3735 including a message explaining that the payment has been completed, superimposed on the field of view image 7717B on the monitor 130B. The processor 210B can also place a virtual object having the same appearance as the notification image 3735 at any position within the field of view area 15B in the second virtual space 2711B. Alternatively, the processor 210B can include the notification image 3735 at any position within the field of view image 7717B when generating the field of view image 7717B. By viewing field of view image 7717B, user 5B recognizes that avatar object 6B has been placed again in third virtual space 6811B after the live performance ended. User 5B further recognizes that payment of the fee for using avatar object 7106B has been completed by viewing notification image 3735. User 5B further recognizes that, because tag object 2733B is no longer placed in virtual left hand 7131LB in third virtual space 6811B, no further fee for using avatar object 7106B will be charged.

[0380] In step S7016, processor 210B determines a reward for the creator of avatar object 7106B. Specifically, processor 210B determines the reward based on the billing information registered in tag object 2733B. Because billing information is registered based on the performance of avatar object 7106B, determining the reward based on the billing information is equivalent to determining the reward based on the performance of avatar object 7106B. In the example of FIG. 76, billing information indicating a charge based on the amount of time avatar object 7106B is used in second virtual space 2711B is registered. Therefore, processor 210B determines a reward based on the amount of time avatar object 7106B is used in second virtual space 2711B.

[0381] In step S7017, processor 210B executes processing to pay the determined remuneration to the creator of avatar object 7106B. Processor 210B, for example, notifies server 600 of the determined remuneration. After executing settlement, server 600 pays the notified remuneration to the creator of avatar object 7106B. Server 600 can pay the remuneration to the user of avatar object 7106B each time a live performance in which avatar object 7106B is used ends. Alternatively, server 600 can pay a remuneration corresponding to the total usage fee for avatar object 7106B within a certain period (e.g., one month) to the creator of avatar object 7106B in one lump sum after the certain period has elapsed. Processor 210B, rather than server 600, can also pay the remuneration corresponding to the billing information to the creator of avatar object 7106B.

[0382] [Major Advantage of This Embodiment] As described above, processor 210B registers billing information corresponding to the performance of avatar object 7106B in second virtual space 2711B in tag object 2733B. In this way, HMD system 100 can provide user 5B with a suitable billing method in second virtual space 2711B.

[0383] The processor 210B causes the avatar object selected by the user 5B to perform a performance according to the position information and joint rotation directions of the user 5B. This allows the movement of the user 5B to be accurately reflected even for an avatar object whose shape is significantly different from that of the user 5B. In this way, the HMD system 100 does not need to restrict the shapes of the avatar objects that the user 5B can select to a specific shape. Therefore, a variety of avatar objects with different shapes can be prepared in the HMD system 100. As a result, the options for avatar objects that the user 5B can select can be further increased, thereby enhancing the diversity of the performer's live performance.

[0384] When an avatar object created by the creator is used by a performer, the creator of the avatar object can receive a reward according to the performance of the avatar object. Therefore, the HMD system 100 can motivate many creators to create new avatar objects that can be used in the HMD system 100. As a result, the variety of avatar objects available in the HMD system 100 can be further increased. This increases the number of avatar objects that a performer can choose from, thereby increasing the variety of the performer's live performance.

[0385] [Variation] Processor 210B does not necessarily have to register charging information corresponding to avatar use in tag object 2733B. For example, processor 210B can associate charging information corresponding to avatar use with avatar object 7106B and register it in avatar object 7106B. Instead of registering charging information corresponding to avatar use in tag object 2733B, processor 210B can perform any other process related to charging corresponding to avatar use. For example, the other process is to transmit charging information corresponding to avatar use to server 600 without registering it in tag object 2733B.

[0386] Processor 210B can execute processing related to charging corresponding to the fee based on the sales of the live performance in second virtual space 2711B. Processor 210B can further determine a remuneration based on the sales of the live performance in second virtual space 2711B. Processor 210B, for example, calculates a portion of the sales as the fee based on the sales of the live performance. Processor 210B registers charging information indicating the fee based on the sales in tag object 2733B. As a result, the higher the sales of the live performance, the higher the fee charged to user 5B as a usage fee for avatar object 7106B. Furthermore, the higher the sales of the live performance, the higher the remuneration paid to the creator of avatar object 7106B. In this way, HMD system 100 can motivate creators to create popular avatar objects 7106B that will lead to increased sales of the live performance.

[0387] The processor 210B can execute processing related to billing corresponding to a fee based on the content of the second virtual space 2711B in which the avatar object 7106B is used. The processor 210B can also determine a remuneration based on the content of the second virtual space 2711B. For example, the larger the size of the second virtual space 2711B in which the avatar object 7106B is used, the higher the fee calculated as a usage fee for the avatar object 7106B. The processor 210B registers billing information representing the calculated fee in the tag object 2733B. The larger the size of the second virtual space 2711B in which the avatar object 7106B is used, the higher the remuneration paid to the creator of the avatar object 7106B. This allows the HMD system 100 to motivate creators to create popular avatar objects 7106B that can be used in larger second virtual spaces 2711B.

[0388] Processor 210B can generate a usage history of avatar object 7106B in second virtual space 2711B. For example, processor 210B generates a usage history of the time avatar object 7106B was used in second virtual space 2711B. For example, processor 210A generates a usage history of the sales of live performances in which avatar object 7106B was used in second virtual space 2711B. For example, processor 210B generates a history of the second virtual space 2711A in which avatar object 7106B was used. Processor 210B notifies the creator of avatar object 7106B of the generated usage history. By viewing the notified usage history of avatar object 7106B, the creator can grasp details of how avatar object 7106B was used in the second virtual space 2711B. The creator can create a new, better avatar object by referring to the grasped usage history.

[0389] User 5B can also rent avatar object 7106B from the creator. In this case, processor 210B charges user 5B an amount according to the rental period of avatar object 7106B. Processor 210B further determines a remuneration according to the rental period of avatar object 7106B.

[0390] [Embodiment 5] In this embodiment, the processor 210A executes processing related to charging and settlement for the user 5A by a method using the tag object 2733A described in embodiment 1. However, without being limited to this, the processor 210A can also execute processing related to charging and settlement for the user 5A by any method different from the method according to embodiment 1.

[0391] [Request for Second Performance] Fig. 78 is a sequence chart showing an example of processing executed in the HMD system 100. The following describes a series of processing steps in which user 5B (second user) causes avatar object 6B (second avatar) to perform the second performance in virtual space 11B in response to a request for a second performance made by user 5A (first user). The sequence chart shown in Fig. 78 shows the processing steps after step S2614 shown in Fig. 26 is executed. The request for the second performance may be made by user 5C or user 5D.

[0392] FIG. 79 is a diagram showing a second virtual space 2711A and a field of view image 7917A according to an embodiment. The processor 210A notifies the user 5A during a live performance of a message informing the user 5A of a request for a second performance for the avatar object 6B. The processor 210A displays the message on, for example, a tag object 2733A. The message may include text informing the user 5A of the content of the second performance. The second performance includes a content of the second performance and a fee required for user 5A to request the second performance. In the example of FIG. 79, the content of the second performance is a pointing pose, and the fee required to request the pointing pose is 500 yen. The pointing pose is, for example, a performance in which avatar object 6B points at avatar object 6A (first avatar). In this way, the second performance may be a performance directed at avatar object 6 associated with user 5 who requested the second performance.

[0393] After the message is displayed on tag object 2733A, processor 210A moves virtual left hand 1531LA into field of view 15A as shown in FIG. 79(A) in accordance with the movement of user 5A's left hand. Processor 210A displays, for example, field of view image 7917A corresponding to second virtual space 2711A shown in FIG. 79(A) on monitor 130A as shown in FIG. 79(B). User 5A confirms the message displayed on tag object 2733A by viewing field of view image 7917A. This allows user 5A to recognize that if he or she pays an additional charge of 500 yen, he or she can request avatar object 6B to perform the pointing pose.

[0394] After checking the message displayed on tag object 2733A, user 5A performs an operation to request the pointing pose. This operation has been described in embodiment 1 with reference to Figure 34, so it will not be repeated here. In response to the selection of tag object 2733A displaying the message instructing the user to request the second performance, processor 210A detects the operation of user 5A to request the second performance.

[0395] Figure 80 shows second virtual space 2711A and field of view image 8017A according to an embodiment. In step S7801, processor 210A registers charging information corresponding to a request for a second performance in tag object 2733A. In the example of Figure 79, when avatar object 6A selects tag object 2733A with virtual right hand 1531RA, processor 210A newly registers charging information indicating the charging amount (500 yen) required to request a pointing pose in tag object 2733A. The charging amount indicated by the registered charging information may be settled using the method described in embodiment 1.

[0396] After the billing information is registered, processor 210A re-visualizes the billing information on tag object 2733A. Details of this process have been described in embodiment 1 with reference to FIG. 35, and will not be repeated here. Processor 210A displays, for example, field of view image 8017A corresponding to second virtual space 2711A shown in FIG. 80(A) on monitor 130A as shown in FIG. 80(B). By viewing field of view image 8017A, user 5A recognizes that the current total billed amount has increased to 2,000 yen as a result of requesting the pointing pose.

[0397] In step S7802, processor 210A transmits a request for the second performance to computer 200B. More specifically, processor 210A transmits the request to server 600. Server 600 transmits the received information about the second performance to computer 200B through synchronization processing. Note that, as already explained in the first embodiment with reference to FIG. 17, information is transmitted and received between computer 200A and computer 200B via server 600. For this reason, FIG. 78 does not illustrate the processing executed by server 600.

[0398] Figure 81 is a diagram illustrating second virtual space 2711B and field of view image 8117B according to an embodiment. In the example of Figure 81, avatar object 6B faces avatar object 6C (third avatar), and avatar object 6A is located at first position 8176 outside the field of view. In other words, avatar object 6A is not located within field of view 15B. Processor 210B displays field of view image 8117B corresponding to second virtual space 2711B shown in Figure 81(A) on monitor 130A as shown in Figure 81(B), for example.

[0399] In step S7811, processor 210B receives a request for a second performance sent from processor 210A. In step S7812, processor 210B notifies user 5B of information related to the requested second performance. The information related to the requested second performance may be, for example, the content of the requested second performance. Processor 210B may further notify user 5B of information indicating the position in the second virtual space 2711B of the avatar object 6A associated with user 5A who requested the second performance (hereinafter, "first position"). As one example, processor 210B notifies user 5B of the information related to the requested second performance and information indicating the first position 8176 on field of view image 8117B. For example, as shown in FIG. 81(B), processor 210B displays a notification image 8181 and an arrow image 8182 in field of view image 8117B on monitor 130. Processor 210B can also place, in second virtual space 2711B, a virtual object that has the same appearance as notification image 8181. Processor 210B can also place, in second virtual space 2711B, a virtual object that has the same appearance as arrow image 8182.

[0400] 81(B), the notification image 8181 includes information that a second performance has been requested and the details of the requested second performance, thereby enabling user 5B to recognize that a second performance has been requested and the details of the requested second performance.

[0401] As shown in FIG. 81(A), the arrow image 8182 is an image indicating a first position 8176 when the avatar object 6A is not located in the field of view 15B. Specifically, the arrow image 8182 is an image indicating a second direction in the field of view image 8117B, which corresponds to a first direction 8178 from a second position 8177 in the field of view 15B toward the first position 8176 where the avatar object 6A is located. That is, in the example of FIG. 81, the arrow image 8182 is an image of an arrow pointing from left to right in the field of view image 8117B (rightward). By including the arrow image 8182 in the field of view image 8117B, the user 5B can easily recognize, by moving his / her head to the right, that the avatar object 6 (avatar object 6A) associated with the user 5 who requested the second performance is included in the field of view image 8117B.

[0402] Processor 210B may use information identifying avatar object 6A included in the request to identify first position 8176 of avatar object 6A in second virtual space 2711B. This information may be, for example, the ID of user 5A or avatar object 6A. Processor 210B may then determine whether avatar object 6A is located in field of view 15B.

[0403] FIG. 82 illustrates a second virtual space 2711B and a field of view image 8217B according to an embodiment. Assume that user 5B moves his or her head, causing avatar object 6B to face the direction shown in FIG. 82(A). As a result, avatar object 6A is included in field of view 15B, as shown in FIG. 82(A). In other words, processor 210B determines that avatar object 6A is included in field of view 15B. Processor 210B displays field of view image 8117B corresponding to second virtual space 2711B shown in FIG. 82(A) on monitor 130B, including a frame image 8283 surrounding avatar object 6A, as shown in FIG. 82(B). This allows user 5B to easily identify the avatar object 6 associated with user 5 who requested the second performance. Processor 210B can also place a virtual object having the same appearance as frame image 8283 in second virtual space 2711B so as to surround avatar object 6A.

[0404] FIG. 83 is a diagram showing an example of the posture of user 5B according to an embodiment. In step S7813, processor 210B detects a movement of user 5B corresponding to the second performance. User 5B, who has visually recognized notification image 8181 shown in FIG. 81(B), moves his / her body to assume, for example, the posture shown in FIG. 83, i.e., a posture with the right hand thrust forward. At this time, user 5B operates (for example, presses) a button (not shown in FIG. 83) provided on the right controller 300RB. The posture shown in FIG. 83 corresponds to the second performance requested by user 5A, i.e., the pointing pose.

[0405] FIG. 84 illustrates a second virtual space 2711B and a field of view image 8417B according to an embodiment. In step S7814, processor 210B causes avatar object 6B to perform a second performance in response to the detected movement. As an example, processor 210B causes avatar object 6B to perform a pointing pose as shown in FIG. 84(A) in response to the movement of user 5B assuming the posture shown in FIG. 83 and the pressing of a button on right controller 300RB. In detail, processor 210B causes avatar object 6B to perform a movement of extending virtual right hand 1531RB forward and pointing at the extended hand. For example, processor 210B displays field of view image 8417B corresponding to second virtual space 2711B shown in FIG. 84(A) on monitor 130A as shown in FIG. 84(B). By viewing the field of view image 8417B including the virtual right hand 1531RB, the user 5B can recognize that the avatar object 6B is performing the pointing pose.

[0406] When processor 210B has caused avatar object 6B to perform the second performance, it hides notification image 8181 and frame image 8283 in field of view image 8417B, as shown in FIG. 84(B). As an example, processor 210B compares information indicating the posture taken by user 5B with information indicating the posture of the second performance that is associated with the content of the second performance and stored in storage 230. If the comparison shows that the degree of match between the two pieces of information is equal to or greater than a predetermined threshold, processor 210B determines that avatar object 6B has performed the second performance, and hides notification image 8181 and frame image 8283 in field of view image 8417B.

[0407] In step S7815, processor 210B transmits avatar information corresponding to the execution of the second performance to computer 200A. Specifically, processor 210B transmits the avatar information to server 600. Server 600 transmits the received avatar information to computer 200A through synchronization processing.

[0408] 85 is a diagram showing second virtual space 2711A and field of view image 8517A according to an embodiment. In step S7803, processor 210A receives avatar information transmitted from processor 210B in real time. In step S7804, processor 210A causes avatar object 6B to perform a second performance based on the movement information included in the received avatar information. This allows processor 210A to cause avatar object 6B to perform the second performance in accordance with the movement of user 5B shown in FIG. 83.

[0409] The processor 210A causes the avatar object 6A to watch the second performance by the avatar object 6B in the second virtual space 2711A. A field of view image 8517A corresponding to the virtual space 2711A is displayed on the monitor 130A as shown in FIG. 85(B). By viewing the field of view image 8517A, the user 5A recognizes that the avatar object 6B has performed the second performance that the user 5A requested. This causes the user 5A to feel joy that the avatar object 6B has performed the second performance that the user 5A requested. In particular, the user 5A can virtually experience, by paying a fee, a performance by a performer directed at the user, which would be difficult to experience in a real live performance. As a result, the user 5A can feel joy and be more satisfied. In other words, the interest of a live performance in a virtual space can be further increased.

[0410] [Variation] User 5A may be able to request a second performance without specifying the content. In this case, processor 210B receiving the request notifies user 5B only that a second performance has been requested. User 5B then decides on the second performance to be performed by avatar object 6B and assumes a posture corresponding to the second performance.

[0411] The fee required to request a second performance may be set according to the content of the requested second performance. For example, a second performance that is difficult for user 5B to execute, such as one that takes a long time to execute, is complicated, requires a lot of work (operations), or the like, may be charged a high fee.

[0412] The content of the second performance is not limited to the above-mentioned examples. For example, the second performance may involve the avatar object 6B moving from the stage object 1532, approaching the requested avatar object 6, and performing any or the requested content of the performance. The movement of the avatar object 6B from the stage object 1532 may be performed using the movement method described in the second embodiment. Furthermore, the second performance may be a performance that does not target the avatar object 6 associated with the user 5 who requested the second performance. An example of such a performance is jumping on the stage object 1532.

[0413] The notification image 8181 and the arrow image 8182 or the frame image 8283 may be included in the field of view image 17B at a preset timing, rather than immediately after receiving the request. The preset timing may be, for example, the timing at which a period set by user 5B as a period for performing the second performance starts.

[0414] The fee required to request the second performance may be settled, for example, when an operation to request the second performance is performed.

[0415] The request for the second performance may be made, for example, by bidding. In this example, a fee required to request the second performance is not set. The processor 210A accepts input of a fee (hereinafter, "first fee") for the second performance. If the first fee is higher than the fee (second fee) for the second performance input by a user 5 (third user) other than the user 5A, the processor 210A registers charging information representing the first fee in the tag object 2733A. The comparison between the first fee and the second fee may be performed, for example, by the server 600. That is, the processor 210A transmits information indicating the first fee to the server 600. If the first fee is higher than the second fee, the server 600 notifies the computer 200A of this fact. When the processor 210A receives this notification, it registers charging information representing the first fee in the tag object 2733A.

[0416] If the request for the second performance is made through a bidding system, the content and timing of the second performance may be set in advance by user 5B. Alternatively, virtual space 11 may be divided into multiple areas, and bidding may be conducted for each area. In this case, processor 210B tallies the charges for each area and includes in field of view image 17B an image showing the area with the highest tallied charge. In this case, the second performance may be, for example, avatar object 6B moving to the stage object located in the area with the highest tallied charge among the stage objects located in the areas, and performing the first performance.

[0417] If the request for the second performance is made via a bidding system, a minimum bid amount may be set. If both the first amount and the second amount are lower than the minimum bid amount, the server 600 may notify the computer 200B of this fact. Upon receiving this notification, the processor 210B may notify the user 5B that the bid amount did not reach the minimum bid amount. In this case, the user 5B may not have the avatar object 6B perform the second performance. For example, consider an example in which the second performance involves approaching the avatar object 6 associated with the requesting user 5 and performing the requested content. In this example, if the first amount and the second amount are lower than the minimum bid amount, the user 5B may have the avatar object 6B continue the first performance on the stage object 1532 instead of the second performance.

[0418] The request for the second performance may also be made before the start of the first performance by the avatar object 6B. As an example, the request may be made before the user 5A starts the virtual live experience. In this example, the request may be made in the form of so-called crowdfunding. For example, before the start of the virtual live experience, the user 5B invites users 5 who wish to perform a predetermined performance to pay a fee, with the avatar object 6B approaching them. If the fee reaches or exceeds a set amount, the avatar object 6B approaches the avatar object 6 associated with the user 5 who paid the fee during the live performance and performs the predetermined performance.

[0419] [Embodiment 6] In this embodiment, the processor 210A executes processing related to charging and settlement for the user 5A by a method using the tag object 2733A described in embodiment 1. However, without being limited to this, the processor 210A can also execute processing related to charging and settlement for the user 5A by any method different from the method according to embodiment 1.

[0420] Fig. 86 is a sequence chart showing an example of processing executed in the HMD system 100. The following describes a series of processing in which a user 5A (first user) controls the appearance of an avatar object 6A (first avatar) in the virtual space 11A and causes an avatar object 6B (second avatar) to perform a second performance targeting the avatar object 6A. The sequence chart shown in Fig. 86 shows the processing after step S2614 shown in Fig. 26 is executed. Note that the appearance of the avatar object 6 may be controlled by either user 5C or user 5D.

[0421] 87 is a diagram showing a second virtual space 2711A and a field of view image 8717A according to an embodiment. The processor 210A notifies the user 5A of a message during a live broadcast, instructing the user 5A to control the appearance of the avatar object 6A. As an example, controlling the appearance is to make the appearance of the avatar object 6A stand out (emphasize) compared to other avatar objects 6. The processor 210A displays the message in, for example, the tag object 2733A. The message includes text instructing the user 5A to emphasize the appearance and the amount of money required for the user 5A to emphasize the appearance of the avatar object 6A.

[0422] After the message is displayed on tag object 2733A, processor 210A moves virtual left hand 1531LA into field of view 15A as shown in FIG. 87(A) in accordance with the movement of user 5A's left hand. Processor 210A displays, for example, field of view image 8717A corresponding to second virtual space 2711A shown in FIG. 87(A) on monitor 130A as shown in FIG. 87(B). User 5A confirms the message displayed on tag object 2733A by viewing field of view image 8717A. This allows user 5A to recognize that if he or she pays an additional charge of 500 yen, he or she can enhance the appearance of avatar object 6A.

[0423] After checking the message displayed on the tag object 2733A, the user 5A performs an operation to emphasize the appearance of the avatar object 6A. This operation has been described in the first embodiment with reference to Fig. 34, so it will not be repeated here. The processor 210A detects the operation of the user 5A to emphasize the appearance of the avatar object 6A in response to the selection of the tag object 2733A on which the message instructing the user 5A to emphasize the appearance of the avatar object 6A is displayed.

[0424] FIG. 88 is a diagram showing second virtual space 2711A and field of view image 8817A according to an embodiment. In step S8601, processor 210A registers billing information corresponding to highlighting avatar object 6A in tag object 2733A. In the example of FIG. 88, when avatar object 6A selects tag object 2733A with virtual right hand 1531RA, processor 210A newly registers billing information indicating the billing amount (500 yen) required to highlight avatar object 6A in tag object 2733A. The billing amount indicated by the registered billing information may be settled using the method described in embodiment 1.

[0425] After the billing information is registered, the processor 210A re-visualizes the billing information on the tag object 2733A. Details of this process have been described in the first embodiment with reference to FIG. 35, and will not be repeated here. The processor 210A displays, for example, a field of view image 8817A corresponding to the second virtual space 2711A shown in FIG. 88(A) on the monitor 130A as shown in FIG. 88(B). By viewing the field of view image 8817A, the user 5A recognizes that the current total billing amount has increased to 2,000 yen as a result of emphasizing the avatar object 6A.

[0426] FIG. 89 illustrates second virtual space 2711B and field of view image 8117B according to an embodiment. In step S8602, processor 210A makes the appearance of avatar object 6A stand out more than avatar objects 6 (third avatars, hereinafter referred to as "other avatar objects 6") associated with viewers 5 (third users) other than user 5A. As an example, processor 210A controls the color of the appearance of avatar object 6A so that avatar object 6A stands out more than the other avatar objects 6. In detail, processor 210A changes the color of the appearance of avatar object 6A by applying texture 4137A to avatar object 6A, as shown in FIG. 89(A). Texture 4137A is a texture colored in a first color corresponding to the one-time charge (500 yen) made to user 5A for highlighting avatar object 6A. Processor 210A displays, for example, field of view image 8917A corresponding to second virtual space 2711A shown in FIG. 89(A) on monitor 130A as shown in FIG. 89(B).

[0427] The processor 210A may make the avatar object 6A more conspicuous each time the number of times the fee required to highlight the avatar object 6A is charged to the user 5A increases. As an example, the processor 210A may change the color of the appearance of the avatar object 6A to make it more conspicuous each time the fee is charged to the user 5A. For example, if the fee is charged twice, the processor 210A may change the color of the appearance of the avatar object 6A to be more conspicuous than the first color. A texture colored in a certain color may be applied to the avatar object 6A. Furthermore, if the charge amount is paid three times, a texture colored in a third color that is more noticeable than the second color may be applied to the avatar object 6A. The first color may be, for example, copper, the second color may be, for example, silver, and the third color may be, for example, gold.

[0428] In step S8603, processor 210A transmits avatar information (first information) of avatar object 6A whose appearance has been controlled to computer 200B. Specifically, processor 210A transmits the avatar information to server 600. Server 600 transmits the received avatar information to computer 200B by synchronization processing. The avatar information includes first user information indicating user 5A. The first user information may be, for example, an ID indicating user 5A or an ID indicating avatar object 6A.

[0429] FIG. 90 illustrates second virtual space 2711B and field of view image 9017B according to an embodiment. In step S8611, processor 210B receives avatar information transmitted from processor 210A in real time. In step S8612, processor 210B makes the appearance of avatar object 6A stand out more than other avatar objects 6 based on the received avatar information. In particular, processor 210B changes the color of the appearance of avatar object 6A by applying texture 4137A to avatar object 6A based on the received avatar information, as shown in FIG. 90(A). Processor 210B displays, for example, field of view image 9017B corresponding to second virtual space 2711B shown in FIG. 90(A) on monitor 130A, as shown in FIG. 90(B). By viewing field of view image 9017B, user 5B (second user) recognizes that the appearance of avatar object 6A has been changed to make it stand out more than other avatar objects 6. In other words, user 5B recognizes that user 5A has been charged a fee to make avatar object 6A stand out.

[0430] In step S8613, the processor 210B notifies the user 5B of the recommended second performances. Specifically, the processor 210B selects at least one second performance from among the plurality of second performances and notifies the user 5B of the second performance preferentially. As an example, the processor 210B selects a second performance that will result in a charge to the user 5A associated with the appearance-controlled avatar object 6A, and notifies the user 5B of the second performance.

[0431] FIG. 91 is a diagram showing an example of data referenced by processor 210B to notify user 5B of at least one second performance. The data is stored, for example, in storage 230B. The "second performance" column stores information indicating the content of each second performance, such as "pointing pose" or "waving hand" (hereinafter referred to as "second information"). The "first user information" column stores first user information associated with each piece of second information. In FIG. 91, reference symbols 5A, 5C, and 5D, which are used in this specification to refer to "users," are used as examples of first user information.

[0432] The first user information is associated with the second information by the processor 210B before the process shown in FIG. 86 . As an example, after the avatar object 6B performs the second performance, the processor 210B waits for reception of avatar information for a predetermined time from the performance of the second performance. If the avatar information is received within the predetermined time, the processor 210B associates the first user information included in the avatar information with the second information indicating the second performance performed immediately before. In other words, if a viewer user 5 registers billing information to highlight the avatar object 6 associated with the viewer within the predetermined time from the performance of the second performance, the processor 210B sets the immediately preceding second performance as the second performance that led to the billing of the user 5.

[0433] The processor 210B notifies the user 5B of the second information associated with the first user information included in the received avatar information. For example, when avatar information including the first user information "5A" indicating the user 5A is received, the processor 210B references the data shown in FIG. 91, selects the "pointing pose" associated with the first user information "5A," and notifies the user 5B of the second information. That is, the processor 210B notifies the user 5B of the second performance that was previously performed by the avatar object 6B and actually resulted in a charge to the user 5A as a second performance that is likely to result in a charge. As shown in FIG. 90(B), the processor 210B displays the field of view image 9017B on the monitor 130B, including a notification image 9084 including text indicating the content of the selected second performance. The processor 210B can also place a virtual object having the same appearance as the notification image 9084 in the second virtual space 2711B. By viewing the notification image 9084, the user 5B can recognize the second performance that is likely to lead to charges being incurred by the user 5A.

[0434] In step S8614, processor 210B detects a movement of user 5B corresponding to the second performance. In step S8615, processor 210B causes avatar object 6B to perform the second performance in accordance with the detected movement of user 5B. Processor 210B causes avatar object 6B to perform a pointing pose in accordance with the movement of user 5B. Details of the pointing pose and details of the movement of user 5B to cause avatar object 6B to perform the pointing pose have already been described in embodiment 5 with reference to Figures 83 and 84, and will not be repeated here.

[0435] When processor 210B causes avatar object 6B to perform the second performance, processor 210B hides notification image 9084 in field of view image 17B. As an example of this processing, processor 210B may perform processing similar to the processing described in embodiment 5 for hiding notification image 8181 and frame image 8283 in field of view image 17B.

[0436] In step S8616, processor 210B transmits avatar information corresponding to the execution of the second performance to computer 200A. Specifically, processor 210B transmits the avatar information to server 600. Server 600 transmits the received avatar information to computer 200A by synchronization processing.

[0437] 92 is a diagram showing second virtual space 2711A and field of view image 9217A according to one embodiment. In step S8604, processor 210A receives avatar information transmitted from processor 210B in real time. In step S8605, processor 210A causes avatar object 6B to perform a second performance based on the movement information included in the received avatar information. This allows processor 210A to cause avatar object 6B to perform the second performance in accordance with the movement of user 5B.

[0438] The processor 210A causes the avatar object 6A to view the second performance by the avatar object 6B in the second virtual space 2711A. For example, the processor 210A displays a field of view image 9217A corresponding to the second virtual space 2711A shown in FIG. 92(A) on the monitor 130A as shown in FIG. 92(B). By viewing the field of view image 8517A, the user 5A recognizes that the avatar object 6B has performed the second performance that the user 5A requested. As a result, the user 5A recognizes that the avatar object 6B has performed the second performance by paying a fee to make the appearance of the avatar object 6A associated with the user 5A more prominent than the avatar objects 6A of the other viewers, and feels delighted. In other words, the live performance in the virtual space can be made more interesting. Furthermore, the user 5A may wish for the second performance directed at the user 5A to be performed again and may be willing to pay an additional fee to make the avatar object 6A more prominent. This allows the user 5B to earn more income.

[0439] Now consider a case where user 5B causes avatar object 6B to perform the notified second performance (recommended second performance). This second performance is the second performance that led user 5A to pay a fee when performed in the past, in other words, it is a second performance that user 5A prefers. From user 5A's perspective, by paying a fee, the appearance of avatar object 6A associated with user 5A has become more prominent than the avatar objects 6B of other viewers, and user 5A feels that avatar object 6B has performed the second performance that user 5A prefers. This makes user 5A feel even more pleased. This also increases the likelihood that user 5A will pay a fee to make avatar object 6A more prominent.

[0440] [Modification] The control of the appearance of the avatar object 6A is not limited to the above-described example of changing the color of the avatar object 6A, as long as the control of the appearance makes the avatar object 6A stand out from the avatar object 6B. For example, the processor 210A may change the size or shape of the avatar object 6A. The processor 210A may also make the avatar object 6A stand out by controlling the appearance of other avatar objects 6. For example, the processor 210A may make other avatar objects 6 transparent for a certain period of time.

[0441] The condition for selecting the second performance to be notified to user 5B is not limited to the above-mentioned example, i.e., "a second performance that leads to a charge." In particular, processor 210B may further narrow down the second performances selected by the above condition using other conditions. The other conditions may be, for example, "a second performance that user 5B is good at." In other words, when there are multiple second performances that have led to a charge for user 5A, processor 210B may select a second performance that user 5B is good at and notify user 5B of it. The second performance that user 5B is good at may be set in advance by user 5B.

[0442] Furthermore, the other condition may be, for example, "a second performance that does not lower the ratings of other users 5." In other words, if there are multiple second performances that have resulted in charges for user 5A, processor 210B may exclude from the options any second performances whose rating values, indicating ratings when performed in past live shows, are below a threshold. The rating values ​​are calculated, for example, based on the individual ratings given by each viewer user. The individual ratings are values ​​input by viewer user 5 performing a predetermined operation with their right or left hand, and the values ​​are transmitted to computer 200B. For example, processor 210B calculates the rating value by averaging the received values. As described above, it is possible to prevent the rating of avatar object 6B from being lowered or the avatar object 6B from being criticized online as a result of performing a second performance that results in charges.

[0443] User 5 on the audience side may input comments on the second performance along with individual evaluations. Processor 210A may place a bulletin board object (not shown) in second virtual space 2711A and display the comments on the bulletin board object.

[0444] Furthermore, when selecting the second performance to notify the user 5B, the processor 210B may refer to the gender ratio, age distribution, etc. of the users 5 on the audience side. For example, the processor 210B may refer to the past rankings of the second performances that led to the billing of the user 5A. Suppose there is a second performance whose evaluation value, indicating the evaluation when performed on Eve, is below a threshold. The evaluation value of this second performance is below the threshold in a live performance for women only, but is above the threshold in a live performance for men only. Here, the male-to-female ratio of users 5 who are viewers of this live performance is assumed to be 80% male. In this case, processor 210B may determine that this second performance is well-received by men, and notify user 5B of this without excluding it from the options.

[0445] [Embodiment 7] In this embodiment, the processor 210A executes processing related to charging and settlement for the user 5A by a method using the tag object 2733A described in embodiment 1. However, without being limited to this, the processor 210A can also execute processing related to charging and settlement for the user 5A by any method different from the method according to embodiment 1.

[0446] FIG. 93 is a sequence chart showing an example of processing executed in the HMD system 100. A series of processes for controlling the performance of a live performance based on input from user 5A (first user) will be described below. The sequence chart shown in FIG. 93 shows the processing after step S2613 shown in FIG. 26 is executed. In this embodiment, lighting control will be described as an example of performance control performed by user 5A. However, the performance control is not limited to lighting control. For example, the performance control may include sound control. In this example, processor 210A controls at least one of lighting and sound based on input from user 5A. Furthermore, the performance control of the live performance may be performed by user 5C or user 5D.

[0447] 94A and 94B are diagrams illustrating second virtual space 2711A and field of view image 9417A according to an embodiment. As shown in Fig. 94A, a lighting object 9491, which is a type of virtual object, is placed in second virtual space 2711A. As an example, processor 210A may place lighting object 9491 in second virtual space 2711A together with stage object 1532.

[0448] In step S9301, processor 210A causes avatar object 6B (second avatar) to perform a first performance based on the movement information included in the received avatar information. In step S9302, processor 210A executes a first lighting performance. The first lighting performance is a lighting performance that user 5B set in advance before the start of the live performance. Information for executing the first lighting performance is transmitted from computer 200B to computer 200A via server 600. Computer 200A may receive the information together with avatar information for placing avatar object 6B in second virtual space 2711A, for example, and execute the first lighting performance based on the information in step S9302.

[0449] 94(A), the processor 210A functions as the virtual object control module 1427 and causes a light ray object 9492 to be emitted from a lighting object 9491 based on information for executing a first lighting performance. The processor 210A may move the lighting object 9491 based on the information. This allows the processor 210A to move the light ray object 9492. Note that the lighting object 9491 may be placed in the second virtual space 2711A at the same time as the placement of the stage object 1532, for example.

[0450] The processor 210A notifies the user 5A of a message informing the user 5A of the lighting control during the live broadcast. The processor 210A displays the message in, for example, the tag object 2733A. The message includes text informing the user 5A of the lighting control and the fee required for the user 5A to perform the live lighting control.

[0451] After the message is displayed on tag object 2733A, processor 210A moves virtual left hand 1531LA into field of view 15A as shown in FIG. 94(A) in accordance with the movement of user 5A's left hand. Processor 210A displays, for example, field of view image 9417A corresponding to second virtual space 2711A shown in FIG. 94(A) on monitor 130A as shown in FIG. 94(B). User 5A confirms the message displayed on tag object 2733A by viewing field of view image 9417A. This allows user 5A to recognize that if he or she pays an additional fee of 1,000 yen, he or she will be able to control the lighting of the live performance.

[0452] After checking the message displayed on the tag object 2733A, the user 5A performs an operation to control the lighting himself / herself. This operation has been described in the first embodiment with reference to Fig. 34, and will not be repeated here. The processor 210A detects an operation by the user 5A to control the lighting in response to the selection of the tag object 2733A on which the message providing guidance on lighting control is displayed.

[0453] FIG. 95 is a diagram showing second virtual space 2711A and field of view image 9517A according to an embodiment. In step S9303, processor 210A registers billing information corresponding to lighting control in tag object 2733A. In the example of FIG. 95, when avatar object 6A (first avatar) selects tag object 2733A with virtual right hand 1531RA, processor 210A newly registers billing information indicating the billing amount (1,000 yen) required for lighting control in tag object 2733A. The billing amount indicated by the registered billing information may be settled using the method described in embodiment 1.

[0454] After the billing information is registered, the processor 210A visualizes the billing information again on the tag object 2733A. Details of this process have been described in the first embodiment with reference to FIG. 35, and will not be repeated here. The processor 210A displays, for example, a field of view image 9517A corresponding to the second virtual space 2711A shown in FIG. 95(A) on the monitor 130A as shown in FIG. 95(B). By viewing the field of view image 9517A, the user 5A recognizes that the current total billed amount has increased to 2,500 yen by controlling the lighting.

[0455] FIG. 96 illustrates a second virtual space 2711A and a field of view image 9617A according to an embodiment. In step S9304, the processor 210A moves the avatar object 6A to a first position for lighting control. The processor 210A, functioning as the virtual object generation module 1421, generates a lighting control object 9693 and places it in front of the avatar object 6A that has moved to the first position. The lighting control object 9693 is used by the user 5C to cause the processor 210A to execute processing for controlling lighting effects. The lighting control object 9693 includes, as an example, partial objects such as a button, dial, or slider that accept operations on the avatar object 6A. The processor 210A controls the lighting effects by accepting operations on the partial objects. For example, the processor 210A can change the color of a light ray object 9492 or move a lighting object 9491 in response to the accepted operation. The lighting control object 9693 may be placed at a predetermined position (the position shown in FIG. 96(A)) determined based on the first position before the avatar object 6A moves. For example, the lighting control object 9693 may be placed at the predetermined position at the same time as the stage object 1532 is placed.

[0456] Processor 210A displays, for example, field of view image 9617A corresponding to second virtual space 2711A shown in Fig. 96(A) on monitor 130A as shown in Fig. 96(B). By viewing field of view image 9617A, user 5A recognizes that avatar object 6A has moved to the first position and recognizes lighting control object 9693. In other words, user 5A recognizes that he or she can now control lighting.

[0457] 97 is a diagram showing second virtual space 2711A and field of view image 9717A according to an embodiment. After moving to a first position, user 5A performs an operation to control lighting. User 5A moves their right hand so that the tip of virtual right hand 1531RA approaches lighting control object 9693. In step S9305, processor 210A detects the movement of user 5A's right hand based on the output of right controller 300RA. In step S9306, processor 210A moves virtual right hand 1531RA in second virtual space 2711A in accordance with the movement of user 5A's right hand so that the tip of virtual right hand 1531RA approaches lighting control object 9693.

[0458] In step S9307, processor 210A manipulates lighting control object 9693 in accordance with the movement of virtual right hand 1531RA. In particular, when the tip of virtual right hand 1531RA and a partial object included in lighting control object 9693 have a first positional relationship, processor 210A detects that the tip of virtual right hand 1531RA has collided with the partial object. The first positional relationship means, for example, that the distance between the tip of virtual right hand 1531RA and the partial object is shorter than a first distance. Alternatively, a collision area defined by the tip of virtual right hand 1531RA and a collision area set for the partial object at least partially collide. Processor 210A detects that the partial object has been manipulated by virtual right hand 1531RA based on the collision between the tip of virtual right hand 1531RA and the partial object.

[0459] In step S9308, processor 210A executes a second lighting performance in response to an operation on lighting control object 9693, i.e., an operation on a partial object. The second lighting performance is a lighting performance different from the first lighting performance, performed under the control of user 5A. For example, processor 210A changes light ray object 9492 colored in a first color to light ray object 9792 colored in a second color, as shown in FIG. 97(A), in response to the operation on the partial object. Processor 210A displays, for example, field of view image 9717A corresponding to second virtual space 2711A shown in FIG. 97(A) on monitor 130A, as shown in FIG. 97(B). By viewing field of view image 9717A, user 5A confirms that the second lighting performance in response to his or her operation is being executed in second virtual space 2711A.

[0460] Processor 210A transmits information for executing the second lighting performance to server 600. Server 600 transmits the received information to computers 200 of all users 5. This enables processor 210 of each computer 200 to execute the second lighting performance.

[0461] Figure 98 is a diagram showing second virtual space 2711C and field of view image 9817C according to an embodiment. Upon receiving information for executing a second lighting performance, processor 210C generates UI panel 9894 and places it in second virtual space 2711C. For example, processor 210C displays field of view image 9817C corresponding to second virtual space 2711C shown in Figure 98(A) on monitor 130C as shown in Figure 98(B). User 5C (third user) recognizes that UI panel 9894 has been placed in second virtual space 2711C by viewing field of view image 9817C.

[0462] The UI panel 9894 is a type of UI object and is used by the user 5C to cause the processor 210C to execute processing for selecting a lighting effect. The UI panel 9894 includes options 9895 and 9896 arranged on the front side of the UI panel 9894. The options 9895 and 9896 are lighting effects that are selected when the options are selected by the user 5C. The option 9895 is an item for selecting a first lighting performance, in other words, a default lighting performance, and the option 9896 is an item for selecting a second lighting performance, in other words, a lighting performance controlled by the user 5A. In other words, the user 5C can select whether or not to watch the first performance of the avatar object 6B with the second lighting performance.

[0463] In the example of FIG. 98, option 9895 is selected. Therefore, processor 210C executes a first lighting rendering in second virtual space 2711C. As shown in FIG. 98(B), processor 210C may make the selected option 9895 and the unselected option 9896 different colors. This allows user 5C to recognize that option 9895 has been selected. Note that, because the first lighting rendering is a default lighting rendering, option 9895 may be automatically selected when UI panel 9894 is placed in second virtual space 2711C.

[0464] When user 5C wishes to execute the second lighting performance in second virtual space 2711C, he or she performs an operation to change the lighting performance. As an example, user 5C moves his or her right hand so that the tip of virtual right hand 1531RC approaches option 9896. Processor 210C detects the movement of user 5C's right hand based on the output of right controller 300RC. Then, processor 210C moves virtual right hand 1531RC in second virtual space 2711C in accordance with the movement of user 5C's right hand so that the tip of virtual right hand 1531RC approaches option 9896.

[0465] Based on the movement of the right hand of user 5C, processor 210C moves virtual right hand 1531RC in second virtual space 2711C so as to bring the tip of virtual right hand 1531RC closer to option 9896. Based on the collision between the tip of virtual right hand 1531RC and option 9896, processor 210C detects that option 9896 has been selected by virtual right hand 1531RC.

[0466] FIG. 99 is a diagram illustrating second virtual space 2711C and field of view image 9917C according to an embodiment. In response to detecting the selection of option 9896, processor 210C executes a second lighting effect in second virtual space 2711C. For example, processor 210C displays field of view image 9917C corresponding to second virtual space 2711C shown in FIG. 99(A) on monitor 130C as shown in FIG. 99(B). By viewing field of view image 9917C, user 5C recognizes that option 9896 has been selected and that the second lighting effect has begun in second virtual space 2711C. When processor 210C executes the second lighting effect in second virtual space 2711C, it notifies computer 200A of this via server 600.

[0467] FIG. 100 illustrates a second virtual space 2711A and a field of view image 10017A according to an embodiment. In step S9309, processor 210A notifies user 5A of the number of users 5 watching the first performance of avatar object 6B with the second lighting effect. Specifically, processor 210A notifies user 5A of the number of notifications received from other computers 200 indicating that the second lighting effect has been performed as the number of users. For example, processor 210A includes notification image 10085 containing text indicating the number of users in field of view image 10017A corresponding to second virtual space 2711A shown in FIG. 100(A). Then, processor 210A displays field of view image 10017A on monitor 130A as shown in FIG. 100(B). Processor 210A can also place a virtual object having the same appearance as notification image 10085 in second virtual space 2711A. By viewing the field of view image 10017A, the user 5A recognizes the number of people watching the first performance of the avatar object 6B with the second lighting effect.

[0468] Figure 101 illustrates second virtual space 2711A and field of view image 10117A according to one embodiment. In Figure 101(A), the live performance in second virtual space 2711A has ended. With the end of the live performance, avatar objects 6B and 6C have left second virtual space 2711A. Avatar objects 6A and 6D remain in second virtual space 2711A after the end of the live performance.

[0469] In step S9310, the processor 210A changes the charge amount in the charging information registered in the tag object 2733A based on the evaluations of the second lighting performance by other users 5. The evaluations may be, for example, the number of users 5 who watched the first performance of the avatar object 6B with the second lighting performance. For example, the processor 210A may decrease the charge amount in the charging information registered in the tag object 2733A as the number of users increases.

[0470] Processor 210A displays, for example, field of view image 10117A corresponding to second virtual space 2711A shown in FIG. 101(A) on monitor 130A as shown in FIG. 101(B). By viewing field of view image 10117A, user 5A recognizes that the live performance has ended. User 5A further recognizes that the total amount charged to user 5A has changed from 2,500 yen (see FIG. 95(B)) to 2,300 yen. In other words, user 5A recognizes that the amount charged for lighting control has been reduced due to the evaluation of the lighting control that user 5A performed.

[0471] As described above, the processor 210A allows the user 5A to control the live performance in accordance with the registration of billing information. In this way, the HMD system 100 allows the user 5A to virtually experience the live performance produced by the audience, which is not possible in a real live performance. As a result, the live performance in the virtual space can be made more interesting. Furthermore, since the control of the live performance requires payment of a fee, the number of users 5 who control the per...

Claims

1. Computer, definition means for defining a virtual space including a first area for providing a virtual experience to a first user; an arrangement means for arranging, in the first area, a plurality of avatars, each of which is associated with a plurality of users including the first user, including a first avatar associated with the first user, and a second avatar associated with a second user; a display means for displaying, in the first area, an image for identifying the first avatar associated with the first user who requested the performance from the plurality of avatars when the first user requests the second user to perform the performance (excluding the request to call the second user), together with an image showing the content of the performance; A program that functions as a

2. a notification means for notifying the second user of the first position where the first avatar associated with the first user who requested the performance is located when the first avatar associated with the second user is located at a first position outside a field of view of the second avatar associated with the second user; The program according to claim 1, wherein the program functions as follows:

3. A definition means for defining a virtual space including a first area for providing a virtual experience to a first user; an arrangement means for arranging, in the first area, a plurality of avatars, each of which is associated with a plurality of users including the first user, including a first avatar associated with the first user, and a second avatar associated with a second user; a display means for displaying, in the first area, an image for identifying the first avatar associated with the first user who requested the performance from the plurality of avatars when the first user requests the second user to perform the performance (excluding the request to call the second user), together with an image showing the content of the performance; An information processing system comprising:

Images