Program, and System

Abstract

To provide a program, method and information terminal device which can improve amusement.SOLUTION: A program comprises the steps of: receiving data for display for displaying a virtual space where a character is arranged; specifying any of plural types of view modes different in modes for moving a view point in the virtual space; and displaying on a display unit an image from the view point that moves in the mode according to the view mode specified by the specifying step among the images in the virtual space based on the data for display. The plural types of view modes include, as the view modes that can move the view point according to the input mode with respect to an input unit from a user, the first view mode that may restrict movement of the view point within a prescribed range where the character may be arranged, and the second view mode that permits movement of the view point within the prescribed range where the character may be arranged but may make the character non-display when the view point is moved within the prescribed range.SELECTED DRAWING: Figure 30

Description

Technical Field

[0001] The present invention relates to a program , oh and system .

Background Art

[0002] Patent Document 1 describes an information processing system that distributes video data for displaying a live video on a viewing terminal. In the information processing system, a video specified by the video data distributed from a content server is displayed on the viewing terminal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the system of Patent Document 1, it is not possible to change the display mode of the video, the control method of the viewing point, etc. For this reason, there is a problem that the system lacks user preferences.

[0005] The present invention has been conceived in view of such circumstances, and an object thereof is to provide a program , oh and system that can improve user preferences.

Means for Solving the Problems

[0006] According to an aspect of an embodiment shown in the present disclosure, computer a program executed in is provided. The program computer , among a plurality of types of viewing modes having different modes for moving the viewing point in the imaginary space where characters are arranged temporary identifies any one.specification means and temporary Among the images in the imaginary space, specification means an image from a viewpoint that moves in a manner corresponding to the viewing mode specified by represent is displayed on the display unit display means and function as the plurality of types of viewing modes include 、 a first viewing mode that can restrict the movement of the viewpoint within a predetermined range where the character can be arranged, and a second viewing mode that allows the movement of the viewpoint within the predetermined range but can make the character non-displayed when the viewpoint is moved within the predetermined range.

Effect of the Invention

[0007] According to the present invention, the interest can be improved.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Embodiments for Carrying Out the Invention

[0009] The system according to the present disclosure is a system for providing games to a plurality of users. Hereinafter, the system will be described with reference to the drawings. It should be noted that the present invention is not limited to these examples, but is defined by the claims, and all modifications within the meaning and scope equivalent to the claims are intended to be included in the present invention. In the following description, the same reference numerals are given to the same elements in the description of the drawings, and duplicate descriptions will not be repeated.

[0010] <Overview of the operation of System 1> FIG. 1 is a diagram showing an overview of System 1 according to the present embodiment. System 1 includes a plurality of user terminals 100 (computers), a server 200, a game play terminal 300 (external device, second external device), and a distribution terminal 400 (external, first external device). In FIG. 1, as an example of a plurality of user terminals 100, user terminals 100A to 100C, in other words, three user terminals 100 are shown, but the number of user terminals 100 is not limited to the illustrated example. Also, in the present embodiment, when there is no need to distinguish between user terminals 100A to C, they are described as "user terminal 100". The user terminal 100, the game play terminal 300, and the distribution terminal 400 are connected to the server 200 via the network 2. The network 2 is composed of the Internet and various mobile communication systems constructed by wireless base stations (not shown). Examples of this mobile communication system include, for example, so-called 3G, 4G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can be connected to the Internet by a predetermined access point.

[0011] (Overview of the game) In this embodiment, as an example of the game provided by system 1 (hereinafter referred to as "this game"), a game mainly played by the user of game play terminal 300 will be described. Hereinafter, the user of game play terminal 300 will be referred to as the "player". The player (performer) advances the game by operating the characters appearing in this game as an example. Also, in this game, the user of user terminal 100 plays a role of assisting the progress of the game by the player. Details of this game will be described later. Note that the game provided by system 1 may be any game in which a plurality of users participate, and is not limited to this example.

[0012] (Game play terminal 300) The game play terminal 300 advances the game in response to an input operation by the player. Also, the game play terminal 300 sequentially distributes in real time to the server 200 the information generated by the player's game play (hereinafter referred to as "game progress information").

[0013] (Server 200) The server 200 transmits the game progress information (second data) received in real time from the game play terminal 300 to the user terminal 100. Also, the server 200 mediates the transmission and reception of various information among the user terminal 100, the game play terminal 300, and the distribution terminal 400.

[0014] (Distribution terminal 400) The distribution terminal 400 generates operation instruction data (first data) in response to an input operation by the user of the distribution terminal 400, and distributes the operation instruction data to the user terminal 100 via the server 200. The operation instruction data is data for playing a video on the user terminal 100, and specifically, data for operating the characters appearing in the video.

[0015] In this embodiment, as an example, the user of the distribution terminal 400 is a player of this game. Also, as an example, the video played based on the operation instruction data on the user terminal 100 is a video in which the character operated by the player in the game moves. "Movement" means moving at least a part of the character's body and includes speech. Therefore, the operation instruction data according to this embodiment includes, for example, voice data for making the character speak and motion data for moving the character's body.

[0016] As an example, the operation instruction data is transmitted to the user terminal 100 after the game ends. Details of the operation instruction data and the video played based on the operation instruction data will be described later.

[0017] (User Terminal 100) The user terminal 100 receives game progress information in real time and uses this information to generate and display a game screen. In other words, the user terminal 100 reproduces the game screen of the game being played by the player through real-time rendering. As a result, the user of the user terminal 100 can view the same game screen as the game screen that the player is viewing while playing the game at almost the same timing as the player.

[0018] Also, the user terminal 100 generates information for assisting the progress of the game by the player in response to an input operation by the user, and transmits this information to the game play terminal 300 via the server 200. Details of this information will be described later.

[0019] Also, the user terminal 100 receives operation instruction data from the distribution terminal 400, and uses this operation instruction data to generate and play a video (image). In other words, the user terminal 100 renders and plays the operation instruction data.

[0020] <Hardware Configuration of System 1> FIG. 2 is a diagram showing the hardware configuration of the user terminal 100. FIG. 3 is a diagram showing the hardware configuration of the server 200. FIG. 4 is a diagram showing the hardware configuration of the game play terminal 300. FIG. 5 is a diagram showing the hardware configuration of the distribution terminal 400.

[0021] (User Terminal 100) In this embodiment, as an example, an example in which the user terminal 100 is realized as a smartphone will be described. However, the user terminal 100 is not limited to a smartphone. For example, the user terminal 100 may be realized as a feature phone, a tablet computer, a laptop computer (so-called notebook computer), or a desktop computer. Further, the user terminal 100 may be a game device suitable for game play.

[0022] As shown in FIG. 2, the user terminal 100 includes a processor 10, a memory 11, a storage 12, a communication interface (IF) 13, an input / output IF 14, a touch screen 15 (display unit), a camera 17, and a distance measuring sensor 18. These components included in the user terminal 100 are electrically connected to each other by a communication bus. Note that the user terminal 100 may include an input / output IF 14 that can connect a display (display unit) configured separately from the user terminal 100 body instead of or in addition to the touch screen 15.

[0023] Also, as shown in FIG. 2, the user terminal 100 may be configured to be communicable with one or more controllers 1020. The controller 1020 establishes communication with the user terminal 100 according to a communication standard such as Bluetooth (registered trademark), for example. The controller 1020 may have one or more buttons or the like, and transmits an output value based on a user input operation on the buttons or the like to the user terminal 100. Further, the controller 1020 may have various sensors such as an acceleration sensor and an angular velocity sensor, and transmits output values of the various sensors to the user terminal 100.

[0024] Note that instead of or in addition to the user terminal 100 including the camera 17 and the distance measuring sensor 18, the controller 1020 may have the camera 17 and the distance measuring sensor 18.

[0025] For example, at the start of a game, it is desirable for the user terminal 100 to cause the user who uses the controller 1020 to input user identification information such as the user's name or login ID via the controller 1020. Thereby, the user terminal 100 can associate the controller 1020 with the user, and based on the transmission source (controller 1020) of the received output value, it can identify which user the output value belongs to.

[0026] When the user terminal 100 communicates with a plurality of controllers 1020, by each user grasping each controller 1020, it is possible to realize multiplayer on the single user terminal 100 without communicating with other devices such as the server 200 via the network 2. Also, by each user terminal 100 communicating with each other according to a wireless standard such as the wireless LAN (Local Area Network) standard (communicating without going through the server 200), it is also possible to realize multiplayer locally with a plurality of user terminals 100. When realizing the above-mentioned multiplayer locally with a single user terminal 100, the user terminal 100 may further include at least a part of various functions described later provided by the server 200. Also, when realizing the above-mentioned multiplayer locally with a plurality of user terminals 100, the plurality of user terminals 100 may separately include various functions described later provided by the server 200.

[0027] Note that even when realizing the above-mentioned multiplayer locally, the user terminal 100 may communicate with the server 200. For example, information indicating a play result such as a score or win / loss in a certain game and the user identification information may be associated and transmitted to the server 200.

[0028] Further, the controller 1020 may be configured to be detachable from the user terminal 100. In this case, a coupling portion with the controller 1020 may be provided on at least one of the surfaces of the housing of the user terminal 100. When the user terminal 100 and the controller 1020 are coupled by wire via the coupling portion, the user terminal 100 and the controller 1020 transmit and receive signals via the wire.

[0029] As shown in FIG. 2, the user terminal 100 may receive the attachment of a storage medium 1030 such as an external memory card via the input / output IF 14. Thereby, the user terminal 100 can read the programs and data recorded on the storage medium 1030. The program recorded on the storage medium 1030 is, for example, a game program.

[0030] The user terminal 100 may store a game program acquired by communicating with an external device such as the server 200 in the memory 11 of the user terminal 100, or may store a game program acquired by reading from the storage medium 1030 in the memory 11.

[0031] As described above, the user terminal 100 includes a communication IF 13, an input / output IF 14, a touch screen 15, a camera 17, and a distance measuring sensor 18 as an example of a mechanism for inputting information to the user terminal 100. Each of the above-described units as an input mechanism can be regarded as an operation unit configured to receive a user's input operation.

[0032] For example, when the operation unit is composed of at least one of the camera 17 and the distance measuring sensor 18, the operation unit detects an object 1010 near the user terminal 100, and specifies an input operation from the detection result of the object. As an example, a user's hand as the object 1010, a marker with a predetermined shape, etc. are detected, and the input operation is specified based on the color, shape, movement, or type of the object 1010 obtained as the detection result. More specifically, when the user's hand is detected from the captured image of the camera 17 in the user terminal 100, the user terminal 100 specifies and accepts a gesture (a series of movements of the user's hand) detected based on the captured image as the user's input operation. Note that the captured image may be a still image or a moving image.

[0033] Alternatively, when the operation unit is composed of the touch screen 15, the user terminal 100 specifies and accepts the user's operation performed on the input unit 151 of the touch screen 15 as the user's input operation. Alternatively, when the operation unit is composed of the communication IF13, the user terminal 100 specifies and accepts a signal (for example, an output value) transmitted from the controller 1020 as the user's input operation. Alternatively, when the operation unit is composed of the input / output IF14, the user terminal 100 specifies and accepts a signal output from an input device (not shown) different from the controller 1020 connected to the input / output IF14 as the user's input operation.

[0034] (Server 200) The server 200 may be a general-purpose computer such as a workstation or a personal computer, for example. The server 200 includes a processor 20, a memory 21, a storage 22, a communication IF 23, and an input / output IF 24. These components included in the server 200 are electrically connected to each other by a communication bus.

[0035] (Game play terminal 300) The game play terminal 300 may be, for example, a general-purpose computer such as a personal computer. The game play terminal 300 includes a processor 30, a memory 31, a storage 32, a communication IF 33, and an input / output IF 34. These components included in the game play terminal 300 are electrically connected to each other by a communication bus.

[0036] As shown in FIG. 4, the game play terminal 300 according to the present embodiment is included in an HMD (Head Mounted Display) set 1000 as an example. That is, it can be expressed that the HMD set 1000 is included in the system 1, and it can also be expressed that the player plays a game using the HMD set 1000. Note that the device for the player to play the game is not limited to the HMD set 1000. As an example, the device may be any device capable of allowing the player to virtually experience the game. Further, the device may be realized as a smartphone, a feature phone, a tablet computer, a laptop computer (so-called notebook personal computer), or a desktop computer. Further, the device may be a game device suitable for game play.

[0037] The HMD set 1000 includes, in addition to the game play terminal 300, an HMD 500, an HMD sensor 510, a motion sensor 520, a display 530, and a controller 540. The HMD 500 includes a monitor 51, a gaze sensor 52, a first camera 53, a second camera 54, a microphone 55, and a speaker 56. The controller 540 may include the motion sensor 520.

[0038] The HMD500 can be worn on the player's head and provide a virtual space to the player during operation. More specifically, the HMD500 respectively displays an image for the right eye and an image for the left eye on the monitor 51. When each eye of the player views the respective image, the player can recognize the image as a three-dimensional image based on the binocular parallax. The HMD500 can include either a so-called head-mounted display equipped with a monitor or a head-mounted device that can be worn with a terminal having a monitor such as a smartphone.

[0039] The monitor 51 is realized, for example, as a non-transmissive display device. In one aspect, the monitor 51 is disposed on the main body of the HMD500 so as to be located in front of both eyes of the player. Therefore, when the player views the three-dimensional image displayed on the monitor 51, the player can immerse himself in the virtual space. In one aspect, the virtual space includes, for example, a background, an object that the player can operate, and an image of a menu that the player can select. In one aspect, the monitor 51 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor provided in a so-called smartphone or other information display terminal.

[0040] In another aspect, the monitor 51 can be realized as a transmissive display device. In this case, the HMD500 can be an open type such as a glasses type instead of a sealed type that covers the player's eyes as shown in FIG. 1. The transmissive monitor 51 can be configured as a non-transmissive display device temporarily by adjusting its transmittance. The monitor 51 may include a configuration for simultaneously displaying a part of the image constituting the virtual space and the real space. For example, the monitor 51 may display an image of the real space captured by a camera mounted on the HMD500, or may make the real space visible by setting a part of the transmittance to be high.

[0041] In a certain aspect, the monitor 51 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, the monitor 51 may be configured to integrally display the image for the right eye and the image for the left eye. In this case, the monitor 51 includes a high-speed shutter. The high-speed shutter operates to alternately display the image for the right eye and the image for the left eye so that the image is recognized only by one of the eyes.

[0042] In a certain aspect, the HMD 500 includes a plurality of light sources (not shown). Each light source is realized, for example, by an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 510 has a position tracking function for detecting the movement of the HMD 500. More specifically, the HMD sensor 510 reads a plurality of infrared rays emitted by the HMD 500 and detects the position and inclination of the HMD 500 in the real space.

[0043] In another aspect, the HMD sensor 510 may be realized by a camera. In this case, the HMD sensor 510 can detect the position and inclination of the HMD 500 by executing image analysis processing using the image information of the HMD 500 output from the camera.

[0044] In another aspect, the HMD 500 may be provided with a sensor (not shown) as a position detector instead of or in addition to the HMD sensor 510. The HMD 500 can detect its own position and inclination using the sensor. For example, when the sensor is an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor, the HMD 500 can detect its own position and inclination using any of these sensors instead of the HMD sensor 510. As an example, when the sensor provided in the HMD 500 is an angular velocity sensor, the angular velocity sensor detects the angular velocity around the three axes of the HMD 500 in the real space over time. The HMD 500 calculates the temporal change of the angle around the three axes of the HMD 500 based on each angular velocity, and further calculates the inclination of the HMD 500 based on the temporal change of the angle.

[0045] The gaze sensor 52 detects the direction in which the player's right and left eyes are directed. That is, the gaze sensor 52 detects the player's line of sight. The detection of the direction of the line of sight is realized, for example, by a known eye tracking function. The gaze sensor 52 is realized by a sensor having the eye tracking function. In a certain aspect, it is preferable that the gaze sensor 52 includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 52 may be, for example, a sensor that irradiates infrared light onto the player's right and left eyes and detects the rotation angle of each eyeball by receiving the reflected light from the cornea and iris with respect to the irradiated light. The gaze sensor 52 can detect the player's line of sight based on the detected rotation angles.

[0046] The first camera 53 photographs the lower part of the player's face. More specifically, the first camera 53 photographs the player's nose and mouth, etc. The second camera 54 photographs the player's eyes and eyebrows, etc. The player-side housing of the HMD 500 is defined as the inside of the HMD 500, and the housing of the HMD 500 on the side opposite to the player is defined as the outside of the HMD 500. In a certain aspect, the first camera 53 may be arranged outside the HMD 500, and the second camera 54 may be arranged inside the HMD 500. The images generated by the first camera 53 and the second camera 54 are input to the game play terminal 300. In another aspect, the first camera 53 and the second camera 54 may be realized as a single camera, and this single camera may photograph the player's face.

[0047] The microphone 55 converts the player's speech into an audio signal (electrical signal) and outputs it to the game play terminal 300. The speaker 56 converts the audio signal into sound and outputs it to the player. In another aspect, the HMD 500 may include earphones instead of the speaker 56.

[0048] The controller 540 is connected to the game play terminal 300 either by wire or wirelessly. The controller 540 receives input of commands from the player to the game play terminal 300. In one aspect, the controller 540 is configured to be grippable by the player. In another aspect, the controller 540 is configured to be wearable on a part of the player's body or clothing. In yet another aspect, the controller 540 may be configured to output at least any one of vibration, sound, and light based on a signal transmitted from the game play terminal 300. In yet another aspect, the controller 540 receives an operation from the player to control the position and movement of an object placed in the virtual space.

[0049] In one aspect, the controller 540 includes a plurality of light sources. Each light source is realized, for example, by an LED that emits infrared rays. The HMD sensor 510 has a position tracking function. In this case, the HMD sensor 510 reads a plurality of infrared rays emitted by the controller 540 and detects the position and inclination of the controller 540 in the real space. In another aspect, the HMD sensor 510 may be realized by a camera. In this case, the HMD sensor 510 can detect the position and inclination of the controller 540 by executing image analysis processing using the image information of the controller 540 output from the camera.

[0050] In certain situations, the motion sensor 520 is attached to the player's hand to detect the movement of the player's hand. For example, the motion sensor 520 detects the rotation speed, rotation count, etc. of the hand. The detected signal is sent to the game play terminal 300. The motion sensor 520 is provided, for example, on the controller 540. In certain situations, the motion sensor 520 is provided, for example, on the controller 540 configured to be grippable by the player. In another situation, for safety in the real space, the controller 540 is worn on the player's hand like a glove so that it does not easily fly away. In yet another situation, a sensor not worn by the player may detect the movement of the player's hand. For example, the signal of a camera that captures the player may be input to the game play terminal 300 as a signal representing the player's motion. The motion sensor 520 and the game play terminal 300 are connected to each other wirelessly as an example. In the case of wireless connection, the communication form is not particularly limited, and for example, Bluetooth or other known communication methods are used.

[0051] The display 530 displays an image similar to the image displayed on the monitor 51. Thereby, a user other than the player wearing the HMD 500 can also view an image similar to that of the player. The image displayed on the display 530 does not have to be a three-dimensional image, and may be an image for the right eye or an image for the left eye. Examples of the display 530 include a liquid crystal display and an organic EL monitor.

[0052] The game play terminal 300 operates a character to be an operation target of the player and advances the game based on various information acquired from each part of the HMD 500, the controller 540, and the motion sensor 520. Here, "operation" includes moving each part of the body, changing the posture, changing the facial expression, moving, speaking, touching or moving an object arranged in the virtual space, using a weapon, tool, etc. held by the character, and the like. That is, in this game, when the player moves each part of the body, the character also moves each part of the body in the same manner as the player. Also, in this game, the character speaks the content spoken by the player. In other words, in this game, the character is an avatar object that acts as a clone of the player. As an example, at least a part of the operation of the character may be executed by an input to the controller 540 by the player.

[0053] In the present embodiment, as an example, the motion sensor 520 is attached to both hands of the player, both feet of the player, the waist of the player, and the head of the player. The motion sensor 520 attached to both hands of the player may be provided in the controller 540 as described above. Also, the motion sensor 520 attached to the head of the player may be provided in the HMD 500. The motion sensor 520 may be further attached to both elbows and both knees of the user. By increasing the number of motion sensors 520 attached to the player, the movement of the player can be more accurately reflected in the character. Also, instead of attaching the motion sensor 520 to each part of the body, the player may wear a suit to which one or more motion sensors 520 are attached. That is, the method of motion capture is not limited to the example using the motion sensor 520.

[0054] (Distribution terminal 400) The distribution terminal 400 may be a mobile terminal such as a smartphone, a PDA (Personal Digital Assistant), or a tablet computer. Further, the distribution terminal 400 may be a so-called stationary terminal such as a desktop personal computer.

[0055] As shown in FIG. 5, the distribution terminal 400 includes a processor 40, a memory 41, a storage 42, a communication IF 43, an input / output IF 44, and a touch screen 45. Note that the distribution terminal 400 may include an input / output IF 44 that can connect a display (display unit) configured separately from the distribution terminal 400 itself, instead of or in addition to the touch screen 45.

[0056] The controller 1021 may have a physical input mechanism such as one or more buttons, levers, sticks, wheels, etc. The controller 1021 transmits an output value based on an input operation input by an operator (player in this embodiment) of the distribution terminal 400 to the input mechanism to the distribution terminal 400. Further, the controller 1021 may have various sensors such as an acceleration sensor and an angular velocity sensor, and may transmit output values of the various sensors to the distribution terminal 400. The above output values are received by the distribution terminal 400 via the communication IF 43.

[0057] The distribution terminal 400 may include a camera and a distance measuring sensor (both not shown). Instead of or in addition to the distribution terminal 400 including them, the controller 1021 may have a camera and a distance measuring sensor.

[0058] As described above, the distribution terminal 400 includes a communication IF 43, an input / output IF 44, and a touch screen 45 as an example of a mechanism for inputting information to the distribution terminal 400. Each of the above-described parts as an input mechanism can be regarded as an operation unit configured to receive a user's input operation.

[0059] When the operation unit is constituted by the touch screen 45, the distribution terminal 400 identifies and accepts the user operation performed on the input unit 451 of the touch screen 45 as the user input operation. Alternatively, when the operation unit is constituted by the communication IF 43, the distribution terminal 400 identifies and accepts the signal (for example, output value) transmitted from the controller 1021 as the user input operation. Alternatively, when the operation unit is constituted by the input / output IF 44, the distribution terminal 400 identifies and accepts the signal output from an input device (not shown) connected to the input / output IF 44 as the user input operation.

[0060] <Hardware Components of Each Device> The processors 10, 20, 30, 40 control the overall operations of the user terminal 100, the server 200, the game play terminal 300, and the distribution terminal 400, respectively. The processors 10, 20, 30, 40 include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). The processors 10, 20, 30, 40 read programs from the storages 12, 22, 32, 42, which will be described later, respectively. Then, the processors 10, 20, 30, 40 expand the read programs in the memories 11, 21, 31, 41, which will be described later, respectively. The processors 10, 20, 30 execute the expanded programs.

[0061] The memories 11, 21, 31, and 41 are main memory devices. The memories 11, 21, 31, and 41 are composed of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). The memory 11 provides a working area for the processor 10 by temporarily storing the programs and various data read by the processor 10 from the storage 12 described later. The memory 11 also temporarily stores various data generated while the processor 10 is operating according to the program. The memory 21 provides a working area for the processor 20 by temporarily storing the various programs and data read by the processor 20 from the storage 22 described later. The memory 21 also temporarily stores various data generated while the processor 20 is operating according to the program. The memory 31 provides a working area for the processor 30 by temporarily storing the various programs and data read by the processor 30 from the storage 32 described later. The memory 31 also temporarily stores various data generated while the processor 30 is operating according to the program. The memory 41 provides a working area for the processor 40 by temporarily storing the programs and various data read by the processor 40 from the storage 42 described later. The memory 41 also temporarily stores various data generated while the processor 40 is operating according to the program.

[0062] In this embodiment, the programs executed by the processors 10 and 30 may be game programs of this game. In this embodiment, the program executed by the processor 40 may be a distribution program for realizing the distribution of operation instruction data. Further, the processor 10 may further execute a viewing program for realizing the playback of video.

[0063] In this embodiment, the program executed by the processor 20 may be at least any one of the above-described game program, distribution program, and viewing program. The processor 20 executes at least any one of the game program, distribution program, and viewing program in response to a request or the like from at least any one of the user terminal 100, the game play terminal 300, and the distribution terminal 400. Note that the distribution program and the viewing program may be executed in parallel.

[0064] That is, the game program may be a program that realizes a game through the cooperation of the user terminal 100, the server 200, and the game play terminal 300. The distribution program may be a program that realizes the distribution of operation instruction data through the cooperation of the server 200 and the distribution terminal 400. The viewing program may be a program that realizes the playback of a video through the cooperation of the user terminal 100 and the server 200.

[0065] The storages 12, 22, 32, and 42 are auxiliary storage devices. The storages 12, 22, 32, and 42 are configured by storage devices such as flash memories or HDDs (Hard Disk Drives). Various data related to games, for example, are stored in the storages 12 and 32. Various data related to the distribution of operation instruction data are stored in the storage 42. In addition, various data related to the playback of videos are stored in the storage 12. At least a part of various data related to each of the game, the distribution of operation instruction data, and the playback of videos may be stored in the storage 22.

[0066] The communication IFs 13, 23, 33, and 43 control the transmission and reception of various data in the user terminal 100, the server 200, the game play terminal 300, and the distribution terminal 400, respectively. The communication IFs 13, 23, 33, and 43 control communication using, for example, communication via a wireless LAN (Local Area Network), a wired LAN, a wireless LAN, or Internet communication via a mobile phone line network, and communication using short-range wireless communication or the like.

[0067] The input / output IFs 14, 24, 34, and 44 are interfaces for the user terminal 100, the server 200, the game play terminal 300, and the distribution terminal 400 to receive data input and output data, respectively. The input / output IFs 14, 24, 34, and 44 may perform data input and output via USB (Universal Serial Bus) or the like. The input / output IFs 14, 24, 34, and 44 may include physical buttons, cameras, microphones, speakers, mice, keyboards, displays, sticks, levers, and the like. Further, the input / output IFs 14, 24, 34, and 44 may include a connection part for transmitting and receiving data to and from peripheral devices.

[0068] The touch screen 15 is an electronic component combining an input part 151 and a display part 152 (display). The touch screen 45 is an electronic component combining an input part 451 and a display part 452. The input parts 151 and 451 are, as an example, touch-sensitive devices, and are constituted by, for example, a touch pad. The display parts 152 and 452 are constituted by, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like.

[0069] The input parts 151 and 451 have a function of detecting the position where a user's operation (mainly physical contact operations such as a touch operation, a slide operation, a swipe operation, a pinch-in / pinch-out operation, and a tap operation) is input with respect to the input surface, and transmitting information indicating the position as an input signal. The input parts 151 and 451 only need to include a touch sensing part (not shown). The touch sensing part may adopt any method such as a capacitance method or a resistive film method.

[0070] Although not shown, the user terminal 100 and the distribution terminal 400 may each include one or more sensors for specifying the holding postures of the user terminal 100 and the distribution terminal 400. This sensor may be, for example, an acceleration sensor or an angular velocity sensor.

[0071] When the user terminal 100 and the distribution terminal 400 are equipped with sensors, the processors 10 and 40 can each identify the holding postures of the user terminal 100 and the distribution terminal 400 from the outputs of the sensors and perform processing according to the holding postures. For example, when the user terminal 100 and the distribution terminal 400 are held vertically, the processors 10 and 40 may perform vertical screen display that causes a vertically long image to be displayed on the display units 152 and 452. On the other hand, when the user terminal 100 and the distribution terminal 400 are held horizontally, horizontal screen display that causes a horizontally long image to be displayed on the display unit may be performed. In this way, the processors 10 and 40 may each be able to switch between vertical screen display and horizontal screen display according to the holding postures of the user terminal 100 and the distribution terminal 400.

[0072] <Functional Configuration of System 1> FIG. 6 is a block diagram showing the functional configurations of the user terminal 100, the server 200, and the HMD set 1000 included in System 1. FIG. 7 is a block diagram showing the functional configuration of the distribution terminal 400 shown in FIG. 6.

[0073] The user terminal 100 has a function as an input device that receives a user's input operation and a function as an output device that outputs game images and sounds. The user terminal 100 functions as a control unit 110 and a storage unit 120 through the cooperation of a processor 10, a memory 11, a storage 12, a communication IF 13, an input / output IF 14, a touch screen 15, and the like.

[0074] The server 200 has a function of mediating the transmission and reception of various information among the user terminal 100, the HMD set 1000, and the distribution terminal 400. The server 200 functions as a control unit 210 and a storage unit 220 through the cooperation of a processor 20, a memory 21, a storage 22, a communication IF 23, and an input / output IF 24.

[0075] The HMD set 1000 (game play terminal 300) has functions as an input device for receiving a player's input operations, as an output device for outputting game images and sounds, and for transmitting game progress information to the user terminal 100 in real time via the server 200. The HMD set 1000 functions as a control unit 310 and a storage unit 320 through the cooperation of the processor 30, memory 31, storage 32, communication IF 33, input / output IF 34 of the game play terminal 300, and the HMD 500, HMD sensor 510, motion sensor 520, and controller 540, etc.

[0076] The distribution terminal 400 has a function of generating operation instruction data and transmitting the operation instruction data to the user terminal 100 via the server 200. The distribution terminal 400 functions as a control unit 410 and a storage unit 420 through the cooperation of the processor 40, memory 41, storage 42, communication IF 43, input / output IF 44, and touch screen 45, etc.

[0077] (Data stored in the storage unit of each device) The storage unit 120 stores a game program 131 (program), game information 132, and user information 133. The storage unit 220 stores a game program 231, game information 232, user information 233, and user list 234. The storage unit 320 stores a game program 331, game information 332, and user information 333. The storage unit 420 stores a user list 421, motion list 422, and distribution program 423 (program, second program).

[0078] The game programs 131, 231, and 331 are game programs executed by the user terminal 100, the server 200, and the HMD set 1000, respectively. The game is realized by the cooperation of each device based on the game programs 131, 231, and 331. Note that the game programs 131 and 331 may be stored in the storage unit 220 and downloaded to the user terminal 100 and the HMD set 1000, respectively. In this embodiment, the user terminal 100 renders the data received from the distribution terminal 400 and plays the video based on the game program 131. In other words, the game program 131 is also a program for playing the video using the video instruction data distributed from the distribution terminal 400. The program for playing the video may be different from the game program 131. In this case, the storage unit 120 stores a program for playing the video separately from the game program 131.

[0079] The game information 132, 232, and 332 are data referred to when the user terminal 100, the server 200, and the HMD set 1000 execute the game program, respectively. The user information 133, 233, and 333 are data related to the accounts of the users of the user terminal 100. The game information 232 is the game information 132 of each user terminal 100 and the game information 332 of the HMD set 1000. The user information 233 is the user information of the players included in the user information 133 of each user terminal 100 and the user information 333. The user information 333 is the user information 133 of each user terminal 100 and the user information of the player.

[0080] User list 234 and user list 421 are lists of users who participated in the game. User list 234 and user list 421 may include, in addition to the list of users who participated in the most recent game play by the player, the list of users who participated in each game play before the game play. Motion list 422 is a list of a plurality of pre-created motion data. The motion list 422 is, for example, a list in which motion data is associated with each piece of information (for example, motion name) for identifying each motion. The distribution program 423 is a program for realizing the distribution of operation instruction data for playing a video at the user terminal 100 to the user terminal 100.

[0081] (Functional Configuration of Server 200) The control unit 210 comprehensively controls the server 200 by executing the game program 231 stored in the storage unit 220. For example, the control unit 210 mediates the transmission and reception of various information among the user terminal 100, the HMD set 1000, and the distribution terminal 400.

[0082] The control unit 210 functions as a communication mediation unit 211, a log generation unit 212, and a list generation unit 213 according to the description of the game program 231. The control unit 210 can also function as other functional blocks (not shown) for mediating the transmission and reception of various information related to game play and the distribution of operation instruction data, and for supporting the progress of the game.

[0083] The communication mediation unit 211 mediates the transmission and reception of various types of information among the user terminal 100, the HMD set 1000, and the distribution terminal 400. For example, the communication mediation unit 211 transmits the game progress information received from the HMD set 1000 to the user terminal 100. The game progress information includes data indicating the movement of a character operated by a player, the parameters of the character, the items and weapons possessed by the character, information on enemy characters, and other such information. The server 200 transmits the game progress information to the user terminals 100 of all users participating in the game. In other words, the server 200 transmits the common game progress information to the user terminals 100 of all users participating in the game. As a result, in each of the user terminals 100 of all users participating in the game, the game progresses in the same way as in the HMD set 1000.

[0084] Also, for example, the communication mediation unit 211 transmits the information received from any of the user terminals 100 for assisting the progress of the game by the player to the other user terminals 100 and the HMD set 1000. As an example, the information may be item information indicating an item for the player to advantageously progress the game and provided to the player (character). The item information includes information (such as a user name, user ID, etc.) indicating the user who provided the item. Further, the communication mediation unit 211 may mediate the distribution of the operation instruction data from the distribution terminal 400 to the user terminal 100.

[0085] The log generation unit 212 generates a game progress log based on the game progress information received from the HMD set 1000. The list generation unit 213 generates a user list 234 after the end of the game play. Although details will be described later, each user in the user list 234 is associated with a tag indicating the content of the support provided to the player by that user. The list generation unit 213 generates a tag based on the game progress log generated by the log generation unit 212 and associates it with the corresponding user. Note that the list generation unit 213 may also associate, as a tag, the content of the support provided to the player by each user, which is input by an operator of the game or the like using a terminal device such as a personal computer, with the corresponding user. Thereby, the content of the support provided by each user becomes more detailed. Note that the user terminal 100 transmits information indicating the user to the server 200 based on the operation of the user when the user participates in the game. For example, the user terminal 100 transmits the user ID input by the user to the server 200. That is, the server 200 holds information indicating each user for all users participating in the game. The list generation unit 213 may generate the user list 234 using this information.

[0086] (Functional Configuration of HMD Set 1000) The control unit 310 comprehensively controls the HMD set 1000 by executing the game program 331 stored in the storage unit 320. For example, the control unit 310 advances the game according to the game program 331 and the operation of the player. Further, while advancing the game, the control unit 310 communicates with the server 200 as necessary to transmit and receive information. The control unit 310 may directly communicate with the user terminal 100 to transmit and receive information without going through the server 200.

[0087] The control unit 310 functions as an operation reception unit 311, a display control unit 312, a UI control unit 313, an animation generation unit 314, a game progress unit 315, a virtual space control unit 316, and a reaction processing unit 317 according to the description of the game program 331. The control unit 310 can also function as other functional blocks (not shown) for controlling characters appearing in the game, etc., according to the nature of the game to be executed.

[0088] The operation reception unit 311 detects and receives the input operations of the player. The operation reception unit 311 receives the signals input from the HMD 500, the motion sensor 520, the controller 540, etc., discriminates what input operation has been made, and outputs the result to each element of the control unit 310.

[0089] The UI control unit 313 controls the user interface (hereinafter referred to as UI) images to be displayed on the monitor 51, the display 530, etc. The UI images are tools for the player to perform the necessary inputs for the progress of the game on the HMD set 1000, or tools for obtaining the information output during the progress of the game from the HMD set 1000. The UI images include, but are not limited to, for example, icons, buttons, lists, menu screens, etc.

[0090] The animation generation unit 314 generates animations showing the motions of various objects based on the control modes of the various objects. For example, the animation generation unit 314 may generate animations that express the state of an object (for example, the player's avatar object) moving as if it were actually there, moving its mouth, changing its expression, etc.

[0091] The game progress unit 315 advances the game based on the game program 331, the input operations by the player, and the actions of the avatar object according to the input operations. For example, when the avatar object performs a predetermined action, the game progress unit 315 performs a predetermined game process. Also, for example, the game progress unit 315 may receive information representing the user's operation on the user terminal 100 and perform a game process based on the user's operation. Further, the game progress unit 315 generates game progress information according to the progress of the game and transmits it to the server 200. The game progress information is transmitted to the user terminal 100 via the server 200. Thereby, the progress of the game in the HMD set 1000 is shared at the user terminal 100. In other words, the progress of the game in the HMD set 1000 and the progress of the game in the user terminal 100 are synchronized.

[0092] The virtual space control unit 316 performs various controls regarding the virtual space provided to the player according to the progress of the game. As an example, the virtual space control unit 316 generates various objects and arranges them in the virtual space. Also, the virtual space control unit 316 arranges a virtual camera in the virtual space. Further, the virtual space control unit 316 operates various objects arranged in the virtual space according to the progress of the game. Also, the virtual space control unit 316 controls the position and inclination of the virtual camera arranged in the virtual space according to the progress of the game.

[0093] The display control unit 312 outputs a game screen in which the processing results executed by the above-described respective elements are reflected to the monitor 51 and the display 530. The display control unit 312 may display, as the game screen, an image based on the field of view from the virtual camera arranged in the virtual space on the monitor 51 and the display 530. Also, the display control unit 312 may include the animation generated by the animation generation unit 314 in the game screen. Further, the display control unit 312 may superimpose and draw the above-described UI image controlled by the UI control unit 313 on the game screen.

[0094] The reaction processing unit 317 receives feedback on the player's game play reaction by the user of the user terminal 100, and outputs this to the player. In the present embodiment, for example, the user terminal 100 can create a comment (message) addressed to the avatar object based on the user's input operation. The reaction processing unit 317 receives the comment data of the comment and outputs this. The reaction processing unit 317 may display the text data corresponding to the user's comment on the monitor 51 and the display 530, or may output the voice data corresponding to the user's comment from a speaker (not shown). In the former case, the reaction processing unit 317 may superimpose and draw an image corresponding to the text data (that is, an image including the content of the comment) on the game screen.

[0095] (Functional configuration of user terminal 100) The control unit 110 comprehensively controls the user terminal 100 by executing the game program 131 stored in the storage unit 120. For example, the control unit 110 advances the game according to the game program 131 and the user's operation. Also, while advancing the game, the control unit 110 communicates with the server 200 as necessary to transmit and receive information. The control unit 110 may directly perform the transmission and reception of information with the HMD set 1000 without going through the server 200.

[0096] The control unit 110 functions as an operation reception unit 111, a display control unit 112, a UI control unit 113, an animation generation unit 114, a game progress unit 115, a virtual space control unit 116, and a video playback unit 117 according to the description of the game program 131. The control unit 110 can also function as other functional blocks (not shown) for the progress of the game according to the nature of the game being executed.

[0097] The operation reception unit 111 detects and accepts the user's input operation on the input unit 151. The operation reception unit 111 determines what input operation has been made from the actions exerted by the user on the console via the touch screen 15 and other input / output IFs 14, and outputs the result to each element of the control unit 110.

[0098] For example, the operation reception unit 111 accepts an input operation on the input unit 151, detects the coordinates of the input position of the input operation, and identifies the type of the input operation. As the type of the input operation, the operation reception unit 111 identifies, for example, a touch operation, a slide operation, a swipe operation, a pinch-in / pinch-out operation, and a tap operation, etc. Also, when the continuously detected input is interrupted, the operation reception unit 111 detects that the touch input has been released from the touch screen 15.

[0099] The UI control unit 113 controls the UI image to be displayed on the display unit 152 in order to construct the UI according to at least one of the user's input operation and the received game progress information. The UI image is a tool for the user to perform the input necessary for the progress of the game on the user terminal 100, or a tool for the user to obtain the information output during the progress of the game from the user terminal 100. The UI image includes, but is not limited to, for example, icons, buttons, lists, menu screens, etc.

[0100] The animation generation unit 114 generates an animation showing the motion of various objects based on the control modes of the various objects.

[0101] The game progress unit 115 progresses the game based on the game program 131, the received game progress information, and input operations by the user. When the game progress unit 115 performs a predetermined game process due to an input operation by the user, it transmits information regarding the game process to the HMD set 1000 via the server 200. As a result, the predetermined game process is shared in the HMD set 1000. In other words, the progress of the game in the HMD set 1000 and the progress of the game in the user terminal 100 are synchronized. The predetermined game process is, for example, a process of providing an item to an avatar object, and in this example, the information regarding the game process is the item information described above.

[0102] The virtual space control unit 116 performs various controls regarding the virtual space provided to the user according to the progress of the game. As an example, the virtual space control unit 116 generates various objects and arranges them in the virtual space. Also, the virtual space control unit 116 arranges a virtual camera in the virtual space. Further, the virtual space control unit 116 operates various objects arranged in the virtual space according to the progress of the game, specifically, the received game progress information. Also, the virtual space control unit 316 controls the position and inclination of the virtual camera arranged in the virtual space according to the progress of the game, specifically, the received game progress information.

[0103] The display control unit 112 outputs a game screen in which the processing results executed by the above-described respective elements are reflected to the display unit 152. The display control unit 112 may display, as the game screen, an image based on the field of view from the virtual camera arranged in the virtual space provided to the user on the display unit 152. Also, the display control unit 112 may include the animation generated by the animation generation unit 114 in the game screen. Further, the display control unit 112 may superimpose and draw the above-described UI image controlled by the UI control unit 113 on the game screen. In any case, the game screen displayed on the display unit 152 is the same game screen as the game screens displayed on other user terminals 100 and the HMD set 1000.

[0104] The video playback unit 117 analyzes (renders) the operation instruction data received from the distribution terminal 400 and plays back the video.

[0105] (Functional Configuration of Distribution Terminal 400) The control unit 410 comprehensively controls the distribution terminal 400 by executing a program (not shown) stored in the storage unit 420. For example, the control unit 410 generates operation instruction data according to the program and the operations of the user (player in this embodiment) of the distribution terminal 400, and distributes it to the user terminal 100. In addition, the control unit 410 communicates with the server 200 as necessary to transmit and receive information. The control unit 410 may directly communicate with the user terminal 100 without going through the server 200 to transmit and receive information.

[0106] The control unit 410 functions as a communication control unit 411, a display control unit 412, an operation reception unit 413, an audio reception unit 414, a motion identification unit 415, and an operation instruction data generation unit 416 according to the description of the program. The control unit 410 can also function as other functional blocks (not shown) for generating and distributing operation instruction data.

[0107] The communication control unit 411 controls the transmission and reception of information with the server 200 or the user terminal 100 via the server 200. As an example, the communication control unit 411 receives a user list 421 from the server 200. In addition, as an example, the communication control unit 411 transmits operation instruction data to the user terminal 100.

[0108] The display control unit 412 outputs various screens to the display unit 452 on which the processing results executed by each element are reflected. As an example, the display control unit 412 displays a screen including the received user list 234. In addition, as an example, the display control unit 412 displays a screen including a motion list 422 for allowing the player to select motion data for operating an avatar object included in the operation instruction data to be distributed.

[0109] The operation reception unit 413 detects and receives the player's input operation on the input unit 151. The operation reception unit 111 determines what input operation has been made from the actions exerted by the player on the console via the touch screen 45 and other input / output IFs 44, and outputs the result to each element of the control unit 410.

[0110] For example, the operation reception unit 413 receives an input operation on the input unit 451, detects the coordinates of the input position of the input operation, and identifies the type of the input operation. As the type of the input operation, the operation reception unit 413 identifies, for example, a touch operation, a slide operation, a swipe operation, a pinch-in / pinch-out operation, and a tap operation, etc. Further, when the continuously detected input is interrupted, the operation reception unit 413 detects that the contact input has been released from the touch screen 45.

[0111] The voice reception unit 414 receives the voice generated around the distribution terminal 400 and generates the voice data of the voice. As an example, the voice reception unit 414 receives the voice uttered by the player and generates the voice data of the voice.

[0112] The motion identification unit 415 identifies the motion data selected by the player from the motion list 422 according to the player's input operation.

[0113] The operation instruction data generation unit 416 generates operation instruction data. As an example, the operation instruction data generation unit 416 generates operation instruction data including the generated voice data and the identified motion data.

[0114] Note that the functions of the HMD set 1000, server 200, and user terminal 100 shown in FIG. 6, and the functions of the distribution terminal 400 shown in FIG. 7 are merely examples. Each of the devices of the HMD set 1000, server 200, user terminal 100, and distribution terminal 400 may include at least a part of the functions provided in other devices. Further, yet another device other than the HMD set 1000, server 200, user terminal 100, and distribution terminal 400 may be a component of the system 1, and the other device may execute a part of the processing in the system 1. That is, the computer that executes the game program in the present embodiment may be any one of the HMD set 1000, server 200, user terminal 100, and distribution terminal 400, or may be realized by a combination of a plurality of these devices.

[0115] <Control Processing of Virtual Space> FIG. 8 is a flowchart showing an example of the flow of control processing of the virtual space provided to the player and the virtual space provided to the user of the user terminal 100. FIG. 9 is a diagram showing a virtual space 600A provided to the player and a field-of-view image viewed by the player according to an embodiment. FIG. 10 is a diagram showing a virtual space 600B provided to the user of the user terminal 100 and a field-of-view image viewed by the user according to an embodiment. Hereinafter, when there is no need to distinguish between the virtual spaces 600A and 600B, they will be referred to as "virtual space 600".

[0116] In step S1, the processor 30, as the virtual space control unit 316, defines the virtual space 600A shown in FIG. 9. The processor 30 defines the virtual space 600A using virtual space data (not shown). The virtual space data may be stored in the game play terminal 300, may be generated by the processor 30 based on the game program 331, or may be acquired by the processor 30 from an external device such as the server 200.

[0117] The virtual space 600 has, as an example, an all-sky spherical structure that covers the entire 360-degree direction of a point defined as the center. In FIGS. 9 and 10, for the sake of simplicity of explanation, the upper half of the celestial sphere of the virtual space 600 is illustrated.

[0118] In step S2, the processor 30, as the virtual space control unit 316, arranges the avatar object 610 (character) in the virtual space 600A. The avatar object 610 is an avatar object associated with the player and operates according to the input operations of the player.

[0119] In step S3, the processor 30, as the virtual space control unit 316, arranges other objects in the virtual space 600A. In the example of FIG. 9, the processor 30 arranges the objects 631 to 634. Other objects may include, for example, character objects (so-called non-player characters, NPCs) that operate according to the game program 331, operation objects such as virtual hands, and objects imitating animals, plants, artifacts, natural objects, etc. that are arranged according to the progress of the game.

[0120] In step S4, the processor 30, as the virtual space control unit 316, arranges the virtual camera 620A in the virtual space 600A. The processor 30, as an example, arranges the virtual camera 620A at the position of the head of the avatar object 610.

[0121] In step S5, the processor 30 displays the view image 650 on the monitor 51 and the display 530. The processor 30 defines a viewing area 640A that is the view from the virtual camera 620A in the virtual space 600A according to the initial position and inclination of the virtual camera 620A. Then, the processor 30 defines a view image 650 corresponding to the viewing area 640A. The processor 30 outputs the view image 650 to the monitor 51 and the display 530 to display the view image 650 on the HMD 500 and the display 530.

[0122] In the example of FIG. 9, as shown in FIG. 9(A), since a part of the object 634 is included in the viewing area 640A, the viewing image 650 includes a part of the object 634 as shown in FIG. 9(B).

[0123] In step S6, the processor 30 transmits the initial arrangement information to the user terminal 100 via the server 200. The initial arrangement information is information indicating the initial arrangement positions of various objects in the virtual space 600A. In the example of FIG. 9, the initial arrangement information includes the initial arrangement position information of the avatar object 610 and the objects 631 to 634. The initial arrangement information can also be expressed as one of the game progress information.

[0124] In step S7, the processor 30 controls the virtual camera 620A according to the movement of the HMD500 as the virtual space control unit 316. Specifically, the processor 30 controls the direction and tilt of the virtual camera 620A according to the movement of the HMD500, that is, the posture of the player's head. As will be described later, when the player moves the head (changes the posture of the head), the processor 30 moves the head of the avatar object 610 in accordance with this movement. The processor 30 controls the direction and tilt of the virtual camera 620A so that, for example, the direction of the line of sight of the avatar object 610 coincides with the direction of the line of sight of the virtual camera 620A. In step S8, the processor 30 updates the viewing image 650 in response to the change in the direction and tilt of the virtual camera 620A.

[0125] In step S9, the processor 30 moves the avatar object 610 according to the movement of the player as the virtual space control unit 316. As an example, the processor 30 moves the avatar object 610 in the virtual space 600A in response to the player's movement in the real space. In addition, the processor 30 moves the head of the avatar object 610 in the virtual space 600A in response to the player's movement of the head in the real space.

[0126] In step S10, as the virtual space control unit 316, the processor 30 moves the virtual camera 620A so as to follow the avatar object 610. That is, even when the avatar object 610 moves, the virtual camera 620A is always at the position of the head of the avatar object 610.

[0127] The processor 30 updates the field of view image 650 in accordance with the movement of the virtual camera 620A. That is, the processor 30 updates the field of view area 640A in accordance with the posture of the player's head and the position of the virtual camera 620A in the virtual space 600A. As a result, the field of view image 650 is updated.

[0128] In step S11, the processor 30 transmits the operation instruction data of the avatar object 610 to the user terminal 100 via the server 200. The operation instruction data here includes at least any one of motion data capturing the player's actions, voice data of the voice spoken by the player, and operation data indicating the content of the input operation to the controller 540 during virtual experience (for example, during game play). When the player is playing a game, the operation instruction data is transmitted to the user terminal 100 as, for example, game progress information.

[0129] The processes of steps S7 to S11 are continuously and repeatedly executed while the player is playing the game.

[0130] In step S21, the processor 10 of the user terminal 100 of user 3 defines the virtual space 600B shown in FIG. 10 as the virtual space control unit 116. The processor 10 defines the virtual space 600B using virtual space data (not shown). The virtual space data may be stored in the user terminal 100, may be generated by the processor 10 based on the game program 131, or may be acquired by the processor 10 from an external device such as the server 200.

[0131] In step S22, the processor 10 receives initial arrangement information. In step S23, as the virtual space control unit 116, the processor 10 arranges various objects in the virtual space 600B according to the initial arrangement information. In the case of the example in FIG. 10, the various objects are the avatar object 610 and the objects 631 to 634.

[0132] In step S24, as the virtual space control unit 116, the processor 10 arranges the virtual camera 620B in the virtual space 600B. As an example, the processor 10 arranges the virtual camera 620B at the position shown in FIG. 10(A).

[0133] In step S25, the processor 10 displays the view image 660 on the display unit 152. The processor 10 defines a viewing area 640B, which is the view from the virtual camera 620B in the virtual space 600B, according to the initial position and inclination of the virtual camera 620B. Then, the processor 10 defines a view image 660 corresponding to the viewing area 640B. The processor 10 causes the view image 660 to be displayed on the display unit 152 by outputting the view image 660 to the display unit 152.

[0134] In the example of FIG. 10, as shown in FIG. 10(A), since the avatar object 610 and the object 631 are included in the viewing area 640B, the view image 660 includes the avatar object 610 and the object 631 as shown in FIG. 10(B).

[0135] In step S26, the processor 10 receives operation instruction data. In step S27, as the virtual space control unit 116, the processor 10 moves the avatar object 610 in the virtual space 600B according to the operation instruction data. In other words, the processor 10 plays back a video in which the avatar object 610 is operating by real-time rendering.

[0136] In step S28, the processor 10 controls the virtual camera 620B as the virtual space control unit 116 according to the user operation received by the operation reception unit 111. In step S29, the processor 10 updates the field of view image 660 in response to the change in the position, orientation, and tilt of the virtual camera 620B in the virtual space 600B. Note that in step S28, the processor 10 may automatically control the virtual camera 620B according to the movement of the avatar object 610, for example, the movement or change in orientation of the avatar object 610. For example, the processor 10 may automatically move the virtual camera 620B or change its orientation and tilt so as to always photograph the avatar object 610 from the front. Also, as an example, the processor 10 may automatically move the virtual camera 620B or change its orientation and tilt so as to always photograph the avatar object 610 from the rear according to the movement of the avatar object 610.

[0137] In this way, in the virtual space 600A, the avatar object 610 operates according to the movement of the player. The movement instruction data indicating this movement is transmitted to the user terminal 100. In the virtual space 600B, the avatar object 610 operates according to the received movement instruction data. As a result, in the virtual space 600A and the virtual space 600B, the avatar object 610 performs the same operation. In other words, the user 3 can visually recognize the operation of the avatar object 610 according to the movement of the player using the user terminal 100.

[0138] <Game Overview> FIG. 11 is a diagram showing another example of the field of view image displayed on the user terminal 100. Specifically, it is a diagram showing an example of the game screen of a game (this game) executed by the system 1 that the player is playing.

[0139] As an example, this game features an avatar object 610 that operates weapons such as guns and knives, and a plurality of enemy objects 671 that are NPCs, which appear in a virtual space 600, and the avatar object 610 is made to engage in battles with the enemy objects 671. Various game parameters such as the physical strength of the avatar object 610, the number of available magazines, the remaining ammunition of the gun, and the remaining number of enemy objects 671 are updated according to the progress of the game.

[0140] This game has a plurality of stages prepared, and the player can clear a stage by fulfilling a predetermined achievement condition associated with each stage. The predetermined achievement conditions may include, for example, conditions established by defeating all of the enemy objects 671 that appear, defeating the boss object among the enemy objects 671 that appear, obtaining a predetermined item, reaching a predetermined position, and so on. The achievement conditions are defined within the game program 131. In this game, in accordance with the content of the game, when the achievement conditions are met, the player clears the stage. In other words, the victory of the avatar object 610 over the enemy object 671 (the win or loss between the avatar object 610 and the enemy object 671) is determined. In contrast, for example, when the game executed by the system 1 is a racing game or the like, when the condition of reaching the goal is met, the ranking of the avatar object 610 is determined.

[0141] In this game, in order to share the virtual space between the HMD set 1000 and a plurality of user terminals 100, game progress information is live-streamed to the plurality of user terminals 100 at predetermined time intervals. As a result, on the touch screen 15 of the user terminal 100 while watching the game, a view image of the viewing area defined by the virtual camera 620B corresponding to the user terminal 100 is displayed. Also, parameter images representing the physical strength of the avatar object 610, the number of available magazines, the remaining ammunition of the gun, the remaining number of enemy objects 671, etc. are superimposed and displayed in the upper right and upper left sections of the view image. This view image can also be expressed as a game screen.

[0142] As described above, the game progress information includes motion data capturing the player's actions, voice data of the voice uttered by the player, and operation data indicating the content of the input operation on the controller 540. These data are, in other words, information for specifying the position, posture, orientation, etc. of the avatar object 610, information for specifying the position, posture, orientation, etc. of the enemy object 671, and information for specifying the position, etc. of other objects (for example, obstacle objects 672, 673). The processor 10 specifies the position, posture, orientation, etc. of each object by analyzing (rendering) the game progress information.

[0143] The game information 132 includes data of various objects such as the avatar object 610, the enemy object 671, and the obstacle objects 672, 673. The processor 10 updates the position, posture, orientation, etc. of each object by using the data and the analysis result of the game progress information. Thereby, the game progresses, and each object in the virtual space 600B moves in the same manner as each object in the virtual space 600A. Specifically, in the virtual space 600B, each object including the avatar object 610 operates based on the game progress information regardless of whether there is an operation from the user to the user terminal 100.

[0144] On the touch screen 15 of the user terminal 100, as an example, the UI images 701 and 702 are displayed superimposed on the view image. The UI image 701 is a UI image that receives an operation for displaying, on the touch screen 15, a UI image 711 that receives an item input operation for assisting the avatar object 610 from the user 3. The UI image 702 is a UI image that receives an operation for displaying, on the touch screen 15, a UI image (described later) that receives an operation from the user 3 for inputting and transmitting a comment to the avatar object 610 (in other words, the player 4). The operations received by the UI images 701 and 702 may be, for example, operations of tapping the UI images 701 and 702.

[0145] When the UI image 701 is tapped, the UI image 711 is displayed superimposed on the field of view image. The UI image 711 includes, for example, a UI image 711A with a magazine icon drawn, a UI image 711B with a first aid kit icon drawn, a UI image 711C with a warning triangle icon drawn, and a UI image 711D with a barricade icon drawn. The item input operation corresponds to, for example, an operation of tapping any of the UI images.

[0146] As an example, when the UI image 711A is tapped, the remaining ammunition of the gun used by the avatar object 610 increases. When the UI image 711B is tapped, the physical strength of the avatar object 610 is restored. When the UI images 711C and 711D are tapped, obstacle objects 672 and 673 that obstruct the movement of the enemy object 671 are arranged in the virtual space. The obstacle objects 672 and 673 may obstruct the movement of the enemy object 671 more than the other.

[0147] The processor 10 transmits item input information indicating that an item input operation has been performed to the server 200. The item input information at least includes information for specifying the type of item specified by the item input operation. The item input information may also include other information about the item, such as information indicating the position where the item is placed. The item input information is transmitted to other user terminals 100 and the HMD set 1000 via the server 200.

[0148] FIG. 12 is a diagram showing another example of the field of view image displayed on the user terminal 100. Specifically, it is a diagram showing an example of the game screen of this game and is a diagram for explaining the communication between the player during the game play and the user terminal 100.

[0149] In the example of Fig. 12(A), the user terminal 100 causes the avatar object 610 to execute speech 691. Specifically, the user terminal 100 causes the avatar object 610 to execute speech 691 according to the voice data included in the game progress information. The content of speech 691 is "There are no bullets!", which was spoken by player 4. That is, the content of speech 691 conveys to each user that due to the magazine being 0 and the number of bullets loaded in the gun being 1, it seems that the means to attack the enemy object 671 will be lost.

[0150] In Fig. 12(A), a speech bubble is used to visually show the speech of the avatar object 610, but actually, voice is output from the speaker of the user terminal 100. Additionally, along with the voice output, the speech bubble shown in Fig. 12(A) (i.e., the speech bubble including the text of the voice content) may be displayed in the field-of-view image. This is the same for speech 692 described later.

[0151] When receiving a tap operation on the UI image 702, the user terminal 100, as shown in Fig. 12(B), overlays and displays the UI images 705 and 706 (message UI) on the field-of-view image. The UI image 705 is a UI image that displays comments for the avatar object 610 (in other words, the player). The UI image 706 is a UI image that receives a comment sending operation from user 3 to send the input comment.

[0152] As an example, when the user terminal 100 receives a tap operation on the UI image 705, it displays a UI image imitating a keyboard (not shown, hereinafter simply referred to as "keyboard") on the touch screen 15. The user terminal 100 displays the text corresponding to the user's input operation on the keyboard on the UI image 705. In the example of Fig. 12(B), the text "Send the magazine" is displayed on the UI image 705.

[0153] After the user inputs text, for example, when the user terminal 100 receives a tap operation on the UI image 706, the user terminal 100 sends comment information including information indicating the input content (text content) and information indicating the user to the server 200. The comment information is sent to other user terminals 100 and the HMD set 1000 via the server 200.

[0154] The UI image 703A is a UI image indicating the user name of the user who sent the comment, and the UI image 704A is a UI image indicating the content of the comment sent by the user. In the example of FIG. 12(B), a user with the user name "BBBBB" uses his or her own user terminal 100 to send comment information with the content "Dangerous!" As a result, the UI image 703A and the UI image 704A are displayed. The UI image 703A and the UI image 704A are displayed on the touch screen 15 of all user terminals 100 participating in this game and on the monitor 51 of the HMD 500. Note that the UI image 703A and 704A may be one UI image. That is, a single UI image may include the user name and the content of the comment.

[0155] In the example of FIG. 12(C), the user with the user name "AAAAA", who is the user of the user terminal 100 shown in FIG. 12, inputs and sends a comment as described above, and as a result, the UI images 703B and 704B are displayed on the touch screen 15. The UI image 703B includes the user name "AAAAA", and the UI image 704B includes the comment "Send the magazine!" input in the example of FIG. 12(B).

[0156] In the example of FIG. 12(C), the user "AAAAA" further inputs a tap operation on the UI image 701, causing the UI image 711 to be displayed on the touch screen 15. The view image 611 is shown after the user inputs a tap operation on the UI image 711A. That is, as a result of transmitting item input information indicating a magazine from the user terminal 100 of the user "AAAAA" to other user terminals 100 and the HMD set 1000, the user terminals 100 and the HMD set 1000 arrange the production object 674 (described later) in the virtual space 600. As an example, after the elapsed time indicated by the item input information has elapsed, the user terminals 100 and the HMD set 1000 execute a production related to the production object 674 and execute a process of activating the effect of the item object.

[0157] In the example of FIG. 12(D), due to the execution of the process of activating the effect of the item object, the number of magazines increases from 0 to 1. As a result, the player speaks "Thank you!" to the user "AAAAA", and the voice data of this speech is transmitted to each user terminal 100. Thereby, each user terminal 100 outputs the voice "Thank you!" as the speech 692 of the avatar object 610.

[0158] As described above, in this game, communication between the user and the avatar object 610 is realized by outputting the speech voice of the avatar object 610 based on the player's speech and inputting comments by each user.

[0159] (Game progress process in the game play terminal 300) FIG. 13 is a flowchart showing an example of the flow of the game progress process executed by the game play terminal 300.

[0160] In step S31, the processor 30, as a game progress unit 315, progresses the game based on the game program 331 and the player's movement. In step S32, the processor 30 generates game progress information and distributes it to the user terminal 100. Specifically, the processor 30 transmits the generated game progress information to each user terminal 100 via the server 200.

[0161] In step S33, when the processor 30 receives item input information (YES in S33), in step S34, based on the item input information, it arranges an item object in the virtual space 600A. As an example, before arranging the item object, the processor 30 arranges an effect object 674 in the virtual space 600A (see Fig. 11(C)). The effect object 674 may be, for example, an object imitating a present box. The processor 30 may execute an effect related to the effect object 674 after the elapsed time indicated by the item input information has elapsed. The effect may be, for example, an animation of the lid of the present box opening. After executing the animation, the processor 30 executes a process of activating the effect of the item object. For example, in the example of Fig. 11(D), an obstacle object 673 is arranged.

[0162] After executing the animation, the processor 30 may arrange an item object corresponding to the tapped UI image in the virtual space 600A. For example, when a tap operation is performed on the UI image 711A, after executing the animation, the processor 30 arranges a magazine object indicating a magazine in the virtual space 600A. Also, when a tap operation is performed on the UI image 711B, after executing the animation, the processor 30 arranges a first aid kit object indicating a first aid kit in the virtual space 600A. The processor 30 may execute a process of activating the effect of the magazine object or the first aid kit object when the avatar object 610 moves to the position of the magazine object or the first aid kit object, for example.

[0163] The processor 30 continues and repeats the processes of steps S31 to S34 until the game ends. When the game ends, for example, when the player inputs a predetermined input operation to end the game (YES in step S35), the process shown in FIG. 13 ends.

[0164] (Game progress processing in the user terminal 100) FIG. 14 is a flowchart showing an example of the flow of game progress processing executed in the user terminal 100.

[0165] In step S41, the processor 10 receives game progress information. In step S42, the processor 10 advances the game based on the game progress information as the game progress unit 115.

[0166] In step S43, when the processor 10 receives an item input operation by the user 3 (YES in step S43), in step S44, the processor 10 consumes virtual currency and arranges the production object 674 in the virtual space 600B. Here, the virtual currency may be purchased (charged for this game) by the user 3 performing a predetermined operation on the processor 10 before participating in the game or during the participation in the game, or may be given to the user 3 when a predetermined condition is satisfied. The predetermined condition may be something necessary for participating in this game, such as clearing a quest in this game, or may be something unnecessary for participating in this game, such as answering a questionnaire. The amount of virtual currency (the ownership amount of virtual currency) is stored in the user terminal 100 as game information 132 as an example.

[0167] In step S45, the processor 10 transmits item input information to the server 200. The item input information is transmitted to the game play terminal 300 via the server 200.

[0168] After the placement of the production object 674, when a predetermined time has elapsed, the processor 10 places the item object in the virtual space 600A. In the example of FIG. 11, the obstacle object 673 is placed. That is, when the user 3 inputs a tap operation on the UI image 711C, a predetermined amount of virtual currency is consumed and the obstacle object 673 is placed.

[0169] The processor 10 continues and repeats the processes of steps S41 to S45 until the game ends. When the game ends, for example, when the player performs a predetermined input operation to end the game, or when the user 3 performs a predetermined input operation to leave the game midway (YES in step S46), the process shown in FIG. 14 ends.

[0170] (Game progress processing in the server 200) FIG. 15 is a flowchart showing an example of the flow of game progress processing executed by the server 200.

[0171] In step S51, the processor 20 receives game progress information from the game play terminal 300. In step S52, the processor 20 updates the game progress log (hereinafter, play log) as the log generation unit 212. The play log is generated by the processor 20, for example, when receiving the initial placement information from the game play terminal 300.

[0172] In step S53, the processor 20 transmits the received game progress information to each user terminal 100.

[0173] In step S54, when receiving item input information from any one of the user terminals 100 (YES in step S54), in step S55, the processor 20 updates the play log as the log generation unit 212. In step S56, the processor 20 transmits the received item input information to the game play terminal 300.

[0174] Processor 20 continues and repeats the processes of steps S51 to S56 until the game ends. When the game ends, for example, when receiving information indicating that the game has ended from the game play terminal 300 (YES in step S57), in step S58, processor 20, as the list generation unit 213, generates a list of users who participated in the game (user list 234) from the play log. Processor 20 stores the generated user list 234 in the server 200.

[0175] FIG. 16 is a diagram showing a specific example of the user list 234. In the "User" column, information indicating each user who participated in the game (for example, user name) is stored. In the "Tag" column, information (tag) generated based on the support provided by each user to the player is stored. In the example of FIG. 16, among the tags stored in the "Tag" column, those without square brackets are information automatically generated by processor 20, and those with square brackets are information manually input by the game operator.

[0176] In the example of FIG. 16, for the user "AAAAA", information such as "Magazine, 10F, Boss, 'Victory over the Boss with the Magazine Present'" is associated. This indicates that, for example, in the boss battle on stage 10F, the user "AAAAA" threw in a magazine, and the avatar object 610 won against the boss with the bullets of the thrown-in magazine.

[0177] Also, for the user "BBBBB", information such as "First Aid Kit, 3F, Mob, 'Recovery Just Before Game Over'" is associated. This indicates that, for example, in the battle against the mob enemies on stage 3F, the user "BBBBB" threw in a first aid kit, and as a result, the health of the avatar object 610 recovered just before the health reached 0 (game over).

[0178] In addition, for user "CCCCC", information such as "barricade", "5F", "low-level enemy", and "stopped two zombies with a barricade" is associated. This indicates that, for example, in the battle against low-level enemies on the stage of "5F", user "CCCCC" deployed a barricade (obstacle object 672 in Fig. 11), and as a result, successfully stopped the feet of two low-level enemies.

[0179] In the example of Fig. 16, for each user name of user 3, one piece of support provided is associated. However, for the user names of user 3 who provided support multiple times, tags for each instance of the multiple supports are associated. In the user list 234, it is preferable that these respective tags are distinguishable. This enables a player who refers to the user list 421 using the distribution terminal 400 after the game ends to accurately understand the details of each support.

[0180] <Distribution of operation instruction data> (Distribution process in distribution terminal 400) Fig. 17 is a flowchart showing an example of the flow of the distribution process executed by the distribution terminal 400. Fig. 18 is a diagram showing a specific example of the screen displayed on the distribution terminal 400. Fig. 19 is a diagram showing another specific example of the screen displayed on the distribution terminal.

[0181] In step S61, the processor 40, acting as the operation reception unit 413, receives a first operation for displaying a list of users (user list 234) who participated in the game. The download screen 721 shown in Fig. 18(A) is a screen for downloading the user list 234 from the server 200 and displaying it on the display unit 452. As an example, the download screen 721 is the screen displayed immediately after an activation operation of an application that executes the distribution process shown in Fig. 17 is input to the distribution terminal 400.

[0182] The download screen 721 includes, as an example, UI images 722 and 723. The UI image 722 accepts an operation for downloading the user list 234, that is, the first operation described above. The first operation may be, for example, an operation of tapping the UI image 722. The UI image 723 accepts an operation for ending the application. The operation may be, for example, an operation of tapping the UI image 723.

[0183] When a tap operation on the UI image 722 is accepted, in step S62, the processor 40, as the communication control unit 411, acquires (receives) the user list 234 from the server 200. In step S63, the processor 40, as the display control unit 412, causes the display unit 452 to display the user list 234. Specifically, the processor 40 causes the display unit 452 to display a user list screen generated based on the user list 234. The user list screen may be, as an example, the user list screen 731 shown in FIG. 18(B). The user list screen 731 consists of record images corresponding to each record in the user list 234. In the example of FIG. 18(B), record images 732A to 732C are described as the record images, but the number of record images is not limited to three. In the example of FIG. 18(B), when the number of records in the user list 234 is more than three (that is, the number of users participating in the game is more than three), the player can cause the display unit 452 to display other record images by inputting an operation of scrolling the screen (for example, a drag operation or a flick operation) on the touch screen 45.

[0184] As an example, the record images 732A to 732C each include a user name 733A to 733C, tag information 734A to 734C, and an icon 735A to 735C. Hereinafter, when there is no need to distinguish between the record images 732A to 732C, the user names 733A to 733C, the tag information 734A to 734C, and the icons 735A to 735C, they are described as "record image 732", "user name 733", "tag information 734", and "icon 735", respectively.

[0185] The user name 733 is information indicating each user who participated in the game and is stored in the "User" column in the user list 234. The tag information 734 is information indicating the tags associated with each piece of information indicating each user who participated in the game in the user list 234. For example, the record image 732A includes "AAAAA" as the user name 733A. Therefore, the record image 732A includes, as the tag information 734A, "Magazine, 10F, Boss, 'Win against the Boss with a Magazine Present'" associated with "AAAAA" in the user list 234. The icon 735 is, for example, an image pre-set by the user.

[0186] Note that the processor 40 may store the received user list in the distribution terminal 400 (user list 421 in FIG. 7). The download screen 721 may include a UI image (not shown) for displaying the user list 421 on the display unit 452. In this example, when the UI image is tapped, the processor 40 does not download the user list 234, but reads out the user list 421, generates a user list screen from the user list 421, and causes it to be displayed on the display unit 452.

[0187] In step S64, the processor 40, as the operation reception unit 413, receives a second operation for selecting any one of the users included in the user list screen 731. As an example, the second operation may be an operation of tapping any one of the record images 732 on the user list screen 731. In the example of FIG. 18(B), the player inputs a tap operation on the record image 732A. That is, the player selects the user "AAAAA" as the user who distributes the operation instruction data.

[0188] When a tap operation on the record image 732 is received, in step S65, the processor 40 causes the display control unit 412 to display the motion list 422 on the display unit 452. Specifically, the processor 40 causes the display unit 452 to display a motion list screen generated based on the motion list 422. The motion list screen may be, for example, the motion list screen 741 shown in FIG. 19. The motion list screen 741 consists of record images corresponding to each record in the motion list 422. In the example of FIG. 19, record images 742A to 742C are described as record images, but the number of record images is not limited to three. In the example of FIG. 19, when the number of records in the motion list 422 is more than four, the player can cause the display unit 452 to display other record images by inputting an operation for scrolling the screen (for example, a drag operation or a flick operation) on the touch screen 45.

[0189] As an example, the record images 742A to 742C each include a motion name 743A to 743C, a motion image 744A to 744C, and a UI image 745A to 745C. Hereinafter, when there is no need to distinguish between the record images 742A to 742C, the motion names 743A to 743C, the motion images 744A to 744C, and the UI images 745A to 745C, they will be described as "record image 7432", "motion name 743", "motion image 744", and "UI image 745", respectively.

[0190] Motion name 743 is information for identifying the motion stored in motion list 422. Motion image 744 is an image generated from the motion data associated with each motion name in motion list 422. As an example, processor 40 includes, in motion image 744 and record image 742, an image of avatar object 610 taking the first pose in each motion data. Motion image 744 may be a UI image that accepts a predetermined operation by the player (for example, a tap operation on motion image 744). When receiving the predetermined operation, processor 40 may play a motion video in which avatar object 610 operates based on the motion data. When the motion video ends, processor 40 may automatically redisplay motion list screen 741.

[0191] Note that record image 742 may include, instead of motion image 744, a UI image including text such as "Play Motion".

[0192] In step S66, processor 40 receives, as operation reception unit 413, a third operation for selecting a motion. The third operation may be, as an example, a tap operation on UI image 745. That is, UI image 745 accepts an operation for selecting the motion data corresponding to each record image 742. By receiving the third operation, processor 40 identifies, as motion identification unit 415, the motion data selected by the player.

[0193] In step S67, processor 40 receives the player's voice input while playing, as display control unit 412 and voice reception unit 414, a motion video in which avatar object 610 operates based on the selected motion data.

[0194] FIG. 20 is a diagram showing a specific example of voice input by player 4. As shown in FIG. 20, while player 4 is playing motion video 810A, it is inputting uttered voice 820A. This uttered voice 820A is an uttered voice addressed to user 3 (hereinafter, user 3A) whose user name is "AAAAA". That is, in the example of FIG. 20, player 4 selects user 3A (the first user) in step S64 and creates operation instruction data addressed to the user 3A. It is assumed that the user terminal 100 used by user 3A is user terminal 100A.

[0195] Since the uttered voice 820A is an uttered voice addressed to user 3A, it is an uttered voice based on the content of the assistance given by the user 3A to the avatar object 610 (in other words, player 4). Specifically, in the boss battle at stage 10F, user 3A inserted a magazine, and avatar object 610 won against the boss with the bullets of the inserted magazine. For this reason, the uttered voice 820A has the content of "Thank you for presenting the magazine in the boss battle! The timing was perfect too! Thanks to Mr. AAAAA, I was able to clear it!". Thus, it is preferable that the uttered voice includes the content of the assistance given by user 3 in the game and gratitude to user 3.

[0196] In a certain situation, before starting voice input, that is, before inputting the third operation to distribution terminal 400, player 4 creates the uttered content addressed to user 3. In another situation, the uttered content addressed to user 3 may be automatically generated by processor 40. Also, processor 40 may superimpose and display the tag associated with user 3 selected by the second operation on motion video 810A.

[0197] Processor 40 converts the received voice into voice data. In step S68, processor 40, as operation instruction data generation unit 416, generates operation instruction data including the voice data and the motion data of the selected motion.

[0198] In step S69, as the communication control unit 411, the processor 40 distributes the generated operation instruction data to the user terminal 100 (the first computer) of the selected user 3 (user 3A in the example of FIG. 20). FIG. 21 is a diagram showing still another specific example of the screen displayed on the distribution terminal 400. After executing step S68, as the display control unit 412, the processor 40 causes the display unit 452 to display the distribution screen. The distribution screen may be, for example, the distribution screen 751 shown in FIG. 21(A). The distribution screen 751 includes a UI image 752 and a motion image 753A. Further, the distribution screen 751 may include information indicating the user who is the distribution destination of the operation instruction data, as shown in FIG. 21(A).

[0199] The UI image 752 accepts an operation for distributing the operation instruction data to the selected user 3. The operation may be, for example, a tap operation on the UI image 752. The motion image 753A is a UI image that accepts an operation for playing a video based on the generated operation instruction data, that is, a video based on the operation instruction data generated for user 3A. The operation may be, for example, a tap operation on the motion image 753A. Note that the UI image that accepts an operation for playing the generated video is not limited to the motion image 753A. For example, it may be a UI image including text such as "Play Video". When the video ends, the processor 40 may automatically redisplay the distribution screen 751.

[0200] The distribution screen 751 preferably further includes a UI image that accepts an operation for returning to accepting voice input. The operation may be, for example, a tap operation on the UI image. By including the UI image in the distribution screen 751, the player 4 can perform voice input again, for example, when the voice input fails, such as when the content to be spoken is misspoken. Note that the UI image may be a UI image that accepts an operation for returning to selecting motion data.

[0201] When the processor 40 receives a tap operation on the UI image 752, it transmits operation instruction data to the server 200 together with information indicating the user 3A. Based on the information indicating the user 3A, the server 200 identifies the user terminal 100 that is the destination of the operation instruction data and transmits the operation instruction data to the identified user terminal 100 (i.e., the user terminal 100A).

[0202] When the transmission of the operation instruction data is completed, as an example, the processor 40 may cause the display unit 452 to display a distribution completion screen 761 shown in FIG. 21(B). The distribution completion screen 761 includes, as an example, UI images 762 and 763. Further, the distribution completion screen 761 may include text indicating that the transmission of the operation instruction data has been completed, as shown in FIG. 21(B).

[0203] The UI image 762 receives an operation for starting the creation of operation instruction data for another user 3. The operation may be, for example, an operation of tapping the UI image 762. When the processor 40 receives the tap operation, it causes the user list screen to be displayed again on the display unit 452. That is, when the tap operation is received, the distribution process returns to step S63. At this time, the processor 40 may generate a user list screen based on the user list 421 stored in the distribution terminal 400 and cause the display unit 452 to display it. The UI image 763 receives an operation for ending the application. The operation may be, for example, an operation of tapping the UI image 763. When the operation is received, the distribution process ends.

[0204] In the example described with reference to FIGS. 20 and 21, as shown in FIG. 21(C), the distribution terminal 400 transmits the operation instruction data of the video for the user 3A (the user 3 with the user name "AAAAA") only to the user terminal 100A.

[0205] FIG. 22 is a diagram showing another specific example of voice input by player 4. As shown in FIG. 22, while player 4 is playing motion video 810B, it is inputting spoken voice 820B. This spoken voice 820B is a spoken voice addressed to user 3 (hereinafter, user 3B) whose user name is "BBBBB". That is, in the example of FIG. 22, player 4 inputs a tap operation on record image 732B corresponding to user 3B at step S64 and creates operation instruction data addressed to user 3B. It is assumed that the user terminal 100 used by user 3B is user terminal 100B.

[0206] Since the spoken voice 820B is a spoken voice addressed to user 3B, it is a spoken voice based on the content of the assistance provided by the user 3B to the avatar object 610 (in other words, player 4). Specifically, in the battle between the minion enemies on stage 3F, user "BBBBB" throws a first aid kit, and as a result, the physical strength of avatar object 610 is about to reach 0 (game over) but the physical strength is restored. Therefore, the content of the spoken voice 820B is "Thanks to the first aid kit presented by Mr. BBBBB, I didn't lose the game on 3F. Thank you so much!"

[0207] FIG. 23 is a diagram showing yet another specific example of the screen displayed on the distribution terminal 400. The distribution screen 751 shown in FIG. 23(A) includes a UI image 752 and a motion image 753B. When the motion image 753B receives a tap operation, it plays a video based on the operation instruction data generated for user 3B.

[0208] When a tap operation on the UI image 752 is received, the processor 40 transmits the operation instruction data to the server 200 together with the information indicating user 3B. The server 200 identifies the user terminal 100 as the transmission destination of the operation instruction data based on the information indicating user 3B, and transmits the operation instruction data to the identified user terminal 100 (that is, user terminal 100B).

[0209] In the example described with reference to FIGS. 22 and 23, as shown in FIG. 23(C), the distribution terminal 400 transmits the operation instruction data of the video addressed to user 3B (user 3 with the user name "BBBBB") only to the user terminal 100B.

[0210] As described above, the content of the voice based on the voice data included in the operation instruction data is based on the content of the support provided by user 3 to player 4 in the most recent game participation. Since the content of this support varies for each user 3, the content of the voice also varies for each user 3. That is, after the game ends, operation instruction data including voices of different contents is transmitted to at least some of the user terminals 100 of the users 3 who participated in the game.

[0211] Also, the motion of the avatar object 610 in the example of FIG. 22 is different from the motion in the example of FIG. 20. That is, player 4 selects different motion data in generating the operation instruction data addressed to user 3B than when generating the operation instruction data addressed to user 3A. Specifically, in step S66, player 4 inputs a tap operation to the UI image 745B to select the motion data corresponding to the record image 742B. In this way, player 4 can make the motion data included in the operation instruction data different for each user 3.

[0212] Then, the operation instruction data for each user 3, which includes voice data of different contents for each user 3 and the motion data selected for each user 3, is transmitted only to the user terminal 100 of each user 3. In other words, unique (one-of-a-kind) operation instruction data is transmitted to each of the user terminals 100 of the selected users 3.

[0213] FIG. 24 is a diagram showing an overview of the transmission of game progress information from the game play terminal 300 to the user terminal 100. While the operation instruction data for video playback in the user terminal 100 is unique for each user terminal 100, as shown in FIG. 24, during game execution, the game progress information transmitted to the user terminals 100 of all users 3 participating in the game is common among the respective user terminals 100. That is, the operation instruction data included in the game progress information is also common among the respective user terminals 100. Thus, it can be said that the operation instruction data for video playback and the operation instruction data for advancing the game are different data in terms of differences and destinations between the user terminals 100.

[0214] (Video playback process in user terminal 100) FIG. 25 is a flowchart showing an example of the flow of the video playback process executed in the user terminal 100.

[0215] In step S71, the processor 10 receives the operation instruction data as the video playback unit 117. In step S72, the processor 10 notifies the user 3 of the reception of the operation instruction data as the video playback unit 117. As an example, the processor 10 notifies the user 3 of the reception of the operation instruction data by at least any one of the display on the display unit 152 of the notification image, the reproduction of the notification sound from a speaker (not shown), and the lighting or flashing of a lighting unit (not shown) composed of an LED (light-emitting diode) or the like.

[0216] In step S73, the processor 10, as the operation reception unit 111, receives a first playback operation for playing a video. The first playback operation may be, for example, an operation of tapping a notification image. In step S74, the processor 10, as the video playback unit 117, renders the operation instruction data and plays the video. The processor 10 may, for example, start an application for playing this game and play the video, or start a separate video playback application and play the video. Hereinafter, the video will be referred to as the "thank you video".

[0217] FIG. 26 is a diagram showing a specific example of the playback of the thank you video. Specifically, it is a diagram showing an example of the playback of the thank you video on the user terminal 100 of the user 3A. In the thank you video 910A played on the user terminal 100, the avatar object 610 is speaking the voice 920A while performing a certain motion. In other words, the processor 10 outputs the voice 920A from a speaker (not shown) while playing the thank you video 910A including the avatar object 610 that performs a certain motion.

[0218] The motion in the thank you video 910A is based on the motion data selected by the player 4 in the generation of the operation instruction data for the user 3A, and the voice 920A is based on the voice data generated from the spoken voice 820A input by the player 4 in the generation of the operation instruction data. That is, the voice 920A is a voice including the content of the support provided by the user 3A in the game and the gratitude for the support. In this way, by inputting the first playback operation, the user 3A can view the thank you video in which the avatar object 610 speaks the content of the support provided by the user 3A in the game and the gratitude for the support.

[0219] As an example, after the playback of the thank-you video 910A ends, the user terminal 100 may display at least one UI image on the touch screen 15. The UI image may be, for example, a UI image that accepts an operation to play the thank-you video 910A again, or a UI image that accepts an operation to transition to another screen, or a UI image that accepts an operation to end the application.

[0220] Also, as an example, during the playback of the thank-you video 910A, the user terminal 100 may display at least one UI image on the touch screen 15. The UI image may be, for example, a plurality of UI images that respectively accept operations to temporarily stop, end, or change the playback scene of the thank-you video 910A being played.

[0221] Note that these UI images displayed during the playback of the thank-you video 910A and after the playback of the thank-you video 910A ends do not include a UI image for responding to the avatar object 610. That is, in the thank-you video 910A according to this embodiment, there is no means for responding to the avatar object 610.

[0222] FIG. 27 is a diagram showing another specific example of the playback of the thank-you video. Specifically, it is a diagram showing an example of the playback of the thank-you video on the user terminal 100 of the user 3B. In the thank-you video 910B played on the user terminal 100, the avatar object 610 is speaking the voice 920B while performing a certain motion. In other words, the processor 10 outputs the voice 920B from a speaker (not shown) while playing the thank-you video 910B including the avatar object 610 that performs a certain motion.

[0223] The motion in the thank-you video 910B is based on the motion data selected by the player 4 in generating the operation instruction data addressed to the user 3B. The voice 920B is based on the voice data generated from the spoken voice 820B input by the player 4 in generating the operation instruction data. Therefore, in the example of FIG. 27, the motion performed by the avatar object 610 is different from the motion in the example of FIG. 26. Also, the voice 920B is a voice that includes the content of the assistance provided by the user 3B in the game and the gratitude for the assistance. Therefore, in the example of FIG. 27, the content of the voice 920B is different from the content of the voice 920A in the example of FIG. 26.

[0224] Thus, after the game ends, the thank-you video received by at least some of the user terminals 100 of the users 3 who participated in the game is a video in which the speech content of the avatar object 610 is different for each user 3.

[0225] Note that the processor 10 may cause the UI image 930 including content that encourages participation in the next game to be superimposed on and displayed on the video 910. The UI image 930 may be distributed together with the operation instruction data, or may be stored in the user terminal 100 as the game information 132.

[0226] In the above embodiment, an example has been described in which the avatar object 610 that operates in real time according to the movement of the performer is operated within the same virtual space 600B as the virtual space 600A defined by the game play terminal 300, and the view image 660 of the viewing area 640B corresponding to the position, orientation, and tilt of the virtual camera 620B arranged in the virtual space 600B is displayed on the touch screen 15 of the user terminal 100. At this time, the position, orientation, etc. of the virtual camera 620B are changed according to a swipe operation, a pinch-in operation / a pinch-out operation, etc. on the touch screen 15 of the user terminal 100. However, the change in the position, orientation, etc. of the virtual camera 620B is not limited to this.

[0227] For example, a camera object 630 that changes its position, orientation, and tilt in response to the operation of a switcher (not shown) by the operator or performer of the system 1 is placed in the virtual space 600A (see Fig. 28(A)), and the position, orientation, etc. of the virtual camera 620B placed in the virtual space 600B may be changed according to the changes in the position, orientation, etc. of the camera object 630. Note that the camera object 630 is displayed only on the monitor 51 of the HMD 500 connected to the game play terminal 300 so that the performer can view it. As a result, the performer can speak, etc. from the perspective of the camera object.

[0228] Also, the space in which the avatar object 610 or the like operates may be not limited to the virtual space but may be the real space. As the real space, for example, it may be a space specified by the acquired image captured by the camera (imaging unit) 17. In this case, a space (hereinafter also referred to as an augmented reality space) in which the avatar object 610 or the like is arranged with respect to the acquired image from the camera 17 is generated, and the virtual camera 620B is arranged in the augmented reality space, and the position, orientation, etc. of the virtual camera 620B may be changed according to, for example, the change in the position and orientation of the camera 17.

[0229] Furthermore, the method of changing the position, orientation, etc. of the virtual camera 620B may be selectable by the user. For example, a plurality of types of viewing modes may be provided as selectable viewing modes for the user terminal 100, and when the TV mode (third viewing mode) is selected in the user terminal 100, the position, orientation, etc. of the virtual camera 620B may be changed in conjunction with the camera object 630 whose position, orientation, etc. change in response to the operation of a switch by the administrator or performer. When the normal mode (first viewing mode) is selected in the user terminal 100, the position, orientation, etc. of the virtual camera 620B may be changed in response to a swipe operation by the user. When the AR mode (second viewing mode) is selected, a space in which the avatar object 610, etc. is arranged with respect to the image acquired from the camera 17 (hereinafter, also referred to as an augmented reality space) may be generated, and the virtual camera 620B may be arranged in the augmented reality space, and the position, orientation, etc. of the virtual camera 620B may be changed in response to the change in the position, orientation, etc. of the camera 17. This allows the viewpoint of the image displayed on the touch screen 15 to be changed according to the user's preferences.

[0230] In the following, an example in which avatar object 610 etc. moves in virtual space 600B or augmented reality space and the position, orientation, etc. of virtual camera 620B are changed will be described for each viewing mode. Specifically, with reference to Figs. 28 to 35, an example will be described in which a user plays a game of rock-paper-scissors against avatar object 610 moving live in virtual space 600B or augmented reality space (hereinafter also referred to as a rock-paper-scissors game).

[0231] (Normal mode and TV mode) FIG. 28(A) is a diagram for schematically explaining a virtual space 600A defined on the game play terminal 300 side during the progress of a rock-paper-scissors game. In FIG. 28(A), an avatar object 610, option objects 281 to 283, and a wreath object 284 that reminds one of a wreath are arranged in the virtual space 600A. The avatar object 610 moves according to the motion of the player (performer) performing live and speaks the voice of the performer. A virtual camera 620A is arranged at the position of the head of the avatar object 610. Although not shown, other objects such as an object that decorates the surroundings of the avatar object 610, an audience object corresponding to the audience, a background object, and a floor object are arranged in the virtual space 600A.

[0232] Also, each of the option objects 281 to 283 is composed of a rod portion and a disk portion at the tip. An illustration representing rock, scissors, and paper is drawn on the surface of the disk portion. Each of the option objects 281 to 283 can grip the rod portion according to the gripping motion by the player. The operation (motion) of gripping the option objects 281 to 283, etc. by the player is detected by a controller 540 provided with a motion sensor 520 that detects the movement of the hand (hand motion (for example, the position of each finger, the shape and direction of the grip, etc.) attached to the player's hand. The player grabs the option object corresponding to the hand (rock, scissors, paper) that is finally shown and raises it upward, so that the hand shown by the avatar object 610 is determined.

[0233] The game play terminal 300 live-streams game progress information, including information (display data) such as the positions, postures, and orientations of at least the avatar object 610 and the option objects 281 to 283 among the objects arranged in the virtual space 600A, and audio data (collectively also referred to as operation instruction data), etc., to a plurality of user terminals 100 at predetermined time intervals. The game progress information distributed to the plurality of user terminals 100 is common information among the plurality of user terminals 100. Thereby, at least the operations of the avatar object 610 and the option objects 281 to 283 can be shared (synchronized) among the game play terminal 300 and each of the plurality of user terminals 100. The objects shared among the game play terminal 300 and the plurality of user terminals 100 may be only the avatar object 610 as long as it includes at least the avatar object 610 that is live-played by an actor among the objects arranged in the virtual space 600A, or may include other objects other than the avatar object 610 and the option objects 281 to 283. Also, the data distributed from the game play terminal 300 to the plurality of user terminals 100 includes not only game progress information, but also initial arrangement information, game information 132, and information for specifying the types (rock, scissors, paper) of the option objects determined as the moves of the avatar object 610, etc. Further, the data includes viewpoint information that can specify the position, orientation, and tilt of the camera object 630, that is, the viewpoint.

[0234] A virtual camera 620B is arranged in the virtual space 600B defined in the user terminal 100. The viewpoint of the camera object 630 changes according to the operation of a switch by an operator or the like, and in the user terminal 100 in which the TV mode is selected, the position, orientation, and tilt of the virtual camera 620B are changed based on the viewpoint information of the camera object 630.

[0235] Since the camera object 630 is displayed on the monitor 51 of the HDM500, the performer can direct their line of sight at the camera object 630. When the TV mode is selected, the virtual camera 620B can be automatically positioned at a location where the avatar object 610 is easily visible from the front, and the video displayed on the display unit 152 is likely to be a view of the avatar object 610 from the front.

[0236] In the user terminal 100 in the normal mode, the position, orientation, etc. of the virtual camera 620B change according to operations by the user such as a swipe operation, a pinch-in operation / pinch-out operation, etc., rather than the viewpoint information. For example, the position of the virtual camera 620B is moved left and right by a swipe operation in the left-right direction, the vertical orientation of the virtual camera 620B is changed vertically by a swipe operation in the up-down direction, the position of the virtual camera 620B is moved outward (i.e., reduced display) toward the outside of the virtual space 600B by a pinch-in operation, and the position of the virtual camera 620B is moved inward (or to the center, the avatar object 610) toward the inside of the virtual space 600B (i.e., enlarged display) by a pinch-out operation. Also, this enables the user to view the objects arranged in the virtual space 600B from a desired viewpoint.

[0237] (AR mode) FIG. 28(B) is a diagram for schematically explaining a virtual space 600C for augmented reality (hereinafter also referred to as the AR virtual space 600C) defined on the user terminal 100 side when the AR mode is selected and superimposed on the acquired image. In the example of FIG. 28(B), among the objects arranged in the virtual space 600A, only the avatar object 610 and the option objects 281 to 283 are arranged. The AR virtual space 600C only needs to have at least no background object arranged. For example, in addition to the avatar object 610 and the option objects 281 to 283, a wreath object 284 may also be arranged.

[0238] In the user terminal 100, a flat surface such as a floor surface is specified by analyzing the acquired image captured by the camera 17. The view image in the virtual space 600C for AR is arranged at a predetermined position (for example, the center position of the flat surface) of the flat surface. The virtual camera 620B is arranged at a position where the virtual space 600C for AR is viewed from the same direction as the shooting direction of the camera 17 when the acquired image is captured. As a result, an image in the augmented reality space obtained by superimposing the view image 640C from the virtual camera 620B on the acquired image from the camera 17 is displayed on the display unit 152. The position, orientation, etc. of the virtual camera 620B are changed according to changes in the position, orientation, etc. of the camera 17. The acquired image changes according to changes in the position, orientation, etc. of the camera 17, and the view image 640C changes according to changes in the position, orientation, etc. of the virtual camera 620B linked to the position, orientation, etc. of the camera 17.

[0239] In the user terminal 100 in which the AR mode is selected, in the augmented reality space where an object including the avatar object 610 is arranged with respect to the acquired image obtained from the camera 17, the avatar object 610 and the like operate. Thereby, it is possible to give the user an impression (sense of presence) as if the avatar object 610 and the like are actually operating in the real space in front of the user's eyes.

[0240] (Flowchart) FIG. 29 is a flowchart showing an example of the processing flow of the user terminal 100 when the viewing mode is switched based on an input operation or the like on the touch screen 15. Note that the switching of the viewing mode includes the reset (restart) of the AR mode.

[0241] In step S81, the processor 10 of the user terminal 100 determines whether the viewing mode has been switched to the TV mode or the normal mode based on an input operation on the touch screen 15. When it is determined that the mode has been switched to the TV mode or the normal mode, the process proceeds to step S82. When it is not determined that the mode has been switched to the TV mode or the normal mode, that is, when it is determined that the mode has been switched to the AR mode or the AR mode has been reset, the process proceeds to step S88.

[0242] In step S82, the processor 10 defines a virtual space 600B that is the same as the virtual space 600A shown in FIG. 28(A). In step S83, the processor 10 arranges the same objects in the virtual space 600B as the various objects arranged in the virtual space 600A. At this time, the objects are arranged so as to have the same positional relationship as the various objects arranged in the virtual space 600A.

[0243] In step S84, the processor 10 determines whether the viewing mode has been switched to the TV mode. When it is determined that the mode has been switched to the TV mode, the process proceeds to step S85, and based on the viewpoint information transmitted from the server 200 in response to an operation of the switcher by the operator of the system 1 or the like, the virtual camera 620B is arranged in the virtual space 600B. As a result, the position, orientation, etc. of the virtual camera 620B coincide with the position, orientation, etc. of the camera object 630.

[0244] When it is determined in step S84 that the mode has not been switched to the TV mode, the processor 10 regards it as being switched to the normal mode and proceeds to step S86. In step S86, the processor 10 arranges the virtual camera 620B at a position corresponding to the position where the virtual camera 620B was arranged before the viewing mode was switched. Specifically, if the viewing mode before the switch is the TV mode, the arrangement of the virtual camera 620B in the virtual space 600B is maintained. On the other hand, if the viewing mode before the switch is the AR mode, the position where the virtual camera 620B was arranged in the AR virtual space 600C is specified, and the virtual camera 620B is arranged at the same position as the specified position in the virtual space 600B. Thereby, when switching to the normal mode, the viewpoint before the switch can be maintained.

[0245] In step S87, the processor 10 displays the view image 660 on the display unit 152. The processor 10 defines a viewing area 640B, which is the viewing field from the virtual camera 620B in the virtual space 600B, according to the position and inclination of the virtual camera 620B. Then, the processor 10 defines a view image 660 corresponding to the viewing area 640B. The processor 10 causes the display unit 152 to display the view image 660 by outputting the view image 660 to the display unit 152. When the process of step S87 is completed, the process returns.

[0246] On the other hand, when switched to the AR mode, in step S88, the processor 10 analyzes the acquired image captured and acquired by the camera 17 using techniques related to image recognition and spatial recognition. Thereby, for example, a three-dimensional space is specified from the acquired image, and a flat surface portion (for example, a horizontal portion, specifically, a floor surface) and the size (width) of the flat surface portion are specified.

[0247] Next, in step S89, the processor 10 sets the placement position for placing (overlaying) the view image in the AR virtual space 600C on the acquired image based on the identified flat surface portion. The processor 10 sets a predetermined position of the flat surface portion (for example, the center position of the flat surface portion) as the position for placing the view image in the AR virtual space 600C. Thereby, the placement position of the view image is associated with the predetermined position of the flat surface portion.

[0248] In step S90, the processor 10 defines the AR virtual space 600C shown in FIG. 28(B). In step S91, the processor 10 places some of the various objects arranged in the virtual space 600A in the AR virtual space 600C. In the case of FIG. 28(B), the some objects are the avatar object 610 and the option objects 281 to 283. In FIG. 28(B), the avatar object 610 and the option objects 281 to 283 are arranged so as to have the same positional relationship as in FIG. 28(A). Even when the wreath object 284 is arranged as another object, the wreath object 284 is arranged so as to have the same positional relationship as in FIG. 28(A).

[0249] In step S92, the processor 10 identifies, for example, the shooting direction (orientation, tilt, etc.) of the camera 17 when the acquired image is taken, and places the virtual camera 620B at a position where the AR virtual space 600C is viewed from the same direction as the shooting direction. Specifically, when the shooting direction is a substantially horizontal direction, the virtual camera 620B is placed at a position where the AR virtual space 600C is viewed from the horizontal direction (for example, FIG. 35(A)), and when the shooting direction is a direction looking down from an obliquely upward direction, the virtual camera 620B is placed at a position where the AR virtual space 600C is viewed from an obliquely upward direction.

[0250] Thus, when switched to the AR mode, the position of the virtual camera 620B may be different from the position before the mode switch. As a result, the view image in the AR mode may also be different from the view image in the TV mode or the normal mode. As a result, it becomes possible to make the view image in the AR mode stand out from the TV mode or the normal mode.

[0251] In step S93, the processor 10 causes the display unit 152 to display an image in the augmented reality space obtained by superimposing the view image 640C from the virtual camera 620B on the acquired image from the camera 17. At this time, the processor 10 superimposes the view image 640C at a predetermined position on the acquired image set in step S89 with a display size corresponding to the size of the flat surface portion specified in step S88, generates an image in the augmented reality space, and displays it. When the process of step S93 is completed, the process returns.

[0252] FIG. 30 is a flowchart showing an example of the processing flow of the user terminal 100 after the viewing mode is switched. The process is executed each time operation instruction data is received, at predetermined time intervals, or the like.

[0253] In step S101, the processor 10 receives operation instruction data. In step S102, the processor 10 moves the avatar object 610 or the like in the virtual space 600B or the augmented reality space according to the operation instruction data. In other words, the processor 10 reproduces, by real-time rendering, a video in which the avatar object 610 or the like is operating in the virtual space 600B or the augmented reality space.

[0254] In step S103, the processor 10 determines whether the TV mode is selected based on the processing result of step S81. When it is determined that the TV mode is selected, the process proceeds to step S104, and the virtual camera 620B is controlled based on the viewpoint information transmitted from the server 200 in response to the operation of the switcher by the operator of the system 1 or the like. If it is not determined in step S103 that the TV mode is selected, the processor 10 proceeds to step S106 and determines whether the normal mode is selected based on the processing result of step S81.

[0255] When it is determined that the normal mode is selected, the process proceeds to step S107, and it is determined whether a movement operation for moving the virtual camera 620B (viewpoint), such as a swipe operation or a pinch-in operation / pinch-out operation, from the user has been received. When it is not determined in step S107 that a movement operation has been received, the process proceeds to step S105 without moving the virtual camera 620B.

[0256] On the other hand, when it is determined in step S107 that a movement operation has been received, in step S108, the destination of the movement of the virtual camera 620B (viewpoint) is specified according to the mode of the movement operation. Specifically, in the case of a swipe operation in the left-right direction, the destination in the operation direction of the swipe operation (left-right, and the distance from the avatar object 610 is substantially constant) is specified. In the case of a pinch-in operation, the destination of the movement to the outside of the virtual space 600B (the direction in which the distance from the avatar object 610 increases) is specified according to the distance reduced for the two points targeted by the pinch-in operation. In the case of a pinch-out operation, the destination of the movement to the inside of the virtual space 600B (the direction in which the distance from the avatar object 610 decreases) is specified according to the distance increased for the two points targeted by the pinch-out operation. The movement distance of the virtual camera 620B (viewpoint) is, for example, a distance proportional to the distance of the swipe operation, the distance reduced by the pinch-in operation, the distance increased by the pinch-out operation, etc. In the case of a swipe operation in the up-down direction, the orientation of the virtual camera 620B can be changed in the up-down direction.

[0257] In step S109, it is determined whether or not the specified destination of the movement of the virtual camera 620B is within a predetermined range. The range within which the performer playing the avatar object 610 can actually move or operate is, for example, predetermined to be within a predetermined range (for example, a range within a radius of 2 m) from a predetermined position on the floor surface (or stage, etc.) where the performer actually exists. The predetermined position may be, for example, a predetermined position that is approximately the central position on the floor surface or the like. The predetermined range in step S109 is the range in the virtual spaces 600A to 600C corresponding to the range predetermined as the range within which the performer can actually move or operate. In other words, the predetermined range is the range within which the avatar object 610 played by the performer can move or operate, and it can also be said that it is a range in which the avatar object 610 basically cannot go out of the predetermined range. Here, with reference to FIG. 31(A), a specific example of the predetermined range in the virtual space 600B will be described.

[0258] FIG. 31(A) shows an example of a predetermined range 670 when the virtual space 600B in the normal mode is viewed from directly above (ceiling, above). In FIG. 31(A), the outer edge of the virtual space 600B is shown by a dotted line, the predetermined range 670 is shown in gray, and the position of the avatar object 610 is shown by a black circle. The predetermined range 670 during the normal mode is a cylindrical (columnar) range that extends to the ceiling at approximately the center within the virtual space 600B, as illustrated in FIG. 31(A), and basically, the avatar object 610 can move or operate within the predetermined range 670.

[0259] Returning to step S109, when it is not determined that the destination of the identified virtual camera 620B is within the predetermined range 670, in step S110, control is performed to move the virtual camera 620B to the destination identified in step S108 according to the mode of the movement operation. Note that the direction of the virtual camera 620B in the present embodiment is controlled in the direction in which the avatar object 610 is arranged when the movement control is performed, but it is not limited thereto, and it may be controlled in the direction of a predetermined position (for example, a predetermined position) within the predetermined range.

[0260] FIG. 31(B) shows an example of movement when the destination of the virtual camera 620B identified according to the movement operation in the normal mode is outside the predetermined range 670. In FIG. 31(B), it is shown that when a pinch-out operation is performed in a state where the viewpoint of the virtual camera 620B (shown as a point in the figure) is at position a, control is performed to move it to position b according to the pinch-out operation. Thus, when the destination of the virtual camera 620B identified according to the movement operation is outside the predetermined range 670, the viewpoint of the virtual camera 620B can be moved to the identified destination.

[0261] On the other hand, when it is determined that the destination of the virtual camera 620B specified in step S109 is within the predetermined range 670, there is a possibility that the viewpoint of the virtual camera 620B will move inside the avatar object 610 that can move or operate within the predetermined range 670. In this case, there is a risk of problems such as displaying the inside of the avatar object 610 that the user would not originally want to view. Therefore, during the normal mode of the present embodiment, when it is determined that the destination of the virtual camera 620B specified in step S109 is within the predetermined range, control is performed to restrict movement into the predetermined range 670. Specifically, in step S111, control is performed to move the virtual camera 620B to the intersection of the path to the destination specified in step S108 and the outer edge (outer peripheral surface) of the predetermined range 670.

[0262] Figures 31(C) and 31(D) show examples of movement when the destination of the virtual camera 620B specified according to the movement operation in the normal mode is within the predetermined range 670. In Figure 31(C), a case is illustrated where a pinch-out operation is performed in a state where the viewpoint of the virtual camera 620B is at position a, and as a result, position c, which is within the predetermined range 670 in step S109 according to the pinch-out operation, is specified as the destination. In this case, as shown in Figure 31(D), control is performed to move the virtual camera 620B to the intersection of the path indicated by the dotted arrow to position c, which is the destination specified in step S108, and the outer edge (outer peripheral surface) of the predetermined range 670. Thereby, in the normal mode, while enabling the virtual camera 620B to be moved relatively freely by a movement operation, when the destination of the virtual camera 620B specified according to the movement operation is within the predetermined range 670, movement into the predetermined range 670 is restricted to avoid the above problems, and the virtual camera 620B can be moved to a position outside the predetermined range 670 and closest to the specified destination.

[0263] When the processing of step S104, step S110, or step S111 is completed, the process proceeds to step S105. In step S105, in response to a change in the position, orientation, etc. of the virtual camera 620B, the processor 10 updates the view image 660. Thereby, it is possible to synchronize the operations of the avatar object 610 and the selection objects 281 to 283 in the virtual space 600A on the game play terminal 300 side with the operations of the avatar object 610 and the selection objects 281 to 283 in the virtual space 600B on the user terminal 100 side. When the processing of step S105 is completed, the process returns.

[0264] In step S106, if it is not determined that the normal mode is selected, the processor 10 assumes that the AR mode is selected and proceeds to step S112. In step S112, the processor 10 identifies the position, orientation, and tilt, etc. of the camera 17 based on a gyro sensor, a GPS signal, etc., and controls the virtual camera 620B in synchronization with the changes in the position, orientation, and tilt, etc. of the camera 17.

[0265] In step S113, it is determined whether or not the position of the virtual camera 620B moved in step S112, etc. is within a predetermined range defined in advance within the AR virtual space 600C. The predetermined range in step S113 is a range in the AR virtual space 600C corresponding to a range defined in advance as a range within which the performer can actually move or operate, and in this embodiment, it is assumed to be the same range as the predetermined range in step S109 during the normal mode. Here, with reference to FIG. 32(A), a specific example of the predetermined range within the AR virtual space 600C will be described.

[0266] FIG. 32(A) shows an example of a predetermined range 680 when the AR virtual space 600C in the AR mode is viewed from directly above (ceiling, above). In FIG. 32(A), the outer edge of the virtual space 600C is shown by a dotted line, the predetermined range 680 is shown in gray, and the position of the avatar object 610 is shown by a black circle. The predetermined range 680 during the AR mode is a cylindrical (columnar) range that extends to the ceiling at approximately the center within the AR virtual space 600C as illustrated in FIG. 32(A), and basically, the avatar object 610 can move or operate within the predetermined range 680.

[0267] Return to step S113. When it is not determined that the position of the virtual camera 620B is within the predetermined range 680, in step S114, the processor 10 updates the acquired image and the view image. As a result, the acquired image changes according to changes in the position of the camera 17 and the like, and the view image changes according to changes in the position of the virtual camera 620B and the like. For example, when the camera 17 is turned to the left as if photographing the left side surface of the avatar object 610, the view image of the avatar object 610 and the like as seen from the left side surface is updated for the acquired image captured and acquired by the camera 17.

[0268] On the one hand, when it is determined in step S113 that the position of the viewpoint of the virtual camera 620B is within the predetermined range 680, there is a risk of problems such as displaying the inside of the avatar object 610 that the user would not normally want to view, similar to the normal mode. However, during the AR mode, since it is a mode in which the virtual camera 620B is moved according to changes in the position of the camera 17 etc., if the movement is restricted as in the normal mode, there is a risk of giving the user a sense of discomfort. Therefore, during the AR mode of the present embodiment, when it is determined that the position of the virtual camera 620B is within the predetermined range 680, in step S115, a setting is made to make the avatar object 610 arranged in the AR virtual space 600C non-displayed, and the process proceeds to step S114. As a result, by bringing the camera 17 closer to the avatar object 610, an image from a viewpoint within the predetermined range 680 can be displayed and viewed, but the avatar object 610 becomes non-displayed.

[0269] FIGS. 32(B) and 32(C) show examples of the case where the position of the virtual camera 620B in the AR mode is within the predetermined range 680. In FIG. 32(B), in a state where the viewpoint of the virtual camera 620B is at position a, when the user who owns the user terminal 100 moves to a position corresponding to within the predetermined range 680, an example where the viewpoint of the virtual camera 620B moves to position c within the predetermined range 680 is illustrated. In this case, as shown in FIG. 32(C), although the viewpoint of the virtual camera 620B moves to position c, the avatar object 610 becomes non-displayed when the viewpoint position of the virtual camera 620B enters within the predetermined range 680. In FIG. 32(C), it is shown that the avatar object 610 is non-displayed by a dotted line. Thereby, during the AR mode, while changing the acquired image according to changes in the position of the camera 17 etc. and making it possible to change the field-of-view image according to changes in the position of the virtual camera 620B etc., when the position of the virtual camera 620B is within the predetermined range 680, the above problem can be avoided by making the avatar object 610 non-displayed.

[0270] While the viewpoint position of the virtual camera 620B is within the predetermined range 680, the avatar object 610 is not displayed, but the voice from the performer is continuously output. Also, when the viewpoint position of the virtual camera 620B changes from within the predetermined range 680 to outside the predetermined range 680, the process of step S115 is not performed and the non-display state is released, so that the avatar object 610 becomes displayable.

[0271] (Display example) Figs. 33 to 35 are diagrams showing display examples of the display unit 152 of the user terminal 100 during the progress of the rock-paper-scissors game. Figs. 33 to 35 show display examples when the same operation instruction data is received. Specifically, Fig. 33 shows a display example when the normal mode is set, Fig. 34 shows a display example when the TV mode is set, and Fig. 35 shows a display example when the AR mode is set.

[0272] (Display example in normal mode) In the normal mode, the processor 10 defines a virtual space 600B that synchronizes with the virtual space 600A in the game play terminal 300, and arranges objects such as the avatar object 610, the option objects 281 to 283, the wreath object 284, the viewer object, and the background object within the virtual space 600B. Further, the processor 10 operates the avatar object 610, etc. based on the game progress information within the virtual space 600B, and displays the view image from the virtual camera 620B on the display unit 152. For this reason, on the display unit 152, the wreath object 284 is displayed behind the avatar object 610, and the floor object and the background object are displayed.

[0273] The avatar object 610 in FIG. 33(A) shows a scene where voice is being output along with a motion such as thinking, e.g., "What should I come up with? Which one should I choose?" Also, during the rock-paper-scissors game, various UIs, game information, etc. are displayed on the display unit 152. In FIG. 33(A), for example, a selection UI 180, a rule explanation display 181, an item input icon 185, a comment input icon 186, etc. are superimposed and displayed on the image within the virtual space 600B. Also, at the lower left of the display unit 152, a comment display unit 184 is provided for displaying the comment content input to the avatar object 610.

[0274] The rule explanation display 181 is an image for explaining the rules of the rock-paper-scissors game. In FIG. 33(A), for example, messages such as "· A total of 3 battles!", "· Press the button within the time limit!", "Win 50P", "Draw 10P", "Lose 0P" are displayed.

[0275] The selection UI 180 includes button icons for the user to select the hand to play, and is a UI for accepting an input operation (tap operation) to the button icons. For this reason, as shown in FIG. 33(A), the selection UI 180 is displayed with higher priority than the avatar object 610, the comment display unit 184, the item input icon 185, the comment input icon 186, etc. In FIG. 33(A), a message such as "Choose the hand to play!" is displayed, and below it, button icons corresponding to rock, scissors, and paper are displayed from the left. This enables the user to recognize that they can intuitively select the hand to play by touching the button icons. Also, on the left side of the message "Choose the hand to play!", the time limit (here "10") for selecting the hand to play is displayed, and on the right side of the message, an "×" button for erasing the display of the selection UI 180 itself is displayed.

[0276] Even if the user has not finished selecting a move, the user can cancel the selection UI 180 by selecting the "×" button. This is because when the selection UI 180 is displayed, the avatar object 610 (character), the comment display section 184, the item input icon 185, etc. are hidden, making it difficult for the user to see the images of the avatar object 610 and the like. Note that as a result of canceling the selection UI 180, if the user does not select a move within the time limit, the user's move may be automatically determined. Also, in a state where the selection UI 180 is canceled, an icon (such as a message icon like "Return to the game") for displaying the selection UI 180 again may be displayed, and the selection UI 180 may be redisplayed by an input operation on the icon, or the selection UI 180 may be redisplayed according to the instructions of the performer, or the selection UI 180 may be redisplayed due to the fulfillment of some condition (for example, the time limit reaches a specific value).

[0277] The item input icon 185 is an icon that triggers the input of an item to be given to the avatar object 610. When an input operation on the item input icon 185 is received, a window for selecting and inputting from multiple types of items is displayed. As described above, the selection UI 180 is displayed with priority over the item input icon 185. Therefore, when the selection UI 180 is displayed overlapping the item input icon 185, the input operation on the item input icon 185 may not be effectively received unless the display of the selection UI 180 is canceled.

[0278] The comment input icon 186 is an icon that triggers the input of a comment addressed to the avatar object 610. When an input operation on the comment input icon 186 is received, a window for inputting and sending a comment is displayed. Note that the comment input icon 186 is displayed at a position where the selection UI 180 is not superimposed. Similar to the item input icon 185, an input operation on the comment input icon 186 may not be effectively received unless the display of the selection UI 180 is cleared. In the comment display section 184, a predetermined number of the latest comments addressed to the avatar object 610 are displayed together with the user names in order.

[0279] Figure 33(B) is an example display when the user selects a hand to play. In Figure 33(B), an example where "Gu" is selected is shown. When "Gu" is selected, as shown in Figure 33(B), the button icon of "Gu" in the selection UI 180 is updated to a selected state (a state surrounded by a double circle). Note that in Figure 33(B), since the time limit is "7", it means that the user has decided on the hand to play with 7 seconds remaining. In contrast, for the avatar object 610 in Figure 33(B), a scene is shown where it grabs the option objects 281 to 283 and is thinking, such as "Which one should I choose? Should I choose Choki or Pa?" (motion), and a voice is being output. After the time limit has elapsed, the avatar object 610 raises one of the option objects 281 to 283 upward along with a call such as "Janken hoi" at the timing of the performer.

[0280] FIG. 33(C) is a display example when the avatar object 610 extends its hand after the time limit has elapsed. In FIG. 33(C), the avatar object 610 presents the option object 282 of "scissors". Thereby, it is notified that the avatar object 610 has shown "scissors". When the avatar object 610 extends its hand, information for specifying the type (rock, scissors, paper) of the option object determined as the hand to be extended by the avatar object 610 is received. Based on the information, the processor 10 determines the win or loss of the user by comparing it with the hand shown by the user. In FIG. 33, since user A has selected "rock", it is determined that user A has won. As a result, a message such as "win" is displayed above the avatar object 610.

[0281] In the normal mode, the position, orientation, etc. of the virtual camera 620B are controlled according to a swipe operation, a pinch-in operation / pinch-out operation, etc. by the user. For this reason, when a leftward swipe operation is performed, the processor 10 moves the position of the virtual camera 620B leftward around the avatar object 610. When a rightward swipe operation is performed, the processor 10 moves the position of the virtual camera 620B rightward around the avatar object 610. When an upward swipe operation is performed, the processor 10 changes the orientation of the virtual camera 620B upward. When a downward swipe operation is performed, the processor 10 changes the orientation of the virtual camera 620B downward.

[0282] Furthermore, when a pinch-in operation is performed, the processor 10 moves the position of the virtual camera 620B outward toward the outside of the virtual space 600B. When a pinch-out operation is performed, the processor 10 moves the position of the virtual camera 620B inward (or the center, avatar object 610) toward the inside of the virtual space 600B. Thereby, it becomes possible to display the view image 660 from the user's preferred viewpoint.

[0283] Note that at the timing of Fig. 33(B), compared with the state of Fig. 33(A), a state is shown in which the avatar object 610 changes its orientation to the left according to the movement of the performer. At the timing of Fig. 33(C), compared with the state of Fig. 33(A), a state is shown in which the avatar object 610 changes its orientation to the right according to the movement of the performer. At this time, the user can perform a swipe operation to capture the avatar object 610 from a preferred perspective. At the timing of Fig. 33(B), an example of a display in which the virtual camera 620B is moved to the left by a swipe operation from left to right, thereby capturing the right front part of the avatar object 610, is shown. Also, at the timing of Fig. 33(C), an example of a display in which the virtual camera 620B is moved to the right by a swipe operation from right to left, thereby capturing the left front part of the avatar object 610, is shown.

[0284] Fig. 33(D) is an example of a display when a pinch-out operation is performed with the chest of the avatar object 610 as the origin in the display state of Fig. 33(A). As shown in Fig. 33(D), objects such as the avatar object 610 and the flower wreath object 284 are enlarged and displayed in accordance with the movement of the pinch-out operation, centering on the chest of the avatar object 610 that is the origin of the pinch-out operation. Conversely, when a pinch-in operation is performed in the display state of Fig. 33(D), objects such as the avatar object 610 and the flower wreath object 284 are reduced and displayed in accordance with the movement of the pinch-in operation. Note that even when enlarged or reduced, the virtual camera 620B can be moved left and right and its orientation can be changed up and down while maintaining its size (while maintaining the distance to the avatar object 610). Also, even when enlarged or reduced, icons such as the selection UI 180 are not enlarged or reduced and are displayed in the same state as in Fig. 33(A), so that the user can perform operations on the icons and check comments.

[0285] FIG. 33(E) is a display example when, in the display state of FIG. 33(D), a pinch-out operation is further performed and the destination of movement of the virtual camera 620B specified in step S108 falls within the predetermined range 670. When the destination of movement falls within the predetermined range 670, as described above, control is performed to move the virtual camera 620B to the intersection of the path to the destination specified in step S108 and the outer edge (outer peripheral surface) of the predetermined range 670. Thereby, while avoiding problems such as displaying the inside of the avatar object 610 that the user would not originally want to view, as shown in FIG. 33(E), the avatar object 610 can be viewed closely.

[0286] (Display example in TV mode) FIG. 34 is a display example when the TV mode is set, and FIGS. 34(A) to 34(C) show display examples at the same timings as FIGS. 33(A) to 33(C), respectively. As can be seen from FIGS. 34(B) and 34(C), the avatar object 610, similar to FIG. 33, changes its orientation to the left at the timing of FIG. 34(B) and changes its orientation to the right at the timing of FIG. 34(C), compared to the state of FIG. 34(A).

[0287] Server 200 distributes the viewpoint information of camera object 630 according to the operation of the switcher by the operator of system 1 or the like to a plurality of user terminals 100. In TV mode, the position, orientation, etc. of virtual camera 620B are controlled based on the viewpoint information so as to coincide with the position, orientation, etc. of camera object 630. FIG. 34(B) is a display example when the arrangement of camera object 630 is switched to a position on the right side of the front of virtual space 600A, and accordingly avatar object 610 changes its orientation toward camera object 630. FIG. 34(C) is a display example when the arrangement of camera object 630 is switched to a position on the left side of the front of virtual space 600A, and accordingly avatar object 610 changes its orientation toward camera object 630. That is, when the orientation of avatar object 610 is changed so as to coincide with the line of sight of camera object 630, a visual field image 660 is displayed on touch screen 15 from the viewpoint of viewing avatar object 610 from the front. Therefore, when the TV mode is selected, the user can view the image from the viewpoint considered to be the best by the operator or the like. Also, since the operator or the like will not set the viewpoint to a position where the inside of avatar object 610 that the user would not originally want to view is displayed, it is not necessary to perform movement restriction or non-display control of the viewpoint based on a predetermined range 670 or the like during the TV mode.

[0288] (Display example in AR mode) FIG. 35 is a display example when the AR mode is set, and FIGS. 35(A) to 35(C) each show a display example at the same timing as FIGS. 33(A) to 33(C) and FIGS. 34(A) to 34(C). FIG. 35(B) is a display example when the position of the camera 17 (i.e., the position of the user terminal 100) is moved about 40° to the right around the avatar object 610 from the front position of the AR virtual space 600C at the timing of displaying FIG. 33(B). FIG. 35(C) is a display example when the position of the camera 17 is further moved about 50° to the right around the avatar object 610 from the front position of the AR virtual space 600C at the timing of displaying FIG. 33(C). In FIGS. 35(A) to 35(C), an image of the augmented reality space in which the avatar object 610, the option objects 281 to 283, the flower wreath object 284, etc. are arranged with respect to the acquired image (an image of a room where a picture is decorated on the wall and a desk is placed) acquired by the camera 17 is displayed on the touch screen 15.

[0289] Specifically, FIG. 35(A) shows, for example, a display example when the avatar object 610 in the virtual space 600A is in the state shown in FIG. 28(A). The acquired image is an image taken from the horizontal direction. For this reason, the virtual camera 620B is arranged at a position looking at the AR virtual space 600C from the horizontal direction. As a result, in FIG. 35(A), the avatar object 610 is viewed from the front. As shown in FIG. 35(A), the avatar object 610 is displayed in a standing state at a substantially central position of the floor, which is a flat part of the acquired image. Based on the avatar object 610, the option objects 281 to 283 are arranged in the right front of the avatar object 610 (the same positional relationship as in FIG. 28(A)), and the flower wreath object 284 is arranged behind the avatar object 610 (the same positional relationship as in FIG. 28(A)).

[0290] In the AR mode, the position, orientation, etc. of the virtual camera 620B are controlled according to changes in the position, orientation, etc. of the camera 17. Therefore, when the position of the camera 17 (i.e., the position of the user terminal 100) is changed about 40° to the right from the position in the front of the AR virtual space 600C, the processor 10 changes the position of the virtual camera 620B about 40° to the right. Also, when the position of the camera 17 is further changed about 50° to the right from the position in the front of the AR virtual space 600C, the processor 10 further changes the position of the virtual camera 620B about 50° to the right. As a result, when the position of the virtual camera 620B is changed about 40° to the right, the display example shown in FIG. 35(B) is switched to, and when the position of the virtual camera 620B is further changed about 50° (total 90°) to the right, the display example shown in FIG. 35(C) is switched to.

[0291] FIG. 35(D) is a display example when the camera 17 is moved to a position (a position outside the predetermined range 680) close to the outer edges of the predetermined range 680 in the display state of FIG. 35(A). Objects such as the avatar object 610 and the flower wreath object 284 are enlarged and displayed according to the position (movement) of the camera 17. Conversely, when the camera 17 is moved in a direction away from the avatar object 610 in the display state of FIG. 35(D), objects such as the avatar object 610 and the flower wreath object 284 are reduced and displayed according to the position (movement) of the camera 17. Note that even when enlarged or reduced, icons such as the selection UI 180 are not enlarged or reduced and are displayed in the same state as in FIG. 35(A) as illustrated in FIG. 35(D), so that the user can operate the icons and check comments.

[0292] FIG. 35(E) is a display example when, in the display state of FIG. 35(D), the camera 17 is further moved in a direction approaching the avatar object 610 and the virtual camera 620B corresponding to the camera 17 is within the predetermined range 680. When the virtual camera 620B is within the predetermined range 680, as described above, control is performed to make the avatar object 610 arranged in the AR virtual space 600C invisible at step S115. For this reason, as shown in FIG. 35(E), the avatar object 610 becomes invisible, and only the flower wreath object 284 arranged behind it is displayed. In this way, in the AR mode, while moving the virtual camera 620B corresponding to the position of the camera 17 regardless of whether it is within the predetermined range 680, when the position of the virtual camera 620B is within the predetermined range 680, the above problem can be avoided by making the avatar object 610 invisible.

[0293] (Mode Switching) The AR mode transition icon 182a is an icon for transitioning to the AR mode. The TV mode transition icon 182b is an icon for transitioning to the TV mode. In the normal mode, the AR mode transition icon 182a and the TV mode transition icon 182b are superimposed and displayed in the normal state in the upper right of the view image 660 (see FIG. 33). For this reason, by detecting a touch operation or the like on the TV mode transition icon 182b, the viewing mode can be switched from the normal mode to the TV mode. Also, by detecting a touch operation or the like on the AR mode transition icon 182a, the viewing mode can be switched from the normal mode to the AR mode.

[0294] In the TV mode, the AR mode transition icon 182a is displayed in the normal state at the same position as in the normal mode, and the TV mode transition icon 182b is displayed in the emphasized state (e.g., black and white inversion display) at the same position as in the normal mode (see FIG. 34). By the display in the emphasized state, it is notified that the TV mode is in progress. If the AR mode transition icon 182a is tapped, the viewing mode is switched from the TV mode to the AR mode. On the other hand, if the TV mode transition icon 182b displayed in the emphasized state is tapped, the viewing mode is switched from the TV mode to the normal mode. Also, in the TV mode, by performing, for example, a slide operation, a swipe operation, a pinch-in / pinch-out operation, etc. on an area other than the area where the AR mode transition icon 182a and the TV mode transition icon 182b are displayed, the viewing mode is also switched from the TV mode to the normal mode.

[0295] In the AR mode, the AR mode transition icon 182a is displayed in the emphasized state (e.g., black and white inversion display) at the upper right position (the same position as in the normal mode) of the image in the augmented reality space, and an AR mode reset icon 182c is displayed in the normal state instead of the TV mode transition icon 182b below the AR mode transition icon 182a (see FIG. 35). By the display of the AR mode transition icon 182a in the emphasized state, it is notified that the AR mode is in progress, and by the display of the AR mode reset icon 182c in the normal state, it is notified that the AR mode can be reset. If the AR mode transition icon 182a displayed in the emphasized state is tapped, the viewing mode is switched from the AR mode to the normal mode. On the other hand, if the AR mode reset icon 182c is tapped, the flat surface such as the floor surface is identified again, and the view image in the AR virtual space 600C is arranged at a predetermined position on the flat surface.

[0296] That is, in this embodiment, in the normal mode, an icon corresponding to a viewing mode different from the normal mode is displayed. When the icon is touched, the viewing mode corresponding to the icon is shifted to, and the icon is displayed in an emphasized manner. When the icon displayed in the emphasized manner in the shifted-to viewing mode is touched, the normal mode is restored. Also, when the icon displayed in the normal manner in the shifted-to viewing mode is touched, the mode corresponding to the icon is shifted to. Thereby, the user can switch the viewing mode while recognizing which mode the currently selected viewing mode is.

[0297] When switched to the TV mode, based on the viewpoint information transmitted from the server 200, the virtual camera 620B is arranged in the virtual space 600B. As a result, the position, orientation, etc. of the virtual camera 620B coincide with the position, orientation, etc. of the camera object 630.

[0298] When switched to the AR mode, the shooting direction (orientation, tilt, etc.) of the camera 17 when the acquisition image was taken is specified, and the virtual camera 620B is arranged at a position where the AR virtual space 600C is viewed from the same direction as the shooting direction. Specifically, when the shooting direction is substantially horizontal, the virtual camera 620B is arranged at a position where the AR virtual space 600C is viewed from the horizontal direction. On the other hand, when the shooting direction is a direction looking down from obliquely above, the virtual camera 620B is arranged at a position where the AR virtual space 600C is viewed from obliquely above.

[0299] When switched to the normal mode, the virtual camera 620B is placed at a position corresponding to the position where the virtual camera 620B was placed before the mode switch. Specifically, if the viewing mode before the switch is the TV mode, the placement of the virtual camera 620B in the virtual space 600B is maintained. On the other hand, if the viewing mode before the switch is the AR mode, the position where the virtual camera 620B was placed in the AR virtual space 600C is specified, and the virtual camera 620B is placed at the same position as the specified position in the virtual space 600B. When the viewing mode before the switch is the AR mode and the position where the virtual camera 620B was placed in the AR virtual space 600C is within the predetermined range 670, the virtual camera 620B is placed at the nearest position outside the predetermined range 670 from the position where the virtual camera 620B was placed. Note that this is not limited to this. When the viewing mode before the switch is the AR mode and the position where the virtual camera 620B was placed in the AR virtual space 600C is within the predetermined range 670, for example, while placing the virtual camera 620B at the same position as the position where the virtual camera 620B was placed in the AR virtual space 600C, the avatar object 610 is made non-displayed (while other objects are displayed) even during the normal mode until the virtual camera 620B moves outside the predetermined range 670 after the switch to the normal mode, and the avatar object 610 may be displayed after moving the virtual camera 620B outside the predetermined range 670 in response to a movement operation.

[0300] <Effects of the present embodiment> According to this embodiment, the avatar object 610 is a character that operates in real time according to the movements of the performer, and the movement instruction data transmitted from the server 200 is data for displaying the virtual space in which the avatar object 610 is arranged. In the user terminal 100, any one of a plurality of types of viewing modes with different modes for moving the virtual camera 620B (viewpoint) is specified, and an image corresponding to the specified viewing mode is displayed on the display unit 152. In addition, a normal mode for moving the viewpoint according to a movement operation on the touch screen 15 and an AR mode for moving the viewpoint according to an operation for moving the user terminal 100 (camera 17) itself are provided. In the normal mode, the movement of the viewpoint within a predetermined range 670 where the avatar object 610 can be arranged is restricted, and in the AR mode, the movement of the viewpoint within a predetermined range 680 is allowed, but when the viewpoint is moved within the predetermined range 680, the avatar object 610 is made non-displayed. Thereby, it becomes possible to move the viewpoint according to the viewing mode, and the interest is improved. In addition, problems such as displaying the inside of the avatar object 610 that the user would not originally want to view can be avoided.

[0301] Further, since the AR mode is a mode for moving the virtual camera 620B according to a change in the position of the user terminal 100 (camera 17), if the movement is restricted in the same way as in the normal mode, it will give the user a sense of discomfort. However, when the viewpoint is moved within the predetermined range 680 while allowing movement within the predetermined range 680, the avatar object 610 is made non-displayed. Thereby, it is possible to avoid the above problems while reducing the sense of discomfort even during the AR mode.

[0302] Also, in the AR mode, an image in the augmented reality space in which the avatar object 610 and the like are arranged is displayed for the acquired image acquired by the camera 17. Thereby, it is possible to give the user an impression (a sense of presence) as if the avatar object 610 and the like are actually operating in the real space in front of the user's eyes.

[0303] It is also possible to switch to a TV mode in which an image within the virtual space 600B is displayed from the same viewpoint as that of the camera object 630 arranged in the virtual space 600A. As a result, the user can view an image from the viewpoint considered to be the best by the switcher on the distribution side.

[0304] In addition, the predetermined ranges 670 and 680 correspond to predetermined ranges in the virtual spaces 600A to 600C that are within a predetermined range (for example, a range within a radius of 2 m) from a predetermined position on the floor where the performer actually exists. Thereby, the processing load can be reduced as compared with, for example, a case where the predetermined range 670 or the like varies according to the position of the avatar object 610. Also, the predetermined range 670 in the normal mode and the predetermined range 680 in the AR mode are defined to be the same range. Thereby, the processing load can be reduced and the performer can be prevented from being confused as compared with, for example, a case where the range is made different according to the mode.

[0305] In addition, both the normal mode and the AR mode display an image from a viewpoint that is moved according to operations, movements, etc. from the same user who views the distribution image including the avatar object 610. For this reason, while it is possible for the user to view the objects arranged in the virtual space from the desired viewpoint, it is also possible to display different images according to the viewing mode, improving the interest.

[0306] Furthermore, in the TV mode in the present embodiment, the same viewpoint as that of the camera object 630 is specified based on the viewpoint information transmitted from the server 200. Thereby, an increase in the data amount can be prevented.

[0307] Further, according to this embodiment, the operation instruction data includes motion data input by the performer who plays the avatar object 610. In the normal mode, an image within the virtual space 600B where the avatar object 610 is placed is displayed from the viewpoint according to the user's operation. In the AR mode, an image within the augmented reality space where the avatar object 610 is placed is displayed for the acquired image acquired by the camera 17. In the TV mode, an image within the virtual space 600B where the avatar object 610 is placed is displayed from the same viewpoint as the viewpoint of the camera object 630 placed in the virtual space 600A. Thereby, even though receiving the same motion data (operation instruction data), it becomes possible to display different images according to the viewing mode.

[0308] Furthermore, according to this embodiment, an icon for switching the viewing mode is displayed on the touch screen 15. For this reason, it becomes possible to switch the viewing mode by a simple operation such as tapping the icon.

[0309] <Modification Example> Modification examples of the embodiment described above are listed below.

[0310] (1) In the above embodiment, an example has been described in which both the predetermined range 670 in the normal mode and the predetermined range 680 in the AR mode are ranges that are predetermined as ranges within which the performer can move from a predetermined position. However, the predetermined range 670 in the normal mode and the predetermined range 680 in the AR mode are not limited to the predetermined ranges, and may be ranges based on the current position of the performer (which is also the avatar object 610) (for example, a range within 1 m from the current position). In this case, the positions of the predetermined range 670 in the normal mode and the predetermined range 680 in the AR mode can move according to the current position of the performer. Thereby, the predetermined range can be suppressed to as narrow a range as possible.

[0311] In addition, in the above-described embodiment, an example in which the predetermined range 670 in the normal mode and the predetermined range 680 in the AR mode are the same range has been described. However, the present invention is not limited to this, and the predetermined range 670 in the normal mode and the predetermined range 680 in the AR mode may be defined as different ranges. For example, the predetermined range 670 in the normal mode may be defined as a range wider than the predetermined range 680 in the AR mode. Thereby, during the AR mode, the avatar object 610 can be visually recognized from a closer position, and the interest can be improved. Further, for example, the predetermined range 670 in the normal mode may be defined as a range that is previously defined as a range within which the performer can move from a predetermined position, and the predetermined range 680 in the AR mode may be defined as a range based on the current position of the performer. Thereby, the predetermined range during the AR mode can be made as narrow as possible, and the avatar object 610 can be visually recognized from a closer position. The motivation to watch in the AR mode can be enhanced.

[0312] (2) In the above embodiment, the content in which the avatar object 610 appears may be regularly distributed daily or at a predetermined time on a predetermined day of the week. In addition, decorative objects (texture images) such as clothes and accessories are associated with the avatar object 610. For example, on each part of the upper body, lower body, hands, feet, neck, earlobes, eye areas, etc. of the character that becomes the avatar object 610, decorative objects corresponding to clothes and accessories (necklaces, earrings, glasses, etc.) are pre-associated. Under such a premise, the distributor may change, for example, the display mode of a specific object (for example, earrings) associated with the avatar object 610 for each distribution. As a result, the user, who is the viewer, can experience that the display mode of a specific object changes to a different one for each distribution regardless of whether there is a request from the user. As a result, a sense of reality of the avatar object 610 can be given to the user. Also, in the case of the AR mode, the viewpoint can be moved to a position where a specific object associated with the avatar object 610 can be displayed, while in the case of the normal mode, the viewpoint may be made immovable to a position where a specific object associated with the avatar object 610 can be displayed. An example will be described with reference to FIG. 36.

[0313] FIG. 36 shows an example in which a character imitating a woman wearing a hat appears in the virtual space 600B as the avatar object 610. Decorative objects such as a one-piece dress that the character wears all over the body, a hat that covers the head, and piercings that are attached to the ears are associated with the avatar object 610. In FIG. 36, the piercings among the decorative objects associated with the avatar object 610 correspond to specific objects. Therefore, the display mode of the piercings is set to a different display mode for each distribution.

[0314] First, referring to FIG. 36(A), the normal mode will be described. FIG. 36(A-1) is a display example when the avatar object 610 faces forward and raises the index finger of the right hand forward. FIG. 36(A-2) is a display example when the character faces diagonally forward to the right and points forward with the index finger of the right hand. In the normal mode, the position, orientation, and tilt of the virtual camera 620B are changed according to swipe operations, pinch-in operations, pinch-out operations, etc. However, the piercing is covered by a hat corresponding to another object, and no matter how swipe operations or the like are performed during the normal mode, the virtual camera 620B (viewpoint) cannot be moved to a position where the piercing can be seen (for example, a position where FIGS. 36(B-3) and 36(B-4) described later can be displayed). For example, the predetermined range 670 in the normal mode may be defined to include all positions where the piercing (specific part) can be seen.

[0315] On the other hand, the predetermined range 680 in the AR mode may be defined not to include at least a part of the positions where the piercing (specific part) can be seen. Referring to FIG. 36(B), the AR mode will be described. FIGS. 36(B-1) and 36(B-2) respectively show display examples during the AR mode at the same timing as FIGS. 36(A-1) and 36(A-2). In the AR mode, among the images of the augmented reality space in which the avatar object 610 is arranged with respect to the acquired image obtained by photographing with the camera 17, the image from the virtual camera 620B is displayed on the display unit 152. Also, the position, orientation, tilt, etc. of the virtual camera 620B are changed according to the actual position, orientation, tilt, etc. of the camera 17.

[0316] Therefore, if the position and orientation of the camera 17 are adjusted so that the left ear is looked up from diagonally below the left side of the avatar object 610, the virtual camera 620B moves to a position where the piercing can be seen, and the image of the piercing is displayed on the display unit 152 (see (B-3) of FIG. 36(B)). That is, the display mode of the piercing can be confirmed by selecting the AR mode and adjusting the position, orientation, etc. of the camera 17. Thereby, the user can be motivated to watch in the AR mode, and thus the interest can be improved. If the position and orientation of the camera 17 are adjusted so that the left half of the character is looked up from the feet of the character, an image of the left half including the feet is displayed on the display unit 152 as shown in (B-4) of FIG. 36(B). When approaching the avatar object 610 too much and the virtual camera 620B (viewpoint) enters the predetermined range 680, the avatar object 610 becomes invisible. Therefore, it is possible to provide the user with the fun of being able to capture the piercing from any position and angle without entering the predetermined range 680.

[0317] As described above, in the AR mode, by adjusting the position, orientation, etc. of the camera 17, the virtual camera 620B can be moved to a position where the piercing associated with the avatar object 610 can be displayed (for example, a position where it is looked up from the neck of the character, a position where it is looked up from the feet of the character, etc.). As a result, the piercing can be displayed, and the user can confirm the display mode of the piercing. On the other hand, in the normal mode, as described above, no matter how the operation is performed, the virtual camera 620B cannot be moved to a position where the piercing associated with the character can be displayed (for example, a position where it is looked up from the feet of the character, etc.). As a result, the piercing, which is a specific object, cannot be displayed, and the user cannot confirm the display mode of the piercing.

[0318] In FIG. 36, a piercing is exemplified as a specific object among the decorative objects associated with the avatar object 610, and an example of changing the display mode of the piercing was described. However, the specific object may be an object associated with the avatar object 610, and is not limited to a shirt or a piercing, but may be an object such as socks, pants, hairpins, gloves, hats, underwear, etc. Further, the specific object is not limited to a decorative object, and may be, for example, an object related to the character itself (for example, hairstyle, nails, etc.).

[0319] Note that regardless of the predetermined range in each of the normal mode and the AR mode, in the case of the AR mode, the viewpoint can be moved to a position where a specific object associated with the avatar object 610 can be displayed, and in the case of the normal mode, the viewpoint cannot be moved to a position where a specific object associated with the avatar object 610 can be displayed. The movable range of the virtual camera 620B (viewpoint) may be controlled. Further, the specific object in this case is not limited to an object whose display mode is changed for each distribution, and may be an object whose display mode is not changed and is uniform.

[0320] Further, when the virtual camera 620B (viewpoint) moves to a specific position corresponding to a part of the position where a specific object can be seen during the AR mode, the avatar object 610 may be made non-displayed. For example, a predetermined range 680 during the AR mode may include the specific position.

[0321] Further, the avatar object 610 may be defined as having a part that cannot be captured from the viewpoint of the virtual camera 620B regardless of the viewing mode. For example, in the avatar object 610 shown in FIG. 36, for example, the part corresponding to the right ear opposite to the left ear with the piercing is hidden by the brim of the hat, and it may be defined that it cannot be captured from the viewpoint of the virtual camera 620B in any viewing mode.

[0322] (3) In the above embodiment, as the predetermined range in the normal mode and the AR mode, a cylindrical range was exemplified. However, it is not limited to this as long as it is a range including the shape of the avatar object 610, and it may be a range having a shape where the lower part is wider than the upper part (for example, a tapered shape), or it may be a range corresponding to the shape of the avatar object 610 itself.

[0323] (4) In the TV mode in the above embodiment, since the operator or the like does not set the viewpoint at a position where the inside of the avatar object 610 that is not originally intended for the user to view is displayed, an example where movement restriction or non-display control of the viewpoint based on the predetermined range 670 or the like is not performed was described. However, for example, in the terminal on the distribution side (for example, the distribution terminal 400, the game play terminal 300, etc.), when the operator accidentally sets the viewpoint within the range corresponding to the predetermined range 670 or the like, a process of notifying the operator to that effect may be performed. Also, a process for restricting the operator from setting the viewpoint within the range corresponding to the predetermined range 670 or the like may be performed.

[0324] Also, in the terminal on the distribution side (for example, the distribution terminal 400, the game play terminal 300, etc.), when the performer goes out of the predetermined range corresponding to the predetermined range 670 or the like, or when the distance to the predetermined range becomes within a predetermined distance (for example, 30 cm), a process of notifying the performer and the operator to that effect may be performed.

[0325] Also, on the user terminal 100 side, when the avatar object 610 goes out of the predetermined range, a process for changing the setting so that the position of the avatar object 610 becomes a predetermined position within the predetermined range may be performed, and the avatar object 610 may be forcibly moved (sideways display) to a position within the predetermined range.

[0326] (5) In the above-described embodiment, in the normal mode, the movement of the viewpoint within a predetermined range 670 where the avatar object 610 can be arranged is restricted, and in the AR mode, the movement of the viewpoint within a predetermined range 680 is allowed, but when the viewpoint is moved within the predetermined range 680, the avatar object 610 is made invisible. However, in the AR mode, the movement of the viewpoint within a predetermined range 670 where the avatar object 610 can be arranged may be restricted, and in the normal mode, the movement of the viewpoint within a predetermined range 680 may be allowed, but when the viewpoint is moved within the predetermined range 680, the avatar object 610 may be made invisible.

[0327] (6) In the above-described embodiment, a predetermined range is provided corresponding to the avatar object 610, and in the normal mode, the movement of the virtual camera 620B (viewpoint) within the predetermined range is restricted, and in the AR mode, while the movement within the predetermined range is allowed, the avatar object 610 is made invisible. However, the object for which such control is performed is not limited to the avatar object 610 played by the performer, and may be applied to other objects. Also, for objects other than the avatar object 610 played by the performer (for example, the wreath object 284, etc.), a predetermined range is provided corresponding to each of the objects, and regardless of the viewing mode, the movement of the virtual camera 620B (viewpoint) within the predetermined range is allowed, while the object corresponding to the predetermined range into which the virtual camera 620B (viewpoint) has entered may be made invisible. Alternatively, for objects other than the avatar object 610 played by the performer (for example, the wreath object 284, etc.), the movement of the virtual camera 620B (viewpoint) within the predetermined range may be restricted regardless of the viewing mode.

[0328] (7) In the above-described embodiment, an example in which, in the AR mode, the avatar object 610 and the selection objects 281 to 283 are arranged in the acquired image obtained from the camera 17 has been described. However, in the AR mode, the sizes of the avatar object 610 and the selection objects 281 to 283 arranged with respect to the acquired image may be adjustable according to a user operation.

[0329] When the display size at the time of arranging the view image including the avatar object 610 and the selection objects 281 to 283 is determined according to the size of the flat portion specified by analyzing the acquired image, depending on the analysis result, the avatar object 610 or the like may be extremely large or conversely small with respect to the acquired image. In such a case, by accepting a user operation and adjusting the display size of the avatar object 610 or the like to an appropriate display size or a display size preferred by the user with respect to the acquired image, it is possible to avoid the inconvenience of giving a sense of discomfort to the display of the avatar object 610 or the like.

[0330] (8) In the above-described embodiment, an example in which the augmented reality space is a space in which only some objects in the virtual space 600A are arranged with respect to the acquired image from the camera 17 has been described. However, the object arranged with respect to the acquired image from the camera 17 is not limited to this as long as it includes the avatar object 610 played by the player, and may be all the objects arranged in the virtual space 600A. That is, the augmented reality space may be a space in which the entire same virtual space as the virtual space 600A is arranged with respect to the acquired image from the camera 17. Specifically, an augmented reality space may be generated in which the entire virtual space 600A described above is miniaturized and arranged with respect to the acquired image from the camera 17, and the avatar object 610 or the like may be operated within the augmented reality space.

[0331] (9) In the above embodiment, the viewing mode is switched according to the user's operation. However, the viewing mode may be specified on the operator side according to the type of content and the time zone, and mode information capable of specifying the viewing mode may be included in the data distributed from the game play terminal 300 to a plurality of user terminals 100. The user terminal 100 may be configured to view the distributed content in the viewing mode based on the mode information. Further, information for requesting the operator to switch to the TV mode may be distributed, and the viewing mode may be set to the TV mode during the period when the information is being distributed. Thereby, for example, during the period when the avatar object 610 changes its clothes or the like, the direction of the camera object 630 can be changed to a direction in which the avatar object 610 cannot be photographed, and the inconvenience that the scene of the clothes change is displayed on the user terminal 100 can be avoided. Note that even during the period when the information is being distributed, switching of the viewing mode may be permitted for specific users such as premium users.

[0332] (10) An example has been described in which the acquired image constituting the augmented reality space in the above embodiment is an image captured in real time by the camera 17. However, the acquired image constituting the augmented reality space is not limited to an image captured in real time, and may be an image captured in the past. Further, the acquired image constituting the augmented reality space may be an image provided from a service providing device (server) that provides images captured at various locations in the real space (including panoramic images and the like). In this case, the image provided may be, for example, an image of the location where the user terminal 100 is present, or may be an image of the location designated by the user with respect to the map when playing a so-called location game in which the user moves on the map.

[0333] Also, in the generation of the augmented reality space, an example was described in which the acquired image was used as it was, and a character or the like was superimposed thereon and displayed on the touch screen 15. However, with regard to the acquired image used for generating the augmented reality space, predetermined processing / editing may be performed, and a character or the like may be superimposed on the image subjected to the processing / editing and displayed on the touch screen 15. The predetermined processing / editing may be, for example, processing such as enlargement / reduction in response to an input operation, or may be processing such as gradation, sharpness, color correction, and special effects in response to the current situation (for example, time, date, time, etc.).

[0334] In the above-described embodiment, as a method of displaying an image in an augmented reality space in which some objects in the virtual space 600A are arranged on the acquired image, an example was shown with reference to FIGS. 28(B) and 35. However, as a method of displaying such an image in the augmented reality space, any method may be used as long as it arranges and operates an avatar object 610 or the like on the acquired image acquired by the camera 17. For example, in the AR virtual space 600C, all the objects (viewer object, floor object, background object, etc.) are arranged in the same manner as in the virtual space 600A, and then the display settings of other objects (viewer object and decoration object) other than the avatar object 610 and the option objects 281 to 283 are set to non-display. Only the avatar object 610 and the option objects 281 to 283 are included as the view image from the virtual camera 620B, and no other objects are included. The view image may be superimposed on the acquired image from the camera 17 to generate an image in the augmented reality space. In this case, when an input operation to the AR mode transition icon 182a highlighted is received, for the objects (viewer object, floor object, background object, etc.) arranged in the AR virtual space 600C but set to non-display, the display setting is switched so that the avatar object 610 and the like are operated in the same virtual space as the virtual space 600A instead of the augmented reality space. Thereby, the processing at the time of normal mode transition or the like can be simplified and the processing load can be reduced.

[0335] (11) The display mode of the avatar object 610 in the above embodiment may vary for each user according to, for example, the type of acquisition image used for generating the augmented reality space (for example, a room image, a shopping mall image, a landscape image). For example, even if the game progress information transmitted from the game play terminal 300 side is the same, for a user terminal with an acquisition image being a room image, the clothing of the avatar object 610 may be pajamas, and for a user terminal with an acquisition image being a mountain image, the clothing of the avatar object 610 may be mountain climbing clothes. In this case, the clothing data of the avatar object 610 may be stored in advance in a plurality of types on each user terminal 100 side. Note that the object for which the display mode of the avatar object 610 is made different is not limited to clothing, and may be a hairstyle, skin color, degree of makeup, etc. Thereby, without increasing the processing load on the game play terminal 300 side that outputs the game progress information, the variations of the augmented reality space can be increased (substantially infinite depending on the acquisition image), so that it is possible to attract users.

[0336] Alternatively or in addition, the display mode of the avatar object 610 may be made different from the display mode displayed on the user terminals of other users according to the payment by the user during the game. Also, the display mode of the avatar object 610 may be made different according to the user ranking updated according to the result of a live game or the like.

[0337] (12) In step S92 of FIG. 29 in the above embodiment, an example in which the virtual camera 620B is arranged in the same direction as the shooting direction (direction, inclination, etc.) of the camera 17 when the acquisition image is shot has been described. However, the position, orientation, etc. for initially arranging the virtual camera 620B are not limited to this, and may be those set according to the type of content, those set based on the analysis result in step S88, those set at predetermined positions, orientations, etc.

[0338] Also, after the virtual camera 620B is initially arranged in step S92, as shown in step S112 of FIG. 30, an example of controlling the virtual camera 620B in synchronization with changes in the position, orientation, inclination, etc. of the camera 17 was described. However, after the initial arrangement, the virtual camera 620B may be controlled to change its position, orientation, inclination, etc. in response to an input operation from the user (e.g., a swipe operation, a touch operation, etc.). Thereby, the viewing area of the virtual camera 620B can be changed in response to the input operation from the user, and as a result, the viewing image arranged on the acquired image can be changed. Specifically, by a swipe operation, an avatar object or the like arranged on the acquired image (e.g., only the avatar object 610, the avatar object 610 and other objects, the entire virtual space) may be rotated or moved. Thereby, for example, even when generating an augmented reality space and operating the avatar object 610 or the like in a narrow space where the position, orientation, etc. of the user terminal 100 cannot be sufficiently adjusted, the avatar object 610 or the like can be rotated and moved by an input operation on the touch screen 15.

[0339] (13) In the above-described embodiment, as the content that progresses in the virtual space 600B or the augmented reality space, a live game in which a player and a user play against each other, such as the rock-paper-scissors game shown in FIGS. 33 to 35, was exemplified. However, the live game that progresses in the virtual space 600B or the augmented reality space is not limited to this. For example, in a puzzle game where the player and the user alternately operate a pachinko machine and compete to see how many objects imitating the fruits launched by the pachinko can be placed on an object imitating a tray arranged in the front (which one can place them without failure), or it may be a participation-type game in which an avatar object operated by a player, soldier objects operated by each of a plurality of users, and enemy objects operated by a game program appear, and the user defeats the enemy objects while protecting the avatar object 610. Also, the content that progresses in the virtual space 600B or the augmented reality space is not limited to live games. For example, it may be viewing-type content for the user to view the state of the avatar object 610 singing live or chatting live.

[0340] (14) In the above-described embodiment, the camera 17 of the user terminal 100 is provided on the surface opposite to the surface on which the display unit 152 is provided, and it is assumed that the acquired image displayed on the display unit 152 when being photographed by the camera 17 is included in (matches) the real scenery directly viewed by the user. However, the camera 17 of the user terminal 100 is not limited to being provided on the surface opposite to the surface on which the display unit 152 is provided. It may be provided on the surface on which the display unit 152 is provided (such as one that allows the avatar object 610 etc. to be arranged and operated on the acquired image including the user himself), or it may be provided separately from the user terminal 100 so as to be communicable with the user terminal 100.

[0341] (15) In the above embodiment, the video in which the avatar object 610 is operating is displayed on the display unit 152 based on the operation instruction data distributed by real-time rendering. However, the video may be displayed based on, for example, 360° video data or the like distributed by the video distribution method. In the case of 360° video data, an image from a viewpoint controlled according to the viewing mode may be displayed.

[0342] <Supplementary Note> The matters described in each of the above embodiments are appended below.

[0343] (Supplementary Note 1): According to an aspect of an embodiment shown in the present disclosure, a program executed in an information terminal device (user terminal 100) including a processor, a memory, an input unit, and a display unit, the processor receiving display data (operation instruction data) for displaying a virtual space in which a character is arranged (step S26), specifying any one of a plurality of types of viewing modes having different modes for moving a viewpoint in the virtual space (steps S81, S84, S103, S106), and displaying, on the display unit, an image from a viewpoint that moves in a manner corresponding to the viewing mode specified in the specifying step among the images in the virtual space based on the display data (steps S85 to S87, S92, S93, S104, S105, S110, S111, S114), the plurality of types of viewing modes being viewing modes in which the viewpoint can be moved according to an input mode to the input unit from a user, including a first viewing mode (normal mode) that can restrict the movement of the viewpoint within a predetermined range where the character can be arranged, and a second viewing mode (AR mode) that allows the movement of the viewpoint within a predetermined range where the character can be arranged but can make the character non-displayed when the viewpoint is moved within the predetermined range.

[0344] (Supplementary Note 2): (Supplementary Note 1): In the input unit, the input unit includes a first input unit (touch screen 15) that specifies an input operation on the display unit, and a second input unit (gyro sensor, GPS, etc.) that specifies the position of the information terminal device. The first viewing mode is a viewing mode in which the viewpoint can be moved according to the operation mode specified by the first input unit, and the second viewing mode is a viewing mode in which the viewpoint can be moved according to the position specified by the second input unit.

[0345] (Supplementary Note 3): (Supplementary Note 2): In the information terminal device, the information terminal device includes an imaging unit (camera 17). The second viewing mode is a viewing mode (AR mode) in which an image from a viewpoint corresponding to the position specified by the second input unit is displayed among the images in the augmented reality space where the character based on the display data is arranged on the acquired image acquired by the imaging unit.

[0346] (Supplementary Note 4): (In any one of Supplementary Notes 1 to 3), the display data includes viewpoint information for specifying a predetermined viewpoint (the same viewpoint as the viewpoint of the camera object 630) regardless of the user's operation mode. The plurality of types of viewing modes include a viewing mode (TV mode) in which an image from a predetermined viewpoint specified based on the viewpoint information is displayed among the images in the virtual space based on the display data.

[0347] (Supplementary Note 5): (In any one of Supplementary Notes 1 to 4), the character is a character that operates in real time according to the movement of the performer, and the predetermined range is a range predetermined as a range within which the performer can move from a predetermined position.

[0348] (Supplementary Note 6): (In any one of Supplementary Notes 1 to 4), the character is a character that operates in real time according to the movement of the performer, and the predetermined range is a predetermined range based on the current position of the performer.

[0349] (Appendix 7): In any one of (Appendix 1) to (Appendix 6), the predetermined range includes a position where a specific part of the character can be displayed when the viewpoint moves.

[0350] (Appendix 8): In any one of (Appendix 1) to (Appendix 7), the predetermined range in the first viewing mode and the predetermined range in the second viewing mode are the same range.

[0351] (Appendix 9): In any one of (Appendix 1) to (Appendix 7), the predetermined range in the first viewing mode is defined as a wider range than the predetermined range in the second viewing mode.

[0352] (Appendix 10): In any one of (Appendix 1) to (Appendix 9), both the first viewing mode and the second viewing mode are viewing modes that display an image from a viewpoint moved according to an input mode to the input unit from the same user who views the image displayed on the display unit.

[0353] (Appendix 11): In any one of (Appendix 1) to (Appendix 10), a predetermined specific object is associated with the character, and the display mode of the specific object is changed regardless of whether there is a request from the user who owns the information terminal device. When it is the second viewing mode, the viewpoint can be moved to a position where the specific object associated with the character can be displayed. When it is the first viewing mode, the viewpoint cannot be moved to a position where the specific object associated with the character can be displayed.

[0354] (Appendix 12): According to an aspect of an embodiment shown in the present disclosure, there is provided a method executed in an information terminal device (user terminal 100) including a processor, a memory, an input unit, and a display unit, the method including: receiving display data (operation instruction data) for displaying a virtual space in which characters are arranged (S26); specifying any one of a plurality of types of viewing modes having different modes for moving a viewpoint within the virtual space (S81, S84, S103, S106); and displaying, on the display unit, an image from a viewpoint that moves in a manner corresponding to the viewing mode specified in the specifying step, among the images in the virtual space based on the display data (S85 to S87, S92, S93, S104, S105, S110, S111, S114). The plurality of types of viewing modes include: a first viewing mode (normal mode) that can be moved in a viewpoint within a predetermined range where the character can be arranged and that can restrict the movement of the viewpoint within the predetermined range according to an input mode to the input unit from the user; and a second viewing mode (AR mode) that allows the movement of the viewpoint within the predetermined range where the character can be arranged and that can make the character non-displayed when the viewpoint is moved within the predetermined range.

[0355] (Appendix 13): According to an aspect of an embodiment shown in the present disclosure, there is provided an information terminal device (user terminal 100), comprising a storage unit that stores a program, and a control unit that controls the operation of the information processing device by executing the program. The control unit performs the steps of: receiving display data (operation instruction data) for displaying a virtual space in which characters are arranged (step S26); specifying any one of a plurality of types of viewing modes having different modes for moving a viewpoint within the virtual space (steps S81, S84, S103, S106); and displaying, on the display unit, an image from a viewpoint that moves in a manner corresponding to the viewing mode specified in the specifying step, among the images in the virtual space based on the display data (steps S85 to S87, S92, S93, S104, S105, S110, S111, S114). The plurality of types of viewing modes include: a first viewing mode (normal mode) that can restrict the movement of the viewpoint within a predetermined range where the character can be arranged, as a viewing mode in which the viewpoint can be moved according to an input mode to the input unit from the user; and a second viewing mode (AR mode) that allows the movement of the viewpoint within a predetermined range where the character can be arranged, but can make the character non-displayed when the viewpoint is moved within the predetermined range.

[0356] [Example of Realization by Software] The control blocks of the user terminal 100, the server 200, the game play terminal 300 (HMD set 1000), and the distribution terminal 400 (particularly the control units 110, 210, 310, 410) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

[0357] In the latter case, the user terminal 100, the server 200, the game play terminal 300 (HMD set 1000), and the distribution terminal 400 are each provided with a computer that executes program instructions of software that realizes each function. This computer includes, for example, one or more processors and a computer-readable recording medium that stores the above program. Then, in the above computer, when the above processor reads and executes the above program from the above recording medium, the object of the present invention is achieved. As the above processor, for example, a CPU (Central Processing Unit) can be used. As the above recording medium, "non-transitory tangible media" such as ROM (Read Only Memory), tapes, disks, cards, semiconductor memories, programmable logic circuits, etc. can be used. Further, it may further include a RAM (Random Access Memory) for expanding the above program. Further, the above program may be supplied to the above computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. Note that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

[0358] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

Explanation of Reference Numerals

[0359] 1 System, 2 Network, 3, 3A, 3B User (First User), 4 Player (Performer), 10, 20, 30, 40 Processor, 11, 21, 31, 41 Memory, 12, 22, 32, 42 Storage, 13, 23, 33, 43 Communication IF, 14, 24, 34, 44 Input / Output IF, 15, 45 Touch Screen, 17 Camera, 18 Distance Measuring Sensor, 51 Monitor, 52 Gaze Sensor, 53 First Camera, 54 Second Camera, 55 Microphone, 56 Speaker, 100, 100A, 100B, 100C User Terminal (Computer, First Computer, First Information Processing Device), 110, 210, 310, 410 Control Unit (First Control Unit, Second Control Unit), 111, 311, 413 Operation Reception Unit, 112, 312, 412 Display Control Unit, 113, 313 UI Control Unit, 114, 314 Animation Generation Unit, 115, 315 Game Progression Unit, 116, 316 Virtual Space Control Unit, 117 Video Playback Unit, 120, 220, 320, 420 Memory Unit (First Memory Unit, Second Memory Unit), 131, 231, 331 Game Program (Program, First Program), 132, 232, 332 Game Information, 133, 233, 333 User Information, 151, 451 Input Unit, 152, 452 Display Unit (Display), 200 Server, 211 Communication Mediation Unit, 212 Log Generation Unit, 213 List Generation Unit, 234, 421 User List, 300 Game Play Terminal (External Device, Second External Device), 317 Reaction Processing Unit, 400 Distribution Terminal (External, First External Device, Computer, Second Information Processing Device), 411 Communication Control Unit, 414 Voice Reception Unit, 415 Motion Identification Unit, 416 Motion Instruction Data Generation Unit, 422 Motion List, 423 Distribution Program (Program, Second Program), 540, 1020, 1021 Controller, 500 HMD, 510 HMD Sensor, 520 Motion Sensor, 530 Display, 600A, 600B Virtual Space, 610 Avatar Object (Character), 620A, 620B Virtual Camera, 631, 632, 633, 634 Object, 640A, 640B Field of View Region, 650, 660 Field of View Image, 671 Enemy Object, 672, 673 Obstacle Object, 674 Effect Object, 691, 692 Utterance, 701, 702, 703A, 70B,704A, 704B, 705, 706, 711, 711A, 711B, 711C, 711D, 722, 723, 745, 745A, 745B, 745C, 752, 762, 763, 930, 2011, 2022, 2031, 2032, 2033, 2034, 2037, 2038, 2051, 2063, 2072, 2073, 2075 UI images (message UI, UI), 721 download screen, 731 user list screen (list), 732, 732A, 732B, 732C, 742, 742A, 742B, 742C record images, 733, 733A, 733B, 733C user names, 734, 734A, 734B, 734C tag information, 735, 735A, 735B, 735C icons, 741 motion list screen (options), 743, 743A, 743B, 743C motion names, 744, 744A, 744B, 744C, 753 motion images, 751 delivery screen, 761 delivery completion screen, 810A, 810B motion videos, 820A, 820B spoken voices, 910A, 910B videos, 920A, 920B voices, 1000 HMD set, 1010 object, 1030 storage medium, 2010 home screen, 2020 ranking screen, 2021 title image, 2026, 2026A, 2026B rankings, 2027, 2027A, 2027B charging amounts, 2028, 2028A, 2028B transmission counts, 2029 sent notification, 2030 previous transmission date, 2035 detailed display area, 2036 scroll bar, 2040 detailed screen, 2050, 2060 preparation screens, 2052 text, 2053, 2061 selection images, 2054A, 2054B, 2054C, 2062A, 2062B, 2062C, 2062D, 2062E, 2062F options, 2070 voice input screen, 2074 tag images,

Claims

1. A program executed in an information terminal device including a processor, a memory, an input unit, and a display unit, wherein the processor is caused to: receive display data for displaying a virtual space in which characters are arranged; specify any one of a plurality of types of viewing modes having different modes for moving a viewpoint within the virtual space; display, on the display unit, an image from a viewpoint that moves in a manner corresponding to the viewing mode specified in the specifying step, among the images in the virtual space based on the display data; the plurality of types of viewing modes include a first viewing mode that can move the viewpoint according to an input mode to the input unit from a user and can restrict the movement of the viewpoint within a predetermined range where the character can be arranged, and a second viewing mode that allows the movement of the viewpoint within a predetermined range where the character can be arranged but can make the character non-displayed when the viewpoint is moved within the predetermined range; the character is a character that operates in real time according to the movement of an actor; the predetermined range is a range predetermined as a range within which an actor can move from a predetermined position. A program.

2. The input unit includes a first input unit that specifies an input operation on the display unit and a second input unit that specifies the position of the information terminal device; the first viewing mode is a viewing mode in which the viewpoint can be moved according to an operation mode specified by the first input unit; The second viewing mode is a viewing mode in which the viewpoint can be moved according to the position specified by the second input unit. The program according to claim 1.

3. The information terminal device includes an imaging unit; the second viewing mode is a viewing mode in which an image from a viewpoint corresponding to the position specified by the second input unit is displayed, among the images in the augmented reality space in which the character is arranged based on the display data for the acquired image acquired by the imaging unit. The program according to claim 2.

4. The display data includes viewpoint information for specifying a predetermined viewpoint regardless of the operation mode of the user. The program according to any one of claims 1 to 3, wherein the plurality of types of viewing modes includes a viewing mode for displaying an image from a predetermined viewpoint specified based on the viewpoint information among the images in the virtual space based on the display data.

5. The program according to any one of claims 1 to 4, wherein the predetermined range includes a position where a specific part of the character can be displayed when the viewpoint moves.

6. The program according to any one of claims 1 to 5, wherein the predetermined range in the first viewing mode and the predetermined range in the second viewing mode are the same range.

7. The program according to any one of claims 1 to 5, wherein the predetermined range in the first viewing mode is defined as a wider range than the predetermined range in the second viewing mode.

8. The program according to claim 1, wherein both the first viewing mode and the second viewing mode are viewing modes for displaying an image from a viewpoint moved according to an input mode with respect to the input unit from the same user who views the image displayed on the display unit.

9. A specific object is associated with the character in advance, the display mode of the specific object is changed regardless of whether there is a request from the user who owns the information terminal device, in the case of the second viewing mode, the viewpoint can be moved to a position where the specific object associated with the character can be displayed, the program according to any one of claims 1 to 8, wherein in the case of the first viewing mode, the viewpoint cannot be moved to a position where the specific object associated with the character can be displayed.

10. A method executed in an information terminal device including a processor, a memory, an input unit, and a display unit, the method comprising: the information terminal device receiving display data for displaying a virtual space in which a character is arranged; specifying any one of a plurality of types of viewing modes having different modes for moving a viewpoint in the virtual space; displaying, on the display unit, an image from a viewpoint that moves in a manner corresponding to the viewing mode specified by the specifying step among the images in the virtual space based on the display data. The plurality of types of viewing modes include a first viewing mode that can restrict the movement of the viewpoint within a predetermined range where the character can be arranged, as a viewing mode in which the viewpoint can be moved according to the input mode for the input unit from the user, and a second viewing mode that allows the movement of the viewpoint within the predetermined range where the character can be arranged but can hide the character when the viewpoint is moved within the predetermined range. The character is a character that operates in real time according to the movement of the performer. The predetermined range is a range that is predetermined as a range within which the performer can move from a predetermined position.

11. An information terminal device including a processor, a memory, an input unit, and a display unit, comprising a storage unit that stores a program, and a control unit that controls the operation of the information terminal device by executing the program. The control unit receives display data for displaying a virtual space in which a character is arranged, identifies any one of a plurality of types of viewing modes with different modes for moving the viewpoint within the virtual space, and executes a step of displaying, on the display unit, an image from the viewpoint that moves in a manner corresponding to the viewing mode identified in the identifying step, among the images in the virtual space based on the display data. The plurality of types of viewing modes include a first viewing mode that can restrict the movement of the viewpoint within a predetermined range where the character can be arranged, as a viewing mode in which the viewpoint can be moved according to the input mode for the input unit from the user, and a second viewing mode that allows the movement of the viewpoint within the predetermined range where the character can be arranged but can hide the character when the viewpoint is moved within the predetermined range. The character is a character that operates in real time according to the movement of the performer. The predetermined range is a range that is predetermined as a range within which the performer can move from a predetermined position.

Images