program

A computer-executed program assists live performances by managing parts and providing interactive virtual spaces, addressing inefficiencies and enhancing performance progression and viewer engagement.

JP7591354B2Active Publication Date: 2024-11-28COLOPL
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2020072162
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-04-14
Publication Date
2024-11-28
Estimated Expiration
2040-04-14

AI Technical Summary

Technical Problem

Live performances often depend on the performer's ability and experience, leading to potential inefficiencies and disruptions, necessitating a program that assists in the progression and management of such events.

Method used

A program executed on a computer that allows a processor to select parts of a live performance and switch to the next part when conditions are met, displaying a virtual space with moving characters and relevant information to assist the performer, and broadcasting this to viewers.

Benefits of technology

The program effectively supports the smooth progression of live performances by providing real-time assistance and enhancing viewer engagement through interactive virtual environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007591354000001
    Figure 0007591354000001
  • Figure 0007591354000002
    Figure 0007591354000002
  • Figure 0007591354000003
    Figure 0007591354000003
Patent Text Reader

Abstract

To provide a program, computer, and method for assisting progress of a live.SOLUTION: It is switched to a previously determined next part among a plurality of types of parts constituting a live of a performer by establishing a progress condition. Display data is live distributed, the display data allowing a virtual space video containing a character moving at real time according to a motion of a performer proceeding a current part to be displayed by a terminal device possessed by an audience. A video containing a character among the virtual space is displayed in a display area which can be viewed by the performer, and information useful for the performer for proceeding the current part is displayed.SELECTED DRAWING: Figure 29
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention is To Regarding. [Background technology]

[0002] Conventionally, there have been known systems that distribute live video to users' communication terminals via a network (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-120098 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the progress of the live broadcast is left to the performers and depends on the performers' abilities, experience, etc. For this reason, there is a risk that problems may occur, such as the live broadcast not being able to proceed properly and efficiently, and there is room for improvement.

[0005] The present invention has been devised in view of the above circumstances, and its object is to provide a program that supports the progress of a live performance. M The aim is to provide. [Means for solving the problem]

[0006] According to one aspect of one embodiment of the present disclosure, ,Ko A program executed on a computer, the program causing a processor to select from among a plurality of parts constituting a live performance by a performer: Part switching clause When the condition is met, the system switches to the next part, which is determined in advance. Switching Method The viewer can watch the video of the virtual space, including the character that moves in real time according to the movement of the performer who is performing the current part. of Terminal equipment NiraEve broadcast Delivery Method and displaying an image including a character in a virtual space in a display area visible to the performer, and displaying information useful to the performer when progressing through the current part. Display means and , and function To make. Effect of the Invention

[0007] According to the present invention, the progress of a live performance can be supported. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of an overview of a system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a user terminal according to an embodiment. [Diagram 3] FIG. 2 is a block diagram showing an example of a hardware configuration of a server according to an embodiment. [Figure 4] 1 is a block diagram showing an example of a hardware configuration of a game playing terminal according to an embodiment. [Diagram 5] FIG. 2 is a block diagram showing an example of a hardware configuration of a distribution terminal according to an embodiment. [Figure 6] FIG. 2 is a block diagram showing an example of a functional configuration of a user terminal, a server, and an HMD set according to an embodiment. [Figure 7] FIG. 2 is a block diagram showing an example of a functional configuration of a distribution terminal according to an embodiment. [Figure 8] 11 is a flowchart illustrating a part of a process executed in a user terminal and a game playing terminal according to an embodiment. [Figure 9] 1A to 1C are diagrams illustrating an example of a virtual space provided to a player and a field of view image viewed by the player according to an embodiment. [Figure 10] A figure showing an example of a virtual space provided to a user of a user terminal and a field of view image viewed by the user according to one embodiment. [Figure 11]FIG. 11 is a diagram showing another example of a field of view image visually recognized by a user of a user terminal. [Figure 12] A figure showing yet another example of a field of view image visually recognized by a user of a user terminal. [Figure 13] 1 is a flowchart illustrating a part of a process executed in a game playing terminal according to an embodiment. [Figure 14] 10 is a flowchart illustrating a part of a process executed in a user terminal according to an embodiment. [Figure 15] 11 is a flowchart illustrating a part of a process executed in a server according to an embodiment. [Figure 16] FIG. 2 illustrates an example of a list of users who have joined a game, according to one embodiment. [Figure 17] 11 is a flowchart illustrating a part of a process executed in a distribution terminal according to an embodiment. [Figure 18] FIG. 13 shows a specific example of a screen displayed on a distribution terminal according to an embodiment. [Figure 19] FIG. 13 is a diagram showing another specific example of a screen displayed on a distribution terminal according to an embodiment. [Figure 20] FIG. 13 illustrates one example of voice input by a player according to one embodiment. [Figure 21] 13A to 13C are diagrams showing yet another specific example of a screen displayed on a distribution terminal according to an embodiment and an overview of distribution of operation instruction data. [Figure 22] FIG. 13 illustrates another example of voice input by a player according to one embodiment. [Diagram 23] 13A to 13C are diagrams showing yet another specific example of a screen displayed on a distribution terminal according to an embodiment and an overview of distribution of operation instruction data. [Figure 24] A diagram showing an overview of the transmission of game progress information from a game playing terminal to a user terminal according to an embodiment. [Diagram 25] 10 is a flowchart illustrating a part of a process executed in a user terminal according to an embodiment. [Figure 26] FIG. 13 is a diagram showing a specific example of video playback. [Figure 27] FIG. 11 is a diagram showing another specific example of video playback. [Figure 28] 1 shows an example of a display on the monitor of a game playing terminal during live streaming. [Figure 29] 1 shows an example of a display on the monitor of a game playing terminal during live streaming. [Diagram 30] 11A and 11B are diagrams illustrating an example of a display on a touch screen of a user terminal during live distribution. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The system according to the present disclosure is a system for providing a game to multiple users. The system will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is indicated by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included in the present invention. In the following description, the same elements in the description of the drawings are given the same reference numerals, and redundant description will not be repeated.

[0010] <System 1 Operation Overview> FIG. 1 is a diagram showing an overview of a system 1 according to this embodiment. The system 1 includes a plurality of user terminals 100 (computers), a server 200, a game play terminal 300 (external device, second external device), and a distribution terminal 400 (external, first external device). Note that in FIG. 1, user terminals 100A to 100C, in other words, three user terminals 100, are shown as an example of the plurality of user terminals 100, but the number of user terminals 100 is not limited to the example shown. In this embodiment, when there is no need to distinguish between the user terminals 100A to C, they are referred to as "user terminals 100". The user terminals 100, the game play terminals 300, and the distribution terminals 400 are connected to the server 200 via a network 2. The network 2 is composed of the Internet and various mobile communication systems constructed by wireless base stations not shown. Examples of such mobile communication systems include so-called 3G and 4G mobile communication systems, LTE (Long Term Evolution), and wireless networks (such as Wi-Fi (registered trademark)) that can be connected to the Internet via a predetermined access point.

[0011] (Game Overview) In this embodiment, a game that is mainly played by a user of a game play terminal 300 will be described as an example of a game provided by the system 1 (hereinafter, this game). Hereinafter, the user of the game play terminal 300 will be referred to as a "player". As an example, the player (actor) progresses through the game by operating a character that appears in the game. In addition, in this game, the user of the user terminal 100 plays a role of supporting the player's progress through the game. Details of this game will be described later. Note that the game provided by the system 1 may be a game in which multiple users participate, and is not limited to this example.

[0012] (Game Play Terminal 300) The game play terminal 300 progresses the game in response to input operations by the player. Furthermore, the game play terminal 300 sequentially distributes information generated by the player's game play (hereinafter, game progress information) to the server 200 in real time.

[0013] (Server 200) The server 200 transmits game progress information (second data) received in real time from the game play terminal 300 to the user terminal 100. The server 200 also mediates the transmission and reception of various types of information between the user terminal 100, the game play terminal 300, and the distribution terminal 400.

[0014] (Distribution terminal 400) The distribution terminal 400 generates action instruction data (first data) in response to an input operation by the user of the distribution terminal 400, and distributes the action instruction data to the user terminal 100 via the server 200. The action instruction data is data for playing a moving image on the user terminal 100, and specifically, is data for causing a character appearing in the moving image to perform a movement.

[0015] In this embodiment, as an example, the user of the distribution terminal 400 is a player of the game. Also, as an example, the video played on the user terminal 100 based on the action instruction data is a video of a character operated by the player in the game moving. The "movement" refers to moving at least a part of the character's body, and includes speaking. For this reason, the action instruction data according to this embodiment includes, for example, voice data for making the character speak, and motion data for moving the character's body.

[0016] As an example, the action instruction data is transmitted to the user terminal 100 after the game ends. Details of the action instruction data and the video played based on the action instruction data will be described later.

[0017] (User terminal 100) The user terminal 100 receives game progress information in real time, and generates and displays a game screen using the information. In other words, the user terminal 100 reproduces the game screen of the game being played by the player by real-time rendering. This allows the user of the user terminal 100 to view the same game screen as the player views while playing the game, at approximately the same timing as the player.

[0018] Furthermore, the user terminal 100 generates information for assisting the player in progressing through the game in response to input operations by the user, and transmits this information to the game playing terminal 300 via the server 200. The information will be described in detail later.

[0019] Furthermore, the user terminal 100 receives motion instruction data from the distribution terminal 400, and generates and plays back a moving image (video) using the motion instruction data. In other words, the user terminal 100 renders and plays back the motion instruction data.

[0020] <Hardware configuration of System 1> Fig. 2 is a diagram showing the hardware configuration of a user terminal 100. Fig. 3 is a diagram showing the hardware configuration of a server 200. Fig. 4 is a diagram showing the hardware configuration of a game playing terminal 300. Fig. 5 is a diagram showing the hardware configuration of a streaming terminal 400.

[0021] (User terminal 100) In this embodiment, as an example, the user terminal 100 is realized as a smartphone, but the user terminal 100 is not limited to a smartphone. For example, the user terminal 100 may be realized as a feature phone, a tablet computer, a laptop computer (a so-called notebook computer), or a desktop computer. The user terminal 100 may also be a game device suitable for playing games.

[0022] 2, the user terminal 100 includes a processor 10, a memory 11, a storage 12, a communication interface (IF) 13, an input / output IF 14, a touch screen 15 (display unit), a camera 17, and a distance measurement sensor 18. These components of the user terminal 100 are electrically connected to each other via a communication bus. Note that the user terminal 100 may include an input / output IF 14 to which a display (display unit) configured separately from the user terminal 100 main body can be connected instead of or in addition to the touch screen 15.

[0023] 2, the user terminal 100 may be configured to be able to communicate with one or more controllers 1020. The controller 1020 establishes communication with the user terminal 100 in accordance with a communication standard such as Bluetooth (registered trademark). The controller 1020 may have one or more buttons, etc., and transmits output values ​​based on user input operations on the buttons, etc., to the user terminal 100. The controller 1020 may also have various sensors, such as an acceleration sensor and an angular velocity sensor, and transmits output values ​​of the various sensors to the user terminal 100.

[0024] Instead of or in addition to the user terminal 100 being equipped with the camera 17 and the distance measurement sensor 18, the controller 1020 may have the camera 17 and the distance measurement sensor 18.

[0025] For example, at the start of a game, the user terminal 100 desirably causes the user using the controller 1020 to input user identification information, such as the user's name or login ID, via the controller 1020. This enables the user terminal 100 to link the controller 1020 with the user, and to identify which user the output value belongs to based on the source (controller 1020) of the received output value.

[0026] When the user terminal 100 communicates with multiple controllers 1020, each user holds a controller 1020, and the single user terminal 100 can realize a multiplay without communicating with other devices such as the server 200 via the network 2. Also, the user terminals 100 can realize a multiplay locally by connecting to each other via a wireless standard such as a wireless LAN (Local Area Network) standard (connecting to communicate without going through the server 200). When the above-mentioned multiplay is realized locally by a single user terminal 100, the user terminal 100 may further include at least a part of various functions of the server 200, which will be described later. Also, when the above-mentioned multiplay is realized locally by multiple user terminals 100, the multiple user terminals 100 may include various functions of the server 200, which will be described later, in a distributed manner.

[0027] Even when the above-mentioned multiplay is realized locally, the user terminal 100 may communicate with the server 200. For example, information indicating a play result such as the score or win / loss in a certain game may be associated with user identification information and transmitted to the server 200.

[0028] Furthermore, the controller 1020 may be configured to be detachable from the user terminal 100. In this case, a coupling unit for the controller 1020 may be provided on at least one surface of the housing of the user terminal 100. When the user terminal 100 and the controller 1020 are coupled to each other via the coupling unit by a wire, the user terminal 100 and the controller 1020 transmit and receive signals via the wire.

[0029] 2, the user terminal 100 may accept the attachment of a storage medium 1030 such as an external memory card via the input / output IF 14. This allows the user terminal 100 to read programs and data recorded in the storage medium 1030. The program recorded in the storage medium 1030 is, for example, a game program.

[0030] The user terminal 100 may store in the memory 11 of the user terminal 100 a game program acquired by communicating with an external device such as the server 200, or may store in the memory 11 a game program acquired by reading it from the storage medium 1030.

[0031] As described above, the user terminal 100 includes the communication IF 13, the input / output IF 14, the touch screen 15, the camera 17, and the distance measurement sensor 18 as examples of mechanisms for inputting information to the user terminal 100. Each of the above-mentioned units as input mechanisms can be considered as an operation unit configured to receive an input operation by the user.

[0032] For example, when the operation unit is composed of at least one of the camera 17 and the distance measuring sensor 18, the operation unit detects an object 1010 in the vicinity of the user terminal 100 and identifies an input operation from the detection result of the object. As an example, a user's hand, a marker of a predetermined shape, or the like is detected as the object 1010, and the input operation is identified based on the color, shape, movement, or type of the object 1010 obtained as the detection result. More specifically, when the user's hand is detected from an image captured by the camera 17, the user terminal 100 identifies a gesture (a series of movements of the user's hand) detected based on the captured image as the user's input operation and accepts it. Note that the captured image may be a still image or a video.

[0033] Alternatively, when the operation unit is configured with the touch screen 15, the user terminal 100 identifies and accepts a user operation performed on the input unit 151 of the touch screen 15 as the user's input operation. Alternatively, when the operation unit is configured with the communication IF 13, the user terminal 100 identifies and accepts a signal (e.g., an output value) transmitted from the controller 1020 as the user's input operation. Alternatively, when the operation unit is configured with the input / output IF 14, the user terminal 100 identifies and accepts a signal output from an input device (not shown) connected to the input / output IF 14 and different from the controller 1020 as the user's input operation.

[0034] (Server 200) The server 200 may be, for example, a general-purpose computer such as a workstation or a personal computer. The server 200 includes a processor 20, a memory 21, a storage 22, a communication IF 23, and an input / output IF 24. These components of the server 200 are electrically connected to each other by a communication bus.

[0035] (Game Play Terminal 300) The game play terminal 300 may be, for example, a general-purpose computer such as a personal computer. The game play terminal 300 comprises a processor 30, a memory 31, a storage 32, a communication IF 33, and an input / output IF 34. These components of the game play terminal 300 are electrically connected to each other via a communication bus.

[0036] As shown in FIG. 4, the game play terminal 300 according to the present embodiment is, for example, included in an HMD (Head Mounted Display) set 1000. In other words, it can be expressed that the HMD set 1000 is included in the system 1, and it can also be expressed that the player plays the game using the HMD set 1000. Note that the device for the player to play the game is not limited to the HMD set 1000. For example, the device may be any device that allows the player to virtually experience the game. The device may be realized as a smartphone, a feature phone, a tablet computer, a laptop computer (a so-called notebook computer), or a desktop computer. The device may also be a game device suitable for game play.

[0037] In addition to the game playing terminal 300, the HMD set 1000 includes an HMD 500, an HMD sensor 510, a motion sensor 520, a display 530, and a controller 540. The HMD 500 includes a monitor 51, a gaze sensor 52, a first camera 53, a second camera 54, a microphone 55, and a speaker 56. The controller 540 may include the motion sensor 520.

[0038] The HMD500 is worn on the head of a player and can provide the player with a virtual space during operation. More specifically, the HMD500 displays an image for the right eye and an image for the left eye on a monitor 51. When each eye of the player views the respective image, the player can recognize the image as a three-dimensional image based on the parallax between the two eyes. The HMD500 can include both a so-called head-mounted display equipped with a monitor and a head-mounted device to which a terminal having a monitor such as a smartphone can be attached.

[0039] The monitor 51 is realized, for example, as a non-transmissive display device. In one aspect, the monitor 51 is disposed on the main body of the HMD 500 so as to be located in front of both eyes of the player. Therefore, when the player visually recognizes the three-dimensional image displayed on the monitor 51, the player can be immersed in the virtual space. In one aspect, the virtual space includes, for example, images of a background, an object that the player can operate, and a menu that the player can select. In one aspect, the monitor 51 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor provided in a so-called smartphone or other information display terminal.

[0040] In another aspect, the monitor 51 may be realized as a transmissive display device. In this case, the HMD 500 may be an open type such as a glasses type, instead of a closed type that covers the player's eyes as shown in FIG. 1. The transmissive monitor 51 may be temporarily configurable as a non-transmissive display device by adjusting the transmittance. The monitor 51 may include a configuration that simultaneously displays a part of an image that constitutes a virtual space and the real space. For example, the monitor 51 may display an image of the real space captured by a camera mounted on the HMD 500, or may make the real space visible by setting the transmittance of a part of the monitor 51 high.

[0041] In one aspect, the monitor 51 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, the monitor 51 may be configured to integrally display an image for the right eye and an image for the left eye. In this case, the monitor 51 includes a high-speed shutter. The high-speed shutter operates to alternately display an image for the right eye and an image for the left eye so that the image is recognized by only one of the eyes.

[0042] In a certain aspect, the HMD 500 includes a plurality of light sources (not shown). Each light source is realized, for example, by an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 510 has a position tracking function for detecting the movement of the HMD 500. More specifically, the HMD sensor 510 reads a plurality of infrared rays emitted by the HMD 500 and detects the position and inclination of the HMD 500 in the real space.

[0043] In another aspect, the HMD sensor 510 may be realized by a camera. In this case, the HMD sensor 510 can detect the position and inclination of the HMD 500 by performing image analysis processing using image information of the HMD 500 output from the camera.

[0044] In another aspect, the HMD 500 may include a sensor (not shown) as a position detector instead of or in addition to the HMD sensor 510. The HMD 500 may detect the position and tilt of the HMD 500 itself using the sensor. For example, if the sensor is an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor, the HMD 500 may detect the position and tilt of the HMD 500 itself using any of these sensors instead of the HMD sensor 510. As an example, if the sensor included in the HMD 500 is an angular velocity sensor, the angular velocity sensor detects the angular velocity of the HMD 500 around three axes in real space over time. The HMD 500 calculates the change in angle of the HMD 500 around the three axes over time based on each angular velocity, and further calculates the tilt of the HMD 500 based on the change in angle over time.

[0045] The gaze sensor 52 detects the direction in which the gaze of the player's right eye and left eye are directed. That is, the gaze sensor 52 detects the gaze of the player. The detection of the gaze direction is realized, for example, by a known eye tracking function. The gaze sensor 52 is realized by a sensor having the eye tracking function. In a certain aspect, the gaze sensor 52 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 52 may be, for example, a sensor that irradiates the right eye and left eye of the player with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris in response to the irradiated light. The gaze sensor 52 can detect the gaze of the player based on each detected rotation angle.

[0046] The first camera 53 photographs the lower part of the player's face. More specifically, the first camera 53 photographs the player's nose, mouth, etc. The second camera 54 photographs the player's eyes, eyebrows, etc. The housing on the player's side of the HMD 500 is defined as the inside of the HMD 500, and the housing on the opposite side of the HMD 500 from the player is defined as the outside of the HMD 500. In one aspect, the first camera 53 may be disposed outside the HMD 500, and the second camera 54 may be disposed inside the HMD 500. Images generated by the first camera 53 and the second camera 54 are input to the game play terminal 300. In another aspect, the first camera 53 and the second camera 54 may be realized as a single camera, and the player's face may be photographed by this single camera.

[0047] The microphone 55 converts the player's speech into an audio signal (electrical signal) and outputs it to the game play terminal 300. The speaker 56 converts the audio signal into sound and outputs it to the player. In another aspect, the HMD 500 may include earphones instead of the speaker 56.

[0048] The controller 540 is connected to the game play terminal 300 by wire or wirelessly. The controller 540 accepts input of commands from the player to the game play terminal 300. In one aspect, the controller 540 is configured to be held by the player. In another aspect, the controller 540 is configured to be attached to a part of the player's body or clothing. In yet another aspect, the controller 540 may be configured to output at least one of vibration, sound, and light based on a signal transmitted from the game play terminal 300. In yet another aspect, the controller 540 accepts an operation from the player to control the position and movement of an object placed in a virtual space.

[0049] In one aspect, the controller 540 includes a plurality of light sources. Each light source is realized, for example, by an LED that emits infrared rays. The HMD sensor 510 has a position tracking function. In this case, the HMD sensor 510 reads a plurality of infrared rays emitted by the controller 540 and detects the position and tilt of the controller 540 in the real space. In another aspect, the HMD sensor 510 may be realized by a camera. In this case, the HMD sensor 510 can detect the position and tilt of the controller 540 by performing an image analysis process using image information of the controller 540 output from the camera.

[0050] In one aspect, the motion sensor 520 is attached to the player's hand and detects the movement of the player's hand. For example, the motion sensor 520 detects the rotation speed, number of rotations, etc. of the hand. The detected signal is sent to the game play terminal 300. The motion sensor 520 is provided, for example, in the controller 540. In one aspect, the motion sensor 520 is provided, for example, in the controller 540 configured to be held by the player. In another aspect, for safety in the real space, the controller 540 is attached to something that is not easily blown away by being worn on the player's hand, such as a glove type. In yet another aspect, a sensor that is not worn by the player may detect the movement of the player's hand. For example, a signal from a camera that captures the player may be input to the game play terminal 300 as a signal representing the player's action. The motion sensor 520 and the game play terminal 300 are connected to each other wirelessly, for example. In the case of wireless communication, the communication form is not particularly limited, and for example, Bluetooth or other known communication methods are used.

[0051] The display 530 displays an image similar to the image displayed on the monitor 51. This allows users other than the player wearing the HMD 500 to view the same image as the player. The image displayed on the display 530 does not need to be a three-dimensional image, and may be an image for the right eye or an image for the left eye. Examples of the display 530 include a liquid crystal display and an organic EL monitor.

[0052] The game play terminal 300 moves the character operated by the player based on various information acquired from each part of the HMD 500, the controller 540, and the motion sensor 520, and progresses the game. The "movement" here includes moving each part of the body, changing posture, changing facial expression, moving, speaking, touching or moving objects arranged in the virtual space, and using weapons, tools, etc. held by the character. That is, in this game, when the player moves each part of the body, the character also moves each part of the body in the same way as the player. Also, in this game, the character speaks the contents of what the player has spoken. In other words, in this game, the character is an avatar object that acts as an alter ego of the player. As an example, at least a part of the character's movement may be executed by input to the controller 540 by the player.

[0053] In this embodiment, the motion sensors 520 are attached to the player's hands, both feet, waist, and head, as an example. The motion sensors 520 attached to the player's hands may be provided in the controller 540, as described above. The motion sensor 520 attached to the player's head may be provided in the HMD 500. The motion sensors 520 may further be attached to the user's elbows and knees. By increasing the number of motion sensors 520 attached to the player, the player's movements can be more accurately reflected in the character. Also, instead of attaching the motion sensors 520 to various parts of the body, the player may wear a suit to which one or more motion sensors 520 are attached. In other words, the method of motion capture is not limited to the example using the motion sensor 520.

[0054] (Distribution terminal 400) The distribution terminal 400 may be a mobile terminal such as a smartphone, a PDA (Personal Digital Assistant), or a tablet computer, etc. The distribution terminal 400 may also be a so-called stationary terminal such as a desktop personal computer.

[0055] 5, the distribution terminal 400 includes a processor 40, a memory 41, a storage 42, a communication IF 43, an input / output IF 44, and a touch screen 45. Note that instead of or in addition to the touch screen 45, the distribution terminal 400 may include an input / output IF 44 to which a display (display unit) configured separately from the distribution terminal 400 main body can be connected.

[0056] The controller 1021 may have a physical input mechanism such as one or more buttons, levers, sticks, wheels, etc. The controller 1021 transmits to the streaming terminal 400 an output value based on an input operation input to the input mechanism by an operator of the streaming terminal 400 (a player in this embodiment). The controller 1021 may also have various sensors such as an acceleration sensor and an angular velocity sensor, and may transmit output values ​​of the various sensors to the streaming terminal 400. The above-mentioned output values ​​are received by the streaming terminal 400 via the communication IF 43.

[0057] The broadcasting terminal 400 may include a camera and a distance measuring sensor (both not shown). Instead of or in addition to the broadcasting terminal 400, the controller 1021 may include a camera and a distance measuring sensor.

[0058] As described above, the streaming terminal 400 includes the communication IF 43, the input / output IF 44, and the touch screen 45 as an example of a mechanism for inputting information to the streaming terminal 400. Each of the above-mentioned units as an input mechanism can be considered as an operation unit configured to receive an input operation by a user.

[0059] When the operation unit is configured with touch screen 45, distribution terminal 400 identifies and accepts a user operation performed on input unit 451 of touch screen 45 as a user input operation. Alternatively, when the operation unit is configured with communication IF 43, distribution terminal 400 identifies and accepts a signal (e.g., an output value) transmitted from controller 1021 as a user input operation. Alternatively, when the operation unit is configured with input / output IF 44, distribution terminal 400 identifies and accepts a signal output from an input device (not shown) connected to the input / output IF 44 as a user input operation.

[0060] <Hardware components of each device> Processors 10, 20, 30, and 40 control the overall operation of user terminal 100, server 200, game play terminal 300, and distribution terminal 400, respectively. Processors 10, 20, 30, and 40 include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). Processors 10, 20, 30, and 40 read programs from storages 12, 22, 32, and 42, respectively, which will be described later. Then, processors 10, 20, 30, and 40 load the read programs into memories 11, 21, 31, and 41, which will be described later, respectively. Processors 10, 20, and 30 execute the loaded programs.

[0061] The memories 11, 21, 31, and 41 are main storage devices. The memories 11, 21, 31, and 41 are composed of storage devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The memory 11 provides a working area for the processor 10 by temporarily storing a program and various data read by the processor 10 from a storage 12 described later. The memory 11 also temporarily stores various data generated while the processor 10 is operating according to the program. The memory 21 provides a working area for the processor 20 by temporarily storing various programs and data read by the processor 20 from a storage 22 described later. The memory 21 also temporarily stores various data generated while the processor 20 is operating according to the program. The memory 31 provides a working area for the processor 30 by temporarily storing various programs and data read by the processor 30 from a storage 32 described later. The memory 31 also temporarily stores various data generated while the processor 30 is operating according to the program. The memory 41 temporarily stores programs and various data that the processor 40 reads from the storage 42 (described later), thereby providing a working area for the processor 40. The memory 41 also temporarily stores various data that the processor 40 generates while it is operating according to the programs.

[0062] In this embodiment, the program executed by the processors 10 and 30 may be a game program of the game. In this embodiment, the program executed by the processor 40 may be a distribution program for realizing distribution of action instruction data. Furthermore, the processor 10 may further execute a viewing program for realizing playback of a video.

[0063] In this embodiment, the program executed by the processor 20 may be at least one of the above-mentioned game program, distribution program, and viewing program. The processor 20 executes at least one of the game program, distribution program, and viewing program in response to a request from at least one of the user terminal 100, the game play terminal 300, and the distribution terminal 400. The distribution program and the viewing program may be executed in parallel.

[0064] That is, the game program may be a program that realizes a game through cooperation between the user terminal 100, the server 200, and the game playing terminal 300. The distribution program may be a program that realizes distribution of action instruction data through cooperation between the server 200 and the distribution terminal 400. The viewing program may be a program that realizes playback of a video through cooperation between the user terminal 100 and the server 200.

[0065] The storages 12, 22, 32, and 42 are auxiliary storage devices. The storages 12, 22, 32, and 42 are configured with storage devices such as flash memory or HDDs (Hard Disk Drives). The storages 12 and 32 store, for example, various data related to games. The storage 42 stores various data related to the distribution of action instruction data. The storage 12 stores various data related to the playback of moving images. The storage 22 may store at least a portion of the various data related to games, the distribution of action instruction data, and the playback of moving images.

[0066] The communication IFs 13, 23, 33, and 43 respectively control the transmission and reception of various data in the user terminal 100, the server 200, the game playing terminal 300, and the distribution terminal 400. The communication IFs 13, 23, 33, and 43 control, for example, communication via a wireless LAN (Local Area Network), Internet communication via a wired LAN, a wireless LAN, or a mobile phone network, and communication using short-range wireless communication, etc.

[0067] The input / output IFs 14, 24, 34, and 44 are interfaces for the user terminal 100, the server 200, the game playing terminal 300, and the distribution terminal 400 to receive data input and output data, respectively. The input / output IFs 14, 24, 34, and 44 may input and output data via a Universal Serial Bus (USB) or the like. The input / output IFs 14, 24, 34, and 44 may include physical buttons, a camera, a microphone, a speaker, a mouse, a keyboard, a display, a stick, a lever, and the like. The input / output IFs 14, 24, 34, and 44 may also include a connection section for transmitting and receiving data to and from peripheral devices.

[0068] The touch screen 15 is an electronic component that combines an input unit 151 and a display unit 152 (display). The touch screen 45 is an electronic component that combines an input unit 451 and a display unit 452. The input units 151 and 451 are, for example, touch-sensitive devices, and are configured, for example, by a touchpad. The display units 152 and 452 are, for example, liquid crystal displays, organic EL (Electro-Luminescence) displays, or the like.

[0069] The input units 151 and 451 have a function of detecting a position where a user operation (mainly a physical contact operation such as a touch operation, a slide operation, a swipe operation, and a tap operation) is input on the input surface, and transmitting information indicating the position as an input signal. The input units 151 and 451 may be provided with a touch sensing unit (not shown). The touch sensing unit may adopt any type such as a capacitive type or a resistive film type.

[0070] Although not shown, the user terminal 100 and the broadcasting terminal 400 may each include one or more sensors for identifying the holding posture of the user terminal 100 and the broadcasting terminal 400. The sensors may be, for example, an acceleration sensor or an angular velocity sensor.

[0071] When the user terminal 100 and the distribution terminal 400 are equipped with sensors, the processors 10 and 40 can respectively identify the holding orientation of the user terminal 100 and the distribution terminal 400 from the output of the sensor and perform processing according to the holding orientation. For example, when the user terminal 100 and the distribution terminal 400 are held vertically, the processors 10 and 40 can respectively perform a portrait screen display in which a vertically elongated image is displayed on the display units 152 and 452. On the other hand, when the user terminal 100 and the distribution terminal 400 are held horizontally, the processors 10 and 40 can respectively perform a landscape screen display in which a horizontally elongated image is displayed on the display unit. In this way, the processors 10 and 40 can respectively switch between a portrait screen display and a landscape screen display depending on the holding orientation of the user terminal 100 and the distribution terminal 400.

[0072] <Functional configuration of System 1> 6 is a block diagram showing the functional configuration of the user terminal 100, the server 200, and the HMD set 1000 included in the system 1. FIG 7 is a block diagram showing the functional configuration of the distribution terminal 400 shown in FIG.

[0073] The user terminal 100 functions as an input device that accepts input operations by the user, and as an output device that outputs images and sounds of the game. The user terminal 100 functions as a control unit 110 and a storage unit 120 through cooperation of the processor 10, memory 11, storage 12, communication IF 13, input / output IF 14, touch screen 15, etc.

[0074] The server 200 has a function of mediating transmission and reception of various information among the user terminal 100, the HMD set 1000, and the distribution terminal 400. The server 200 functions as a control unit 210 and a storage unit 220 through cooperation of the processor 20, the memory 21, the storage 22, the communication IF 23, the input / output IF 24, and the like.

[0075] The HMD set 1000 (game playing terminal 300) has a function as an input device that accepts input operations by the player, a function as an output device that outputs game images and sounds, and a function of transmitting game progress information in real time to the user terminal 100 via the server 200. The HMD set 1000 functions as a control unit 310 and a storage unit 320 through cooperation of the processor 30, memory 31, storage 32, communication IF 33, input / output IF 34 of the game playing terminal 300, as well as the HMD 500, HMD sensor 510, motion sensor 520, controller 540, etc.

[0076] The distribution terminal 400 has a function of generating action instruction data and transmitting the action instruction data to the user terminal 100 via the server 200. The distribution terminal 400 functions as a control unit 410 and a storage unit 420 through cooperation of the processor 40, the memory 41, the storage 42, the communication IF 43, the input / output IF 44, the touch screen 45, and the like.

[0077] (Data stored in the memory of each device) The storage unit 120 stores a game program 131 (program), game information 132, and user information 133. The storage unit 220 stores a game program 231, game information 232, user information 233, and a user list 234. The storage unit 320 stores a game program 331, game information 332, and user information 333. The storage unit 420 stores a user list 421, a motion list 422, and a delivery program 423 (program, second program).

[0078] The game programs 131, 231, and 331 are game programs executed by the user terminal 100, the server 200, and the HMD set 1000, respectively. The game is realized by each device operating in cooperation based on the game programs 131, 231, and 331. The game programs 131 and 331 may be stored in the storage unit 220 and downloaded to the user terminal 100 and the HMD set 1000, respectively. In this embodiment, the user terminal 100 renders data received from the distribution terminal 400 based on the game program 131 and plays a video. In other words, the game program 131 is also a program for playing a video using video instruction data distributed from the distribution terminal 400. The program for playing the video may be different from the game program 131. In this case, the storage unit 120 stores a program for playing the video separately from the game program 131.

[0079] The game information 132, 232, and 332 are data that the user terminal 100, the server 200, and the HMD set 1000 refer to when executing a game program, respectively. The user information 133, 233, and 333 are data related to the account of the user of the user terminal 100. The game information 232 is the game information 132 of each user terminal 100 and the game information 332 of the HMD set 1000. The user information 233 is the user information 133 of each user terminal 100 and the user information of the player included in the user information 333. The user information 333 is the user information 133 of each user terminal 100 and the user information of the player.

[0080] The user list 234 and the user list 421 are lists of users who have participated in the game. The user list 234 and the user list 421 may include a list of users who have participated in the most recent game play by the player, as well as a list of users who have participated in each game play prior to the game play. The motion list 422 is a list of multiple motion data created in advance. The motion list 422 is, for example, a list in which information identifying each motion (for example, a motion name) is associated with each piece of motion data. The distribution program 423 is a program for realizing the distribution of action instruction data for playing a video on the user terminal 100 to the user terminal 100.

[0081] (Functional configuration of server 200) The control unit 210 executes a game program 231 stored in the storage unit 220 to comprehensively control the server 200. For example, the control unit 210 mediates the transmission and reception of various types of information among the user terminal 100, the HMD set 1000, and the distribution terminal 400.

[0082] The control unit 210 functions as a communication intermediation unit 211, a log generation unit 212, and a list generation unit 213 in accordance with the description of the game program 231. The control unit 210 can also function as other functional blocks (not shown) to intermediate the transmission and reception of various information related to game play and distribution of action instruction data, and to support the progress of the game.

[0083] The communication mediation unit 211 mediates the transmission and reception of various information between the user terminal 100, the HMD set 1000, and the distribution terminal 400. For example, the communication mediation unit 211 transmits game progress information received from the HMD set 1000 to the user terminal 100. The game progress information includes data indicating the movement of a character operated by a player, the parameters of the character, items and weapons held by the character, enemy characters, and other information. The server 200 transmits the game progress information to the user terminals 100 of all users participating in the game. In other words, the server 200 transmits common game progress information to the user terminals 100 of all users participating in the game. As a result, the game progresses in the user terminals 100 of all users participating in the game in the same manner as in the HMD set 1000.

[0084] Also, for example, the communication intermediation unit 211 transmits information for supporting the player's progress in the game, received from any one of the user terminals 100, to the other user terminals 100 and the HMD set 1000. As an example, the information may be item information indicating an item that allows the player to progress in the game advantageously and is provided to the player (character). The item information includes information indicating the user who provided the item (user name, user ID, etc.). The communication intermediation unit 211 may also intermediate the distribution of action instruction data from the distribution terminal 400 to the user terminal 100.

[0085] The log generating unit 212 generates a log of the game progress based on the game progress information received from the HMD set 1000. The list generating unit 213 generates a user list 234 after the game play ends. Although details will be described later, each user in the user list 234 is associated with a tag indicating the content of the support provided by that user to the player. The list generating unit 213 generates a tag based on the log of the game progress generated by the log generating unit 212 and associates it with the corresponding user. Note that the list generating unit 213 may associate the content of the support provided by each user to the player, which is input by the game operator or the like using a terminal device such as a personal computer, with the corresponding user as a tag. This makes the content of the support provided by each user more detailed. Note that when a user participates in a game, the user terminal 100 transmits information indicating the user to the server 200 based on the user's operation. For example, the user terminal 100 transmits the user ID input by the user to the server 200. That is, the server 200 holds information indicating each user for all users participating in the game. The list generation unit 213 may generate the user list 234 using this information.

[0086] (Functional configuration of HMD set 1000) The control unit 310 executes the game program 331 stored in the storage unit 320 to comprehensively control the HMD set 1000. For example, the control unit 310 progresses the game in accordance with the game program 331 and the player's operation. Furthermore, while the game is progressing, the control unit 310 communicates with the server 200 as necessary to transmit and receive information. The control unit 310 may transmit and receive information directly to and from the user terminal 100 without going through the server 200.

[0087] The control unit 310 functions as an operation reception unit 311, a display control unit 312, a UI control unit 313, an animation generation unit 314, a game progression unit 315, a virtual space control unit 316, and a reaction processing unit 317 in accordance with the description of the game program 331. The control unit 310 can also function as other functional blocks (not shown) for controlling characters appearing in the game in accordance with the nature of the game being executed.

[0088] The operation reception unit 311 detects and receives input operations by the player. The operation reception unit 311 receives signals input from the HMD 500, the motion sensor 520, the controller 540, etc., determines what kind of input operation has been performed, and outputs the result to each element of the control unit 310.

[0089] The UI control unit 313 controls user interface (hereinafter, UI) images to be displayed on the monitor 51, the display 530, etc. The UI images are tools that allow the player to input necessary for the progress of the game to the HMD set 1000, or tools that allow the player to obtain information output during the progress of the game from the HMD set 1000. The UI images are, but are not limited to, for example, icons, buttons, lists, menu screens, etc.

[0090] The animation generation unit 314 generates animations showing the motions of various objects based on the control modes of the various objects. For example, the animation generation unit 314 may generate animations that express the manner in which an object (e.g., an avatar object of a player) moves as if it were actually there, moves its mouth, changes its facial expression, etc.

[0091] The game progression unit 315 progresses the game based on the game program 331, an input operation by the player, and an action of the avatar object in response to the input operation. For example, the game progression unit 315 performs a predetermined game process when the avatar object performs a predetermined action. For example, the game progression unit 315 may receive information representing an operation of a user at the user terminal 100 and perform game processing based on the operation of the user. The game progression unit 315 generates game progression information according to the progress of the game and transmits it to the server 200. The game progress information is transmitted to the user terminal 100 via the server 200. As a result, the progress of the game at the HMD set 1000 is shared in the user terminal 100. In other words, the progress of the game at the HMD set 1000 and the progress of the game at the user terminal 100 are synchronized.

[0092] The virtual space control unit 316 performs various controls related to the virtual space provided to the player according to the progress of the game. As an example, the virtual space control unit 316 generates various objects and places them in the virtual space. The virtual space control unit 316 also places a virtual camera in the virtual space. The virtual space control unit 316 also operates the various objects placed in the virtual space according to the progress of the game. The virtual space control unit 316 also controls the position and inclination of the virtual camera placed in the virtual space according to the progress of the game.

[0093] The display control unit 312 outputs a game screen reflecting the results of the processing executed by each of the above-mentioned elements to the monitor 51 and the display 530. The display control unit 312 may display an image based on a field of view from a virtual camera arranged in a virtual space as a game screen on the monitor 51 and the display 530. The display control unit 312 may also include an animation generated by the animation generation unit 314 in the game screen. The display control unit 312 may also draw the above-mentioned UI image controlled by the UI control unit 313 by superimposing it on the game screen.

[0094] The reaction processor 317 receives feedback on the reaction of the user of the user terminal 100 to the gameplay of the player, and outputs this to the player. In this embodiment, for example, the user terminal 100 can create a comment (message) addressed to an avatar object based on an input operation of the user. The reaction processor 317 receives comment data of the comment, and outputs it. The reaction processor 317 may display text data corresponding to the user's comment on the monitor 51 or the display 530, or may output audio data corresponding to the user's comment from a speaker (not shown). In the former case, the reaction processor 317 may draw an image corresponding to the text data (i.e., an image including the content of the comment) by superimposing it on the game screen.

[0095] (Functional configuration of the user terminal 100) The control unit 110 executes the game program 131 stored in the storage unit 120 to comprehensively control the user terminal 100. For example, the control unit 110 progresses the game in accordance with the game program 131 and user operations. Furthermore, while the game is progressing, the control unit 110 communicates with the server 200 as necessary to transmit and receive information. The control unit 110 may transmit and receive information directly to the HMD set 1000 without going through the server 200.

[0096] The control unit 110 functions as an operation reception unit 111, a display control unit 112, a UI control unit 113, an animation generation unit 114, a game progression unit 115, a virtual space control unit 116, and a video playback unit 117 in accordance with the description of the game program 131. The control unit 110 can also function as other functional blocks (not shown) for the progression of the game in accordance with the nature of the game being executed.

[0097] The operation reception unit 111 detects and receives an input operation by the user to the input unit 151. The operation reception unit 111 determines what kind of input operation has been performed from the action performed by the user on the touch screen 15 and the console via the other input / output IF 14, and outputs the result to each element of the control unit 110.

[0098] For example, the operation reception unit 111 receives an input operation on the input unit 151, detects the coordinates of the input position of the input operation, and identifies the type of the input operation. The operation reception unit 111 identifies, for example, a touch operation, a slide operation, a swipe operation, a tap operation, etc., as the type of the input operation. Furthermore, when a continuously detected input is interrupted, the operation reception unit 111 detects that the touch input from the touch screen 15 has been released.

[0099] The UI control unit 113 controls UI images to be displayed on the display unit 152 to construct a UI in response to at least one of a user's input operation and received game progress information. The UI images are tools that allow the user to input necessary for the progress of the game to the user terminal 100, or tools that allow the user to obtain information output during the progress of the game from the user terminal 100. The UI images are, but are not limited to, for example, icons, buttons, lists, menu screens, etc.

[0100] The animation generating unit 114 generates animations showing the motions of various objects based on the control modes of the various objects.

[0101] The game progression unit 115 progresses the game based on the game program 131, the received game progress information, and an input operation by the user. When a predetermined game process is performed by an input operation by the user, the game progression unit 115 transmits information related to the game process to the HMD set 1000 via the server 200. This causes the predetermined game process to be shared in the HMD set 1000. In other words, the progress of the game in the HMD set 1000 and the progress of the game in the user terminal 100 are synchronized. The predetermined game process is, for example, a process of providing an item to an avatar object, and in this example, the information related to the game process is the item information described above.

[0102] The virtual space control unit 116 performs various controls on the virtual space provided to the user according to the progress of the game. As an example, the virtual space control unit 116 generates various objects and places them in the virtual space. The virtual space control unit 116 also places a virtual camera in the virtual space. The virtual space control unit 116 also operates various objects placed in the virtual space according to the progress of the game, specifically, according to the received game progress information. The virtual space control unit 316 also controls the position and inclination of the virtual camera placed in the virtual space according to the progress of the game, specifically, according to the received game progress information.

[0103] The display control unit 112 outputs to the display unit 152 a game screen reflecting the results of the processing executed by each of the above-mentioned elements. The display control unit 112 may display, as a game screen, an image based on a field of view from a virtual camera arranged in a virtual space provided to the user on the display unit 152. The display control unit 112 may also include an animation generated by the animation generation unit 114 in the game screen. The display control unit 112 may also draw the above-mentioned UI image controlled by the UI control unit 113 by superimposing it on the game screen. In any case, the game screen displayed on the display unit 152 is the same game screen as the game screen displayed on the other user terminals 100 and the HMD set 1000.

[0104] The video playback unit 117 analyzes (renders) the action instruction data received from the distribution terminal 400, and plays back the video.

[0105] (Functional configuration of distribution terminal 400) The control unit 410 performs overall control of the distribution terminal 400 by executing a program (not shown) stored in the storage unit 420. For example, the control unit 410 generates action instruction data according to the program and an operation by a user (a player in this embodiment) of the distribution terminal 400, and distributes the action instruction data to the user terminal 100. Furthermore, the control unit 410 communicates with the server 200 as necessary to transmit and receive information. The control unit 410 may transmit and receive information directly to and from the user terminal 100 without going through the server 200.

[0106] The control unit 410 functions as a communication control unit 411, a display control unit 412, an operation reception unit 413, a voice reception unit 414, a motion identification unit 415, and an action instruction data generation unit 416 according to the description of the program. The control unit 410 can also function as other functional blocks (not shown) for generating and distributing action instruction data.

[0107] The communication control unit 411 controls transmission and reception of information to and from the server 200 or the user terminal 100 via the server 200. As an example, the communication control unit 411 receives a user list 421 from the server 200. In addition, as an example, the communication control unit 411 transmits operation instruction data to the user terminal 100.

[0108] The display control unit 412 outputs various screens reflecting the results of the processing executed by each element to the display unit 452. As an example, the display control unit 412 displays a screen including the received user list 234. Also, as an example, the display control unit 412 displays a screen including a motion list 422 for allowing the player to select motion data for moving an avatar object, which is included in the action instruction data to be distributed.

[0109] The operation reception unit 413 detects and receives the input operation of the player to the input unit 151. The operation reception unit 111 determines what kind of input operation has been performed from the action of the player on the touch screen 45 and the console via the other input / output IF 44, and outputs the result to each element of the control unit 410.

[0110] For example, the operation reception unit 413 receives an input operation on the input unit 451, detects the coordinates of the input position of the input operation, and identifies the type of the input operation. The operation reception unit 413 identifies, for example, a touch operation, a slide operation, a swipe operation, a tap operation, etc. as the type of the input operation. Furthermore, when a continuously detected input is interrupted, the operation reception unit 413 detects that the touch input from the touch screen 45 has been released.

[0111] The voice receiving unit 414 receives voice generated around the distribution terminal 400 and generates voice data of the voice. As an example, the voice receiving unit 414 receives voice uttered by a player and generates voice data of the voice.

[0112] The motion identification unit 415 identifies the motion data selected by the player from the motion list 422 in response to the input operation of the player.

[0113] The action instruction data generating unit 416 generates action instruction data. As an example, the action instruction data generating unit 416 generates action instruction data including the generated voice data and the identified motion data.

[0114] The functions of the HMD set 1000, the server 200, and the user terminal 100 shown in FIG. 6 and the functions of the distribution terminal 400 shown in FIG. 7 are merely examples. Each of the HMD set 1000, the server 200, the user terminal 100, and the distribution terminal 400 may have at least a part of the functions of the other devices. Furthermore, a device other than the HMD set 1000, the server 200, the user terminal 100, and the distribution terminal 400 may be a component of the system 1, and the other device may execute a part of the processing in the system 1. That is, the computer that executes the game program in this embodiment may be any of the HMD set 1000, the server 200, the user terminal 100, and the distribution terminal 400, and other devices, or may be realized by a combination of a plurality of these devices.

[0115] <Virtual space control processing> FIG. 8 is a flowchart showing an example of a flow of a control process of a virtual space provided to a player and a virtual space provided to a user of a user terminal 100. FIG. 9 is a diagram showing a virtual space 600A provided to a player and a field of view image visually recognized by the player according to an embodiment. FIG. 10 is a diagram showing a virtual space 600B provided to a user of a user terminal 100 and a field of view image visually recognized by the user according to an embodiment. Hereinafter, when there is no need to distinguish between the virtual spaces 600A and 600B, they will be referred to as "virtual space 600".

[0116] In step S1, the processor 30, functioning as the virtual space control unit 316, defines the virtual space 600A shown in Fig. 9. The processor 30 defines the virtual space 600A using virtual space data (not shown). The virtual space data may be stored in the game play terminal 300, may be generated by the processor 30 based on the game program 331, or may be acquired by the processor 30 from an external device such as the server 200.

[0117] As an example, the virtual space 600 has a spherical structure that covers the entire 360-degree direction from a point defined as the center. In order to avoid complicating the explanation, the upper half of the celestial sphere in the virtual space 600 is illustrated in Fig. 9 and Fig. 10.

[0118] In step S2, processor 30, functioning as virtual space control unit 316, places avatar object 610 (character) in virtual space 600A. Avatar object 610 is an avatar object associated with a player, and moves according to an input operation by the player.

[0119] In step S3, processor 30, functioning as virtual space control unit 316, places other objects in virtual space 600A. In the example of Fig. 9, processor 30 places objects 631 to 634. The other objects may include, for example, a character object (a so-called non-player character, NPC) that operates according to game program 331, an operational object such as a virtual hand, and objects that imitate animals, plants, artificial objects, natural objects, and the like that are placed according to the progress of the game.

[0120] In step S4, the processor 30, functioning as the virtual space control unit 316, places a virtual camera 620A in the virtual space 600A. The processor 30 places the virtual camera 620A at the position of the head of the avatar object 610, for example.

[0121] In step S5, the processor 30 displays the field of view image 650 on the monitor 51 and the display 530. The processor 30 defines a field of view area 640A, which is a field of view from the virtual camera 620A in the virtual space 600A, according to the initial position and inclination of the virtual camera 620A. The processor 30 then defines a field of view image 650 corresponding to the field of view area 640A. The processor 30 outputs the field of view image 650 to the monitor 51 and the display 530, thereby causing the HMD 500 and the display 530 to display the field of view image 650.

[0122] In the example of FIG. 9, as shown in FIG. 9(A), a portion of object 634 is included in field of view area 640A, so field of view image 650 includes a portion of object 634 as shown in FIG. 9(B).

[0123] In step S6, the processor 30 transmits the initial position information to the user terminal 100 via the server 200. The initial position information is information indicating the initial positions of various objects in the virtual space 600A. In the example of FIG. 9, the initial position information includes information on the initial positions of the avatar object 610 and the objects 631 to 634. The initial position information can also be expressed as one piece of game progress information.

[0124] In step S7, the processor 30, as the virtual space control unit 316, controls the virtual camera 620A according to the movement of the HMD 500. Specifically, the processor 30 controls the direction and inclination of the virtual camera 620A according to the movement of the HMD 500, that is, the attitude of the head of the player. As described later, when the player moves his / her head (changes the attitude of the head), the processor 30 moves the head of the avatar object 610 in accordance with this movement. For example, the processor 30 controls the direction and inclination of the virtual camera 620A so that the direction of the line of sight of the avatar object 610 and the direction of the line of sight of the virtual camera 620A coincide with each other. In step S8, the processor 30 updates the field of view image 650 in response to the change in the direction and inclination of the virtual camera 620A.

[0125] In step S9, processor 30, functioning as virtual space control unit 316, moves avatar object 610 in response to the movement of the player. As an example, processor 30 moves avatar object 610 in virtual space 600A in response to the movement of the player in real space. Also, processor 30 moves the head of avatar object 610 in virtual space 600A in response to the movement of the head of the player in real space.

[0126] In step S10, processor 30, functioning as virtual space control unit 316, moves virtual camera 620A to follow avatar object 610. That is, virtual camera 620A is always at the position of the head of avatar object 610 even if avatar object 610 moves.

[0127] Processor 30 updates field of view image 650 in response to the movement of virtual camera 620A. That is, processor 30 updates field of view area 640A in response to the pose of the player's head and the position of virtual camera 620A in virtual space 600A. As a result, field of view image 650 is updated.

[0128] In step S11, processor 30 transmits action instruction data of avatar object 610 to user terminal 100 via server 200. The action instruction data here includes at least any of motion data capturing the player's actions during the virtual experience (e.g., during game play), audio data of the player's voice, and operation data indicating the contents of an input operation on controller 540. When the player is playing a game, the action instruction data is transmitted to user terminal 100 as, for example, game progress information.

[0129] The processing of steps S7 to S11 is continuously and repeatedly executed while the player is playing the game.

[0130] In step S21, the processor 10 of the user terminal 100 of the user 3, as the virtual space control unit 116, defines the virtual space 600B shown in FIG. 10. The processor 10 defines the virtual space 600B by using virtual space data (not shown). The virtual space data may be stored in the user terminal 100, may be generated by the processor 10 based on the game program 131, or may be acquired by the processor 10 from an external device such as the server 200.

[0131] In step S22, the processor 10 receives the initial placement information. In step S23, the processor 10, functioning as the virtual space control unit 116, places various objects in the virtual space 600B in accordance with the initial placement information. In the example of FIG. 10, the various objects are the avatar object 610 and objects 631 to 634.

[0132] In step S24, the processor 10, functioning as the virtual space control unit 116, places a virtual camera 620B in the virtual space 600B. As an example, the processor 10 places the virtual camera 620B at the position shown in FIG.

[0133] In step S25, the processor 10 displays the field of view image 660 on the display unit 152. The processor 10 defines a field of view area 640B, which is a field of view from the virtual camera 620B in the virtual space 600B, according to the initial position and inclination of the virtual camera 620B. Then, the processor 10 defines a field of view image 660 corresponding to the field of view area 640B. The processor 10 outputs the field of view image 660 to the display unit 152, thereby causing the display unit 152 to display the field of view image 660.

[0134] In the example of FIG. 10, avatar object 610 and object 631 are included in field of view area 640B as shown in FIG. 10(A), so field of view image 660 includes avatar object 610 and object 631 as shown in FIG. 10(B).

[0135] In step S26, the processor 10 receives the action instruction data. In step S27, the processor 10, as the virtual space control unit 116, moves the avatar object 610 in the virtual space 600B in accordance with the action instruction data. In other words, the processor 10 plays back a video of the avatar object 610 moving by real-time rendering.

[0136] In step S28, the processor 10, as the virtual space control unit 116, controls the virtual camera 620B in response to the user's operation received as the operation receiving unit 111. In step S29, the processor 10 updates the field of view image 660 in response to changes in the position of the virtual camera 620B in the virtual space 600B and the direction and inclination of the virtual camera 620B. In addition, in step S28, the processor 10 may automatically control the virtual camera 620B in response to the movement of the avatar object 610, for example, the movement of the avatar object 610 and the change in the direction. For example, the processor 10 may automatically move the virtual camera 620B or change the direction and inclination so that the avatar object 610 is always photographed from the front. In addition, as an example, the processor 10 may automatically move the virtual camera 620B or change the direction and inclination so that the avatar object 610 is always photographed from the rear in response to the movement of the avatar object 610.

[0137] In this way, in the virtual space 600A, the avatar object 610 moves in accordance with the movement of the player. Action instruction data indicating this action is transmitted to the user terminal 100. In the virtual space 600B, the avatar object 610 moves in accordance with the received action instruction data. As a result, the avatar object 610 performs the same action in the virtual space 600A and the virtual space 600B. In other words, the user 3 can visually recognize the action of the avatar object 610 in accordance with the action of the player using the user terminal 100.

[0138] <Game Overview> 11 is a diagram showing another example of a field of view image displayed on the user terminal 100. Specifically, it is a diagram showing an example of a game screen of a game (this game) executed by the system 1 and being played by a player.

[0139] As an example, this game is a game in which an avatar object 610 that controls a weapon such as a gun or a knife and a plurality of enemy objects 671 that are NPCs appear in a virtual space 600, and the avatar object 610 fights against the enemy objects 671. Various game parameters such as the vitality of the avatar object 610, the number of usable magazines, the number of bullets remaining in the gun, and the number of remaining enemy objects 671 are updated as the game progresses.

[0140] In this game, a plurality of stages are prepared, and the player can clear each stage by satisfying a predetermined achievement condition associated with each stage. The predetermined achievement condition may include, for example, a condition that is satisfied by defeating all enemy objects 671 that appear, defeating a boss object among the enemy objects 671 that appear, acquiring a predetermined item, reaching a predetermined position, and the like. The achievement condition is defined in the game program 131. In this game, the player clears a stage when an achievement condition is satisfied in accordance with the content of the game, in other words, the victory of the avatar object 610 over the enemy object 671 (the victory or defeat between the avatar object 610 and the enemy object 671) is determined. In contrast, for example, when the game executed by the system 1 is a racing game or the like, the ranking of the avatar object 610 is determined when the condition of reaching the goal is satisfied.

[0141] In this game, in order to share a virtual space between the HMD set 1000 and the multiple user terminals 100, game progress information is live-streamed to the multiple user terminals 100 at predetermined time intervals. As a result, a field of view image of a field of view area defined by the virtual camera 620B corresponding to the user terminal 100 is displayed on the touch screen 15 of the user terminal 100 while watching the game. In addition, parameter images representing the vitality of the avatar object 610, the number of available magazines, the number of remaining bullets in the gun, the number of remaining enemy objects 671, etc. are superimposed and displayed on the upper right and upper left sections of the field of view image. This field of view image can also be expressed as a game screen.

[0142] As described above, the game progress information includes motion data capturing the actions of the player, audio data of the voice uttered by the player, and operation data indicating the contents of input operations to the controller 540. These data are, in other words, information for specifying the position, posture, orientation, etc. of the avatar object 610, information for specifying the position, posture, orientation, etc. of the enemy object 671, and information for specifying the positions, etc. of other objects (e.g., obstacle objects 672, 673). The processor 10 analyzes (renders) the game progress information to specify the position, posture, orientation, etc. of each object.

[0143] The game information 132 includes data on various objects such as an avatar object 610, an enemy object 671, and obstacle objects 672 and 673. The processor 10 updates the position, posture, orientation, and the like of each object using the data and the analysis result of the game progress information. As a result, the game progresses, and each object in the virtual space 600B moves in the same manner as each object in the virtual space 600A. Specifically, in the virtual space 600B, each object including the avatar object 610 moves based on the game progress information, regardless of the presence or absence of an operation from the user on the user terminal 100.

[0144] On the touch screen 15 of the user terminal 100, UI images 701 and 702 are displayed, as an example, superimposed on the field of view image. The UI image 701 is a UI image that accepts an operation for displaying, on the touch screen 15, a UI image 711 that accepts, from the user 3, an item insertion operation for supporting the avatar object 610. The UI image 702 is a UI image that accepts an operation for displaying, on the touch screen 15, a UI image (described later) that accepts, from the user 3, an operation for inputting and transmitting a comment for the avatar object 610 (in other words, the player 4). The operation accepted by the UI images 701 and 702 may be, for example, an operation of tapping the UI images 701 and 702.

[0145] When the UI image 701 is tapped, a UI image 711 is displayed superimposed on the field of view image. The UI images 711 include, for example, a UI image 711A depicting a magazine icon, a UI image 711B depicting a first aid box icon, a UI image 711C depicting a traffic cone icon, and a UI image 711D depicting a barricade icon. The item insertion operation corresponds to, for example, the operation of tapping any of the UI images.

[0146] As an example, when UI image 711A is tapped, the number of remaining bullets in the gun used by avatar object 610 increases. When UI image 711B is tapped, the health of avatar object 610 is restored. When UI images 711C and 711D are tapped, obstacle objects 672 and 673 that impede the movement of enemy object 671 are placed in the virtual space. One of obstacle objects 672 and 673 may be more likely to impede the movement of enemy object 671 than the other.

[0147] The processor 10 transmits item input information indicating that an item input operation has been performed to the server 200. The item input information includes at least information for identifying the type of item specified by the item input operation. The item input information may include other information related to the item, such as information indicating the position where the item is placed. The item input information is transmitted to the other user terminals 100 and the HMD set 1000 via the server 200.

[0148] 12 is a diagram showing another example of the field of view image displayed on the user terminal 100. Specifically, it is a diagram showing an example of the game screen of the present game, and is a diagram for explaining communication between a player and the user terminal 100 during game play.

[0149] In the example of Fig. 12(A), the user terminal 100 causes the avatar object 610 to make an utterance 691. Specifically, the user terminal 100 causes the avatar object 610 to make an utterance 691 in accordance with the voice data included in the game progress information. The content of the utterance 691 is "I'm out of bullets!" uttered by the player 4. That is, the content of the utterance 691 informs each user that the magazine is empty and the number of bullets loaded in the gun is reduced to 1, and therefore, the user is likely to lose the means to attack the enemy object 671.

[0150] In Fig. 12(A), speech bubbles are used to visually indicate the speech of avatar object 610, but in reality, sound is output from the speaker of user terminal 100. In addition to the sound output, the speech bubbles shown in Fig. 12(A) (i.e., speech bubbles containing the text of the sound content) may be displayed in the field of view image. This also applies to utterance 692, which will be described later.

[0151] When a tap operation on the UI image 702 is received, the user terminal 100 displays UI images 705 and 706 (message UIs) superimposed on the field of view image, as shown in Fig. 12(B). The UI image 705 is a UI image that displays a comment on the avatar object 610 (in other words, the player). The UI image 706 is a UI image that receives a comment sending operation from the user 3 to send the input comment.

[0152] For example, when the user terminal 100 receives a tap operation on the UI image 705, the user terminal 100 displays a UI image imitating a keyboard (not shown, hereinafter simply referred to as "keyboard") on the touch screen 15. The user terminal 100 displays text corresponding to the user's input operation on the keyboard on the UI image 705. In the example of FIG. 12(B), the text "I'll send you a magazine" is displayed on the UI image 705.

[0153] After inputting text, for example, when the user terminal 100 receives a tap operation on the UI image 706, the user terminal 100 transmits comment information including information indicating the input content (content of the text) and information indicating the user to the server 200. The comment information is transmitted to other user terminals 100 and the HMD set 1000 via the server 200.

[0154] The UI image 703A is a UI image showing the user name of the user who sent the comment, and the UI image 704A is a UI image showing the content of the comment sent by the user. In the example of FIG. 12(B), a user with the user name "BBBBB" uses his / her user terminal 100 to send comment information with the content "Danger!", and thus the UI image 703A and the UI image 704A are displayed. The UI image 703A and the UI image 704A are displayed on the touch screen 15 of all the user terminals 100 participating in the game and on the monitor 51 of the HMD 500. The UI images 703A and 704A may be one UI image. That is, the user name and the content of the comment may be included in one UI image.

[0155] In the example of Fig. 12(C), a user with the user name "AAAAA", who is a user of the user terminal 100 shown in Fig. 12, has input and sent a comment as described above, and as a result, UI images 703B and 704B are displayed on the touch screen 15. UI image 703B includes the user name "AAAAA", and UI image 704B includes the comment "I'll send you the magazine!", which was input in the example of Fig. 12(B).

[0156] 12C is a view image 611 after the user "AAAAA" further inputs a tap operation on the UI image 701, causes the UI image 711 to be displayed on the touch screen 15, and inputs a tap operation on the UI image 711A. That is, as a result of the user terminal 100 of the user "AAAAAA" transmitting item insertion information indicating a magazine to the other user terminals 100 and the HMD set 1000, the user terminal 100 and the HMD set 1000 place a performance object 674 (described later) in the virtual space 600. As an example, the user terminal 100 and the HMD set 1000 execute a performance related to the performance object 674 after the elapsed time indicated in the item insertion information has elapsed, and execute a process of activating the effect of the item object.

[0157] In the example of FIG. 12(D), the number of magazines is increased from 0 to 1 by executing a process to activate the effect of the item object. As a result, the player utters "Thank you!" to the user "AAAAA", and the voice data of this utterance is transmitted to each user terminal 100. As a result, each user terminal 100 outputs the voice "Thank you!" as the utterance 692 of the avatar object 610.

[0158] As described above, in this game, communication between the users and the avatar object 610 is realized by outputting the speech of the avatar object 610 based on the speech of the player and inputting comments by each user.

[0159] (Game Progression Processing in the Game Playing Terminal 300) FIG. 13 is a flow chart showing an example of the flow of a game progression process executed on the game playing terminal 300.

[0160] In step S31, the processor 30, as the game progression unit 315, progresses the game based on the game program 331 and the movement of the player. In step S32, the processor 30 generates game progress information and distributes it to the user terminals 100. Specifically, the processor 30 transmits the generated game progress information to each user terminal 100 via the server 200.

[0161] In step S33, when processor 30 receives the item insertion information (YES in S33), in step S34, processor 30 places an item object in virtual space 600A based on the item insertion information. As an example, processor 30 places effect object 674 in virtual space 600A before placing the item object (see FIG. 11(C)). The effect object 674 may be, for example, an object resembling a gift box. As an example, processor 30 may execute an effect related to effect object 674 after the elapse of the time indicated in the item insertion information. The effect may be, for example, an animation in which the lid of the gift box opens. After the execution of the animation, processor 30 executes a process of activating the effect of the item object. For example, in the example of FIG. 11(D), an obstacle object 673 is placed.

[0162] After the animation is executed, the processor 30 may place an item object corresponding to the tapped UI image in the virtual space 600A. For example, when a tap operation is performed on the UI image 711A, the processor 30 places a magazine object indicating a magazine in the virtual space 600A after the animation is executed. Also, when a tap operation is performed on the UI image 711B, the processor 30 places a first aid box object indicating a first aid box in the virtual space 600A after the animation is executed. For example, when the avatar object 610 moves to the position of the magazine object or the first aid box object, the processor 30 may execute a process of activating an effect of the magazine object or the first aid box object.

[0163] Processor 30 continues and repeats the processes of steps S31 to S34 until the game ends. When the game ends, for example, when the player inputs a predetermined input operation for ending the game (YES in step S35), the process shown in FIG. 13 ends.

[0164] (Game Progression Processing in User Terminal 100) FIG. 14 is a flowchart showing an example of the flow of a game progression process executed on the user terminal 100.

[0165] In step S41, the processor 10 receives game progress information. In step S42, the processor 10, functioning as the game progress unit 115, progresses the game based on the game progress information.

[0166] In step S43, when the processor 10 accepts an item insertion operation by the user 3 (YES in step S43), in step S44, the processor 10 consumes the virtual currency and places the effect object 674 in the virtual space 600B. Here, the virtual currency may be purchased (charged for the game) by the user 3 performing a predetermined operation on the processor 10 before or during participation in the game, or may be granted to the user 3 when a predetermined condition is satisfied. The predetermined condition may be a condition that requires participation in the game, such as clearing a quest in the game, or a condition that does not require participation in the game, such as answering a questionnaire. The amount of virtual currency (amount of virtual currency owned) is stored in the user terminal 100 as game information 132, for example.

[0167] In step S45, the processor 10 transmits item input information to the server 200. The item input information is transmitted to the game playing terminal 300 via the server 200.

[0168] When a predetermined time has elapsed after placing the effect object 674, the processor 10 places an item object in the virtual space 600A. In the example of Fig. 11, an obstacle object 673 is placed. That is, when the user 3 inputs a tap operation on the UI image 711C, a predetermined amount of virtual currency is consumed and the obstacle object 673 is placed.

[0169] Processor 10 continues and repeats the processes of steps S41 to S45 until the game ends. When the game ends, for example, when the player performs a predetermined input operation for ending the game, or when user 3 performs a predetermined input operation for leaving the game midway (YES in step S46), the process shown in FIG. 14 ends.

[0170] (Game Progression Processing in Server 200) FIG. 15 is a flowchart showing an example of the flow of a game progression process executed by the server 200.

[0171] In step S51, the processor 20 receives game progress information from the game play terminal 300. In step S52, the processor 20, functioning as the log generating unit 212, updates a log of the game progress (hereinafter, a play log). Note that the play log is generated by the processor 20, for example, when initial placement information is received from the game play terminal 300.

[0172] In step S53, the processor 20 transmits the received game progress information to each user terminal 100.

[0173] In step S54, if item input information has been received from any of the user terminals 100 (YES in step S54), in step S55, the processor 20 updates the play log as the log generating unit 212. In step S56, the processor 20 transmits the received item input information to the game play terminal 300.

[0174] The processor 20 continues and repeats the processes of steps S51 to S56 until the game ends. When the game ends, for example, when information indicating that the game has ended is received from the game play terminal 300 (YES in step S57), in step S58, the processor 20, as the list generation unit 213, generates a list of users who participated in the game (user list 234) from the play log. The processor 20 stores the generated user list 234 in the server 200.

[0175] FIG. 16 is a diagram showing a specific example of user list 234. The "User" column stores information (e.g., user name) indicating each user who participated in the game. The "Tag" column stores information (tag) generated based on the support provided by each user to the player. In the example of FIG. 16, among the tags stored in the "Tag" column, those without quotation marks are information automatically generated by processor 20, and those with quotation marks are information manually input by the administrator of the game.

[0176] 16, the user "AAAAA" is associated with the information of magazine, 10th floor, boss, and "defeat the boss by gifting the magazine." This indicates that, for example, in a boss battle on the stage 10th floor, the user "AAAAAA" inserted a magazine, and the avatar object 610 defeated the boss with the bullets from the inserted magazine.

[0177] Additionally, the user “BBBBB” is associated with the information: first aid kit, 3F, weak enemy, and “recovered just before game over.” This indicates that, for example, in a battle with a weak enemy on stage 3F, the user “BBBBB” throws in a first aid kit, and as a result, the avatar object 610’s health is recovered just before it reaches 0 (game over).

[0178] Additionally, the user "CCCCC" is associated with information such as barricade, 5F, and weak enemies, and "two zombies stopped by barricade." This indicates that, for example, in a battle with weak enemies on the 5F stage, the user "CCCCC" put in a barricade (obstacle object 672 in FIG. 11), and as a result, succeeded in stopping two weak enemies.

[0179] 16, one support provided is associated with the user name of each user 3, but the user name of a user 3 who provided support multiple times is associated with tags for each of the multiple support times. It is preferable that each tag is distinguished in the user list 234. This allows a player who refers to the user list 421 using the distribution terminal 400 after the game ends to accurately understand the content of each support.

[0180] <Distribution of operation instruction data> (Distribution process in distribution terminal 400) Fig. 17 is a flowchart showing an example of the flow of a distribution process executed in distribution terminal 400. Fig. 18 is a diagram showing a specific example of a screen displayed in distribution terminal 400. Fig. 19 is a diagram showing another specific example of a screen displayed in the distribution terminal.

[0181] In step S61, processor 40, functioning as operation receiving unit 413, receives a first operation for displaying a list of users who have participated in the game (user list 234). Download screen 721 shown in Fig. 18(A) is a screen for downloading user list 234 from server 200 and displaying it on display unit 452. Download screen 721 is, as an example, a screen displayed immediately after an operation for starting an application that executes the distribution process shown in Fig. 17 is input to distribution terminal 400.

[0182] The download screen 721 includes, for example, UI images 722 and 723. The UI image 722 accepts an operation for downloading the user list 234, that is, the first operation described above. The first operation may be, for example, an operation of tapping the UI image 722. The UI image 723 accepts an operation for ending an application. The operation may be, for example, an operation of tapping the UI image 723.

[0183] When a tap operation on the UI image 722 is accepted, in step S62, the processor 40, functioning as the communication control unit 411, acquires (receives) the user list 234 from the server 200. In step S63, the processor 40, functioning as the display control unit 412, causes the display unit 452 to display the user list 234. Specifically, the processor 40 causes the display unit 452 to display a user list screen generated based on the user list 234. As an example, the user list screen may be a user list screen 731 shown in FIG. 18(B). The user list screen 731 is made up of record images corresponding to the respective records in the user list 234. In the example of FIG. 18(B), record images 732A to 732C are described as record images, but the number of record images is not limited to three. In the example of FIG. 18(B), if the number of records in the user list 234 is greater than three (i.e., the number of users participating in the game is greater than three), the player can display other record images on the display unit 452, for example, by inputting an operation to scroll the screen (e.g., a drag operation or a flick operation) on the touch screen 45.

[0184] As an example, the record images 732A to 732C include user names 733A to 733C, tag information 734A to 734C, and icons 735A to 735C, respectively. Hereinafter, when there is no need to distinguish between the record images 732A to 732C, the user names 733A to 733C, the tag information 734A to 734C, and the icons 735A to 735C, they will be referred to as "record image 732," "user name 733," "tag information 734," and "icon 735," respectively.

[0185] The user name 733 is information indicating each user who participated in the game, which is stored in the "user" column in the user list 234. The tag information 734 is information indicating a tag associated with each piece of information indicating each user who participated in the game in the user list 234. For example, the record image 732A includes "AAAAA" as the user name 733A. Therefore, the record image 732A includes, as the tag information 734A, "Magazine, 10F, Boss, 'Defeat the boss by presenting a magazine'," which is associated with "AAAAA" in the user list 234. The icon 735 is, for example, an image set in advance by the user.

[0186] The processor 40 may store the received user list in the distribution terminal 400 (user list 421 in FIG. 7). The download screen 721 may include a UI image (not shown) for displaying the user list 421 on the display unit 452. In this example, when the UI image is tapped, the processor 40 does not download the user list 234, but reads out the user list 421, generates a user list screen from the user list 421, and displays it on the display unit 452.

[0187] In step S64, the processor 40, functioning as the operation receiving unit 413, receives a second operation for selecting one of the users included in the user list screen 731. The second operation may be, for example, an operation of tapping one of the record images 732 on the user list screen 731. In the example of FIG. 18(B), the player inputs a tap operation on the record image 732A. That is, the player selects the user "AAAAA" as the user to whom the action instruction data is to be distributed.

[0188] When the tap operation on the record image 732 is accepted, in step S65, the processor 40, as the display control unit 412, causes the motion list 422 to be displayed on the display unit 452. Specifically, the processor 40 causes the display unit 452 to display a motion list screen generated based on the motion list 422. The motion list screen may be, for example, a motion list screen 741 shown in FIG. 19. The motion list screen 741 is made up of record images corresponding to each record in the motion list 422. In the example of FIG. 19, record images 742A to 742C are shown as record images, but the number of record images is not limited to three. In the example of FIG. 19, when the number of records in the motion list 422 is more than four, the player can display other record images on the display unit 452 by inputting, for example, an operation of scrolling the screen (for example, a drag operation or a flick operation) to the touch screen 45.

[0189] As an example, record images 742A to 742C include motion names 743A to 743C, motion images 744A to 744C, and UI images 745A to 745C, respectively. Hereinafter, when there is no need to distinguish between record images 742A to 742C, motion names 743A to 743C, motion images 744A to 744C, and UI images 745A to 745C, they will be referred to as "record image 7432," "motion name 743," "motion image 744," and "UI image 745," respectively.

[0190] The motion name 743 is information for identifying a motion stored in the motion list 422. The motion image 744 is an image generated from the motion data associated with each motion name in the motion list 422. As an example, the processor 40 includes an image of the avatar object 610 taking an initial posture in each motion data in the record image 742 as the motion image 744. The motion image 744 may be a UI image that accepts a predetermined operation by the player (for example, a tap operation on the motion image 744). When the processor 40 accepts the predetermined operation, it may play a motion video in which the avatar object 610 moves based on the motion data. When the motion video ends, the processor 40 may automatically redisplay the motion list screen 741.

[0191] Note that, instead of the motion image 744, the record image 742 may include a UI image including the text "motion playback", for example.

[0192] In step S66, processor 40, functioning as operation receiving unit 413, receives a third operation for selecting a motion. The third operation may be, for example, a tap operation on UI image 745. That is, UI image 745 receives an operation for selecting motion data corresponding to each record image 742. Having received the third operation, processor 40, functioning as motion identifying unit 415, identifies the motion data selected by the player.

[0193] In step S67, the processor 40, functioning as the display control unit 412 and the voice receiving unit 414, receives a voice input from the player while playing back a motion moving image in which the avatar object 610 moves based on the selected motion data.

[0194] FIG. 20 is a diagram showing a specific example of voice input by the player 4. As shown in FIG. 20, the player 4 inputs a speech voice 820A while playing back a motion video 810A. This speech voice 820A is a speech voice addressed to a user 3 (hereinafter, user 3A) whose user name is "AAAAA." That is, in the example of FIG. 20, the player 4 selects the user 3A (first user) in step S64 and creates action instruction data addressed to the user 3A. It is assumed that the user terminal 100 used by the user 3A is the user terminal 100A.

[0195] The speech voice 820A is addressed to the user 3A, and is therefore based on the content of the support that the user 3A provided to the avatar object 610 (in other words, the player 4). Specifically, the user 3A inserts a magazine in a boss battle on stage 10F, and the avatar object 610 defeats the boss with the bullets from the magazine. Therefore, the speech voice 820A is "Thank you for giving me the magazine in the boss battle! Your timing was perfect! Thanks to you, AAAAA, I was able to clear it!" In this way, it is preferable that the speech voice includes the content of the support that the user 3 provided in the game and gratitude to the user 3.

[0196] In one aspect, the player 4 creates a message to be sent to the user 3 before starting voice input, i.e., before inputting the third operation to the distribution terminal 400. In another aspect, the message to be sent to the user 3 may be automatically generated by the processor 40. The processor 40 may also display a tag associated with the user 3 selected by the second operation superimposed on the motion video 810A.

[0197] The processor 40 converts the received voice into voice data. In step S68, the processor 40, functioning as the motion instruction data generating unit 416, generates motion instruction data including the voice data and the motion data of the selected motion.

[0198] In step S69, the processor 40, functioning as the communication control unit 411, distributes the generated action instruction data to the user terminal 100 (first computer) of the selected user 3 (user 3A in the example of FIG. 20). FIG. 21 is a diagram showing yet another specific example of a screen displayed on the distribution terminal 400. After executing step S68, the processor 40, functioning as the display control unit 412, causes the display unit 452 to display a distribution screen. As an example, the distribution screen may be a distribution screen 751 shown in FIG. 21(A). The distribution screen 751 includes a UI image 752 and a motion image 753A. Furthermore, the distribution screen 751 may include information indicating a user to which the action instruction data is to be distributed, as shown in FIG. 21(A).

[0199] The UI image 752 accepts an operation for delivering the action instruction data to the selected user 3. The operation may be, for example, a tap operation on the UI image 752. The motion image 753A is a UI image for accepting an operation for playing a video based on the generated action instruction data, that is, a video based on the action instruction data generated for the user 3A. The operation may be, for example, a tap operation on the motion image 753A. Note that the UI image for accepting an operation for playing the generated video is not limited to the motion image 753A. For example, it may be a UI image including the text "Play video". The processor 40 may automatically redisplay the delivery screen 751 when the video ends.

[0200] It is preferable that the distribution screen 751 further includes a UI image that accepts an operation to return to receiving voice input. The operation may be, for example, a tap operation on the UI image. By including the UI image on the distribution screen 751, the player 4 can retry voice input when the voice input fails, for example, when the player 4 makes a mistake in what to say. Note that the UI image may be a UI image that accepts an operation to return to selecting motion data.

[0201] When a tap operation on the UI image 752 is received, the processor 40 transmits the action instruction data together with information indicating the user 3A to the server 200. The server 200 identifies the user terminal 100 to which the action instruction data is to be transmitted based on the information indicating the user 3A, and transmits the action instruction data to the identified user terminal 100 (i.e., the user terminal 100A).

[0202] When the transmission of the operation instruction data is completed, the processor 40 may, for example, cause the display unit 452 to display a delivery completion screen 761 shown in Fig. 21(B). For example, the delivery completion screen 761 includes UI images 762 and 763. Furthermore, as shown in Fig. 21(B), the delivery completion screen 761 may include text indicating that the transmission of the operation instruction data is completed.

[0203] The UI image 762 accepts an operation for starting the creation of action instruction data addressed to another user 3. The operation may be, for example, an operation of tapping the UI image 762. When the processor 40 accepts the tap operation, the processor 40 causes the display unit 452 to display the user list screen again. That is, when the processor 40 accepts the tap operation, the distribution process returns to step S63. At this time, the processor 40 may generate a user list screen based on the user list 421 stored in the distribution terminal 400 and display the user list screen on the display unit 452. The UI image 763 accepts an operation for ending an application. The operation may be, for example, an operation of tapping the UI image 763. When the processor 40 accepts the operation, the distribution process ends.

[0204] In the example described with reference to Figures 20 and 21, as shown in Figure 21 (C), the distribution terminal 400 transmits action instruction data for a video addressed to user 3A (user 3 whose user name is "AAAAA") only to user terminal 100A.

[0205] FIG. 22 is a diagram showing another specific example of voice input by the player 4. As shown in FIG. 22, the player 4 inputs a speech voice 820B while playing back a motion video 810B. This speech voice 820B is a speech voice addressed to a user 3 (hereinafter, user 3B) whose user name is "BBBBB." That is, in the example of FIG. 22, the player 4 inputs a tap operation on the record image 732B corresponding to the user 3B in step S64 to create action instruction data addressed to the user 3B. It is assumed that the user terminal 100 used by the user 3B is the user terminal 100B.

[0206] The speech voice 820B is addressed to the user 3B, and is therefore based on the content of the support that the user 3B provided to the avatar object 610 (in other words, the player 4). Specifically, in a battle with a weak enemy on the 3F stage, the user "BBBBB" throws in a first aid kit, and as a result, the avatar object 610's strength is restored just before it becomes 0 (the game is over). Therefore, the speech voice 820B has the content "Thanks to the first aid kit that BBBBB gave me, I didn't get a game over on the 3F. Thank you so much!"

[0207] Fig. 23 is a diagram showing yet another specific example of a screen displayed on distribution terminal 400. Distribution screen 751 shown in Fig. 23(A) includes UI image 752 and motion image 753B. When a tap operation is received, motion image 753B plays a video based on action instruction data generated for user 3B.

[0208] When a tap operation on the UI image 752 is received, the processor 40 transmits the action instruction data together with information indicating the user 3B to the server 200. The server 200 identifies the user terminal 100 to which the action instruction data is to be transmitted based on the information indicating the user 3B, and transmits the action instruction data to the identified user terminal 100 (i.e., the user terminal 100B).

[0209] In the example described with reference to Figures 22 and 23, as shown in Figure 23(C), the distribution terminal 400 transmits action instruction data for a video addressed to user 3B (user 3 whose user name is "BBBBB") only to user terminal 100B.

[0210] As described above, the content of the voice based on the voice data included in the action instruction data is based on the content of the support that the user 3 provided to the player 4 when participating in the most recent game. Since the content of the support differs for each user 3, the content of the voice will be different for each user 3. In other words, after the game ends, action instruction data including voices with different content is transmitted to at least some of the user terminals 100 of the users 3 who participated in the game.

[0211] Also, the motion of the avatar object 610 in the example of Fig. 22 is different from the motion in the example of Fig. 20. That is, in generating the action instruction data for the user 3B, the player 4 selects motion data different from that in generating the action instruction data for the user 3A. Specifically, in step S66, the player 4 inputs a tap operation on the UI image 745B to select the motion data corresponding to the record image 742B. In this way, the player 4 can make the motion data included in the action instruction data different for each user 3.

[0212] Then, the action instruction data for each user 3, including the voice data with different contents for each user 3 and the motion data selected for each user 3, is transmitted only to the user terminal 100 of each user 3. In other words, the action instruction data unique to each user terminal 100 is transmitted to each user terminal 100 of the selected user 3.

[0213] FIG. 24 is a diagram showing an overview of transmission of game progress information from a game playing terminal 300 to a user terminal 100. While the action instruction data for playing moving images in the user terminal 100 is unique for each user terminal 100, as shown in FIG. 24, the game progress information transmitted to the user terminals 100 of all users 3 participating in the game during game execution is common to each user terminal 100. In other words, the action instruction data included in the game progress information is also common to each user terminal 100. In this way, it can be said that the action instruction data for playing moving images and the action instruction data for progressing the game are different data in terms of similarities and differences between the user terminals 100 and transmission destinations.

[0214] (Video playback process in user terminal 100) FIG. 25 is a flowchart showing an example of the flow of a video playback process executed by the user terminal 100.

[0215] In step S71, the processor 10 receives the action instruction data as the video playback unit 117. In step S72, the processor 10, as the video playback unit 117, notifies the user 3 of the reception of the action instruction data. As one example, the processor 10 notifies the user 3 of the reception of the action instruction data by at least one of displaying a notification image on the display unit 152, playing a notification sound from a speaker (not shown), and lighting or blinking a lighting unit (not shown) constituted by an LED (light-emitting diode) or the like.

[0216] In step S73, processor 10, as operation receiving unit 111, receives a first playback operation for playing the video. The first playback operation may be, for example, an operation of tapping a notification image. In step S74, processor 10, as video playback unit 117, renders the action instruction data and plays the video. For example, processor 10 may start an application for playing the game and play the video, or may start an application for video playback separate from the application and play the video. Hereinafter, the video will be referred to as a "thank you video."

[0217] 26 is a diagram showing a specific example of playback of a thank you video. Specifically, it is a diagram showing an example of playback of a thank you video on user terminal 100 of user 3A. In thank you video 910A played on user terminal 100, avatar object 610 utters sound 920A while performing a certain motion. In other words, processor 10 causes sound 920A to be output from a speaker (not shown) while playing back thank you video 910A including avatar object 610 performing a certain motion.

[0218] The motion in the thank you video 910A is based on the motion data selected by the player 4 in generating the action instruction data addressed to the user 3A, and the audio 920A is based on the audio data generated from the speech audio 820A input by the player 4 in generating the action instruction data. In other words, the audio 920A is audio including the content of the support provided by the user 3A in the game and gratitude for the support. In this way, the user 3A can watch the thank you video in which the avatar object 610 speaks the content of the support provided by the user 3A in the game and gratitude for the support, by inputting the first playback operation.

[0219] For example, after the playback of thank you video 910A ends, user terminal 100 may display at least one UI image on touch screen 15. The UI image may be, for example, a UI image that accepts an operation to play thank you video 910A again, a UI image that accepts an operation to transition to another screen, or a UI image that accepts an operation to end an application.

[0220] Also, as an example, user terminal 100 may display at least one UI image on touch screen 15 during playback of thank you video 910A. The UI image may be a plurality of UI images each of which accepts an operation to temporarily stop or end thank you video 910A being played, or to change the scene being played.

[0221] Note that these UI images displayed during playback of thank you video 910A and after playback of thank you video 910A has stopped do not include UI images for replying to avatar object 610. That is, thank you video 910A according to this embodiment is not provided with means for replying to avatar object 610.

[0222] 27 is a diagram showing another specific example of playback of a thank you video. Specifically, this is a diagram showing an example of playback of a thank you video on user terminal 100 of user 3B. In thank you video 910B played on user terminal 100, avatar object 610 utters sound 920B while performing a certain motion. In other words, processor 10 causes sound 920B to be output from a speaker (not shown) while playing back thank you video 910B including avatar object 610 performing a certain motion.

[0223] The motion in the thank you video 910B is based on the motion data selected by the player 4 when generating the action instruction data addressed to the user 3B, and the audio 920B is based on the audio data generated from the speech audio 820B input by the player 4 when generating the action instruction data. Therefore, the motion performed by the avatar object 610 in the example of FIG. 27 is different from the motion in the example of FIG. 26. Also, the audio 920B is audio including the content of the assistance provided by the user 3B in the game and gratitude for the assistance. Therefore, in the example of FIG. 27, the content of the audio 920B is different from the content in the example of FIG. 26. The content of the audio 920A in the example is different from that of the audio 920A in the example.

[0224] In this way, the thank you video received by at least some of the user terminals 100 of the users 3 who participated in the game after the game ends is a video in which the speech content of the avatar object 610 differs for each user 3 .

[0225] The processor 10 may display a UI image 930 including a content encouraging the user to participate in the next game by superimposing it on the video 910. The UI image 930 may be delivered together with the action instruction data, or may be stored in the user terminal 100 as the game information 132.

[0226] In the live distribution game, a virtual space 600B, which is the same as the virtual space 600A defined by the game play terminal 300, is defined by a user terminal (viewing terminal, terminal device) 100. The game play terminal 300 distributes game progress information, including action instruction data that can identify the movement and speech of the performer, to a plurality of user terminals 100 via a server 200. A plurality of types of object data for displaying objects including an avatar object 610 are stored in advance in a memory 11 of the user terminal 100. When an operation for starting a live distribution game is performed, the user terminal 100 identifies the action instruction data from the game progress information distributed from the game play terminal 300 via the server 200, and analyzes (renders) the action instruction data to identify the type, position, posture, etc. of the object. Furthermore, the user terminal 100 places objects including the avatar object 610 in the virtual space 600B using the object data stored in the memory 11 based on the analysis result.

[0227] A field of view image 650 of a field of view area 640A corresponding to the position, orientation, and inclination of a virtual camera 620A placed in a virtual space 600A is displayed on the monitor 51 of the HMD 500. In addition, a field of view image 660 of a field of view area 640B corresponding to the position, orientation, and inclination of a virtual camera 620B placed in a virtual space 600B is displayed on the touch screen 15 of the user terminal 100. A speaker (not shown) provided in the user terminal 100 outputs the voice of the performer based on the action instruction data identified from the game progress information.

[0228] <Game Overview> The game that can be provided by the system 1 according to the present embodiment includes, as a plurality of types of parts constituting the live performance by the performer, a live distribution part performed during a live distribution period allocated before and after the live game period, and a live game part performed during the live game period. The live distribution part is a part where an interaction such as a dialogue can be performed between the user and the avatar object 610 (performer). On the other hand, the live game part is a part where a predetermined game can be performed between the user and the avatar object 610 (performer). The game that can be provided by the system 1 according to the present embodiment is composed of a plurality of types of parts including a live distribution part and a live game part, and progresses by shifting (switching) the parts in a predetermined order on the distribution side and according to the progress of the performer. On the game play terminal 300 side, game progress information including action instruction data for enabling a video of a virtual space including the avatar object 610 that moves in real time according to the movement of the performer to be displayed on the user terminal 100 side is simultaneously live distributed. During live distribution, whether during the live distribution part or the live game part, the user (viewer) can insert an item, input a comment, or the like at an appropriate timing while checking the status of the avatar object 610. For example, in response to a touch operation by the user, an object of an item designated by the user (hereinafter also referred to as a tipping item) can be arranged or displayed (reflected) in the virtual space in which the avatar object 610 is arranged. This allows the user's presence, contribution, and the like to be appealed to performers and other users, and can increase the interest of the game. When a comment is sent, comment information capable of identifying the content of the comment and the name of the user who sent it is sent to the game play terminal 300 via the server. Also, when a tipping item is inserted, insertion information capable of identifying the type of tipping item and the name of the user who inserted it is sent to the game play terminal 300 via the server.

[0229] Games that can be provided by the system 1 generate a virtual space on the user terminal 100 side based on game progress information simultaneously distributed from the game play terminal 300 side, and display images in which avatar objects 610 move in real time in accordance with the movements of the performers. Specific examples will be described below with reference to Figs. 28 to 30. Figs. 28 and 29 show examples of display on the monitor 51 of the HMD 500 on the game play terminal 300 side during live distribution. Fig. 30 shows an example of display on the touch screen 15 of the user terminal 100 during live distribution.

[0230] 28(A), the display area of ​​the monitor 51 of the HMD 500 on the game play terminal 300 is provided with a first display area 51a for displaying an image including an avatar object 610 in the virtual space 600A, and a second display area 51b for displaying an image capable of identifying information useful to the performer as the player in proceeding with the live performance (e.g., a cheat sheet (cue card) necessary for proceeding with the live performance, an event displayed on the user terminal, etc.). The game play terminal 300 performs a display control process of the display area including the first display area 51a and the second display area 51b by the control unit 310 using a program and image data, etc. (e.g., stored in the storage unit 320) for controlling the display of the display area.

[0231] The image displayed in the first display area 51a is a field of view image based on the field of view from a virtual camera for mirror arranged in the virtual space 600A at a position where the avatar object 610 can be viewed from the front. The position and inclination of the virtual camera for mirror may be determined in advance so that the avatar object 610 in the initial state can be viewed from the front, or may vary depending on the position and orientation of the avatar object 610 (performer) so that the avatar object 610 is always viewed from the front, or may be set by another operator on the distribution side. Note that images similar to those shown in Figs. 28 and 29 are displayed on the display 530. Therefore, not only the performer but also other operators on the distribution side can view similar images.

[0232] (Live streaming part) When the performer performs a live distribution start operation using the controller 540, distribution of game progress information (live distribution) is started. Here, an example will be described in which a live distribution part is started at the start of live distribution. In the live distribution part, the game play terminal 300 defines the virtual space 600A as a live distribution space, and places virtual objects including, for example, an avatar object 610 imitating a girl wearing a dress, a banner with the word "Banquet" written on it, and a decorative object (background object) imitating a mountain in the live distribution space as a title logo effect of the distribution content. The first display area 51a in FIG. 28(A) shows a field of view image from the mirror virtual camera at a predetermined timing immediately after the start of the live distribution part. Note that in FIG. 28(B) and FIG. 29, the first display area 51a and the second display area 51b of the monitor 51 of the HMD 500 are illustrated. The first display area 51a in FIG. 28(B) shows a field of view image from the mirror virtual camera at a predetermined timing immediately after the start of the live distribution part.

[0233] The game play terminal 300 detects the body movements of the performer based on the output from each of the HMD 500 and the motion sensor 520 (i.e., captures the motion of the performer), and, based on the motion data generated thereby, positions and moves virtual objects including the avatar object 610 within the live distribution space. The game play terminal 300 also generates action instruction data based on the motion data, audio data, etc., for making the positions, movements, utterances, etc. of the virtual objects including the avatar object 610 and decorative objects specifiable (displayable and outputtable on the user terminal side), and distributes game progress information including the action instruction data to multiple user terminals 100.

[0234] In the live distribution part, each of the multiple user terminals 100 defines the virtual space 600B as a live distribution space and places a virtual camera 620B at a predetermined position in the live distribution space (for example, a position where the avatar object 610 can be captured from the front). The user terminal 100 places virtual objects including the avatar object 610 and decorative objects in the live distribution space, or moves virtual objects that have already been placed, based on model data previously stored in the memory 11 and game progress information distributed from the game play terminal 300. In addition, the user terminal 100 defines a field of view 640B according to the current position and direction of the virtual camera 620B, and displays a field of view image 660 corresponding to the field of view 640B on the touch screen 15 as a live distribution video (see FIG. 30(A) and FIG. 30(B)). The game progress information distributed from the game play terminal 300 side is distributed to the multiple user terminals 100 simultaneously at predetermined time intervals. This allows at least the movements of the avatar object 610 corresponding to the movements of the performer, the speech of the performer, and the like to be shared between the game play terminal 300 and each of the plurality of user terminals 100, and is updated and output at predetermined time intervals.

[0235] 30(A) and 30(B) show display examples including an avatar object 610, a banner reading "Banquet," and a decorative object resembling a mountain. Note that FIG. 30(A) shows a field of view image from virtual camera 620B on user terminal 100 side in the live distribution space at the same timing as FIG. 28(B). FIG. 30(A) shows an image from a different viewpoint compared to FIG. 28(B) due to differences in the position of the virtual camera, but the movement of avatar object 610 is the same.

[0236] During the live distribution period, the performers have, for example, predetermined conversations (greetings, reports on events that have happened nearby, etc.) and notices and explanations of the next part (for example, announcing that a live game will start and an overview of the live game, etc.). In this embodiment, a field of view image from a mirror virtual camera is displayed in the first display area 51a of the monitor 51, and an image that can identify information useful to the performers as the live progresses is displayed in the second display area 51b. Information useful to the performers is notified in the "Current Part" column, "Cue Cards for the Performers" column, "What is Happening on the User's Side" column, and "Next Part" column from above the second display area 51b, as exemplified in FIG. 28(B) and the like.

[0237] The "Current Part" column notifies the current state (condition) in the live distribution. The "Cue Card for Performer" column notifies the contents desired for the performer to say, the actions and gestures desired for the performer to take, the direction desired for the performer to face, etc. The "What is happening on the user's side" column is a column that displays information that can identify the items displayed on the user's terminal at the current timing (also the items being displayed), and for example, the contents occurring on the user's terminal side, the contents being displayed, and the game status (remaining time, speed, etc.) during a live game are notified. The "Next Part" column notifies the next state in the live distribution and the estimated time remaining until the start of the next state, etc. This allows the performer to check the field of view image including the avatar object 610 during the live distribution, while checking the useful information shown in the "Current Part", "Cue Card for Performer", "What is happening on the user's side", and "Next Part" columns, and reflect it in his / her statements and actions. As a result, regardless of the performer's ability, experience, knowledge, etc., the quality and content of the live distribution can be maintained at a certain level, and the live can be carried out appropriately and efficiently.

[0238] FIG. 28(B) is a timing after the start of the live distribution part, so "Opening" is displayed in the "Current Part" column as useful information, "·Talk about anything to liven things up·Inform the performer that today's game is a rock-paper-scissors game" is displayed in the "Cue Card for Performer" column, "·Title logo production of distribution content" is displayed in the "What is happening on the user's side" column, and "Rock-paper-scissors game part, estimated remaining time: 2 minutes 30 seconds" is displayed in the "Next Part" column. This allows the performer to liven up the live space in line with the opening, and to share the fact that the rock-paper-scissors game part will start next, as well as the production and background that the user is watching. As a result, the quality and content of the live distribution can be maintained at a certain level, and the live can be carried out appropriately and efficiently. In this embodiment, an example of notifying the performer of useful information by text will be described, but as long as the performer can be identified, the information is not limited to text, and instead or in addition to this, icon images, animated videos, etc. may be used to notify the performer.

[0239] When the performer completes an explanation during the live distribution part, the performer performs a part transition operation (predetermined button operation, etc.) on the controller 540 to transition to the next part. When the game play terminal 300 detects the part transition operation, it performs processing to end the current opening part (live distribution part) and transition to preparation for the next rock-paper-scissors game part (live game part). When transitioning from a live distribution part to a live game part (transition to a part of a different type or scene), the background, etc. changes significantly, so game progress information including part end information is output. On the other hand, during part transitions within a live distribution part and a live game part (part transitions within the same type), the background, etc. does not change significantly, so the same game progress information as usual is output.

[0240] When the user terminal 100 receives game progress information including part end information, a predetermined transition screen is displayed over the entire display area of ​​the touch screen 15 regardless of the position and orientation of the virtual camera 620B (i.e., the viewing area 640B for each user) when starting display based on the game progress information (see FIG. 30(C)). This makes it possible to easily notify the user that processing to transition to a part is being performed. Display data for displaying the transition screen is stored in advance in the memory 11 of the user terminal 100 when the game program is downloaded or updated. In FIG. 30(C), a still image in which a predetermined pattern is displayed as the background and each of the three characters is taking a predetermined action is exemplified as the transition screen. Note that the transition screen is not limited to a still image, and may be a predetermined moving image. In addition, the type of transition screen can be set in advance for each user terminal 100, and the type of transition screen set for each user terminal 100 may be displayed.

[0241] After performing the part transition operation, the performer prepares for the live distribution of the next part and reconfirms the arrangement and progress. When the part transition operation is performed, an object for the rock-paper-scissors game is placed in the virtual space 600A defined on the game play terminal 300 side. In the live game part of the rock-paper-scissors game, an avatar object 610 and a choice object are placed in the virtual space 600A, and a wreath object reminiscent of a wreath is placed as a background. Therefore, when the part transition operation is performed, an image as shown in FIG. 29(A) is displayed as a background in the first display area 51a of the monitor 51 while a transition screen is being displayed on the user terminal side. This allows the performer to know in advance what kind of atmosphere the rock-paper-scissors game will be played in. Even during the transition screen display, the avatar object 610 is displayed in a manner corresponding to the performer's actions. This allows the performer to check the display manner of the avatar object 610.

[0242] In addition, when a part transition operation is performed, the second display area 51b of the monitor 51 displays information that is useful for preparing for the next live game part of the rock-paper-scissors game and reconfirming the arrangement and progress, while the transition screen is being displayed on the user terminal side. Specifically, as information useful to the performer, "Currently transitioning" is displayed in the "Current part" column, "Cue card for performer" column displays "Rules and precautions will be explained in the next part", "Transition screen being displayed" is displayed in the "What is happening on the user's side" column, and "Explanation of rules, estimated 10 seconds remaining" is displayed in the "Next part" column. This allows the performer to know in advance that it is necessary to first explain the rules and precautions before starting the rock-paper-scissors game. As a result, the performer can also collect knowledge to progress through the next part. When preparations for the next part are complete, the performer performs a part transition operation on the controller 540. When the game play terminal 300 detects a part transition operation, it simultaneously distributes game progress information including part start information for ending the transition screen display and starting the next part. The user terminal 100 that has received the game progress information including the part start information switches from the transition screen to an image based on the game progress information at the time of reception, regardless of the display time for which the transition screen was displayed, etc. This allows the transitioned part to be started at the same time on all user terminals that are viewing, and synchronization can be achieved.

[0243] (Live Game Part) 29(A) to 29(C) are diagrams showing a display example of the monitor 51 of the HMD 500 while the rock-paper-scissors game is in progress, and Fig. 30(D) to Fig. 30(F) are diagrams showing a display example of the touch screen 15 of the user terminal 100 while the rock-paper-scissors game is in progress.

[0244] FIG. 29(A) shows a display example at a predetermined timing during the rule explanation part of the live game part. As shown in first display area 51a of FIG. 29(A), avatar object 610 and choice objects 281-283 (see FIG. 30(D) and the like) are arranged in virtual space 600A defined on the game play terminal 300 side, and a wreath object reminiscent of a wreath is arranged as a background. Other objects arranged in virtual space 600A include objects that decorate the periphery of avatar object 610, viewer objects corresponding to the viewer, background objects, floor objects, and the like. Even in the live game part, avatar object 610 moves in response to the motion of the performer and utters the voice of the performer.

[0245] Each of the option objects 281-283 is composed of a stick part and a disk part at the tip. An illustration representing rock, paper, scissors is drawn on the surface of the disk part. The stick part of each of the option objects 281-283 can be grabbed in accordance with the grabbing motion made by the player. The operation (motion) of the player grabbing the option objects 281-283 etc. is detected by a controller 540 provided with a motion sensor 520 attached to the player's hand and detecting the movement of the hand (hand action (e.g., the position of each finger, the shape and direction of the grip, etc.)). The player grabs the option object corresponding to the final hand (rock, paper, scissors) among the option objects 281-283 and holds it up above, thereby finalizing the hand that the avatar object 610 will make.

[0246] Also, during the rule explanation part, as shown in the second display area 51a of FIG. 29(A), "rule explanation" is displayed in the "current part" column as useful information, "3 rounds in total · Press the button within the time limit · Win 50P Tie 10P Loss 0P · Choose from the icons below" is displayed in the "cue card for performer" column, "· Rule explanation is displayed · Icons are displayed below" is displayed in the "what is happening on the user's side" column, and "1st round hand-out period Estimated remaining: 1 minute 10 seconds" is displayed in the "next part" column. This allows the performer to provide necessary explanations to start the rock-paper-scissors game, and then transition to the hand-out period for the user to decide which hand to play in the first round. As a result, the quality and content of the live distribution can be maintained at a certain level, and the live can be progressed appropriately and efficiently. When the performer has finished the explanation, for example, by outputting a voice such as "Now, let's start the first round," and performing a part transition operation, the performer can transition from the rule explanation part for the first round to the hand-out period part for the first round. Furthermore, even if a part transition operation is performed, since the transition is within a live game part, game progress information similar to normal is output, rather than game progress information including the part end information described above.

[0247] During the live game part, the game play terminal 300 live-distributes game progress information including information for identifying the positions, postures, and directions of objects (avatar object 610, choice objects 281-283, etc.) arranged in the virtual space 600A, and audio data (action instruction data), to the multiple user terminals 100 at predetermined time intervals. The game progress information distributed from the game play terminal 300 to the multiple user terminals 100 includes not only the action instruction data, but also initial placement information, game information 132, distribution time, the type of choice object (rock, scissors, paper) determined as the hand played by the avatar object 610, and information for identifying the current part, scene, and period. As a result, on the user terminal 100 side, a virtual space 600B is defined in which objects such as the avatar object 610, choice objects 281-283, wreath object 284, viewer object, and background object are arranged, similar to the virtual space 600A. Furthermore, on the user terminal 100 side, avatar objects 610 and the like are operated based on the game progress information within the virtual space 600B, and a field of view image from a virtual camera 620B placed at a position or the like according to an input operation by the user is displayed on the touch screen 15. Note that the objects shared between the game play terminal 300 and the multiple user terminals 100 may be only the avatar object 610, as long as it includes at least the avatar object 610 that is performed live by an actor, among the objects placed in the virtual space 600A, or may include objects other than the avatar object 610 and the option objects 281 to 283.

[0248] On the user terminal 100 side, when game progress information including part start information is received, the conversion screen is switched to a display corresponding to the next part based on the game progress information. The position and orientation of the virtual camera 620B at this time (i.e., the viewing area 640B for each user) is set to a position (predetermined position) and orientation (predetermined direction) that views the avatar object 610 from the front, regardless of the position and orientation of the virtual camera 620B before the conversion screen is displayed. Note that the position and orientation of the virtual camera 620B when starting a new part are not limited to this, and may be the same as the position and orientation of the virtual camera 620B before the conversion screen is displayed.

[0249] FIG. 30(D) shows a field of view image from virtual camera 620B on the user terminal 100 side in the live distribution space at the same timing as FIG. 29(A). FIG. 30(D) is an image from a different viewpoint compared to FIG. 29(A) due to differences in the position of the virtual camera, etc., but the movement of the avatar object 610 is the same. In the examples of FIG. 30(D) to FIG. 30(F), it is assumed that virtual camera 620B is disposed at a position where avatar object 610 is viewed from the front. Therefore, as an image of the field of view from virtual camera 620B, as exemplified in FIG. 30(D), avatar object 610 is displayed standing up at approximately the center of touch screen 15, and option objects 281 to 283 are displayed to the right of avatar object 610 based on avatar object 610, and a wreath object 284 and the like are displayed behind avatar object 610.

[0250] During the progress of the rock-paper-scissors game, various UI images and images related to game information are displayed on the touch screen 15. In Fig. 30(D), for example, a selection UI 180, a rule explanation display 181, an item insertion icon 185, a comment input icon 186, and the like are displayed superimposed on the field of view image in the virtual space. Also, a comment display section 184 is provided at the bottom left of the touch screen 15 to display the contents of a comment input to the avatar object 610.

[0251] The rule explanation display 181 is an image for explaining the rules of the rock-paper-scissors game. In Fig. 30(D), for example, messages such as "3 rounds in total!", "Press the button within the time limit!", "Win: 50 points", "Tie: 10 points", and "Loss: 0 points" are displayed. Note that a message such as "First round" is displayed in the upper left corner of the touch screen 15 as information for identifying which scene (for example, which round) in the live game part it is.

[0252] The selection UI 180 includes button icons for the user to select a hand to be played, and is a UI for receiving an input operation (tap operation) to the button icons. In other words, the selection UI 180 can be said to be a UI image for receiving an operation related to the result of the rock-paper-scissors game. In the example of FIG. 30(D), the selection UI 180 is displayed with priority over a field of view image including the avatar object 610, a comment display section 184, an item insertion icon 185, a comment input icon 186, and the like. In FIG. 30(D), a message such as "Choose your hand!" is displayed, and below that, from the left, a button icon corresponding to rock, a button icon corresponding to scissors, and a button icon corresponding to paper are displayed. This allows the user to intuitively recognize that a hand to be played can be selected by a touch operation on the button icons. Also, to the left of the message "Choose a hand!", a time limit (also called a reception period; here, "10") for selecting a hand to be played is displayed, and to the right of the message (top right of the selection UI 180) an "x" button for erasing the display of the selection UI 180 itself is displayed. The rock-paper-scissors game is started with the selection UI 180 displayed, but when the selection UI 180 is set to be hidden at the start of the game by the user's initial settings, for example, the game may be started with the selection UI 180 not displayed (such as the screen exemplified in FIG. 30(I)). The time limit displayed in the selection UI 180 starts counting (starts subtracting) after the acceptance of operations is started by the performer's operation.

[0253] Even if the user has not yet finished selecting the hand to be played (input operation to the button icons corresponding to rock, paper, and scissors), the user can erase the selection UI 180 by selecting the "x" button (touch operation to the "x" button, etc.). This is because, when the selection UI 180 is displayed, if the avatar object 610 (character) is hidden by the selection UI 180 as shown in FIG. 30(D) and the like, the image of the avatar object 610 is not visible to the user (the superimposed portion is not visible). Also, when the selection UI 180 is displayed, as shown in FIG. 30(D) and the like, the comment display section 184, the item insertion icon 185, and the like are hidden, making these images difficult to see (the superimposed portion is not visible), and difficult (or impossible) to operate. The user can view the images that were not visible through the selection UI 180 by erasing the selection UI 180 according to the user's own operation.

[0254] The item input icon 185 is an icon that triggers inputting an item and assigning it to the avatar object 610. When an input operation to the item input icon 185 is accepted, a window for selecting and inputting an item from a plurality of types is displayed. As described above, the selection UI 180 is displayed with priority over the item input icon 185. Therefore, when the selection UI 180 is displayed superimposed on the item input icon 185, the input operation to the item input icon 185 may not be validly accepted unless the display of the selection UI 180 is turned off. Also, even when the selection UI 180 is displayed superimposed on the item input icon 185, the input operation to the item input icon 185 may be validly accepted while the display of the selection UI 180 is maintained.

[0255] The comment input icon 186 is an icon that triggers input of a comment to the avatar object 610. When an input operation to the comment input icon 186 is accepted, a window for inputting and sending a comment is displayed. The comment input icon 186 is displayed at a position where the selection UI 180 is not superimposed, but as with the item input icon 185, an input operation to the comment input icon 18 may not be accepted effectively unless the display of the selection UI 180 is turned off. Even when the selection UI 180 is displayed, an input operation to the comment input icon 18 may be accepted effectively while the display of the selection UI 180 is maintained. In the comment display unit 184, a predetermined number of comments entered to the avatar object 610 are displayed together with the user name in order from the most recent one.

[0256] The item input icon 185 and the comment input icon 186 are icons that accept input operations for the user to interact with the avatar object 610, and can also be considered as interaction images. The comment display unit 184 displays the results of the interaction that the user has performed with the avatar object 610, and can also be considered as an interaction image.

[0257] FIG. 29(B) shows a display example at a predetermined timing (when the remaining time limit is 7 seconds) during the first round hand period part to which the performer has moved by a part transition operation. During the first round hand period part, as shown in the second display area 51b in FIG. 29(B), the "current part" column displays "first round hand period" as useful information, the "cue card for performer" column displays "·Talk about the hand you are planning to play·Notice that reception has started·Information on the remaining time limit" is displayed, the "what is happening on the user's side" column displays "·7 seconds remaining in the time limit" and the "next part" column displays "first round performer hand period remaining: 30 seconds". This allows the performer to give the user something to consider, and by notifying the user of the remaining time limit, encourages the user to operate within the time limit. The first display area 51a in Figure 29(B) displays a field of view image from the mirror virtual camera at a specific timing during the hand-out period part of the first round, and shows, for example, the motion of avatar object 610 pondering, such as "What should I play? Which one should I choose?"

[0258] FIG. 30(E) shows a field of view image from the virtual camera 620B on the user terminal 100 side in the live distribution space at the same timing as FIG. 29(B). FIG. 30(E) also shows a display example when the user selects a hand to play. FIG. 30(E) shows an example in which "rock" is selected. When "rock" is selected, the "rock" button icon in the selection UI 180 is updated to a selected state (a state surrounded by a double circle) as shown in FIG. 30(E). In FIG. 30(E), since the time limit is "7", the user has decided on the hand to play with 7 seconds remaining. In addition, the hand selected by the user in the rock-paper-scissors game is stored in a predetermined area of ​​the memory 11, and the hand to play can be changed by operating the button icon again within the time limit thereafter. In addition, even if the selection UI 180 is erased with a hand to play selected, information that can identify the selected hand is maintained in the memory 11. Furthermore, the hand selected by the user in the rock-paper-scissors game and information capable of identifying the user are transmitted to the game playing terminal 300 via the server.

[0259] Fig. 29(C) shows a display example at a predetermined timing during the performer's hand-out part of the first round, which has been shifted to by the performer's part-shifting operation. During the performer's hand-out part, as shown in the second display area 51b of Fig. 29(C), useful information is displayed in the "Current part" column as "Performer's hand-out in the first round"; in the "Cue card for performer" column as "·Make a determined hand"; in the "What is happening on the user's side" column as "·Waiting for performer"; and in the "Next part" column as "Approximate remaining hand-out period for the second round: 15 seconds." This allows the performer to know when to make a hand within the time limit, and at the performer's timing, the performer (avatar object 610) makes a hand-out by raising one of the option objects 281 to 283 upwards, along with a call such as "Jankenhoi!". In the first display area 51a of FIG. 29(C), a view image from the virtual camera for the mirror at a predetermined timing during the part where the performer puts out his / her hand in the first round is displayed, and for example, the avatar object 610 is shown holding up the option object 282 of "scissors". When the avatar object 610 puts out his / her hand, game progress information including information for identifying the type of option object (rock, scissors, paper) that is confirmed as the hand that the avatar object 610 puts out is output. After the performer puts out his / her hand, for example, the "Cue card for performer" column displays "·Speak your thoughts after the first round" and the "What is happening on the user's side" column is updated to information such as "Displaying the winning / losing effect". "Displaying the winning / losing effect" refers to a state in which the result of the rock-paper-scissors game is displayed on each user terminal being viewed, such as "Win!" (see FIG. 30(F)) for the user who beat the performer, "Lose!" for the user who lost to the performer, and "Tie" for the user who put out the same hand as the performer. This information is specified based on the hand selected by the user in the rock-paper-scissors game and information that can identify the user. As a result, the performer can talk about his / her thoughts and react to the outcome in real time, which increases the interest of the audience in the game.

[0260] FIG. 30(F) shows a field of view image from the virtual camera 620B on the user terminal 100 side in the live distribution space at the same timing as FIG. 29(C). In FIG. 30(F), the avatar object 610 holds up the option object 282 of "scissors". This notifies the user that the avatar object 610 has put out "scissors". When the avatar object 610 puts out its hand, information is received for identifying the type of option object (rock, scissors, paper) that has been confirmed as the hand that the avatar object 610 will put out. Based on the information, the processor 10 compares the hand that the user put out with the hand that the user put out and determines whether the user has won or lost. In FIG. 30, since the user selected "rock", it is determined that the user has won. As a result, a message such as "win" is displayed above the avatar object 610 as a winning or losing performance. Note that the hand that the avatar object 610 puts out in the rock-paper-scissors game is the same for all users, but the hand that the user selects is different for each user. Therefore, the actions of the avatar object 610 etc. performed by the player will be the same across multiple user terminals, but the outcome of the rock-paper-scissors game (win / lose presentation) will differ across multiple user terminals depending on the user's input operations.

[0261] As described above, the performer continues to play the second and third rounds of the rock-paper-scissors game based on the useful information displayed on monitor 51, and then transitions to the live distribution part to summarize the live performance, etc. According to the progress of the performer, live video of the second and subsequent rounds of the rock-paper-scissors game is also displayed on user terminal 100 as shown in Fig. 30(G).

[0262] Fig. 30(I) is a diagram showing an example of a display on the touch screen 15 when an input operation to the "x" button in the selection UI 180 is accepted. Fig. 30(I), like Fig. 30(D), shows a state in which the remaining time limit is "10" seconds immediately after the start of the rock-paper-scissors game and the user has not yet selected a move to play.

[0263] In a rock-paper-scissors game, the performer tends to make motions and speak in order to liven up the game. In particular, immediately after the start of a rock-paper-scissors game, other users are likely to be working on selecting their hand, so that by inputting an operation or comment to the item insertion icon 185, the performer is more likely to respond (by motion or speech). However, in the example of FIG. 30(D), the selection UI 180 is displayed superimposed on the avatar object 610, and some of the movements of the avatar object 610 cannot be seen, so that even if there is a response, it may not be visible. Depending on the position and orientation of the virtual camera 600B, the selection UI 180 may not be superimposed on the avatar object 610, but conversely, the selection UI 180 may be superimposed on the face, body, or entirety of the avatar object 610. Furthermore, in the rock-paper-scissors game, regardless of the position of the virtual camera 600B, as shown in FIG. 30(D), the selection UI 180 is displayed superimposed on the item insertion icon 185 and the comment input icon 186, making it difficult for the user to interact with the avatar object 610 or to check responses.

[0264] Considering such a situation, in the game of this embodiment, even when an operation related to the game result is not accepted, the selection UI 180 can be erased by inputting an operation to the "x" button in the selection UI 180. FIG. 30(I) shows an example of a display on the touch screen 15 when the "x" button in the selection UI 180 is touched and the selection UI 180 is erased in a situation such as FIG. 30(D). This makes it possible to view the avatar object 610, the item input icon 185, and the comment input icon 186 that were not visible due to the selection UI 180. As a result, the user can check the movement (including the response) of the avatar object 610 by his / her own operation, and the situation can be changed to one in which it is easy to perform an interaction and to check the response from the performer.

[0265] Furthermore, when the selection UI 180 is erased, as shown in FIG. 30(I), an icon 187 (such as a message icon such as "return to game") for displaying the selection UI 180 again is displayed on the touch screen 15. The icon 187 has a display size smaller than that of the selection UI 180, and is displayed at a position closer to the edge than the selection UI 180 and at a position where it does not overlap with any other icon. This can prevent inconveniences such as the image including the avatar object 610 becoming invisible due to the icon 187. Note that the remaining time limit may be displayed in a predetermined area of ​​the touch screen 15 (for example, an area within the icon 187) even when the selection UI 180 is erased.

[0266] After the selection UI 180 is erased, the selection UI 180 can be displayed again in response to an operation on the icon 187 by the user, so that the user can select or change the hand to be played. Furthermore, even if the selection UI 180 is erased, the processor 10 can perform a process of automatically displaying the selection UI 180 again in response to the remaining time limit, so that the user can be forced to select the hand to be played or confirm the selected hand. When a hand to be played has not yet been selected, the selection UI 180 is redisplayed in a state in which no hand is selected. In contrast, when a hand to be played has already been selected, the selection UI 180 is redisplayed in a state in which the hand is selected. Note that if the user does not select a hand to be played within the time limit, the hand to be played by the user may be automatically and randomly determined. In this case, for example, when the selection UI 180 is redisplayed, the selection UI 180 may be redisplayed in a state in which the automatically determined hand is selected, and when the time limit expires, the selection UI 180 may be redisplayed in a state in which the automatically determined hand is selected.

[0267] 28(B) and 29, in the above embodiment, an image including an avatar object 610 from the virtual space 600A is displayed in the first display area 51a of the monitor 51 visible to the performer, and information useful to the performer (player) in proceeding with the live show (for example, cheat sheets (cue cards) necessary for proceeding with the live show, events displayed on the user terminal, etc.) is displayed in the second display area 51b. This makes it possible to support the performer proceeding with the live show, and to maintain the quality and content of the live show distribution at a certain level regardless of the performer's ability, experience, knowledge, etc., and to proceed with the live show appropriately and efficiently.

[0268] The useful information includes information reported in the "Current Part" column, which allows the current part being performed to be identified. This allows the performer to be accurately informed even if the performer does not remember the current part. The useful information also includes information reported in the "Cue Card for Performer" column, which allows the performer to identify the action to be prompted. This allows the performer to be prompted to make appropriate statements and take appropriate actions. The useful information also includes information reported in the "Next Part" column, which allows the performer to identify the timing to end the current part and the next part. This allows the live performance to proceed after identifying the timing to end the current part objectively, not based on the performer's physical sensation. This allows the performer to be accurately informed even if the performer does not remember the next part. The useful information also includes information reported in the "What is happening on the user terminal side" column. This allows the performer to grasp what the user is looking at at the same time and react to the content (such as ``Hurry up and put your hands out!''), thereby improving the realism of the live performance. The useful information also includes result information reported in the "What is happening on the user terminal side" column, which is updated taking into account the content of the performer's hand-out. This allows the player to react to each user's results, making the game more interesting.

[0269] Also, useful information is displayed in frames that are displayed in different ways (colors, sizes) for each column (type), as shown in Fig. 29. Fig. 29 shows that the frame display manner differs for each column, depending on the presence or absence of gray, the darkness of gray, etc. This makes it possible to prevent useful information belonging to one type from being misread as information of another type as much as possible.

[0270] <Modification> Modifications of the above-described embodiment are listed below.

[0271] (1) In the above embodiment, the avatar object 610 moves based on signals, i.e., motion data, detected by a sensor included in the HMD 500 worn on the performer's head and the motion sensors 520 worn on the performer's hands, feet, waist, etc. However, the avatar object 610 is not limited to this as long as it moves according to the performer's movements. For example, the avatar object 610 may be displayed on the user terminal 100 side to move the avatar object 610 according to the performer's movements by analyzing images of the performer photographed (captured) from multiple directions using multiple cameras, and the game play terminal 300 side (distribution side) may output action instruction data (game progress information including motion data) for moving the avatar object 610 based on the identification results.

[0272] Also, in the above embodiment, an example was described in which the performer checks objects in the virtual space from the image displayed on the HMD 500 worn on the head. However, this is not limited as long as the performer can check the image in the virtual space. For example, a plurality of display units may be arranged to surround the performer (on all four sides or all eight sides, etc.), and the image in the virtual space from the performer's perspective may be displayed on the display units. In this case, the performer can check the virtual space by looking at the images displayed on the surrounding display units. In this case, the image including the avatar object 610 in the virtual space 600A shown in FIG. 28 and FIG. 29, and information useful to the performer as a player when proceeding with the live performance may be notified (displayed) on one of the display units in the direction in which the performer faces in the initial state (at the start of the live performance), or may be notified (displayed) on a display unit separate from the plurality of display units that display the image in the virtual space. Also, an example has been described in which an image including avatar object 610 in virtual space 600A and information useful to the player / performer as the live performance progresses are displayed on the same display unit, but the images may also be displayed on physically separate display units.

[0273] In the above embodiment, an example was described in which an image including avatar object 610 is displayed in virtual space 600A so that the performer can be seen, and information useful to the performer as the player progresses through the live performance is displayed. However, the present invention is not limited to this, and it is also possible to not display an image including avatar object 610 in virtual space 600A, and instead display information useful to the performer as the player progresses through the live performance.

[0274] (2) In the above embodiment, information useful to the performers as the live show progresses is stored in advance in storage unit 320 or the like in association with the progress of the live show, and the corresponding useful information is read out and displayed as the live show progresses. However, the useful information is not limited to this, and may instead or in addition include information that is input or instructed as the live show progresses by another operator on the distribution side who can view display 530 on which images similar to those in Figures 28 and 29 are displayed.

[0275] In the above embodiment, the performer himself performs a part transition operation (such as a predetermined button operation) to transition to the next part. However, the performer may perform the transition by an operation of another operator on the distribution side, or may perform the transition automatically when a time is measured at the start of each part and a predetermined time corresponding to the part is reached. Also, a different trigger may be set as a trigger for transitioning to the next part from among the above-mentioned multiple types of triggers depending on the type of the current part. For example, the live game part may be transitioned by an operation of another operator on the distribution side, and the live distribution part may be transitioned by the performer himself performing a part transition operation.

[0276] (3) In the “Next Part” column, an example has been described in which the “estimated remaining time” is notified as the timing for ending the current part, but the present invention is not limited to this. It is also possible to notify the “time until the start of the next part” as the timing for switching to the next part.

[0277] (4) In the above embodiment, a rock-paper-scissors game part was given as an example of a specific part, but this is not limited to this and may be a card game (such as Old Maid), a game in which the aim is to complete a mission given by an actor (avatar object 610), or a game in which the player competes for the highest game score provided by the actor (avatar object 610).

[0278] In the above embodiment, the remaining time limit and the like were given as examples of information that would be useful to the user as the game progresses, but this is not limited to this. For example, in the case of a game of whack-a-mole, if the speed at which moles appear increases halfway through the game, the information could be information that enables users to identify that their appearance speed has increased (such as a change in the game situation), or in the case of a game of Old Maid with multiple users, the information could be information that enables users to identify that one user has only one card left (such as the game progress situation).

[0279] In the above embodiment, the hand played by the user was used as an example of useful information regarding the game outcome, but the information is not limited to this and may also be statistical values ​​of the hands played by the users (for example, the number of users who played rock, the number of users who played paper, the number of users who played scissors, etc.).

[0280] (5) In the above embodiment, the information useful to the performer, who is the player, during the progress of the live performance is not limited to that shown in the above embodiment, and may be information regarding the interaction with the user. The useful information may be, for example, when user A sends a comment to the performer, the content of the comment that can be identified based on the comment information and the sender, user A. In this case, the performer can react to user A or read out the comment content to share with other users. The useful information may also be, for example, when user B inserts a tipping item, the type of the tipping item that can be identified based on the insertion information and the inserter, user B. In this case, the performer can react to user B or share with other users that the tipping item has been inserted.

[0281] In the above embodiment, as an example of updating useful information triggered by the performer's actions even during one part, an example was described in which the "Cue Card for Performer" column is updated and the "What is happening on the user's side" column is updated to information such as "Win / lose performance is being displayed" triggered by the performer's hand being put out, in relation to Fig. 29(C). However, useful information may be updated, for example, triggered by the user's actions even during one part (for example, displaying the number of users who put out their hands during the user's hand-putting period part, which is updated by adding up according to the user's hand-putting), or may be updated according to the passage of time or the progress of the live performance.

[0282] (6) In the above embodiment, an example (real-time rendering method) has been described in which an image in a virtual space including an avatar object 610 performed by an actor is generated (rendered) and played on the user terminal 100 side using action instruction data, which is data distributed from an external source, and the user terminal 100 side uses the action instruction data to generate (render) and play a video (image) including the avatar object 610. However, the game play terminal 300 and the distribution terminal 400 may distribute video data for displaying a video in a virtual space including the avatar object 610, and the user terminal 100 may use the video data to play a video (image) including the avatar object 610 (video distribution method). In other words, the image in the virtual space 600B is not limited to being displayed on the user terminal 100 using the real-time rendering method, but may be displayed on the user terminal 100 using a video distribution method that distributes video data consisting of images of multiple frames.

[0283] (7) The display mode of the avatar object 610 in the above embodiment may be different for each user depending on the progress of the game (for example, each time a player wins a rock-paper-scissors game). For example, even if the game progress information transmitted from the game play terminal 300 is the same, the costume of the avatar object 610 may be changed to a costume corresponding to the progress of the game. In this case, a plurality of types of costume data of the avatar object 610 may be stored in advance on each user terminal 100. Note that the subject for which the display mode of the avatar object 610 is to be changed may be not only the costume but also the hairstyle, skin color, degree of makeup, and the like. This makes it possible to increase the variation of the avatar object 610 without increasing the processing load on the game play terminal 300 that outputs the game progress information, thereby attracting users.

[0284] Alternatively or additionally, the display mode of the avatar object 610 may be made different from the display mode displayed on the user terminals of other users in response to a charge made by the user during the game. Also, the display mode of the avatar object 610 may be made different depending on the user's ranking, which is updated according to the game result.

[0285] <Additional Notes> The matters described in the above embodiments will be supplemented below.

[0286] (Appendix 1): According to one aspect of an embodiment disclosed herein, a program is executed on a computer (game playing terminal 300) having a processor, memory, and a display unit, which causes the processor to execute the following steps: switching to a predetermined next part among multiple types of parts constituting a performer's live performance when a progress condition is met (part transition operation (prescribed button operation, etc.)); live-streaming display data (game progress information including action instruction data) for enabling a viewer to display an image of a virtual space including a character that moves in real time in response to the movements of the performer progressing through the current part on a terminal device owned by the viewer; and displaying an image of the virtual space including the character in a display area visible to the performer, and displaying information useful to the performer as he progresses through the current part (see Figures 28 and 29).

[0287] (Appendix 2): In (Supplementary Note 1), the useful information includes information for making it possible to identify the current part (current part field).

[0288] (Appendix 3): In (Supplementary Note 1) or (Supplementary Note 2), the useful information includes information (a cue box for the performer) for enabling identification of an action to be prompted to the performer depending on the current part.

[0289] (Appendix 4): In any of (Supplementary Note 1) to (Supplementary Note 3), the useful information includes information (next part field) for making it possible to identify the timing to end the current part or the timing to switch to the next part.

[0290] (Appendix 5): In any one of (Supplementary Note 1) to (Supplementary Note 4), the useful information includes information (next part field) for making it possible to specify the next part to be switched to when the condition for progression is satisfied.

[0291] (Appendix 6): In any of (Supplementary Note 1) to (Supplementary Note 5), the parts constituting the live broadcast include a specific part (live game part) in which a game is played in which the user who owns the terminal device can participate, and the processor executes a step of displaying information that is useful to the user (a column about what is happening on the user's side) according to the progress of the game during the specific part.

[0292] (Appendix 7): In any of (Appendix 1) to (Appendix 6), the parts constituting the live broadcast include a specific part (live game part) in which a game is played in which the user who owns the terminal device can participate, and the useful information includes information regarding the outcome of the game during the specific part (a section on what is happening on the user's side).

[0293] (Appendix 8): In any of (Appendix 1) to (Appendix 7), the display area visible to the performer has a first display area and a second display area separate from the first display area, and the displaying step displays an image in the virtual space including the character and corresponding to a predetermined viewpoint in the first display area, and displays the useful information in the second display area (Figures 28 and 29).

[0294] (Appendix 9): In any of (Appendix 1) to (Appendix 8), a step of receiving information capable of identifying an interaction performed on the performer in response to an operation on the terminal device by a viewer watching the image of the virtual space is executed, and the useful information includes information regarding the interaction identified based on the information received in the receiving step.

[0295] (Appendix 10): In (Appendix 9), the interaction includes sending a comment to the performer, the information capable of identifying the interaction includes information capable of identifying the comment and the name of the viewer who sent the comment (comment information), and the useful information includes information capable of identifying the comment and the name of the viewer who sent the comment.

[0296] (Appendix 11): In (Appendix 9) or (Appendix 10), the interaction includes selecting an item from a plurality of items and reflecting an object of the selected item in the virtual space, the information capable of identifying the interaction includes information (input information) capable of identifying the selected item and the name of the viewer who selected the item, and the useful information includes information capable of identifying the selected item and the name of the viewer who selected the item.

[0297] (Appendix 12): In any one of (Supplementary Note 1) to (Supplementary Note 11), the useful information includes information capable of identifying an item displayed on the terminal device (a section on what is happening on the user's side).

[0298] (Appendix 13): According to one aspect of one embodiment, a computer (game playing terminal 300) is provided with a memory unit for storing a program, and a control unit for controlling operation of the computer by executing the program, and the control unit executes the steps of: switching to a predetermined next part among multiple types of parts constituting a live performance by a performer when a progress condition is met (part transition operation (prescribed button operation, etc.)); live-streaming display data (game progress information including action instruction data) for enabling a viewer to display an image of a virtual space including a character that moves in real time in response to the movements of the performer progressing through the current part; and displaying an image of the virtual space including the character in a display area visible to the performer, and displaying information useful to the performer when progressing through the current part (see Figures 28 and 29).

[0299] (Appendix 14): According to one aspect of one embodiment, a method is executed on a computer (game playing terminal 300) having a processor, memory, and a display unit, and includes the steps of: switching to a predetermined next part among multiple types of parts that make up a live performance by a performer when a progress condition is met (a part transition operation (such as a specified button operation)); live-streaming display data (game progress information including action instruction data) to enable a viewer to display an image of a virtual space including a character that moves in real time in response to the movements of the performer progressing through the current part on a terminal device owned by the viewer; and displaying an image of the virtual space including the character in a display area visible to the performer, and displaying information useful to the performer as he progresses through the current part (see Figures 28 and 29).

[0300] [Software implementation example] The control blocks (particularly control units 110, 210, 310, 410) of the user terminal 100, server 200, game playing terminal 300 (HMD set 1000), and distribution terminal 400 may be realized by logic circuits (hardware) formed in an integrated circuit (IC chip) or the like, or by software.

[0301] In the latter case, the user terminal 100, the server 200, the game play terminal 300 (HMD set 1000), and the distribution terminal 400 are equipped with a computer that executes instructions of a program, which is software that realizes each function. This computer is equipped with, for example, one or more processors, and a computer-readable recording medium that stores the above program. Then, in the above computer, the processor reads the above program from the above recording medium and executes it, thereby achieving the object of the present invention. The above processor can be, for example, a CPU (Central Processing Unit). The above recording medium can be a "non-transient tangible medium", such as a ROM (Read Only Memory), as well as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, etc. Also, a RAM (Random Access Memory) that expands the above program, etc. may be further equipped. Also, the above program may be supplied to the above computer via any transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

[0302] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0303] 1 system, 2 network, 3, 3A, 3B user (first user), 4 player (actor), 10, 20, 30, 40 processor, 11, 21, 31, 41 memory, 12, 22, 32, 42 storage, 13, 23, 33, 43 communication IF, 14, 24, 34, 44 input / output IF, 15, 45 touch screen, 17 camera, 18 distance measurement sensor, 51 monitor, 52 gaze sensor, 53 first camera, 54 second camera, 55 microphone, 56 speaker, 100, 100A, 100B, 100C user terminal (computer, first computer, first information processing device), 110, 210, 310, 410 control unit (first control unit, second control unit), 111, 311, 413 operation reception unit, 112, 312, 412 Display control unit, 113, 313 UI control unit, 114, 314 Animation generation unit, 115, 315 Game progress unit, 116, 316 Virtual space control unit, 117 Video playback unit, 120, 220, 320, 420 Memory unit (first memory unit, second memory unit), 131, 231, 331 Game program (program, first program), 132, 232, 332 Game information, 133, 233, 333 User information, 151, 451 Input unit, 152, 452 Display unit (display), 200 Server, 211 Communication mediation unit, 212 Log generation unit, 213 List generation unit, 234, 421 User list, 300 Game play terminal (external device, second external device), 317 Response processing unit, 400 Distribution terminal (external, first external device, computer, second information processing device), 411, communication control unit, 414, voice reception unit, 415, motion identification unit, 416, action instruction data generation unit, 422, motion list, 423, distribution program (program, second program), 540, 1020, 1021, controller, 500, HMD, 510, HMD sensor, 520, motion sensor, 530, display, 600A, 600B, virtual space, 610, avatar object (character), 620A, 620B, virtual camera, 631, 632, 633, 634, object, 640A, 640B, field of view area, 650, 660, field of view image, 671, enemy object, 672, 673, obstacle object, 674, performance object, 691, 692 Speech, 701, 702, 703A, 70B,704A, 704B, 705, 706, 711, 711A, 711B, 711C, 711D, 722, 723, 745, 745A, 745B, 745C, 752, 762, 763, 930, 2011, 2022, 2031, 2032, 2033, 2034, 2037, 2038, 2051, 2063, 2072, 2073, 2075 UI image (message UI, UI), 721 Download screen, 731 User list screen (list), 732, 732A, 732B, 732C, 742, 742A, 742B, 742C Record image, 733, 733A, 733B, 733C User name, 734, 734A, 734B, 734C Tag information, 735, 735A, 735B, 735C Icon, 741 Motion list screen (options), 743, 743A, 743B, 743C Motion name, 744, 744A, 744B, 744C, 753 Motion image, 751 Distribution screen, 761 Distribution completion screen, 810A, 810B Motion video, 820A, 820B Speech audio, 910A, 910B Video, 920A, 920B Audio, 1000 HMD set, 1010 Object, 1030 Storage medium,

Claims

1. A program executed on a computer, The computer, a switching means for switching to a next predetermined part among a plurality of parts constituting a live performance by a performer when a part switching condition is satisfied; A distribution means for live-distributing to a terminal device of a viewer an image of a virtual space including a character that moves in real time in response to the movement of a performer who is performing a current part; a display means for displaying an image from a viewpoint in which the character is viewed from the front among images in the virtual space in a display area visible to an actor who controls the character according to his / her own movements, and displaying information useful to the actor who controls the character according to his / her own movements when progressing through a current part; The display means is capable of displaying the useful information so that the performer can see it regardless of the performer's orientation.

2. The computer-readable medium according to claim 1 , wherein the useful information includes information for making a current part identifiable.

3. 3. The program according to claim 1, wherein the useful information includes information for enabling identification of an action to be prompted to the performer depending on a current part.

4. The program according to any one of claims 1 to 3, wherein the useful information includes information for making it possible to identify a timing for ending a current part or a timing for switching to a next part.

5. The program according to any one of claims 1 to 4, wherein the useful information includes information for making it possible to identify a next part to be switched to when the part switching condition is satisfied.

6. The parts constituting the live distribution include a specific part in which a game in which a user who owns the terminal device can participate is played, The program according to any one of claims 1 to 5, which causes the computer to function as a means for displaying information that is beneficial to the user in accordance with the progress of the game during the specific part.

7. The parts constituting the live distribution include a specific part in which a game in which a user who owns the terminal device can participate is played, 7. The program according to claim 1, wherein the useful information includes information regarding the outcome of the game during the specific part.

8. the display area visible to the performer includes a first display area and a second display area separate from the first display area; The program described in any one of claims 1 to 7, wherein the display means displays an image including the character in the virtual space corresponding to a predetermined viewpoint in the first display area, and displays the useful information in the second display area.

9. causing the computer to function as a receiving means for receiving information capable of identifying an interaction performed with the performer in response to an operation on the terminal device by a viewer viewing the video of the virtual space; The program according to any one of claims 1 to 8, wherein the useful information includes information regarding an interaction identified based on the information received by the receiving means.

10. The interaction includes sending a comment to the performer; The information capable of identifying the interaction includes information capable of identifying a comment and a name of a viewer who sent the comment, The program according to claim 9 , wherein the useful information includes information capable of identifying the comment and the name of the viewer who has sent the comment.

11. the interaction includes selecting an item from a plurality of items and reflecting an object of the selected item in the virtual space; The information capable of identifying the interaction includes information capable of identifying a selected item and a name of a viewer who selected the item; 11. The program according to claim 9, wherein the useful information includes information capable of identifying a selected item and a name of a viewer who selected the item.

12. The program according to any one of claims 1 to 11, wherein the useful information includes information that can identify an item displayed on the terminal device.

Citation Information

Patent Citations

  • Information provision system

    JP2012120098A

  • Content distribution system, content distribution method, computer program, content distribution server, and transmission path

    JP2019220932A

  • Distribution server, distribution system, distribution method, and program

    JP2020000644A

  • JPP6665273B