Programs and systems

Abstract

To provide a program and a system capable of improving the understandability of information. [Solution] A computer functions as a display means capable of displaying a virtual space including multiple objects and an audio acquisition means for acquiring the user's audio, and the display means outputs information corresponding to the audio acquired by the audio acquisition means in association with one of the multiple objects.

Description

[Technical Field]

[0001] The present invention relates to a program and a system. [Background technology]

[0002] In a virtual space where users can communicate with other users and send information via avatars, when a user sees another user's avatar, information about that avatar is displayed near the avatar (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-128817 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention provides a program and a system that can improve the understandability of information. [Means for solving the problem]

[0005] According to one aspect of an embodiment of the present disclosure, a computer is caused to function as a display means capable of displaying a virtual space including a plurality of objects and an audio acquisition means for acquiring a user's audio, and the display means outputs information corresponding to the audio acquired by the audio acquisition means in association with one of the plurality of objects. [Effects of the Invention]

[0006] According to the present invention, it is possible to improve the understandability of information. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of an overview of a system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a user terminal according to an embodiment. [Figure 3] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a server according to an embodiment. [Figure 4] FIG. 1 is a block diagram showing an example of a hardware configuration of a game playing terminal according to an embodiment. [Figure 5] FIG. 2 is a block diagram showing an example of a hardware configuration of a distribution terminal according to an embodiment. [Figure 6] FIG. 2 is a block diagram showing an example of a functional configuration of a user terminal, a server, and an HMD set according to an embodiment. [Figure 7] FIG. 2 is a block diagram illustrating an example of a functional configuration of a distribution terminal according to an embodiment. [Figure 8] 10 is a flowchart illustrating a portion of the processing executed in a user terminal and a game playing terminal according to an embodiment. [Figure 9] 1A and 1B are diagrams illustrating an example of a virtual space provided to a player and a field of view image viewed by the player according to an embodiment. [Figure 10] A figure showing an example of a virtual space provided to a user of a user terminal and a field of view image viewed by the user, according to one embodiment. [Figure 11] FIG. 10 is a diagram showing another example of a field of view image visually recognized by a user of a user terminal. [Figure 12] FIG. 10 is a diagram showing yet another example of a field of view image visually recognized by a user of a user terminal. [Figure 13] 10 is a flowchart illustrating a part of a process executed in a game playing terminal according to an embodiment. [Figure 14] 10 is a flowchart illustrating a part of a process executed in a user terminal according to an embodiment. [Figure 15]FIG. 10 is a diagram showing an example of a screen when live commerce images and audio are distributed by multiple avatars. [Figure 16] FIG. 10 is a diagram showing an example of a reflection specification table for specifying a reflection position and a reflection content according to an extracted keyword. [Figure 17] FIG. 10 is a diagram showing an example of a display in which an object in a virtual space is updated by receiving a reaction from a viewer. [Figure 18] FIG. 10 is a diagram for explaining a flowchart of a voice-enabled object-related process. DETAILED DESCRIPTION OF THE INVENTION

[0008] The system according to the present disclosure is a system for providing a game to multiple users. The system will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is defined by the claims, and all modifications within the meaning and scope equivalent to the claims are intended to be included in the present invention. In the following description, the same elements in the drawings will be given the same reference numerals, and redundant explanations will not be repeated.

[0009] <System 1 Operation Overview> FIG. 1 is a diagram illustrating an overview of a system 1 according to this embodiment. The system 1 includes multiple user terminals 100 (computers), a server 200, a game play terminal 300 (external device, second external device), and a distribution terminal 400 (external, first external device). Note that FIG. 1 illustrates user terminals 100A-100C, or in other words, three user terminals 100, as an example of the multiple user terminals 100; however, the number of user terminals 100 is not limited to the illustrated example. In this embodiment, when there is no need to distinguish between the user terminals 100A-C, they will be referred to as "user terminals 100." The user terminals 100, game play terminals 300, and distribution terminal 400 are connected to the server 200 via a network 2. The network 2 is comprised of the Internet and various mobile communication systems established by wireless base stations (not shown). Examples of such mobile communication systems include so-called 3G and 4G mobile communication systems, LTE (Long Term Evolution), and wireless networks (such as Wi-Fi (registered trademark)) that can be connected to the Internet via a predetermined access point.

[0010] (Game Overview) In this embodiment, a game that is primarily played by a user of a game play terminal 300 will be described as an example of a game provided by the system 1 (hereinafter referred to as the game). Hereinafter, a user of a game play terminal 300 will be referred to as a "player." As an example, a player (performer) progresses through the game by operating a character that appears in the game. Furthermore, in the game, a user of a user terminal 100 plays a role in supporting the player's progress through the game. Details of the game will be described later. Note that the game provided by the system 1 may be any game in which multiple users participate, and is not limited to this example.

[0011] (Game play terminal 300) The game play terminal 300 progresses the game in response to input operations by the player. The game play terminal 300 also sequentially distributes information generated by the player's game play (hereinafter referred to as game progress information) to the server 200 in real time.

[0012] (Server 200) The server 200 transmits game progress information (second data) received in real time from the game play terminal 300 to the user terminal 100. The server 200 also mediates the transmission and reception of various types of information between the user terminal 100, the game play terminal 300, and the distribution terminal 400.

[0013] (Distribution terminal 400) The distribution terminal 400 generates action instruction data (first data) in response to an input operation by the user of the distribution terminal 400, and distributes the action instruction data to the user terminal 100 via the server 200. The action instruction data is data for playing a video on the user terminal 100, and more specifically, data for causing a character appearing in the video to perform an action.

[0014] In this embodiment, as an example, the user of the distribution terminal 400 is a player of the game. Also, as an example, the video played on the user terminal 100 based on the action instruction data is a video of a character operated by the player in the game moving. "Movement" refers to moving at least a part of the character's body, and includes speaking. Therefore, the action instruction data according to this embodiment includes, for example, voice data for making the character speak and motion data for moving the character's body.

[0015] As an example, the action instruction data is transmitted to the user terminal 100 after the game has ended. Details of the action instruction data and the video played based on the action instruction data will be described later.

[0016] (User terminal 100) The user terminal 100 receives game progress information in real time and generates and displays a game screen using that information. In other words, the user terminal 100 reproduces the game screen of the game being played by the player through real-time rendering. This allows the user of the user terminal 100 to view the same game screen as the player views while playing the game, at approximately the same time as the player.

[0017] Furthermore, the user terminal 100 generates information to assist the player in progressing through the game in response to input operations by the user, and transmits this information to the game play terminal 300 via the server 200. Details of this information will be described later.

[0018] Furthermore, the user terminal 100 receives the action instruction data from the distribution terminal 400, and generates and plays back a moving image (video) using the action instruction data. In other words, the user terminal 100 renders and plays back the action instruction data.

[0019] <System 1 hardware configuration> Fig. 2 is a diagram showing the hardware configuration of a user terminal 100. Fig. 3 is a diagram showing the hardware configuration of a server 200. Fig. 4 is a diagram showing the hardware configuration of a game playing terminal 300. Fig. 5 is a diagram showing the hardware configuration of a distribution terminal 400.

[0020] (User terminal 100) In this embodiment, an example will be described in which the user terminal 100 is realized as a smartphone, but the user terminal 100 is not limited to a smartphone. For example, the user terminal 100 may be realized as a feature phone, a tablet computer, a laptop computer (a so-called notebook computer), or a desktop computer. The user terminal 100 may also be a game device suitable for playing games.

[0021] 2, the user terminal 100 includes a processor 10, a memory 11, a storage 12, a communication interface (IF) 13, an input / output IF 14, a touch screen 15 (display unit), a camera 17, and a distance measurement sensor 18. These components of the user terminal 100 are electrically connected to each other via a communication bus. Note that instead of or in addition to the touch screen 15, the user terminal 100 may include an input / output IF 14 to which a display (display unit) configured separately from the user terminal 100 main body can be connected.

[0022] 2, the user terminal 100 may be configured to be able to communicate with one or more controllers 1020. The controller 1020 establishes communication with the user terminal 100 in accordance with a communication standard such as Bluetooth (registered trademark). The controller 1020 may have one or more buttons, etc., and transmits output values ​​based on user input operations on the buttons, etc., to the user terminal 100. The controller 1020 may also have various sensors, such as an acceleration sensor and an angular velocity sensor, and transmits output values ​​of the various sensors to the user terminal 100.

[0023] Instead of or in addition to the user terminal 100 being provided with the camera 17 and the distance measurement sensor 18, the controller 1020 may have the camera 17 and the distance measurement sensor 18.

[0024] For example, at the start of a game, the user terminal 100 preferably prompts the user using the controller 1020 to input user identification information such as the user's name or login ID via the controller 1020. This enables the user terminal 100 to associate the controller 1020 with the user, and to identify which user the received output value belongs to based on the sender (controller 1020) of the output value.

[0025] When a user terminal 100 communicates with multiple controllers 1020, each user holds a controller 1020, allowing a single user terminal 100 to realize a multiplay mode without communicating with other devices such as the server 200 via the network 2. Alternatively, the user terminals 100 may connect to each other via a wireless standard such as a wireless local area network (LAN) standard (connecting to each other via a wireless standard without going through the server 200) to realize a multiplay mode locally using multiple user terminals 100. When a single user terminal 100 realizes the above-described multiplay mode locally, the user terminal 100 may further include at least some of the various functions of the server 200, which will be described later. When a plurality of user terminals 100 realize the above-described multiplay mode locally, the plurality of user terminals 100 may include the various functions of the server 200, which will be described later, in a distributed manner.

[0026] Even when the above-described multiplay is realized locally, the user terminal 100 may communicate with the server 200. For example, information indicating the results of a game, such as the score or win / loss, may be associated with user identification information and transmitted to the server 200.

[0027] The controller 1020 may also be configured to be detachable from the user terminal 100. In this case, a coupling unit for the controller 1020 may be provided on at least one surface of the housing of the user terminal 100. When the user terminal 100 and the controller 1020 are coupled to each other via the coupling unit by a wire, the user terminal 100 and the controller 1020 transmit and receive signals via the wire.

[0028] 2, the user terminal 100 may accept the attachment of a storage medium 1030 such as an external memory card via the input / output IF 14. This allows the user terminal 100 to read programs and data recorded on the storage medium 1030. The program recorded on the storage medium 1030 is, for example, a game program.

[0029] The user terminal 100 may store in the memory 11 of the user terminal 100 a game program acquired by communicating with an external device such as the server 200, or may store in the memory 11 a game program acquired by reading it from the storage medium 1030.

[0030] As described above, the user terminal 100 includes the communication IF 13, the input / output IF 14, the touch screen 15, the camera 17, and the distance measurement sensor 18 as examples of mechanisms for inputting information to the user terminal 100. Each of the above-described units as input mechanisms can be considered as an operation unit configured to accept input operations by the user.

[0031] For example, when the operation unit is configured with at least one of the camera 17 and the distance measurement sensor 18, the operation unit detects an object 1010 near the user terminal 100 and identifies an input operation from the detection result of the object. As an example, a user's hand, a marker of a predetermined shape, or the like is detected as the object 1010, and the input operation is identified based on the color, shape, movement, or type of the object 1010 obtained as the detection result. More specifically, when the user's hand is detected from an image captured by the camera 17, the user terminal 100 identifies and accepts a gesture (a series of movements of the user's hand) detected based on the captured image as the user's input operation. Note that the captured image may be a still image or a video.

[0032] Alternatively, if the operation unit is configured with the touch screen 15, the user terminal 100 identifies and accepts a user operation performed on the input unit 151 of the touch screen 15 as the user's input operation. Alternatively, if the operation unit is configured with the communication IF 13, the user terminal 100 identifies and accepts a signal (e.g., an output value) transmitted from the controller 1020 as the user's input operation. Alternatively, if the operation unit is configured with the input / output IF 14, the user terminal 100 identifies and accepts a signal output from an input device (not shown) connected to the input / output IF 14 that is different from the controller 1020 as the user's input operation.

[0033] (Server 200) The server 200 may be, for example, a general-purpose computer such as a workstation or a personal computer. The server 200 includes a processor 20, a memory 21, a storage 22, a communication IF 23, and an input / output IF 24. These components of the server 200 are electrically connected to each other by a communication bus.

[0034] (Game play terminal 300) The game play terminal 300 may be, for example, a general-purpose computer such as a personal computer. The game play terminal 300 comprises a processor 30, memory 31, storage 32, a communication IF 33, and an input / output IF 34. These components of the game play terminal 300 are electrically connected to each other via a communication bus.

[0035] As shown in FIG. 4, the game play terminal 300 according to this embodiment is, for example, included in an HMD (Head Mounted Display) set 1000. In other words, it can be expressed that the HMD set 1000 is included in the system 1, and it can also be expressed that the player plays the game using the HMD set 1000. Note that the device on which the player plays the game is not limited to the HMD set 1000. For example, the device may be any device that can allow the player to virtually experience the game. The device may also be realized as a smartphone, feature phone, tablet computer, laptop computer (so-called notebook computer), desktop computer, or the like. The device may also be a gaming device suitable for game play.

[0036] In addition to the game play terminal 300, the HMD set 1000 includes an HMD 500, an HMD sensor 510, a motion sensor 520, a display 530, and a controller 540. The HMD 500 includes a monitor 51, a gaze sensor 52, a first camera 53, a second camera 54, a microphone 55, and a speaker 56. The controller 540 may include the motion sensor 520.

[0037] The HMD 500 is worn on the player's head and can provide the player with a virtual space during operation. More specifically, the HMD 500 displays an image for the right eye and an image for the left eye on a monitor 51. When each eye of the player views the respective image, the player can recognize the image as a three-dimensional image based on the parallax between the two eyes. The HMD 500 can include both a so-called head-mounted display equipped with a monitor and a head-mounted device to which a smartphone or other terminal equipped with a monitor can be attached.

[0038] The monitor 51 is realized, for example, as a non-transparent display device. In one aspect, the monitor 51 is disposed on the main body of the HMD 500 so as to be positioned in front of both eyes of the player. Therefore, when the player views the three-dimensional image displayed on the monitor 51, the player can be immersed in the virtual space. In one aspect, the virtual space includes, for example, images of a background, objects that the player can control, and menus that the player can select. In another aspect, the monitor 51 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor provided in a so-called smartphone or other information display terminal.

[0039] In another aspect, the monitor 51 may be realized as a transmissive display device. In this case, the HMD 500 may be an open type such as glasses, rather than a closed type that covers the player's eyes as shown in FIG. 1 . The transmissive monitor 51 may be temporarily configured as a non-transmissive display device by adjusting its transmittance. The monitor 51 may be configured to simultaneously display a portion of an image constituting a virtual space and the real space. For example, the monitor 51 may display an image of the real space captured by a camera mounted on the HMD 500, or the transmittance of a portion of the monitor may be set high to make the real space visible.

[0040] In one aspect, the monitor 51 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, the monitor 51 may be configured to display an image for the right eye and an image for the left eye as a single image. In this case, the monitor 51 includes a high-speed shutter. The high-speed shutter operates to alternately display an image for the right eye and an image for the left eye so that the image is recognized by only one of the eyes.

[0041] In one aspect, the HMD 500 includes a plurality of light sources (not shown). Each light source is realized, for example, by an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 510 has a position tracking function for detecting the movement of the HMD 500. More specifically, the HMD sensor 510 reads a plurality of infrared rays emitted by the HMD 500 and detects the position and tilt of the HMD 500 in real space.

[0042] In another aspect, the HMD sensor 510 may be realized by a camera. In this case, the HMD sensor 510 can detect the position and tilt of the HMD 500 by performing image analysis processing using image information of the HMD 500 output from the camera.

[0043] In another aspect, the HMD 500 may include a sensor (not shown) as a position detector instead of or in addition to the HMD sensor 510. The HMD 500 may use the sensor to detect the position and tilt of the HMD 500 itself. For example, if the sensor is an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor, the HMD 500 may use any of these sensors instead of the HMD sensor 510 to detect the position and tilt of the HMD 500 itself. As an example, if the sensor included in the HMD 500 is an angular velocity sensor, the angular velocity sensor detects the angular velocity of the HMD 500 around three axes in real space over time. The HMD 500 calculates changes over time in the angles of the HMD 500 around the three axes based on the angular velocities, and further calculates the tilt of the HMD 500 based on the changes over time in the angles.

[0044] The gaze sensor 52 detects the direction in which the player's right and left eyes are looking. In other words, the gaze sensor 52 detects the player's gaze. Detection of the gaze direction is achieved, for example, by a known eye tracking function. The gaze sensor 52 is achieved by a sensor having the eye tracking function. In some aspects, the gaze sensor 52 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 52 may be, for example, a sensor that irradiates the player's right and left eyes with infrared light and detects the rotation angle of each eyeball by receiving light reflected from the cornea and iris of the irradiated light. The gaze sensor 52 can detect the player's gaze based on the detected rotation angles.

[0045] The first camera 53 captures the lower part of the player's face. More specifically, the first camera 53 captures the player's nose, mouth, etc. The second camera 54 captures the player's eyes, eyebrows, etc. The housing of the HMD 500 on the player's side is defined as the inside of the HMD 500, and the housing of the HMD 500 on the opposite side from the player is defined as the outside of the HMD 500. In one aspect, the first camera 53 may be located outside the HMD 500, and the second camera 54 may be located inside the HMD 500. Images generated by the first camera 53 and the second camera 54 are input to the game play terminal 300. In another aspect, the first camera 53 and the second camera 54 may be implemented as a single camera, and the player's face may be captured by this single camera.

[0046] The microphone 55 converts the player's speech into an audio signal (electrical signal) and outputs it to the game play terminal 300. The speaker 56 converts the audio signal into sound and outputs it to the player. In another aspect, the HMD 500 may include earphones instead of the speaker 56.

[0047] The controller 540 is connected to the game play terminal 300 by wire or wirelessly. The controller 540 accepts commands input from the player to the game play terminal 300. In one aspect, the controller 540 is configured to be held by the player. In another aspect, the controller 540 is configured to be attachable to the player's body or clothing. In yet another aspect, the controller 540 may be configured to output at least one of vibration, sound, and light based on a signal transmitted from the game play terminal 300. In yet another aspect, the controller 540 accepts operations from the player to control the position and movement of an object placed in a virtual space.

[0048] In one aspect, the controller 540 includes a plurality of light sources. Each light source is realized, for example, by an LED that emits infrared light. The HMD sensor 510 has a position tracking function. In this case, the HMD sensor 510 reads a plurality of infrared rays emitted by the controller 540 and detects the position and tilt of the controller 540 in real space. In another aspect, the HMD sensor 510 may be realized by a camera. In this case, the HMD sensor 510 can detect the position and tilt of the controller 540 by performing image analysis processing using image information of the controller 540 output from the camera.

[0049] In one aspect, the motion sensor 520 is attached to the player's hand and detects the player's hand movements. For example, the motion sensor 520 detects the rotation speed, number of rotations, etc. of the hand. The detected signal is sent to the game play terminal 300. The motion sensor 520 is provided, for example, in the controller 540. In one aspect, the motion sensor 520 is provided, for example, in the controller 540 configured to be held by the player. In another aspect, for safety in real space, the controller 540 is attached to a device, such as a glove, that is worn on the player's hand and does not easily fly away. In yet another aspect, a sensor not worn by the player may detect the player's hand movements. For example, a signal from a camera capturing an image of the player may be input to the game play terminal 300 as a signal representing the player's movements. The motion sensor 520 and the game play terminal 300 are connected to each other wirelessly, for example. In the case of wireless communication, the communication method is not particularly limited, and for example, Bluetooth or other known communication methods may be used.

[0050] The display 530 displays an image similar to the image displayed on the monitor 51. This allows users other than the player wearing the HMD 500 to view the same image as the player. The image displayed on the display 530 does not need to be a three-dimensional image, and may be an image for the right eye or an image for the left eye. Examples of the display 530 include a liquid crystal display and an organic EL monitor.

[0051] The game play terminal 300 controls the character controlled by the player to move and progress through the game based on various information acquired from the various components of the HMD 500, the controller 540, and the motion sensor 520. Here, "movement" includes moving various body parts, changing posture, changing facial expressions, moving, speaking, touching or moving objects placed in the virtual space, and using weapons, tools, etc. held by the character. That is, in this game, when the player moves various body parts, the character also moves various body parts in the same way as the player. Also, in this game, the character speaks what the player speaks. In other words, in this game, the character is an avatar object that acts as the player's alter ego. As an example, at least some of the character's movements may be performed by input from the player to the controller 540.

[0052] In this embodiment, the motion sensors 520 are attached to the player's hands, both feet, waist, and head, as an example. The motion sensors 520 attached to the player's hands may be provided in the controller 540, as described above. The motion sensor 520 attached to the player's head may be provided in the HMD 500. The motion sensors 520 may also be attached to the user's elbows and knees. By increasing the number of motion sensors 520 attached to the player, the player's movements can be more accurately reflected on the character. Furthermore, instead of attaching the motion sensors 520 to various parts of the body, the player may wear a suit to which one or more motion sensors 520 are attached. In other words, the method of motion capture is not limited to the example using the motion sensors 520.

[0053] (Distribution terminal 400) The distribution terminal 400 may be a mobile terminal such as a smartphone, a PDA (Personal Digital Assistant), or a tablet computer, or may be a so-called stationary terminal such as a desktop personal computer.

[0054] 5, the distribution terminal 400 includes a processor 40, a memory 41, a storage 42, a communication IF 43, an input / output IF 44, and a touch screen 45. Instead of or in addition to the touch screen 45, the distribution terminal 400 may include the input / output IF 44 to which a display (display unit) configured separately from the distribution terminal 400 main body can be connected.

[0055] The controller 1021 may have a physical input mechanism such as one or more buttons, levers, sticks, wheels, etc. The controller 1021 transmits to the streaming terminal 400 an output value based on an input operation input to the input mechanism by an operator of the streaming terminal 400 (a player in this embodiment). The controller 1021 may also have various sensors such as an acceleration sensor and an angular velocity sensor, and may transmit output values ​​of the various sensors to the streaming terminal 400. The above-mentioned output values ​​are received by the streaming terminal 400 via the communication IF 43.

[0056] The broadcasting terminal 400 may include a camera and a distance measurement sensor (both not shown). Instead of or in addition to the broadcasting terminal 400 including the camera and the distance measurement sensor, the controller 1021 may include the camera and the distance measurement sensor.

[0057] As described above, the streaming terminal 400 includes the communication IF 43, the input / output IF 44, and the touch screen 45 as an example of a mechanism for inputting information to the streaming terminal 400. Each of the above-described units serving as an input mechanism can be considered as an operation unit configured to accept input operations by the user.

[0058] If the operation unit is configured with touch screen 45, distribution terminal 400 identifies and accepts a user operation performed on input unit 451 of touch screen 45 as the user's input operation. Alternatively, if the operation unit is configured with communication IF 43, distribution terminal 400 identifies and accepts a signal (e.g., an output value) transmitted from controller 1021 as the user's input operation. Alternatively, if the operation unit is configured with input / output IF 44, distribution terminal 400 identifies and accepts a signal output from an input device (not shown) connected to input / output IF 44 as the user's input operation.

[0059] <Hardware components of each device> Processors 10, 20, 30, and 40 control the overall operation of user terminal 100, server 200, game play terminal 300, and distribution terminal 400, respectively. Processors 10, 20, 30, and 40 include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). Processors 10, 20, 30, and 40 read programs from storages 12, 22, 32, and 42, respectively, which will be described later. Processors 10, 20, 30, and 40 then load the read programs into memories 11, 21, 31, and 41, which will be described later, respectively. Processors 10, 20, and 30 execute the loaded programs.

[0060] Memories 11, 21, 31, and 41 are main storage devices. Memories 11, 21, 31, and 41 are composed of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). Memory 11 provides a working area for processor 10 by temporarily storing programs and various data that processor 10 reads from storage 12 (described later). Memory 11 also temporarily stores various data generated while processor 10 is operating according to a program. Memory 21 provides a working area for processor 20 by temporarily storing various programs and data that processor 20 reads from storage 22 (described later). Memory 21 also temporarily stores various data generated while processor 20 is operating according to a program. Memory 31 provides a working area for processor 30 by temporarily storing various programs and data that processor 30 reads from storage 32 (described later). Memory 31 also temporarily stores various data generated while processor 30 is operating according to a program. The memory 41 temporarily stores programs and various data that the processor 40 reads from the storage 42 (described later), thereby providing a working area for the processor 40. The memory 41 also temporarily stores various data that the processor 40 generates while operating according to the programs.

[0061] In this embodiment, the program executed by processors 10 and 30 may be a game program for the game. In this embodiment, the program executed by processor 40 may be a distribution program for distributing action instruction data. Processor 10 may also execute a viewing program for playing video.

[0062] In this embodiment, the program executed by processor 20 may be at least one of the above-mentioned game program, distribution program, and viewing program. Processor 20 executes at least one of the game program, distribution program, and viewing program in response to a request from at least one of user terminal 100, game play terminal 300, and distribution terminal 400. The distribution program and the viewing program may be executed in parallel.

[0063] That is, the game program may be a program that realizes a game through cooperation between the user terminal 100, the server 200, and the game play terminal 300. The distribution program may be a program that realizes distribution of action instruction data through cooperation between the server 200 and the distribution terminal 400. The viewing program may be a program that realizes playback of a video through cooperation between the user terminal 100 and the server 200.

[0064] Storages 12, 22, 32, and 42 are auxiliary storage devices. Storages 12, 22, 32, and 42 are configured with storage devices such as flash memory or HDDs (Hard Disk Drives). Storages 12 and 32 store, for example, various data related to games. Storage 42 stores various data related to the distribution of action instruction data. Storage 12 stores various data related to the playback of moving images. Storage 22 may store at least some of the various data related to games, the distribution of action instruction data, and the playback of moving images.

[0065] Communication IFs 13, 23, 33, and 43 respectively control the transmission and reception of various types of data between user terminal 100, server 200, game play terminal 300, and distribution terminal 400. Communication IFs 13, 23, 33, and 43 control, for example, communication via a wireless LAN (Local Area Network), internet communication via a wired LAN, wireless LAN, or mobile phone network, and communication using short-range wireless communication, etc.

[0066] The input / output IFs 14, 24, 34, and 44 are interfaces through which the user terminal 100, the server 200, the game play terminal 300, and the distribution terminal 400 receive data input and output data, respectively. The input / output IFs 14, 24, 34, and 44 may input and output data via a USB (Universal Serial Bus) or the like. The input / output IFs 14, 24, 34, and 44 may include physical buttons, a camera, a microphone, a speaker, a mouse, a keyboard, a display, a stick, a lever, and the like. The input / output IFs 14, 24, 34, and 44 may also include a connection section for transmitting and receiving data to and from peripheral devices.

[0067] The touch screen 15 is an electronic component that combines an input unit 151 and a display unit 152 (display). The touch screen 45 is an electronic component that combines an input unit 451 and a display unit 452. The input units 151 and 451 are, for example, touch-sensitive devices, and are configured, for example, by touchpads. The display units 152 and 452 are, for example, liquid crystal displays, organic EL (Electro-Luminescence) displays, or the like.

[0068] The input units 151 and 451 have a function of detecting the position where a user operation (mainly a physical contact operation such as a touch operation, a slide operation, a swipe operation, or a tap operation) is input on the input surface, and transmitting information indicating the position as an input signal. The input units 151 and 451 may be provided with a touch sensing unit (not shown). The touch sensing unit may be of any type, such as a capacitive type or a resistive type.

[0069] Although not shown, the user terminal 100 and the streaming terminal 400 may each include one or more sensors for identifying the holding posture of the user terminal 100 and the streaming terminal 400. The sensors may be, for example, an acceleration sensor or an angular velocity sensor.

[0070] If the user terminal 100 and the streaming terminal 400 are equipped with sensors, the processors 10 and 40 can identify the holding orientation of the user terminal 100 and the streaming terminal 400, respectively, from the sensor output and perform processing according to the holding orientation. For example, when the user terminal 100 and the streaming terminal 400 are held in portrait orientation, the processors 10 and 40 can set the display units 152 and 452 to display a vertically elongated image in portrait screen mode. On the other hand, when the user terminal 100 and the streaming terminal 400 are held in landscape orientation, the processors 10 and 40 can set the display units 152 and 452 to display a horizontally elongated image in landscape screen mode. In this way, the processors 10 and 40 can switch between portrait screen mode and landscape screen mode depending on the holding orientation of the user terminal 100 and the streaming terminal 400, respectively.

[0071] <Functional configuration of System 1> 6 is a block diagram showing the functional configuration of the user terminal 100, the server 200, and the HMD set 1000 included in the system 1. FIG. 7 is a block diagram showing the functional configuration of the distribution terminal 400 shown in FIG.

[0072] The user terminal 100 functions as an input device that accepts user input operations and as an output device that outputs game images and sounds. The user terminal 100 functions as a control unit 110 and a storage unit 120 through cooperation of a processor 10, a memory 11, a storage 12, a communication IF 13, an input / output IF 14, a touch screen 15, and the like.

[0073] The server 200 has a function of mediating the transmission and reception of various types of information among the user terminal 100, the HMD set 1000, and the distribution terminal 400. The server 200 functions as a control unit 210 and a storage unit 220 through cooperation of the processor 20, the memory 21, the storage 22, the communication IF 23, the input / output IF 24, and the like.

[0074] The HMD set 1000 (game playing terminal 300) functions as an input device that accepts input operations from the player, as an output device that outputs game images and sounds, and as a function that transmits game progress information to the user terminal 100 in real time via the server 200. The HMD set 1000 functions as a control unit 310 and a memory unit 320 through cooperation of the processor 30, memory 31, storage 32, communication IF 33, input / output IF 34 of the game playing terminal 300, as well as the HMD 500, HMD sensor 510, motion sensor 520, and controller 540.

[0075] The distribution terminal 400 has a function of generating action instruction data and transmitting the action instruction data to the user terminal 100 via the server 200. The distribution terminal 400 functions as a control unit 410 and a storage unit 420 through cooperation of the processor 40, memory 41, storage 42, communication IF 43, input / output IF 44, touch screen 45, etc.

[0076] (Data stored in the memory unit of each device) The storage unit 120 stores a game program 131 (program), game information 132, and user information 133. The storage unit 220 stores a game program 231, game information 232, user information 233, and a user list 234. The storage unit 320 stores a game program 331, game information 332, and user information 333. The storage unit 420 stores a user list 421, a motion list 422, and a distribution program 423 (program, second program).

[0077] The game programs 131, 231, and 331 are game programs executed by the user terminal 100, the server 200, and the HMD set 1000, respectively. The game is realized by the cooperative operation of each device based on the game programs 131, 231, and 331. The game programs 131 and 331 may be stored in the storage unit 220 and downloaded to the user terminal 100 and the HMD set 1000, respectively. In this embodiment, the user terminal 100 renders data received from the distribution terminal 400 based on the game program 131 and plays videos. In other words, the game program 131 is also a program for playing videos using video instruction data distributed from the distribution terminal 400. The program for playing the videos may be different from the game program 131. In this case, the storage unit 120 stores the program for playing the videos separately from the game program 131.

[0078] The game information 132, 232, and 332 are data that are referenced by the user terminal 100, the server 200, and the HMD set 1000 when executing a game program, respectively. The user information 133, 233, and 333 are data related to the account of the user of the user terminal 100. The game information 232 is the game information 132 of each user terminal 100 and the game information 332 of the HMD set 1000. The user information 233 is the user information 133 of each user terminal 100 and the user information of the player included in the user information 333. The user information 333 is the user information 133 of each user terminal 100 and the user information of the player.

[0079] The user list 234 and the user list 421 are lists of users who have participated in the game. The user list 234 and the user list 421 may include a list of users who participated in the player's most recent game play, as well as a list of users who participated in each game play prior to that game play. The motion list 422 is a list of multiple motion data created in advance. The motion list 422 is, for example, a list in which information identifying each motion (for example, a motion name) is associated with each piece of motion data. The distribution program 423 is a program for realizing the distribution of action instruction data for playing a video on the user terminal 100 to the user terminal 100.

[0080] (Functional configuration of server 200) The control unit 210 executes the game program 231 stored in the storage unit 220 to comprehensively control the server 200. For example, the control unit 210 mediates the transmission and reception of various types of information among the user terminal 100, the HMD set 1000, and the distribution terminal 400.

[0081] The control unit 210 functions as a communication intermediation unit 211, a log generation unit 212, and a list generation unit 213 in accordance with the description of the game program 231. The control unit 210 can also function as other functional blocks (not shown) to mediate the sending and receiving of various information related to game play and distribution of action instruction data, and to support the progress of the game.

[0082] The communication intermediation unit 211 mediates the transmission and reception of various types of information among the user terminal 100, the HMD set 1000, and the distribution terminal 400. For example, the communication intermediation unit 211 transmits game progress information received from the HMD set 1000 to the user terminal 100. The game progress information includes data indicating the movement of a character controlled by a player, the character's parameters, items and weapons held by the character, enemy characters, and other information. The server 200 transmits the game progress information to the user terminals 100 of all users participating in the game. In other words, the server 200 transmits common game progress information to the user terminals 100 of all users participating in the game. As a result, the game progresses in the same manner as in the HMD set 1000 in each of the user terminals 100 of all users participating in the game.

[0083] Furthermore, for example, the communication intermediation unit 211 transmits information received from one of the user terminals 100 to assist the player in progressing through the game to the other user terminals 100 and the HMD set 1000. As an example, the information may be item information indicating an item that allows the player to progress through the game advantageously and is provided to the player (character). The item information includes information indicating the user who provided the item (user name, user ID, etc.). The communication intermediation unit 211 may also mediate the distribution of action instruction data from the distribution terminal 400 to the user terminal 100.

[0084] The log generation unit 212 generates a game progress log based on the game progress information received from the HMD set 1000. The list generation unit 213 generates a user list 234 after game play ends. As will be described in detail later, each user in the user list 234 is associated with a tag indicating the content of the support provided by that user to the player. The list generation unit 213 generates a tag based on the game progress log generated by the log generation unit 212 and associates it with the corresponding user. Note that the list generation unit 213 may associate the content of the support provided by each user to the player, which is input by a game operator or the like using a terminal device such as a personal computer, with the corresponding user as a tag. This provides more detailed information about the support provided by each user. Note that when a user participates in a game, the user terminal 100 transmits information identifying the user to the server 200 based on the user's operation. For example, the user terminal 100 transmits the user ID input by the user to the server 200. In other words, the server 200 holds information identifying each user for all users participating in the game. The list generation unit 213 may generate the user list 234 using this information.

[0085] (Functional configuration of HMD set 1000) The control unit 310 executes a game program 331 stored in the storage unit 320 to exercise overall control over the HMD set 1000. For example, the control unit 310 progresses the game in accordance with the game program 331 and the player's operations. Furthermore, while the game is progressing, the control unit 310 communicates with the server 200 as necessary to send and receive information. The control unit 310 may also send and receive information directly to and from the user terminal 100 without going through the server 200.

[0086] The control unit 310 functions as an operation reception unit 311, a display control unit 312, a UI control unit 313, an animation generation unit 314, a game progression unit 315, a virtual space control unit 316, and a reaction processing unit 317 in accordance with the description of the game program 331. The control unit 310 can also function as other functional blocks (not shown) to control characters appearing in the game in accordance with the nature of the game being executed.

[0087] The operation reception unit 311 detects and receives input operations from the player. The operation reception unit 311 receives signals input from the HMD 500, the motion sensor 520, the controller 540, etc., determines what kind of input operation has been performed, and outputs the results to each element of the control unit 310.

[0088] The UI control unit 313 controls user interface (hereinafter referred to as UI) images to be displayed on the monitor 51, the display 530, etc. The UI images are tools that allow the player to input necessary for the progress of the game to the HMD set 1000, or to obtain information output during the progress of the game from the HMD set 1000. The UI images are not limited to these, but include, for example, icons, buttons, lists, menu screens, etc.

[0089] The animation generation unit 314 generates animations showing the motion of various objects based on the control modes of the various objects. For example, the animation generation unit 314 may generate animations that express the movement of an object (e.g., a player's avatar object) as if it were actually there, moving its mouth, changing its facial expression, etc.

[0090] The game progression unit 315 progresses the game based on the game program 331, input operations by the player, and actions of the avatar object in response to the input operations. For example, the game progression unit 315 performs predetermined game processing when the avatar object performs a predetermined action. Furthermore, for example, the game progression unit 315 may receive information representing user operations on the user terminal 100 and perform game processing based on the user operations. Furthermore, the game progression unit 315 generates game progression information in accordance with the progress of the game and transmits it to the server 200. The game progress information is transmitted to the user terminal 100 via the server 200. In this way, the progress of the game on the HMD set 1000 is shared with the user terminal 100. In other words, the progress of the game on the HMD set 1000 and the progress of the game on the user terminal 100 are synchronized.

[0091] The virtual space control unit 316 performs various controls related to the virtual space provided to the player in accordance with the progress of the game. As an example, the virtual space control unit 316 generates various objects and places them in the virtual space. The virtual space control unit 316 also places a virtual camera in the virtual space. The virtual space control unit 316 also operates the various objects placed in the virtual space in accordance with the progress of the game. The virtual space control unit 316 also controls the position and tilt of the virtual camera placed in the virtual space in accordance with the progress of the game.

[0092] The display control unit 312 outputs a game screen reflecting the results of the processing executed by the above-mentioned elements to the monitor 51 and the display 530. The display control unit 312 may display an image based on the field of view from a virtual camera placed in virtual space as a game screen on the monitor 51 and the display 530. The display control unit 312 may also include animation generated by the animation generation unit 314 in the game screen. The display control unit 312 may also draw the above-mentioned UI image controlled by the UI control unit 313 so as to be superimposed on the game screen.

[0093] The reaction processing unit 317 receives feedback from the user of the user terminal 100 regarding the reaction of the player to the game play and outputs this to the player. In this embodiment, for example, the user terminal 100 can create a comment (message) addressed to an avatar object based on an input operation by the user. The reaction processing unit 317 receives comment data of the comment and outputs it. The reaction processing unit 317 may display text data corresponding to the user's comment on the monitor 51 or the display 530, or may output audio data corresponding to the user's comment from a speaker (not shown). In the former case, the reaction processing unit 317 may draw an image corresponding to the text data (i.e., an image including the content of the comment) by superimposing it on the game screen.

[0094] (Functional configuration of user terminal 100) The control unit 110 executes a game program 131 stored in the storage unit 120 to exercise overall control over the user terminal 100. For example, the control unit 110 progresses the game in accordance with the game program 131 and user operations. Furthermore, while the game is progressing, the control unit 110 communicates with the server 200 as necessary to send and receive information. The control unit 110 may also send and receive information directly to the HMD set 1000 without going through the server 200.

[0095] The control unit 110 functions as an operation reception unit 111, a display control unit 112, a UI control unit 113, an animation generation unit 114, a game progression unit 115, a virtual space control unit 116, and a video playback unit 117 in accordance with the description of the game program 131. The control unit 110 can also function as other functional blocks (not shown) for the progression of the game in accordance with the nature of the game being executed.

[0096] The operation reception unit 111 detects and receives an input operation by the user to the input unit 151. The operation reception unit 111 determines what kind of input operation has been performed from the action the user has taken on the touch screen 15 and the console via the other input / output IF 14, and outputs the result to each element of the control unit 110.

[0097] For example, the operation reception unit 111 receives an input operation on the input unit 151, detects the coordinates of the input position of the input operation, and identifies the type of the input operation. The operation reception unit 111 identifies, for example, a touch operation, a slide operation, a swipe operation, a tap operation, etc. Furthermore, when a continuously detected input is interrupted, the operation reception unit 111 detects that the touch input from the touch screen 15 has been released.

[0098] The UI control unit 113 controls UI images to be displayed on the display unit 152 to construct a UI in response to at least one of a user's input operation and received game progress information. The UI images are tools that allow the user to input necessary information for the progress of the game to the user terminal 100, or tools that allow the user to obtain information output during the progress of the game from the user terminal 100. The UI images are not limited to these, but include, for example, icons, buttons, lists, menu screens, etc.

[0099] The animation generation unit 114 generates animations showing the motions of various objects based on the control modes of the various objects.

[0100] The game progression unit 115 progresses the game based on the game program 131, the received game progress information, and input operations by the user. When a predetermined game process is performed in response to an input operation by the user, the game progression unit 115 transmits information related to the game process to the HMD set 1000 via the server 200. This causes the predetermined game process to be shared in the HMD set 1000. In other words, the progress of the game in the HMD set 1000 and the progress of the game in the user terminal 100 are synchronized. The predetermined game process is, for example, a process of providing an item to an avatar object, and in this example, the information related to the game process is the item information described above.

[0101] The virtual space control unit 116 performs various controls related to the virtual space provided to the user in accordance with the progress of the game. As an example, the virtual space control unit 116 generates various objects and places them in the virtual space. The virtual space control unit 116 also places a virtual camera in the virtual space. The virtual space control unit 116 also operates the various objects placed in the virtual space in accordance with the progress of the game, specifically, the received game progress information. The virtual space control unit 316 also controls the position and tilt of the virtual camera placed in the virtual space in accordance with the progress of the game, specifically, the received game progress information.

[0102] The display control unit 112 outputs to the display unit 152 a game screen that reflects the results of the processing executed by each of the above-mentioned elements. The display control unit 112 may display, as a game screen, an image based on a field of view from a virtual camera placed in a virtual space provided to the user on the display unit 152. The display control unit 112 may also include animation generated by the animation generation unit 114 in the game screen. The display control unit 112 may also draw the above-mentioned UI image controlled by the UI control unit 113, superimposed on the game screen. In any case, the game screen displayed on the display unit 152 is the same game screen as the game screen displayed on other user terminals 100 and the HMD set 1000.

[0103] The video playback unit 117 analyzes (renders) the action instruction data received from the distribution terminal 400 and plays back the video.

[0104] (Functional configuration of distribution terminal 400) The control unit 410 executes a program (not shown) stored in the storage unit 420 to exercise overall control over the distribution terminal 400. For example, the control unit 410 generates action instruction data in accordance with the program and operations by the user (player in this embodiment) of the distribution terminal 400, and distributes the data to the user terminal 100. The control unit 410 also communicates with the server 200 as necessary to send and receive information. The control unit 410 may also send and receive information directly to and from the user terminal 100 without going through the server 200.

[0105] According to the description of the program, the control unit 410 functions as a communication control unit 411, a display control unit 412, an operation reception unit 413, a voice reception unit 414, a motion identification unit 415, and an action instruction data generation unit 416. The control unit 410 can also function as other functional blocks (not shown) for generating and distributing action instruction data.

[0106] The communication control unit 411 controls transmission and reception of information to and from the server 200 or the user terminal 100 via the server 200. As an example, the communication control unit 411 receives a user list 421 from the server 200. In addition, as an example, the communication control unit 411 transmits operation instruction data to the user terminal 100.

[0107] The display control unit 412 outputs various screens reflecting the results of the processing performed by each element to the display unit 452. As an example, the display control unit 412 displays a screen including the received user list 234. Also, as an example, the display control unit 412 displays a screen including a motion list 422 for allowing the player to select motion data for moving the avatar object, which is included in the action instruction data to be distributed.

[0108] The operation reception unit 413 detects and receives input operations by the player via the input unit 151. The operation reception unit 111 determines what kind of input operation has been performed from the action performed by the player on the touch screen 45 and the console via other input / output IF 44, and outputs the result to each element of the control unit 410.

[0109] For example, the operation acceptance unit 413 accepts an input operation on the input unit 451, detects the coordinates of the input position of the input operation, and identifies the type of the input operation. The operation acceptance unit 413 identifies, for example, a touch operation, a slide operation, a swipe operation, a tap operation, etc. Furthermore, when a continuously detected input is interrupted, the operation acceptance unit 413 detects that the touch input from the touch screen 45 has been released.

[0110] The voice receiving unit 414 receives voice generated around the distribution terminal 400 and generates voice data of the voice. As an example, the voice receiving unit 414 receives voice uttered by a player and generates voice data of the voice.

[0111] The motion specifying unit 415 specifies the motion data selected by the player from the motion list 422 in response to the input operation by the player.

[0112] The action instruction data generating unit 416 generates action instruction data. As an example, the action instruction data generating unit 416 generates action instruction data including the generated voice data and the identified motion data.

[0113] Note that the functions of the HMD set 1000, the server 200, and the user terminal 100 shown in FIG. 6 and the functions of the streaming terminal 400 shown in FIG. 7 are merely examples. Each of the HMD set 1000, the server 200, the user terminal 100, and the streaming terminal 400 may have at least some of the functions of the other devices. Furthermore, a device other than the HMD set 1000, the server 200, the user terminal 100, and the streaming terminal 400 may be a component of the system 1, and the other device may execute some of the processing in the system 1. That is, in this embodiment, the computer that executes the game program may be any of the HMD set 1000, the server 200, the user terminal 100, the streaming terminal 400, and other devices, or may be realized by a combination of multiple devices.

[0114] <Virtual space control processing> FIG. 8 is a flowchart showing an example of the flow of control processing for a virtual space provided to a player and a virtual space provided to a user of user terminal 100. FIG. 9 is a diagram showing a virtual space 600A provided to a player and a field of view image visually recognized by the player according to an embodiment. FIG. 10 is a diagram showing a virtual space 600B provided to a user of user terminal 100 and a field of view image visually recognized by the user according to an embodiment. Hereinafter, when there is no need to distinguish between virtual spaces 600A and 600B, they will be referred to as "virtual space 600."

[0115] In step S1, processor 30, functioning as virtual space control unit 316, defines virtual space 600A shown in FIG. 9. Processor 30 defines virtual space 600A using virtual space data (not shown). The virtual space data may be stored in game play terminal 300, may be generated by processor 30 based on game program 331, or may be acquired by processor 30 from an external device such as server 200.

[0116] As an example, virtual space 600 has a spherical structure that covers the entire 360-degree area from a point defined as the center. 9 and 10 illustrate only the upper half of the celestial sphere in virtual space 600 in order to avoid complicating the explanation.

[0117] In step S2, processor 30, functioning as virtual space control unit 316, places avatar object 610 (character) in virtual space 600A. Avatar object 610 is an avatar object associated with a player, and moves in accordance with an input operation by the player.

[0118] In step S3, processor 30, functioning as virtual space control unit 316, places other objects in virtual space 600A. In the example of Fig. 9, processor 30 places objects 631 to 634. The other objects may include, for example, character objects (so-called non-player characters, NPCs) that operate according to game program 331, controllable objects such as virtual hands, and objects that resemble animals, plants, artificial objects, natural objects, etc. that are placed as the game progresses.

[0119] In step S4, processor 30, functioning as virtual space control unit 316, places virtual camera 620A in virtual space 600A. Processor 30 places virtual camera 620A at the position of the head of avatar object 610, for example.

[0120] In step S5, processor 30 displays field of view image 650 on monitor 51 and display 530. Processor 30 defines field of view area 640A, which is the field of view from virtual camera 620A in virtual space 600A, according to the initial position and tilt of virtual camera 620A. Processor 30 then defines field of view image 650 corresponding to field of view area 640A. Processor 30 outputs field of view image 650 to monitor 51 and display 530, thereby displaying field of view image 650 on HMD 500 and display 530.

[0121] In the example of FIG. 9, as shown in FIG. 9(A), a portion of object 634 is included in field of view area 640A, and therefore field of view image 650 includes a portion of object 634 as shown in FIG. 9(B).

[0122] In step S6, processor 30 transmits initial placement information to user terminal 100 via server 200. The initial placement information is information indicating the initial placement positions of various objects in virtual space 600A. In the example of FIG. 9, the initial placement information includes information on the initial placement positions of avatar object 610 and objects 631 to 634. The initial placement information can also be expressed as one piece of game progress information.

[0123] In step S7, processor 30, functioning as virtual space control unit 316, controls virtual camera 620A in accordance with the movement of HMD 500. Specifically, processor 30 controls the orientation and tilt of virtual camera 620A in accordance with the movement of HMD 500, i.e., the pose of the player's head. As will be described later, when the player moves his / her head (changes the pose of his / her head), processor 30 moves the head of avatar object 610 in accordance with this movement. Processor 30 controls the orientation and tilt of virtual camera 620A, for example, so that the line of sight of avatar object 610 and the line of sight of virtual camera 620A coincide with each other. In step S8, processor 30 updates field of view image 650 in accordance with the change in the orientation and tilt of virtual camera 620A.

[0124] In step S9, processor 30, functioning as virtual space control unit 316, moves avatar object 610 in accordance with the movement of the player. As an example, processor 30 moves avatar object 610 in virtual space 600A in accordance with the player's movement in real space. Furthermore, processor 30 moves the head of avatar object 610 in virtual space 600A in accordance with the player's movement of his / her head in real space.

[0125] In step S10, processor 30, functioning as virtual space control unit 316, moves virtual camera 620A to follow avatar object 610. That is, virtual camera 620A is always located at the head position of avatar object 610, even if avatar object 610 moves.

[0126] Processor 30 updates field of view image 650 in accordance with the movement of virtual camera 620A. That is, processor 30 updates field of view 640A in accordance with the pose of the player's head and the position of virtual camera 620A in virtual space 600A. As a result, field of view image 650 is updated.

[0127] In step S11, processor 30 transmits action instruction data for avatar object 610 to user terminal 100 via server 200. The action instruction data here includes at least any of motion data capturing the player's actions during the virtual experience (e.g., while playing a game), audio data of the player's voice, and operation data indicating the content of an input operation on controller 540. When the player is playing a game, the action instruction data is transmitted to user terminal 100 as, for example, game progress information.

[0128] The processing of steps S7 to S11 is continuously and repeatedly executed while the player is playing the game.

[0129] In step S21, the processor 10 of the user terminal 100 of the user 3 defines the virtual space 600B shown in FIG. 10 as the virtual space control unit 116. The processor 10 defines the virtual space 600B using virtual space data (not shown). The virtual space data may be stored in the user terminal 100, may be generated by the processor 10 based on the game program 131, or may be acquired by the processor 10 from an external device such as the server 200.

[0130] In step S22, processor 10 receives the initial placement information. In step S23, processor 10, functioning as virtual space control unit 116, places various objects in virtual space 600B in accordance with the initial placement information. In the example of Fig. 10, the various objects are avatar object 610 and objects 631 to 634.

[0131] In step S24, the processor 10, functioning as the virtual space control unit 116, places a virtual camera 620B in the virtual space 600B. As an example, the processor 10 places the virtual camera 620B at the position shown in FIG.

[0132] In step S25, processor 10 displays field of view image 660 on display unit 152. Processor 10 defines field of view area 640B, which is the field of view from virtual camera 620B in virtual space 600B, according to the initial position and tilt of virtual camera 620B. Processor 10 then defines field of view image 660 corresponding to field of view area 640B. Processor 10 outputs field of view image 660 to display unit 152, thereby displaying field of view image 660 on display unit 152.

[0133] In the example of Figure 10, as shown in Figure 10(A), avatar object 610 and object 631 are included in field of view area 640B, so field of view image 660 includes avatar object 610 and object 631 as shown in Figure 10(B).

[0134] In step S26, processor 10 receives the action instruction data. In step S27, processor 10, as virtual space control unit 116, moves avatar object 610 in virtual space 600B in accordance with the action instruction data. In other words, processor 10 plays back a video of avatar object 610 moving by real-time rendering.

[0135] In step S28, the processor 10, functioning as the virtual space control unit 116, controls the virtual camera 620B in response to a user operation received by the operation receiving unit 111. In step S29, the processor 10 updates the field of view image 660 in response to changes in the position of the virtual camera 620B in the virtual space 600B and the orientation and tilt of the virtual camera 620B. Note that in step S28, the processor 10 may automatically control the virtual camera 620B in response to a movement of the avatar object 610, such as a movement or change in orientation of the avatar object 610. For example, the processor 10 may automatically move the virtual camera 620B or change its orientation and tilt so that the avatar object 610 is always photographed from the front. As another example, the processor 10 may automatically move the virtual camera 620B or change its orientation and tilt so that the avatar object 610 is always photographed from behind in response to the movement of the avatar object 610.

[0136] In this way, in the virtual space 600A, the avatar object 610 moves in accordance with the player's movements. Action instruction data indicating this movement is transmitted to the user terminal 100. In the virtual space 600B, the avatar object 610 moves in accordance with the received action instruction data. As a result, the avatar object 610 performs the same movement in the virtual space 600A and the virtual space 600B. In other words, the user 3 can visually recognize the movement of the avatar object 610 in accordance with the player's movement using the user terminal 100.

[0137] <Game Overview> 11 is a diagram showing another example of a field of view image displayed on the user terminal 100. Specifically, it is a diagram showing an example of a game screen of the game (this game) executed by the system 1 and played by the player.

[0138] As an example, this game is a game in which an avatar object 610 that controls a weapon such as a gun or a knife and a plurality of enemy objects 671 that are NPCs appear in a virtual space 600, and the avatar object 610 fights against the enemy objects 671. Various game parameters such as the stamina of the avatar object 610, the number of usable magazines, the number of bullets remaining in the gun, and the number of remaining enemy objects 671 are updated as the game progresses.

[0139] The game includes multiple stages, and the player can clear each stage by fulfilling a predetermined achievement condition associated with the stage. The predetermined achievement condition may include, for example, defeating all enemy objects 671 that appear, defeating a boss object among the enemy objects 671 that appear, acquiring a predetermined item, or reaching a predetermined location. The achievement condition is defined in the game program 131. In the game, the player clears a stage when the achievement condition is fulfilled in accordance with the content of the game; in other words, the victory of the avatar object 610 over the enemy object 671 (the victory or defeat between the avatar object 610 and the enemy object 671) is determined. In contrast, for example, if the game executed by the system 1 is a racing game, the ranking of the avatar object 610 is determined when the condition of reaching the goal is fulfilled.

[0140] In this game, in order to share a virtual space between the HMD set 1000 and multiple user terminals 100, game progress information is live-streamed to multiple user terminals 100 at predetermined time intervals. As a result, a field of view image of the field of view area defined by the virtual camera 620B corresponding to the user terminal 100 is displayed on the touch screen 15 of the user terminal 100 while the user is watching the game. In addition, parameter images representing the avatar object 610's vitality, the number of available magazines, the number of remaining bullets in the gun, the number of remaining enemy objects 671, etc. are superimposed and displayed in the upper right and upper left sections of the field of view image. This field of view image can also be referred to as a game screen.

[0141] As described above, the game progress information includes motion data capturing the player's actions, audio data of the voices uttered by the player, and operation data indicating the content of input operations on controller 540. These pieces of data are, in other words, information for specifying the position, posture, orientation, etc. of avatar object 610, information for specifying the position, posture, orientation, etc. of enemy object 671, and information for specifying the positions, etc. of other objects (e.g., obstacle objects 672, 673). Processor 10 analyzes (renders) the game progress information to specify the position, posture, orientation, etc. of each object.

[0142] The game information 132 includes data on various objects such as the avatar object 610, the enemy object 671, and the obstacle objects 672 and 673. The processor 10 uses this data and the analysis results of the game progress information to update the position, posture, orientation, and so on of each object. As a result, the game progresses, and each object in the virtual space 600B moves in the same way as each object in the virtual space 600A. Specifically, in the virtual space 600B, each object including the avatar object 610 moves based on the game progress information, regardless of whether or not a user operates the user terminal 100.

[0143] As an example, UI images 701 and 702 are displayed superimposed on the field of view image on the touch screen 15 of the user terminal 100. The UI image 701 is a UI image that accepts an operation for displaying, on the touch screen 15, a UI image 711 that accepts, from the user 3, an item insertion operation for supporting the avatar object 610. The UI image 702 is a UI image that accepts an operation for displaying, on the touch screen 15, a UI image (described later) that accepts, from the user 3, an operation for inputting and transmitting a comment for the avatar object 610 (in other words, the player 4). The operation accepted by the UI images 701 and 702 may be, for example, an operation of tapping the UI images 701 and 702.

[0144] When UI image 701 is tapped, UI image 711 is displayed superimposed on the field of view image. UI image 711 includes, for example, UI image 711A with a magazine icon, UI image 711B with a first aid kit icon, UI image 711C with a traffic cone icon, and UI image 711D with a barricade icon. An item insertion operation corresponds to, for example, tapping any of the UI images.

[0145] As an example, when UI image 711A is tapped, the number of remaining bullets in the gun used by the avatar object 610 increases. When UI image 711B is tapped, the stamina of the avatar object 610 is restored. When UI images 711C and 711D are tapped, obstacle objects 672 and 673 that hinder the movement of the enemy object 671 are placed in the virtual space. One of the obstacle objects 672 and 673 may hinder the movement of the enemy object 671 more than the other.

[0146] The processor 10 transmits item insertion information indicating that an item insertion operation has been performed to the server 200. The item insertion information includes at least information for identifying the type of item specified by the item insertion operation. The item insertion information may also include other information related to the item, such as information indicating the position where the item is to be placed. The item insertion information is transmitted to other user terminals 100 and the HMD set 1000 via the server 200.

[0147] 12 is a diagram showing another example of the field of view image displayed on the user terminal 100. Specifically, it is a diagram showing an example of the game screen of the game, and is a diagram for explaining communication between the player and the user terminal 100 during game play.

[0148] 12(A), the user terminal 100 causes the avatar object 610 to make an utterance 691. Specifically, the user terminal 100 causes the avatar object 610 to make an utterance 691 in accordance with the voice data included in the game progress information. The content of the utterance 691 is "I'm out of bullets!" uttered by the player 4. In other words, the content of the utterance 691 informs each user that the magazine is empty and the number of bullets loaded in the gun is one, and therefore, the user is likely to lose the means to attack the enemy object 671.

[0149] 12(A) uses a speech bubble to visually indicate the speech of avatar object 610, but in reality, audio is output from the speaker of user terminal 100. Along with the audio output, the speech bubble shown in FIG. 12(A) (i.e., a speech bubble containing the text of the audio content) may be displayed in the field of view image. This also applies to utterance 692, which will be described later.

[0150] When a tap operation on the UI image 702 is received, the user terminal 100 displays UI images 705 and 706 (message UIs) superimposed on the field of view image, as shown in Fig. 12(B). The UI image 705 is a UI image that displays a comment for the avatar object 610 (in other words, the player). The UI image 706 is a UI image that receives a comment sending operation from the user 3 to send the input comment.

[0151] For example, when the user terminal 100 receives a tap operation on the UI image 705, the user terminal 100 displays a UI image resembling a keyboard (not shown, hereinafter simply referred to as "keyboard") on the touch screen 15. The user terminal 100 displays text corresponding to the user's input operation on the keyboard on the UI image 705. In the example of FIG. 12(B), the text "I'll send you a magazine" is displayed on the UI image 705.

[0152] After inputting text, for example, when the user terminal 100 receives a tap operation on the UI image 706, the user terminal 100 transmits comment information including information indicating the input content (content of the text) and information indicating the user to the server 200. The comment information is transmitted to other user terminals 100 and the HMD set 1000 via the server 200.

[0153] UI image 703A is a UI image showing the username of the user who sent the comment, and UI image 704A is a UI image showing the content of the comment sent by that user. In the example of FIG. 12(B), a user with the username "BBBBB" sent comment information saying "Danger!" using his / her user terminal 100, and as a result, UI image 703A and UI image 704A are displayed. UI image 703A and UI image 704A are displayed on the touch screens 15 of all user terminals 100 participating in the game and on the monitor 51 of the HMD 500. Note that UI images 703A and 704A may be a single UI image. In other words, a single UI image may include the username and the content of the comment.

[0154] In the example of Fig. 12(C), a user with the user name "AAAAA", who is a user of the user terminal 100 shown in Fig. 12, has input and sent a comment as described above, and as a result, UI images 703B and 704B are displayed on the touch screen 15. UI image 703B includes the user name "AAAAA", and UI image 704B includes the comment "I'll send you a magazine!" that was input in the example of Fig. 12(B).

[0155] 12(C) is a view image 611 after user "AAAAAA" further inputs a tap operation on UI image 701, causes UI image 711 to be displayed on touch screen 15, and inputs a tap operation on UI image 711A. That is, as a result of item insertion information indicating a magazine being transmitted from user terminal 100 of user "AAAAAA" to other user terminals 100 and HMD set 1000, user terminal 100 and HMD set 1000 place an effect object 674 (described below) in virtual space 600. As an example, after the elapsed time indicated in the item insertion information has elapsed, user terminal 100 and HMD set 1000 execute an effect related to effect object 674 and execute a process to activate the effect of the item object.

[0156] In the example of FIG. 12(D), the number of magazines increases from 0 to 1 by executing a process to activate the effect of the item object. As a result, the player utters "Thank you!" to the user "AAAAA", and the voice data of this utterance is transmitted to each user terminal 100. As a result, each user terminal 100 outputs the voice "Thank you!" as the utterance 692 of the avatar object 610.

[0157] As described above, in this game, communication between the users and the avatar object 610 is realized by the output of speech from the avatar object 610 based on the speech of the player and the input of comments by each user.

[0158] (Game Progression Processing in Game Play Terminal 300) FIG. 13 is a flowchart showing an example of the flow of the game progression process executed by the game play terminal 300.

[0159] In step S31, the processor 30, as the game progression unit 315, progresses the game based on the game program 331 and the movements of the player. In step S32, the processor 30 generates game progress information and distributes it to the user terminals 100. Specifically, the processor 30 transmits the generated game progress information to each user terminal 100 via the server 200.

[0160] In step S33, upon receiving the item insertion information (YES in S33), in step S34, processor 30 places an item object in virtual space 600A based on the item insertion information. As an example, before placing the item object, processor 30 places an effect object 674 in virtual space 600A (see FIG. 11(C)). Effect object 674 may be, for example, an object resembling a gift box. As an example, processor 30 may execute an effect related to effect object 674 after the elapse of the time indicated in the item insertion information. The effect may be, for example, an animation in which the lid of the gift box opens. After the execution of the animation, processor 30 executes processing to activate the effect of the item object. For example, in the example of FIG. 11(D), obstacle object 673 is placed.

[0161] After the animation is executed, processor 30 may place an item object corresponding to the tapped UI image in virtual space 600A. For example, when a tap operation is performed on UI image 711A, processor 30 places a magazine object indicating a magazine in virtual space 600A after the animation is executed. Furthermore, when a tap operation is performed on UI image 711B, processor 30 places a first aid kit object indicating a first aid kit in virtual space 600A after the animation is executed. For example, when avatar object 610 moves to the position of the magazine object or the first aid kit object, processor 30 may execute processing to activate the effect of the magazine object or the first aid kit object.

[0162] Processor 30 continues and repeats the processes of steps S31 to S34 until the game ends. When the game ends, for example, when the player inputs a predetermined input operation to end the game (YES in step S35), the process shown in FIG. 13 ends.

[0163] (Game Progression Processing in User Terminal 100) FIG. 14 is a flowchart showing an example of the flow of the game progression process executed by the user terminal 100.

[0164] In step S41, the processor 10 receives game progress information. In step S42, the processor 10, functioning as the game progression unit 115, progresses the game based on the game progress information.

[0165] In step S43, when processor 10 accepts an item insertion operation by user 3 (YES in step S43), in step S44, processor 10 consumes virtual currency and places effect object 674 in virtual space 600B. Here, virtual currency may be purchased (charged for the game) by user 3 performing a predetermined operation on processor 10 before or during participation in the game, or may be granted to user 3 when a predetermined condition is met. The predetermined condition may be one that requires participation in the game, such as completing a quest in the game, or one that does not require participation in the game, such as answering a questionnaire. The amount of virtual currency (amount of virtual currency owned) is, for example, stored in user terminal 100 as game information 132.

[0166] In step S45, processor 10 transmits item insertion information to server 200. The item insertion information is transmitted to game play terminal 300 via server 200.

[0167] After placing the effect object 674, when a predetermined time has elapsed, the processor 10 places an item object in the virtual space 600A. In the example of Fig. 11, an obstacle object 673 is placed. That is, when the user 3 inputs a tap operation on the UI image 711C, a predetermined amount of virtual currency is consumed and the obstacle object 673 is placed.

[0168] Processor 10 continues and repeats the processing of steps S41 to S45 until the game ends. When the game ends, for example, when a player performs a predetermined input operation to end the game or when user 3 performs a predetermined input operation to leave the game midway (YES in step S46), the processing shown in FIG. 14 ends.

[0169] (Other distribution examples) 11 to 14 illustrate an example of content that broadcasts images and audio of a game featuring an avatar object operated by a player (performer) (moving in response to the performer's movements) and accepts reactions, such as support and comments, from viewers. However, the content provided by system 1 is not limited to this, and may broadcast images and audio of a game featuring not only performers but also avatar objects operated by viewers. Furthermore, the content provided by system 1 is not limited to broadcasting images and audio of a game. For example, the content may broadcast images and audio of an avatar object operated by a performer talking, images and audio of a live performance in which an avatar object operated by a performer sings, or images and audio of live commerce using an avatar object operated by a performer. Furthermore, while FIGS. 11 to 14 illustrate an example in which audio from a player (performer) is broadcast and output from a user terminal, and reactions from viewers' user terminals are reflected in a virtual space, the content may also broadcast images in which information corresponding to audio from the performer and viewers is reflected (output) in the virtual space. Information corresponding to the sound includes, for example, specific images that allow the information corresponding to the sound to be identified, dramatic effects (brightening, vibrating, etc.), etc. In the following, in the case of distributing images and sounds of live commerce, as an example of information corresponding to the sound emitted by the performer user or the viewer user, a specific image that identifies the information (for example, an object image or text image corresponding to the sound) is reflected in a virtual space and distributed.

[0170] Fig. 15 is a diagram showing an example of a screen when live commerce images and audio are distributed by multiple avatars. Fig. 15 shows an example of a screen displayed on the user terminal of a viewer user, in which an avatar object 610a of performer a (user a) corresponding to the first player and an avatar object 610b of performer b (user b) corresponding to the second player introduce and explain a specific product (e.g., a specific vacuum cleaner).

[0171] In the example of FIG. 15(A), the upper bodies of avatar object 610a and avatar object 610b are shown on the left and right sides of the screen. In this case, avatar object 610a and avatar object 610b are placed in virtual space 600A provided to players (here, performers a and b) shown in FIG. 9. Therefore, performer a, who operates avatar object 610a, can see avatar object 610b in field of view image 650 when, for example, he turns to the left. Similarly, performer b, who operates avatar object 610b, can see avatar object 610a in field of view image 650 when, for example, he turns to the right. This allows performer a and performer b to communicate as if they were in the same place, even when they are in different places.

[0172] Furthermore, initial placement information and operation instruction data corresponding to such virtual space 600A are distributed as needed, as shown in Fig. 8. By receiving the distributed information, as shown in Fig. 8, the user terminal defines virtual space 600B (see Fig. 10) corresponding to virtual space 600A as needed, and displays an image from virtual camera 620B on the user terminal as field of view image 650. Furthermore, Fig. 15(A) shows a situation in which performers a and b talk after starting content distribution to explain what kind of product they will be introducing, and then utter a voice such as, "The product we will be introducing today is this product, 'bang!'"

[0173] In the system 1 of this embodiment, when an emphasis word such as "ba-bang" is extracted, for example, an object image corresponding to the number of times the emphasis word is uttered is reflected in the virtual space. When an emphasis word such as "ba-bang" is extracted for the first time by a predetermined speech recognition process or the like, the system 1 displays (outputs) an image object corresponding to the reflection content corresponding to the first emphasis word at a reflection position corresponding to the first emphasis word.

[0174] 16 is a diagram showing an example of a reflection specification table for specifying the reflection position and reflection content according to the extracted keyword. The reflection specification table is a table that is set in advance by the administrator, performer, etc. before the content is distributed, and is stored in storage such as the game play terminal 300. The reflection specification table may be set in advance for each product to be introduced, each performer, or each content type, or may be customized by the administrator, performer, etc. each time before the content is distributed.

[0175] The emphasis words are set to a variety of words that typically attract the viewer's attention, such as "ba-bang," "boom," "what?", and "wow." When an emphasis word is extracted from the speech of a performer, if the performer who uttered the emphasis word does not perform an operation to specify a reflection position (an operation to specify an object), a reflection process determined according to the number of times the emphasis word is uttered is performed. In the example shown in FIG. 16, as shown in the "No emphasis word specified" column (top row), for a title frame object in which a title is displayed as a reflection position, the reflection content is determined to be text such as "ABC vacuum cleaner" the first time, text such as "29,800 yen" the second time, and text such as "100 units in total" the third time. In this embodiment, objects that are reflected (output) in response to uttered speech are also collectively referred to as "speech-responsive objects."

[0176] When the first emphasis word is extracted, as shown in FIG. 15(B), for example, a 3D model 680 (an example of a voice-enabled object) of a vacuum cleaner, which is the product to be introduced, is placed in a predetermined position (for example, in the center of the screen between avatar objects 610a and 610b), and a title frame object 681 (an example of a voice-enabled object) is placed above avatar object 610a, thereby performing a reflection process to reflect the emphasis word in the virtual space. Furthermore, as shown in FIG. 16, title frame object 681 includes text such as "ABC vacuum cleaner!" As a result, when performer a, who is operating avatar object 610a, looks to the left, for example, he or she can see avatar object 610b and the 3D model 680 of the vacuum cleaner in field of view image 650, and when he or she looks up, he or she can see title frame object 681 in field of view image 650. Furthermore, when actor b, who is operating avatar object 610b, looks to the right, for example, he or she can see avatar object 610a and a 3D model of a vacuum cleaner 680 in the field of view image 650, and when actor b looks to the upper right, he or she can see title frame object 681 in the field of view image 650.

[0177] As shown in FIG. 15(C), the 3D model 680 of the vacuum cleaner placed in the virtual space is composed of multiple objects corresponding to, for example, a nozzle portion (tip portion) 680a that sucks up dirt, a hose portion 680b that connects the nozzle to the main body, a dirt collection portion 680c that collects the sucked-up dirt, and a main body portion 680d equipped with a motor. The game play terminal 300 can accept an operation to specify an object by identifying a designation action, such as a performer touching (or pointing) one of the objects that make up the 3D model 680 with a finger or the like, based on, for example, positional information between the avatar object (e.g., a finger) and the 3D model 680. This allows the user to perform actions such as specifying parts of the 3D model 680 of the vacuum cleaner, introducing or explaining it, as if the vacuum cleaner were actually present. Subsequently, the objects placed in the virtual space are updated in response to voices and operations from performers a, b, viewers, the broadcast operator, and the like.

[0178] Next, performer a and performer b conduct live commerce, and Fig. 15(C) shows a situation in which performer a (avatar object 610a) designates an object corresponding to main body portion 680d of 3D model 680 and makes a sound, for example, saying, "This is a cordless vacuum cleaner." Note that Figs. 15(C) to (E) show a situation in which a sound is made to a vacuum cleaner, and therefore illustrate 3D model 680 of the vacuum cleaner, but as described above, avatar object 610a, avatar object 610b, and other objects are placed in the virtual space, and a state in which they are displayed in response to the movements of performer a and performer b is constructed in the virtual space and can be displayed on the user terminal.

[0179] In the system 1 of this embodiment, when a command word such as "this," "that," "that," or "here" is extracted and a part of the 3D model 680 is designated, as shown in the command word designated column (bottom row) in FIG. 16 , a process is performed in which a voice-enabled object containing text corresponding to a series of words (e.g., automatically determined words or sentences) uttered following the command word is associated with the object designated as the reflection position and reflected in the virtual space. Specifically, as shown in FIG. 15(C), a sticky note object 682 (an example of a voice-enabled object) containing text such as "cordless" is placed (output) on an object corresponding to the main body portion 680d designated by performer a, and the reflection process is performed to reflect the text in the virtual space. As shown in FIG. 15(C), identification information (e.g., a circled "a") is added to the sticky note object 682 to identify the user who uttered the word (here, performer a).

[0180] FIG. 15(D) illustrates a situation in which performer b (avatar object 610b) designates an object corresponding to nozzle portion 680a of 3D model 680 and makes a comment, for example, "The suction power from here is amazing." System 1 extracts the instruction word "from here" and performs a process of reflecting an object containing the corresponding text in the virtual space, treating the sentence "the suction power is amazing" uttered following the instruction word as a series of words. Specifically, as shown in FIG. 15(D), a sticky note object 683 (an example of a voice-enabled object) containing the text "the suction power is amazing" is placed (output) on the object corresponding to the nozzle portion designated by performer b, and a reflection process is performed to reflect the text in the virtual space. Also, as shown in FIG. 15(D), identification information (e.g., a circled "b") is added to sticky note object 683 to identify the user who uttered the word (here, performer b). When a new sticky note object is placed (output), processing is performed to change the display mode of the previously placed sticky note object to, for example, a mode that makes it less noticeable (such as gray, shown in a shaded mode in Figure 15).

[0181] FIG. 15(E) shows a situation in which performer a (avatar object 610a) designates an object corresponding to hose portion 680b of 3D model 680 and utters a voice such as, for example, "This is also 'stretchable.'" System 1 extracts the instruction word, such as "this too," and performs a process of reflecting objects containing the corresponding text in the virtual space, using the instruction word followed by the word "stretchable" as a series of words. Specifically, as shown in FIG. 15(E), a sticky note object 684 (an example of a voice-enabled object) containing the text "stretchable" is placed (output) on the object corresponding to hose portion 680b designated by performer a, and a reflection process is performed to reflect the text in the virtual space.

[0182] In the above-described manner, after performers a and b have introduced and explained the functions and features of the product, when it is time to announce the price, for example, FIG. 15(F) shows a situation in which a voice such as "Finally, the price announcement, bang!" is uttered. As described above, when system 1 extracts an emphasis word such as bang without specifying a position, it performs a reflection process, for example, as shown in FIG. 16, to reflect text corresponding to the number of times the word is uttered. When system 1 extracts an emphasis word for the second time, it additionally inserts text such as "29,800 yen" into title frame object 681 as shown in FIG. 15(F) (outputting an updated voice-enabled object). Furthermore, when a comment such as "The number of units offered is..." is subsequently made, when the emphasis word is extracted for the third time, it additionally inserts text such as "100 units in total" into title frame object 681 as shown in FIG. 15(G) (outputting an updated voice-enabled object). The display mode of the additionally inserted text in Figures 15(F) and 15(G) may be highlighted (for example, in blue, at a large size, etc.) Note that Figures 15(F) and 15(G) show the state when title frame object 681 is updated and displayed, and therefore only title frame object 681 is shown, but as described above, a 3D model 680 of a vacuum cleaner and sticky note objects for 3D model 680 are placed in the virtual space, and an image of avatar object 610a and avatar object 610b displaying their movements is constructed in the virtual space and can be displayed on the user terminal.

[0183] Furthermore, when an emphasis word is uttered while designating an object, as shown in the "Designated by Emphasis Word" column (middle row) in FIG. 16, the designated object (including text within the object) is set as the reflection position, and a process of changing the display mode of the object is performed. The process of changing the display mode of the object includes, for example, a process of highlighting the object and changing the display color of the text within the object. FIG. 15(H) illustrates a situation in which, for example, in the state of FIG. 15(E), the performer designates a sticky note object 682 and utters a voice such as "This is really good, isn't it?" As a result, a process of changing the display mode of the sticky note object 682 is performed, and FIG. 15(H) illustrates a state in which the display size of the sticky note object 682 and the text increases and the text color changes from black to red (italicized characters in the figure indicate that they are displayed in red) (an updated voice-enabled object is output). Note that an object changed by the process of changing the display mode may return to its original display mode after a predetermined time (e.g., 10 seconds) has elapsed or when the performer or other person designates the object again.

[0184] The system 1 in this embodiment can accept reactions (examples of responses) from viewers other than the performers to voice-enabled objects such as a 3D product model 680, a title frame object 681, and sticky note objects shown in Fig. 15. Fig. 17 is a diagram showing a display example in which objects in the virtual space are updated by accepting reactions from viewers.

[0185] In the user terminal, when the viewer performs a first operation (for example, a touch operation, a click operation, or a voice output such as "Like") on either the title frame object 681 or the sticky note object to which the viewer wishes to show a favorable impression, first reaction information is transmitted to the game play terminal 300 to identify the object that was the target of the first operation and the fact that the first operation was performed.

[0186] The game play terminal 300 that has received the first reaction information performs processing to display a like object 685 indicating a favorable impression of the object that was the target of the first operation identified from the first reaction information in association with the object (outputs an updated voice-enabled object). FIG. 17(A) shows, as an example, a state in which a like object 685 is displayed as a result of a first operation indicating a favorable impression being performed on a title frame object 681. As shown in FIG. 17(A), the like object 685 is displayed in association with the title frame object 681. Note that when multiple viewers have expressed favorable impressions of the same object, a number of likes to identify the number of likes may be displayed together with the object.

[0187] Furthermore, on the user terminal, for example, the viewer can input a comment (for example, by voice input, keyboard input, etc.) by performing a second operation (for example, a long press operation, a voice output of the comment content while performing a touch operation) on either the title frame object 681 or the sticky note object about which the viewer wants to comment, and by performing a send operation after inputting the comment, the object that was the target of the operation, the comment that was input, and second reaction information for identifying the name of the viewer (user name) who made the comment are sent to the game play terminal 300.

[0188] The game play terminal 300 that has received the second reaction information performs processing to display the comment in association with the object that is the target of the operation identified from the second reaction information (outputting the updated voice-enabled object). FIG. 17(B) shows an example of a comment object 686 that is displayed when a comment is entered for a title frame object 681, one of the objects that has been placed. In FIG. 17(B), a comment object 686 indicating that a comment has been entered is displayed in association with the title frame object 681. The performer, audience, and administrator can operate the comment object 686 to display a list object 687 that displays a list of comments associated with the title frame object 681, as shown in FIG. 17(C). The list object 687 displays the content of the comment together with the name of the user who entered the comment, as exemplified in FIG. 17(C). Note that objects output in response to audio (such as the objects shown in FIG. 15, the objects shown in FIGS. 17(A) and (B), and the object shown in FIG. 17(D) described below) are reflected in the virtual space and can be displayed on all terminals viewing the virtual space, but as shown in FIG. 17(C), a list object 687 displayed by operating a comment object 686 may be reflected only on the terminal that accepted the operation (either the game play terminal or the user terminal) and not on other terminals. Note that the list object 687 may be changed to its original state (FIG. 17(B)) when a predetermined time (for example, 10 seconds) has passed or when it is designated again by the user who performed the operation to display the list object 687.

[0189] Furthermore, on the user terminal, for example, the viewer can input a question (for example, voice input, keyboard input, etc.) by performing a third operation (for example, a double-click operation, a voice output such as "Question, can the nozzle be replaced?" while performing a touch operation) on the part of the product about which the viewer wants to ask a question, and by performing a send operation after inputting the question, the object that was the target of the operation and third reaction information for identifying the content of the input question are sent to the game play terminal 300.

[0190] The game play terminal 300 that has received the third reaction information performs processing to display a question in association with the object that was the target of the operation identified from the third reaction information. FIG. 17(D) shows an example of a question object 688 that is displayed when a question (here, "Can the nozzle be replaced?") is input to the nozzle portion 681a of the 3D model 680. In FIG. 17(D), a question object 688 that displays the question is displayed in association with the nozzle portion 681a of the 3D model 680. The display manner of the question object is different from the display manner of the sticky note object, and therefore is illustrated by a dotted line frame in FIG. 17(D). However, this is not limiting, and the display manner of the question object may be the same as the display manner of the sticky note object. Note that the performer or the like attempts to answer the question when the question object 688 is displayed, and when the answer is complete, the performer (excluding the viewer user) or the like may designate the question object 688 again, thereby resolving the question and hiding the question object 688.

[0191] Figure 18 is a diagram illustrating a flowchart of the voice-enabled object-related processing related to the display of voice-enabled objects, etc., described in Figures 15 to 17. The voice-enabled object-related processing is repeatedly executed at predetermined time intervals (e.g., 2 msec) on a terminal (e.g., game play terminal 300) that defines and enables distribution of a virtual space.

[0192] In step S101, the speech uttered by the performer is analyzed to determine whether a specific keyword has been extracted. A specific keyword is, for example, an emphasis word or a command word shown in FIG. 16. Note that the specific keyword may also include other words. If it is not determined in step S101 that a specific keyword has been extracted, the process proceeds to step S108. On the other hand, if it is determined in step S101 that a specific keyword has been extracted, it is determined in step S102 whether the specific keyword is an emphasis word shown in FIG. 16.

[0193] If it is determined in step S102 that a specific keyword is an emphasis word, it is determined in step S103 whether or not an object has been designated. If it is not determined that an object has been designated, a process of updating and displaying text, etc. according to the number of times the emphasis word has been uttered is performed in step S104. Note that in the first case, a 3D object 680 as a product and a title frame object 681 are reflected (output) in step S103 as shown in FIG. 15(B). Furthermore, in the second or subsequent cases, the title frame object 681 is updated and reflected (output) in step S103 as shown in FIGS. 15(F)(G).

[0194] If it is determined in step S103 that an object has been designated, processing is performed in step S105 to update the display mode of the designated object (including the text within the object) (see, for example, FIG. 15(H)).

[0195] If it is not determined in step S102 that the specific keyword is an emphasis word, it is determined in step S106 whether or not an object has been designated. If it is not determined that an object has been designated, the process proceeds to step S108. On the other hand, if it is determined that an object has been designated, in step S107, a process is performed in which a sticky note object as a voice-enabled object containing text corresponding to a series of words uttered following the instruction word (for example, words automatically determined as single words or sentences) is associated with the designated object and reflected (output) (for example, see FIGS. 15(C) to (E)).

[0196] In step S108, it is determined whether any of the first to third reaction information described above has been received. If it is determined in step S108 that no reaction information has been received, the process proceeds to step S110. On the other hand, if it is determined that any reaction information has been received, the process reflects (outputs) an image (see, for example, FIGS. 17(A) to 17(D)) corresponding to the received reaction information in step S109 (see FIGS. 17(A) to 17(D)).

[0197] In step S110, it is determined whether an operation has been received for an image corresponding to any of the reaction information (see, for example, FIG. 17(C), FIG. 17(D), etc.). If it is not determined in step S110 that an operation has been received for an image corresponding to the reaction information, the voice-enabled object-related processing is terminated. On the other hand, if it is determined in step S110 that an operation has been received for an image corresponding to the reaction information, then in step S111, processing corresponding to the operation (for example, processing to return to FIG. 17(B), processing to hide the question object 688 in FIG. 17(C), etc.) is performed, and the voice-enabled object-related processing is terminated.

[0198] In this content, while avatar object 610a and avatar object 610b introduce and explain the vacuum cleaner, not only when the introduction and explanation is finished but also during the introduction and explanation, depending on a specific operation from the viewer user (for example, an operation on a specific icon), it is possible to transition to a process (including a process for making payment) for making the introduced and explained product available for purchase.

[0199] [Effects of the embodiment] According to the above embodiment, it is possible to display a virtual space including a plurality of objects such as an avatar object, a 3D model composed of a plurality of objects, and sticky note objects, and by acquiring voices from performers and viewers, information corresponding to the voices (for example, text, a like object, a comment object, etc.) is output (displayed) in association with one of the plurality of objects. This allows information corresponding to the voices to be added to the plurality of objects, thereby improving the understandability of the information corresponding to the voices. Furthermore, it is possible to improve the interest of the content. Furthermore, it is possible to provide smooth interaction in real time.

[0200] Furthermore, information corresponding to the voice is associated with any of the plurality of objects for which an instruction from the user has been received, thereby improving operability.

[0201] Furthermore, the information corresponding to the voice may include identification information for identifying the user who uttered the voice, as shown in Fig. 15(C), etc. This makes it possible to know which user associated the information.

[0202] 15(D) and (E), when new information corresponding to a voice is displayed, the format of the information corresponding to a voice that is already being displayed is changed, thereby making it easier to grasp the newly displayed information corresponding to a voice.

[0203] 17 and other figures, responses from other users (e.g., viewers) to the information corresponding to the displayed voice can be received and included in the information corresponding to the voice. This makes it possible to improve the understandability of the information not only for the user who displayed the information corresponding to the voice but also for other users, and also to improve interest and convenience.

[0204] In addition, it is possible to display avatar objects according to the performer's actions, and instructions for the object (such as instructions to display a sticky note object) can be received when the performer points to one of the objects or when the performer speaks a word (see step S107 in FIG. 18). This improves the performer's operability.

[0205] Furthermore, it is possible to display avatar objects according to the performer's actions, and the performer can change the display mode of one of the objects by pointing to the object whose display mode he / she wants to change, or by speaking a word, changing the mode of information corresponding to the voice already displayed for that object (see step S105 in FIG. 18). This improves the performer's operability.

[0206] [Variations] In the above embodiment, an example was shown in which sound-corresponding information such as sticky note objects can be associated with any object placed in the virtual space. However, this is not limited to this, and objects with which sound-corresponding information such as sticky note objects cannot be defined may be defined. Examples of objects with which sound-corresponding information such as sticky note objects cannot be associated include, for example, avatar objects corresponding to the movements of performers, objects constituting the background of the virtual space (e.g., background objects placed as the background of 3D models and avatar objects in FIG. 15(B)), and objects with which sound-corresponding information such as sticky note objects is already associated. Furthermore, instead of or in addition to restricting information on an object-by-object basis, a range within the virtual space with which sound-corresponding information such as sticky note objects cannot be defined may be defined. Examples of the range with which sound-corresponding information such as sticky note objects cannot be defined include a range outside the 3D model or a range outside a predetermined radius from the 3D model. This prevents sound-corresponding information such as sticky note objects from being associated with all kinds of objects, which would make the virtual space difficult to understand. When displaying a sticky note object or the like associated with an object that moves rapidly (such as the arm of an avatar object), the object may be displayed so that it completely follows the movement of the object, but in order to prevent the object from moving around too much and distracting the viewer, the object may be displayed so that it follows the associated object in a somewhat smooth manner.

[0207] In the above embodiment, for example, even when information corresponding to a voice is already associated with a plurality of objects constituting a 3D model, new information corresponding to a voice may be associated, and information corresponding to two or more voices may be displayed for the object. Furthermore, when a predetermined number (e.g., three) or more pieces of information corresponding to a voice are associated with the same object, the predetermined number or more pieces of information corresponding to the voices may be displayed by scrolling within a single sticky object in a predetermined order (in order of most recent association), or may be displayed by switching in a predetermined order at predetermined intervals. This improves understandability. Furthermore, by inputting a specific condition (e.g., sticky objects by a specific performer, question objects by viewers, etc.), objects that meet the condition may be sorted to be displayed. Sticky objects that are highly rated or attracting attention (e.g., favorable impressions) may be displayed preferentially in front. Furthermore, an upper limit may be set on the number of sticky objects that can be associated with the same object.

[0208] In the above embodiment, an example was shown in which a sticky note object or the like is associated with an object in response to an action of pointing at the object, but the action that triggers the associated display of a sticky note object or the like is not limited to the action of pointing at the object. Alternatively, or in addition, the action may include holding the target object in a hand or making contact with the target object (e.g., hitting it). Furthermore, an example was shown in which the text displayed in the sticky note object corresponds to a series of words (e.g., words automatically determined as single words or sentences) uttered following a command word. However, the text is not limited to this, and may be all words uttered following a command word, or may be words uttered during operation of a predetermined operation unit connected to a game play terminal or the like. Furthermore, a sticky note object may be associated with an avatar object corresponding to the performer, or may be associated with an object in the performer's line of sight, when the volume of the voice uttered by the performer suddenly becomes too loud (a volume exceeding a threshold) or too quiet.

[0209] In the above embodiment, an example is shown in which sticky note objects and the like that are associated with an object are always displayed while the content is being distributed, but this is not limited to this, and the objects may be automatically hidden after a predetermined time (e.g., one minute) has elapsed since they were associated.

[0210] 15 in the above embodiment shows an example in which an image of the virtual space viewed from the same viewpoint is displayed without changing the position of the virtual camera on the user terminal, etc. However, since the virtual camera can be controlled in response to an operation on the user terminal as shown in S28 of FIG. 8, an image of the virtual space viewed from a viewpoint corresponding to the operation can be displayed in response to an operation on the user terminal. Furthermore, the avatar objects placed in the virtual space and the 3D models to be described are 3D objects, but the objects displayed as information corresponding to the audio (e.g., sticky note objects) may also be 3D objects (e.g., rectangular parallelepiped objects with the aforementioned text displayed on one side, or three-dimensional character objects) or 2D objects. When the objects displayed as information corresponding to the audio are 3D and 2D objects, the text may be visible or invisible depending on the position of the virtual camera without changing the orientation of the placed object, or the orientation of the object may be changed for each user terminal so that the text is always visible regardless of the position of the virtual camera. When the objects displayed as information corresponding to the audio are 3D objects and 2D objects, they may be enlarged or reduced depending on the distance (near or far) from the virtual camera, may be hidden when the distance is equal to or greater than a first distance (for example, a distance equivalent to 10 m in the virtual space) or when the distance is less than a second distance (for example, a distance equivalent to 30 cm in the virtual space), or may be displayed at a constant size regardless of the distance from the virtual camera.

[0211] In the above embodiment, an example of live distribution of content has been described with reference to Fig. 15 etc., but it is not limited to this. Data for generating a virtual space (including data for displaying sticky note objects etc.) may be saved, and content may be played (distributed) based on that data in response to access from a user. Also, a state in which a sticky note object etc. is displayed just before the end of content may be displayed, and in response to an operation on the sticky note object, a portion of the content in the vicinity of the sticky note object associated therewith (for example, 30 seconds before or after) may be played (distributed). This allows viewers who were unable to watch the live broadcast to be able to enjoy the excitement of the broadcast.

[0212] In the above embodiment, an example was shown in which information based on voice from the viewer, such as a question object, was displayed. In this case, however, for example, normal objects that can be displayed without charging and special objects that can be displayed by charging may be provided, and when a special object is associated, it may be displayed with priority over a normal object.

[0213] [Note] Some of the features of the present invention are summarized below.

[0214] [assignment] The present invention aims to improve the comprehensibility of information.

[0215] [Solution] (1) A computer a display means for displaying a virtual space including a plurality of objects; functioning as a voice acquisition means for acquiring a user's voice; The display means outputs information corresponding to the voice acquired by the voice acquisition means in association with one of the plurality of objects.

[0216] (2) The computer functioning as an instruction receiving means for receiving an instruction from a user regarding any one of the plurality of objects; The display means outputs information corresponding to the voice in association with an object according to the instruction received by the instruction receiving means.

[0217] (3) The information corresponding to the voice may include identification information that identifies the user who uttered the voice.

[0218] (4) When the display means outputs new information corresponding to a voice, the display means can change the format of information corresponding to a voice that has already been output.

[0219] (5) The computer a means for receiving a response from another user to information corresponding to the voice output by the display means; The information corresponding to the voice may include information corresponding to a response received to the information corresponding to the voice.

[0220] (6) The display means is capable of displaying an avatar image according to the user's action; Computer, functioning as an instruction receiving means for receiving an instruction from a user regarding any one of the plurality of objects; The instruction receiving means receives an instruction according to at least one of a manner of the user's movement and a manner of the user's voice.

[0221] (7) The display means is capable of displaying an avatar image according to the user's actions, and is capable of changing the manner of information corresponding to the audio that has already been output according to at least one of the manner of the user's actions and the manner of the audio.

[0222] (8) The virtual space may include areas or objects to which information corresponding to audio cannot be associated.

[0223] (9) When a predetermined number or more pieces of information corresponding to sounds are associated with the same object, the display means outputs the information corresponding to the predetermined number or more of sounds by scrolling it in order, or by switching it at predetermined time intervals.

[0224] (10) a display means for displaying a virtual space including a plurality of objects; a voice acquisition means for acquiring a voice of a user, The display means outputs information corresponding to the voice acquired by the voice acquisition means in association with one of the plurality of objects.

[0225] [Action and effect] According to the above solutions (1) and (10), information corresponding to the voice is added to multiple objects, which makes it possible to improve the comprehension of the information corresponding to the voice, increase interest, and provide smooth interaction in real time.

[0226] According to the above solution (2), operability can be improved.

[0227] According to the above solution (3), it is possible to know which user associated the information.

[0228] According to the above solution (4), it is possible to easily grasp the information corresponding to the newly displayed voice.

[0229] According to the above solution (5), it is possible to improve the understandability of the information not only for the user who displayed the information corresponding to the voice but also for other users, and to improve interest and convenience.

[0230] According to the above solutions (6) and (7), it is possible to improve the operability for the user who operates the avatar image.

[0231] According to the above solution (8), it is possible to avoid the situation where information corresponding to voice is associated with every possible object, making it difficult to understand.

[0232] According to the above solution (9), it is possible to improve the ease of understanding.

[0233] [Software implementation example] The control blocks (particularly control units 110, 210, 310, 410) of the user terminal 100, server 200, game play terminal 300 (HMD set 1000), and distribution terminal 400 may be realized by logic circuits (hardware) formed on an integrated circuit (IC chip) or the like, or by software.

[0234] In the latter case, the user terminal 100, the server 200, the game play terminal 300 (HMD set 1000), and the distribution terminal 400 each include a computer that executes instructions from a program, which is software that realizes each function. This computer includes, for example, one or more processors and a computer-readable recording medium that stores the program. The object of the present invention is achieved when the processor in the computer reads and executes the program from the recording medium. The processor may be, for example, a CPU (Central Processing Unit). The recording medium may be a "non-transitory tangible medium," such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer may also include a RAM (Random Access Memory) for expanding the program. The program may be supplied to the computer via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves). Note that one aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0235] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0236] 1 System, 2 Network, 3, 3A, 3B User (first user), 4 Player (actor), 10, 20, 30, 40 Processor, 11, 21, 31, 41 Memory, 12, 22, 32, 42 Storage, 13, 23, 33, 43 Communication IF, 14, 24, 34, 44 Input / output IF, 15, 45 Touch screen, 17 Camera, 18 Distance measurement sensor, 51 Monitor, 52 Gaze sensor, 53 First camera, 54 Second camera, 55 Microphone, 56 Speaker, 100, 100A, 100B, 100C User terminal (computer, first computer, first information processing device), 110, 210, 310, 410 Control unit (first control unit, second control unit), 111, 311, 413 Operation reception unit, 112, 312, 412 Display control unit, 113, 313 UI control unit, 114, 314 animation generation unit, 115, 315 game progression unit, 116, 316 virtual space control unit, 117 video playback unit, 120, 220, 320, 420 memory unit (first memory unit, second memory unit), 131, 231, 331 game program (program, first program), 132, 232, 332 game information, 133, 233, 333 user information, 151, 451 input unit, 152, 452 display unit (display), 200 server, 211 communication intermediation unit, 212 log generation unit, 213 list generation unit, 234, 421 user list, 300 game play terminal (external device, second external device), 317 reaction processing unit, 400 Distribution terminal (external, first external device, computer, second information processing device), 411 communication control unit, 414 voice reception unit, 415 motion identification unit, 416 action instruction data generation unit, 422 motion list, 423 distribution program (program, second program), 540, 1020, 1021 controller, 500 HMD, 510 HMD sensor, 520 motion sensor, 530 display, 600A, 600B virtual space, 610 avatar object (character), 620A, 620B virtual camera, 631, 632, 633, 634 object, 640A, 640B field of view area, 650, 660 field of view image, 671 enemy object, 672, 673 obstacle object, 674 performance object

Claims

1. Computer, a display means for displaying a virtual space including a plurality of objects; functioning as a voice acquisition means for acquiring a user's voice; The display means outputs information corresponding to the voice acquired by the voice acquisition means in association with one of the plurality of objects.

2. Computer, functioning as an instruction receiving means for receiving an instruction from a user regarding any one of the plurality of objects; 2. The program according to claim 1, wherein the display means outputs the information corresponding to the voice in association with an object corresponding to the instruction received by the instruction receiving means.

3. The program according to claim 1 , wherein the information corresponding to the voice can include identification information for identifying a user who uttered the voice.

4. 2. The program according to claim 1, wherein said display means is capable of changing the appearance of information corresponding to a voice that has already been output when new information corresponding to a voice is output.

5. Computer, a means for receiving a response from another user to information corresponding to the voice output by the display means; The program according to claim 1 , wherein the information corresponding to the voice can include information corresponding to a response received in response to the information corresponding to the voice.

6. the display means is capable of displaying an avatar image according to a movement of the user; Computer, functioning as an instruction receiving means for receiving an instruction from a user regarding any one of the plurality of objects; The program according to claim 1 , wherein the instruction accepting means accepts the instruction in accordance with at least one of a manner of the user's movement and a manner of the user's voice.

7. The program of claim 1, wherein the display means is capable of displaying an avatar image according to the user's actions, and is capable of changing the manner of information corresponding to audio that has already been output according to at least one of the user's action manner and the audio manner.

8. The program according to claim 1 , wherein the virtual space may include an area or object to which information corresponding to a sound cannot be associated.

9. The program according to claim 1, wherein when a predetermined number or more pieces of information corresponding to voices are associated with the same object, the display means outputs the information corresponding to the predetermined number or more of voices by scrolling them in order, or by switching them at predetermined time intervals.

10. a display means for displaying a virtual space including a plurality of objects; a voice acquisition means for acquiring a voice of a user, The display means outputs information corresponding to the voice acquired by the voice acquisition means in association with one of the plurality of objects.

Images