Information processing apparatus, information processing method, program, and information processing system

Abstract

To provide an apparatus, method, program, and system that enables users watching videos to more easily display predetermined reactions to performances within the video.SOLUTION: An information processing apparatus includes a video control unit that displays at least a portion of a video, within a virtual space where a character performing a performance and an object corresponding to a user viewing the performance are positioned, on a display, and a registration unit that registers, in response to a user operation received by the user's terminal, a predetermined performance object associated with the user's action detected by a detection device. When a registered user action is detected by the detection device during the character's performance, the video control unit displays the predetermined performance object within the video.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, a program, and an information processing system. [Background technology]

[0002] It is already known that a video showing a performance such as singing or dancing is distributed to multiple users (viewers), data regarding the reactions of at least some of the users to the performance is obtained, and information corresponding to the reaction data is displayed within the video (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-53313 Summary of the Invention [Problem to be solved by the invention]

[0004] In cases where the user's reaction to a performance can be displayed within the video, as in the invention of Patent Document 1, it is preferable that the user can display a predetermined reaction to the performance, for example a reaction specified by the user, within the video with simpler operations.

[0005] Therefore, an object of the present invention is to provide an information processing device, an information processing method, a program, and an information processing system that enable users watching a video to more easily display predetermined reactions to performances in the video within the video. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present invention comprises a video control unit that causes a display to display a video of at least a portion of a virtual space in which a character performing a performance and an object corresponding to a user watching the performance are placed, and a registration unit that registers a predetermined performance object in association with a user's action detected by a detection device in accordance with a user's operation received by the user's terminal, and is characterized in that when a registered user's action is detected by the detection device while the character is performing a performance, the video control unit causes the predetermined performance object to be displayed within the video. [Effects of the Invention]

[0007] According to one aspect of the present invention, a user watching a video can more easily display a predetermined reaction to a performance in the video within the video, thereby allowing the user to more easily perform an operation to display a predetermined reaction within the video at an appropriate timing during the display period of the video of the performance. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of a display screen of a video showing at least a part of a virtual space according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram showing an example of a screen on which a performance object is displayed within a video. [Figure 3] FIG. 10 is a diagram showing an example of an operation panel for registering a reaction action and a performance object in association with each other. [Figure 4] This is a diagram showing how a specific effect object is displayed in a video when a user performs a reaction action.The diagram on the left shows the real space in which the user is located, and the diagram on the right shows the display screen of the video on which the specific effect object is displayed. [Figure 5] 1 is a diagram showing a configuration of an information processing system according to an embodiment of the present invention. [Figure 6]FIG. 10 is a diagram showing an example of a screen displayed for selecting a detection device. [Figure 7] FIG. 10 is a diagram showing an example of a screen displayed while a reaction action to be registered is being performed. [Figure 8] FIG. 10 is a diagram showing an example of a screen displayed at the end of a reaction action to be registered. [Figure 9] FIG. 10 is a diagram illustrating an example of user information. [Figure 10] FIG. 10 is a diagram illustrating an example of performance information. [Figure 11] FIG. 10 is a diagram showing an example of performance object information. [Figure 12] FIG. 10 is a diagram illustrating an example of registration information. [Figure 13] FIG. 2 is an explanatory diagram illustrating functions of an information processing device according to an embodiment of the present invention. [Figure 14] FIG. 10 is a diagram showing an example in which a performance, a reaction action, and a predetermined performance object are registered in association with each other. [Figure 15] FIG. 2 is a diagram showing an information processing flow according to an embodiment of the present invention. [Figure 16] FIG. 10 is a diagram showing an example of an operation panel according to a modified example of the present invention. [Figure 17] FIG. 10 is an explanatory diagram of functions of an information processing device according to a modified example of the present invention. [Figure 18] This is a diagram showing how, in one variant of the present invention, a specified effect object is displayed in a video when a first user and a second user perform a reaction action.The diagram on the left shows the real space in which the first user and the second user are located, and the diagram on the right shows the display screen of the video on which the specified effect object is displayed. [Figure 19] FIG. 10 is a diagram showing an example of an operation window for a second user to copy the registered contents of a first user in one modified example of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below with reference to preferred embodiments (hereinafter referred to as the present embodiment) shown in the accompanying drawings. Note that the basic data processing technologies (communication / transmission technology, data acquisition technology, data recording technology, data processing / analysis technology, image processing technology, visualization technology, etc.) required to realize the contents of the present embodiment are well-known technologies, and therefore a description thereof will be omitted.

[0010] This embodiment relates to a technology for generating, transmitting, and displaying a video that can be viewed by multiple users simultaneously, and more specifically, to a technology for using a video that displays at least a portion of a virtual space. The virtual space is, for example, a virtual live venue, and includes a first area in which characters as performers are placed, and a second area in which objects corresponding to users watching the performance (hereinafter, "audience objects") are placed. In this embodiment, the virtual space is a three-dimensional space, but is not limited to this and may be a two-dimensional space.

[0011] The video is a video that displays a virtual live event held in the virtual space. In this embodiment, the video can be viewed while playing the game, and a user playing the game can view the video of the virtual live event as an in-game event. In the following explanation, "displaying a video" means expanding the video data delivered during game play, playing the video, and displaying it on a display device, and also includes playing the audio included in the video.

[0012] The virtual live performance may be held at a time specified in a predetermined time schedule in real space, or may be held at a time arbitrarily set by one of multiple users who gather to watch a video of the virtual live performance.

[0013] In the virtual live performance, as shown in FIG. 1, a character C who is a performer performs singing, musical instruments, dancing, acting, acrobatics, MC talk, and the like in a stage area Ar1 which is a first area. In this embodiment, the performance is the singing of a song, and in the virtual live performance, character C sings multiple songs. In the virtual live performance, the singing of each song is performed at a different time period. In other words, the performance in the virtual live performance includes multiple songs, and in the video of the virtual live performance, a video of character C singing one song (hereinafter also referred to as a first performance) can be viewed at a different time period from the video of the singing of the other songs (hereinafter also referred to as a second performance).

[0014] The character C may be, for example, a humanoid character that moves in a predetermined sequence, or may be an avatar that moves in response to the detected movements of an operator in real space detected by motion capture. The number of characters C performing in the stage area Ar1 may be one, or two or more. The characters C on the stage area Ar1 may be replaced as the performance changes.

[0015] Users who are spectators of the virtual live performance (hereinafter also referred to as spectator users) can operate user terminals, which are devices owned by the spectator users, and watch videos of the virtual live performance through the display of the user terminal. The display of the user terminal corresponds to an example of a display device, and may be provided on the user terminal itself, or may be connected to the user terminal via a wired or wireless connection. Displays connected to the user terminal may include general stationary displays as well as HMDs (Head Mounted Displays) such as VR goggles.

[0016] Furthermore, in this embodiment, multiple spectator users can simultaneously watch the same virtual live performance. Specifically, an audience seating area Ar2 is set as a second area in the virtual live performance venue, and each of the multiple spectator users is assigned a seat, with the location of the seat determined for each spectator user. The number of seats, i.e., the number of spectator users who can simultaneously watch the same virtual live performance, may be set to any number. Furthermore, the seats may be assigned randomly, or may be set based on the spectator users' attributes in the game, their game play history (for example, the amount of time they have played the game, the number of points they have earned in the game), etc. Note that viewing the same virtual live performance at the same time may involve the possibility that the timing at which each scene in the video is displayed may differ slightly between audience users (causing a time lag) due to the influence of the communication environment, etc.

[0017] 1, audience avatars D are placed at the audience seating locations as audience objects, and each audience user can watch the virtual live performance from the perspective of their own audience avatar D. In other words, each audience user is provided with a video of the venue as seen from the perspective of their corresponding audience avatar D. However, this is not limited to this, and a video showing a bird's-eye view of a wide area of ​​the virtual live venue, including both the stage area Ar1 and the audience seating area Ar2, may also be provided.

[0018] Furthermore, during the performance, audience users can use cheering goods E such as penlights and glow sticks by operating audience avatars D. The cheering goods E used by an audience user is one of the audience objects, and is displayed in the audience seating area Ar2 at a position corresponding to that audience user, more specifically, at the position of the audience seat assigned to that audience user, while the video is being displayed. As described above, in the audience seating area Ar2 of the virtual live venue, audience objects corresponding to multiple audience users watching the performance of character C, specifically audience avatars D and cheering goods E, are placed, and specifically displayed at the audience seating position of each audience user.

[0019] Furthermore, audience users can react to the performance while it is being performed (specifically, while a song is being sung), and a performance object F based on that action can be displayed in the video, as shown in Fig. 2. A reaction action corresponds to the action of the present invention, and is an action (gesture) that audience users make to express their feelings when watching the performance, to liven up the performance, or to cheer on character C. Examples of reaction actions include taking a distinctive pose, moving an arm or leg in a predetermined trajectory, and turning one's head. A reaction action is detected by a detection device, which will be described later, and when a reaction action is detected, a predetermined performance object F is displayed in the video.

[0020] The production object F is displayed in the video as a means of production to help audience users liven up the virtual live performance, and includes, for example, text information such as messages to character C, images such as illustrations, images with animation, images called "stamps" consisting of decorative text and / or illustrations, and images that resemble character C.

[0021] During the performance, the effect object F is displayed in the audience seating area at a position corresponding to the audience user who performed the reaction operation, more specifically, at a position corresponding to the audience seat of that audience user. Then, by displaying the effect object F in the video, each of the multiple audience users, including the audience user who performed the reaction operation, can watch the video with the effect object. If the performer character C is an avatar of the operator existing in the real world, the operator can watch a video of the audience seating area on a display used by the operator during the performance. In this case, when the operator finds a performance object F that catches his / her eye in the video, he / she may operate the avatar to make a movement in response to the performance object F (for example, waving in the direction of the performance object F).

[0022] When an audience user performs a reaction action to display a predetermined effect object F in the video, the audience user performs an operation to associate and register the reaction action with the predetermined effect object F. Specifically, before character C starts singing a predetermined song (hereinafter, a song to be registered) at the virtual live performance, the audience user operates the user terminal to display the registration operation panel P shown in FIG. 3 on the display of the user terminal. Then, through the displayed registration operation panel P, the audience user specifies a song to be registered from among multiple songs (performances) that character C will sing at the virtual live performance, and also specifies the type of effect object F to be displayed, etc. Here, the effect object F specified by the audience user corresponds to the predetermined effect object F.

[0023] In addition, the audience user performs a reaction action (hereinafter referred to as the action to be registered) that triggers the display of a specific performance object F. The action to be registered is detected by a detection device. When the above series of operations (hereinafter referred to as the registration operations) are performed, the action to be registered, the song to be registered specified by the audience user, and the specific performance object F specified by the audience user are associated and registered. Here, registering the action to be registered, the song to be registered, and the specific performance object F in association with each other means storing these three items in a linked state so that identifying two of the action to be registered, the song to be registered, and the specific performance object F can identify the remaining one.

[0024] After the registration target action, the registration target song, and the predetermined effect object F are associated and registered by the above registration operation, the audience user who performed the registration operation performs the registration target action while the character C is singing the registration target song. This causes the predetermined effect object F associated with the registration target action to be displayed in the video, as shown in Fig. 4. Furthermore, by repeatedly performing the registered registration target action while singing (performing) the registered registration target song, the audience user who performed the registration operation can cause the registered predetermined effect object F to be displayed in the video for a period of time corresponding to the number of times the registration target video has been performed.

[0025] Furthermore, when an audience user performs a new registration operation, the audience user can display the newly registered predetermined effect object F in the video by performing the newly registered registration target action while singing (performing) the newly registered registration target song. More specifically, in this embodiment, by performing the registration operation for each song (each performance) during a virtual live performance, the registration target action and the predetermined effect object F can be associated and registered for each song. This makes it possible to change the registration target action and the predetermined effect object F for each song. For example, if the same registration target action is registered for multiple songs (registration target songs), it is possible to change the predetermined effect object F that is displayed in the video when the registration target action is performed for each song.

[0026] As mentioned above, there may be cases where multiple audience users are watching the same virtual live performance, and in this case, each audience user can perform the registration operation individually. This allows the registration target actions and the predetermined effect object F to be different for each audience user. For example, for the same song, the registration target actions and the predetermined effect object F can be different for each audience user.

[0027] In addition, in this embodiment, when a spectator user performs a reaction action to display a predetermined effect object F in a video, the spectator user may be required to pay a number of points equivalent to the compensation for the predetermined effect object to be displayed. In this case, the number of points held by the spectator user who displayed the effect object F will be reduced by the amount of the compensation.

[0028] There is no particular limit to the number of times that audience users can react and display the effect object F during one performance (e.g., singing one song). Also, there may be a limit on the number of times that each audience user can display the effect object F during one performance.

[0029] The number of points held by each spectator user is managed, and points correspond to a predetermined value. Points can be earned by playing games or increased by paying a fee. The value (predetermined value) displayed in conjunction with the display of the effect object F is not limited to points, but may be money, electronic money, virtual currency, or in-game currency that can be used in the game.

[0030] <<Configuration of an information processing device according to one embodiment of the present invention>> Next, a configuration example of the information processing system 100 according to this embodiment will be described. As shown in Fig. 5, the information processing system 100 is composed of a server computer (hereinafter referred to as server 10), user terminals 12 used by each of a plurality of audience users, and a detection device 16. The server 10 and each user terminal 12 are connected to each other so that they can communicate with each other via a network 14. The network 14 is a communication network consisting of, for example, the Internet or a mobile communication network, and may include a LAN (Local Area Network), a WAN (Wide Area Network), an intranet, Ethernet (registered trademark), etc. The configurations of the user terminal 12, the detection device 16, and the server 10 will be described below.

[0031] (user terminal) The user terminal 12 is a terminal operated by a user (more specifically, a spectator user when watching a video) to play a game and watch a virtual live video as an event within the game. In other words, the user terminal 12 accepts a series of operations performed by the user to play the game, and these operations include operations performed by the spectator user to watch the virtual live video (hereinafter referred to as video viewing operations).

[0032] The user terminal 12 is configured by a personal computer, smartphone, mobile phone, tablet terminal, television receiver capable of inputting information, wearable terminal, or the like. As shown in FIG. 5, the user terminal 12 includes a processor 12a, memory 12b, and communication interface 12c. The configuration of these devices is similar to the processor 10a, memory 10b, and communication interface 10c, which are provided in the server 10 and will be described later. The user terminal 12 also includes an input device 12d including a touch panel, a mouse, a keyboard, and the like, and an output device 12e including a display, a speaker, and the like. The input device 12d may include a microphone 12f as a sound collection device.

[0033] In addition, a program for an OS (Operating System) and an application program for a game (hereinafter referred to as an app) are installed on the user terminal 12. This app includes a program for watching videos. By starting the app on the user terminal 12, it becomes possible to play a game, and by executing the program for watching videos while playing a game, it becomes possible to watch videos of a virtual live performance.

[0034] More specifically, the user terminal 12 receives video data distributed from the server 10 via the network 14. The user terminal 12 then expands the received video data, and a video of the virtual live performance shown in Fig. 1, i.e., a video of a portion of the virtual live performance venue, is displayed on a display or other display device of the user terminal 12. The video displays a character C performing in the stage area Ar1, as well as an audience avatar D and cheering goods E placed in the audience seating area Ar2.

[0035] Furthermore, the user terminal 12 accepts the above-mentioned registration operation by the user. Specifically, when an audience user watching a video performs a predetermined operation on the user terminal 12, the registration operation panel P of Fig. 3 is displayed on the display of the user terminal 12. The audience user can perform the registration operation through the registration operation panel P. In the registration operation, the audience user specifies a song to be registered from among multiple songs sung by the character C in the virtual live performance.

[0036] Furthermore, in the registration operation, a predetermined rendering object F to be displayed in the video is selected from among multiple rendering object candidates presented on the registration operation panel P. At this time, as shown in Fig. 3, for rendering objects that require a fee to be displayed in the video among the multiple rendering object candidates presented on the registration operation panel P, the number of points equivalent to that fee (indicated as "purchase points" in the figure) is displayed.

[0037] Furthermore, in the registration operation, after selecting a detection device 16 for detecting a reaction action to be registered (action to be registered) on the device selection screen S1 shown in Fig. 6, the audience user actually performs the action to be registered, and the selected detection device 16 detects the action to be registered. Note that while the action to be registered is being performed, an action reception screen S2 shown in Fig. 7 is displayed on the display of the user terminal 12, and when the action to be registered is completed, an action completion screen S3 shown in Fig. 8 is displayed.

[0038] After the registration operation has been performed according to the above procedure, the user terminal 12 identifies the operation content, i.e., the designated song to be registered and the predetermined effect object F, as well as the detected action to be registered, and transmits data relating to these three contents (hereinafter, registration data) to the server 10. Upon receiving the registration data, the server 10 associates the designated song to be registered and the predetermined effect object F with the action to be registered and registers them.

[0039] Subsequently, while character C is singing the song to be registered during the virtual live performance, if an audience member watching the video performs the action to be registered, detection device 16 detects the action, and user terminal 12 generates data on the detected reaction action (hereinafter, detected action data) and transmits the detected action data to server 10. Meanwhile, server 10, having received the detected action data, transmits video data of the virtual live performance based on the detected action data. When user terminal 12 unpacks the video data, a video displaying a predetermined effect object F associated with the action to be registered is displayed on user terminal 12, as shown in FIG. 4. At this time, the predetermined effect object F is displayed in the virtual space at a position corresponding to the audience user who performed the action to be registered; specifically, in audience seating area Ar2, it is displayed at or near the seating position of the audience user who performed the action to be registered.

[0040] (Detection device) The detection device 16 is a device used by the spectator user to detect reaction movements made by the spectator user while watching a video. The detection device 16 is, for example, a wearable device that detects the user's body movements, specifically, a smart watch, a motion capture device, and a device equipped with a gyro sensor (angular velocity sensor) or an acceleration sensor. The detection device 16 may also be a camera that captures the user's movements, or a controller that the user holds and operates in their hand. The detection device 16 may be a device separate from the user terminal 12 and used separately from the user terminal 12, or may be mounted on the user terminal 12 and used integrated with the user terminal 12.

[0041] The detection device 16 digitizes the detected user's actions (specifically, reaction actions) and transmits the data (hereinafter, "action data") to the user terminal 12 of the user who performed the action. The action data may be motion data in a BVH (Biovision Hierarchy) file format or video data of the reaction actions. The data may be transmitted via a wired or wireless connection. Examples of wireless transmission methods include Bluetooth (registered trademark), Wi-Fi (registered trademark), infrared communication, and other wireless communication methods.

[0042] (server) The server 10 executes a series of processes related to game play, specifically generating, storing, transmitting and receiving data necessary for game progress and management. The game data processed by the server 10 includes the video data of the virtual live performance described above. That is, the server 10 generates data of the video of the virtual live performance and distributes the video data to the user terminals 12 of multiple spectator users who are simultaneously watching the virtual live performance. Furthermore, when a spectator user watching the video performs a reaction motion, the server 10 generates video data incorporating a predetermined effect object F associated with the motion as a process of displaying the effect object F in the video, and distributes the data to the user terminals 12 of each spectator user.

[0043] The server 10 is an example of a computer, and in this embodiment, is configured by, for example, an SNS server or a server computer for cloud services. Server computers for cloud services include server computers for ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). The server 10 may be a single computer or multiple computers distributed in parallel.

[0044] The configuration of the server 10 will be explained. As shown in FIG. 5, the server 10 has, as hardware devices, a processor 10a, a memory 10b, a communication interface 10c, and a storage 10d.

[0045] The processor 10a may be configured by a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a TPU (Tensor Processing Unit), or an ASIC (Application Specific Integrated Circuit), etc. The memory 10b may be configured by semiconductor memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory).

[0046] The communication interface 10c may be configured by, for example, a network interface card, a communication interface board, etc. The standard of data communication by the communication interface 10c is not particularly limited, and examples include communication by a wireless LAN based on Wi-fi (registered trademark), communication by a 3G to 5G or later generation mobile communication system, or communication based on LTE (Long Term Evolution).

[0047] The storage 10d may be configured by a flash memory, a hard disc drive (HDD), a solid state drive (SSD), a flexible disc (FD), a magneto-optical disc (MO disc), a compact disc (CD), a digital versatile disc (DVD), a secure digital card (SD card), a universal serial bus memory (USB memory), etc. The storage 10d may be built into the server 10, may be attached to the server main body in an external format, or may be a network attached storage (NAS), etc. The storage 10d may also be configured by a third computer communicably connected to the server 10, such as a database server or online storage.

[0048] The storage 10d stores and accumulates various information related to game control, including the viewing of virtual live video. Of the information stored in the storage 10d, the information related to the viewing of virtual live video includes user information, performance information, effect object information, and registration information.

[0049] The user information is stored in the storage 10d for each spectator user, and includes, for example, as shown in Fig. 9, the spectator user's name, account ID, login password and other personal information, whether or not there are other users with whom the user has a friend relationship (corresponding to a predetermined relationship), game play history, in-game status, and the current number of points acquired in the game. Other users with a friend relationship are users who have accepted a request made by the user through the aforementioned app function to become a friend with whom the user wishes to cooperate in playing a game or the like.

[0050] 9, the user information further includes the ID of the virtual live performance that the audience user is watching, the position information of the audience seats assigned to the audience user at the venue of the virtual live performance, information on the audience avatars, and information on cheering goods to be used with the audience avatars, etc. Furthermore, as shown in FIG. 9, the user information also includes information on candidate performance objects that the audience user can display in the video of the virtual live performance.

[0051] The performance information is information about multiple performances performed by character C in the virtual live performance, more specifically, multiple songs sung by character C, and is stored for each performance (song) in storage 10d. As shown in Fig. 10, the performance information includes the song title, identification ID, performance date, scheduled performance start / end times, and the character performing the performance. Although not shown, the performance information may also include information about one or more singing sections (specifically, intro, verse 3, verse 4, chorus, verse 5, etc.) that make up each song, such as information about the start / end times of each section, more specifically, the elapsed time since the start of the performance.

[0052] The rendering object information is information about the rendering object and is stored for each rendering object in storage 10d. As shown in FIG. 11, the rendering object information includes the identification information of the rendering object, information indicating the type and appearance of the rendering object, whether or not a fee is charged for displaying the rendering object in the video, the amount of the fee, and the display mode when displayed in the video. Here, the display mode of the rendering object refers to the size, color, shape, and posture of the rendering object when displayed, the display position in the video, the display time, and whether or not it undergoes movement such as deformation, movement, or rotation. Furthermore, the display mode of the rendering object may be changeable when a predetermined condition is met after it has been set.

[0053] The registration information is information relating to registration performed based on a registration operation by an audience user, and is stored in storage 10d for each audience user (strictly speaking, for each audience user who performed the registration operation). As shown in Fig. 12, the registration information includes the identification ID of the audience user who performed the registration operation, the identification IDs of the songs and performance objects F to be registered specified by the audience user, and the details of the actions to be registered performed by the audience user when performing the registration operation and detected by detection device 16. Furthermore, if one audience user performs a registration operation for each song, the registration information for that audience user is stored for each song.

[0054] Returning to the explanation of the configuration of the server 10, a program for an operating system (OS) and an information processing program for game progression are installed on the server 10. The information processing program for game progression includes a program related to video generation and distribution. The above programs correspond to the "programs" of the present invention and are programs for controlling the server 10. In other words, the execution of the above programs by the processor 10a causes the server 10 to function as the information processing device of the present invention.

[0055] The above-mentioned program for progressing the game may be obtained by reading it from a computer-readable recording medium, or may be obtained (downloaded) via a network such as the Internet or an intranet.

[0056] <<Functions of the information processing device according to this embodiment>> Next, functions of the information processing device according to this embodiment (hereinafter, referred to as information processing device 20) will be described with reference to FIG. As described above, the information processing device 20 is configured by the server 10, and has functions related to the progress of the game. As shown in FIG. 13, the information processing device 20 also has an operation input unit 21, a registration unit 22, and a video control unit 23 as functional units related to the virtual live video. These functional units are realized by cooperation between various hardware devices that configure the server 10 and programs installed on the server 10. Each functional unit will be described in detail below.

[0057] (Operation input section) When the user terminal 12 accepts an operation from an audience user, the operation input unit 21 receives data indicating the operation content from the user terminal 12 and acquires input information related to the user operation. The user operation indicated by the input information acquired by the operation input unit 21 includes an operation to watch a video of a virtual live performance, which includes the above-mentioned registration operation. Furthermore, when a spectator user performs a reaction action, more specifically, an action to be registered, while watching a video, and the detection device 16 detects the action to be registered, the user terminal 12 of the spectator user generates data on the detected reaction action, i.e., detected action data, and transmits it to the server 10. The operation input unit 21 receives the detected action data and acquires information on the reaction action (action to be registered) performed by the spectator user.

[0058] (Registration Department) When the user terminal 12 of the audience user accepts a registration operation, the registration unit 22 executes a registration process in response to the registration operation. In the registration process, the registration unit 22 first identifies the content of the registration operation and the audience user who performed the registration operation from the input information for the registration operation acquired by receiving the registration data via the operation input unit 21. As described above, the content of the registration operation is the music to be registered and the predetermined performance object F designated by the audience user in the registration operation, as well as the action to be registered that the audience user performed at the time of the registration operation and that was detected by the detection device 16.

[0059] The registration unit 22 then registers the content of the registration operation in association with the audience user who performed the operation. At this time, the registration unit 22 registers the predetermined effect object F, the action to be registered (reaction action) by the audience user, and the music to be registered in association with each other. As a result, new registration information is added to the storage 10d of the server 10, which is the information processing device 20.

[0060] Furthermore, if an audience user performs a registration operation for each of two or more songs sung by character C in the virtual live performance, the registration unit 22 performs the above-mentioned registration process for each song. Here, it is assumed that the audience user performs the same reaction action (hereinafter also referred to as a specific action) as the registration target action in two or more registration operations, while specifying different songs to be registered and specifying a different effect object F for each song. In this case, the registration unit 22 performs a registration process for a song corresponding to a first performance (hereinafter referred to as a first registration process) and a registration process for a song corresponding to a second performance (hereinafter referred to as a second registration process) in accordance with the registration operation received by the audience user's user terminal 12 for each song.

[0061] In the first registration process, the registration unit 22 registers a performance object F (hereinafter also referred to as the first performance object) specified in a registration operation in which a song corresponding to the first performance is the song to be registered, in association with the specific action and the song corresponding to the first performance. In the second registration process, the registration unit 22 registers a performance object F (hereinafter also referred to as the second performance object) specified in a registration operation in which a song corresponding to the second performance is the song to be registered, in association with the specific action and the song corresponding to the second performance. The song corresponding to the first performance and the song corresponding to the second performance will be performed (sung) at different times during the virtual live concert. However, there are no particular restrictions on which song is performed first, and it can be decided freely.

[0062] (Video control unit) The video control unit 23 displays video of the virtual live performance, i.e., video of at least a portion of the virtual live performance venue (virtual space), on a display. Specifically, the video control unit 23 generates data for video display (video data) and transmits the generated video data to the user terminal 12 of each audience user. In the user terminal 12 that has received the video data, the video of the virtual live performance, i.e., a video showing at least a portion of the virtual live performance venue where character C as a performer and objects corresponding to multiple audience users (more specifically, audience avatars D, cheering goods E, etc.) are arranged, is displayed on a display as a display device.

[0063] Furthermore, when an audience user who has performed a registration operation performs a registration target action while singing a song to be registered and the action is detected by the detection device 16, the video control unit 23 displays a predetermined effect object F associated with the registration target action in the video. Specifically, when the operation input unit 21 receives detected action data and acquires information about the registration target action, the video display unit 23 identifies the audience user who is the operator of the user terminal 12 that sent the detected action data.

[0064] The video display unit 23 then reads out the registration information of the audience user identified as the sender of the detected action data from the registration information stored in the storage 10d, and identifies a predetermined effect object F associated with the song to be registered and the action to be registered based on the registration information. The video display unit 23 then generates video data for displaying the identified predetermined effect object F within the video, and transmits the data to the user terminal 12 of each audience user. As a result, while the video of the character C singing the song to be registered is displayed on the user terminal 12 of each audience user, the predetermined effect object F is displayed in the audience seat area Ar2 at the position of the audience seat of the audience user who performed the action to be registered (see FIG. 4).

[0065] Furthermore, suppose that the registration unit 22 executes a first registration process for a piece of music corresponding to a first performance and a second registration process for a piece of music corresponding to a second performance in response to a registration operation from one audience user (hereinafter, a specific audience user), and that in these registration processes, a specific action that is the same reaction action is registered as a registration target action. In such a case, if the specific audience user performs a specific action during the first performance by character C and the action is detected by the detection device 16, the video control unit 23 displays, in the video, a first performance object associated with the specific action in the first registration process. Also, if the specific audience user performs a specific action during the second performance by character C and the action is detected by the detection device 16, the video control unit 23 displays, in the video, a second performance object associated with the specific action in the second registration process.

[0066] The display of the first and second performance objects will be described in more detail with reference to the example shown in FIG. 14. FIG. 14 shows the registered target actions and the predetermined performance object F for a registration target song designated by user "U" from among multiple songs sung at the virtual live performance, assuming that the user "U" is a specific audience user. Here, the registration target songs designated by U are songs M1 and M2, which are performances performed (sung) by character C at different times during the virtual live performance, with song M1 corresponding to the first performance and song M2 corresponding to the second performance. Furthermore, the registration target actions R1 and R2 performed by U are both the same reaction action, for example, raising both hands, which corresponds to a specific action. Furthermore, the predetermined performance objects F designated by U are a first performance object F1 (e.g., a predetermined message) and a second performance object F2 (e.g., a predetermined mark), which are different types of performance objects. In the above case, if U performs action R1 while character C is singing song M1 and action R1 is detected by detection device 16, video control unit 23 displays first performance object F1 within the video, specifically at the position of U's seat in audience seat area Ar2. After song M1 has finished being sung, if U performs action R2 that is the same as action R1 while character C is singing song M2 and action R2 is detected by detection device 16, video control unit 23 displays second performance object F2 within the video, specifically at the position of U's seat in audience seat area Ar2.

[0067] <<Information Processing Flow According to This Embodiment>> Next, the information processing flow of this embodiment, specifically the information processing flow relating to video distribution of a virtual live performance, will be described. The information processing flow of this embodiment employs the information processing method of the present invention and proceeds according to the flow shown in Fig. 15. In other words, each step in the information processing flow shown in Fig. 15 (specifically, each step) corresponds to a component of the information processing method of the present invention. The information processing flow described below is merely an example, and unnecessary steps may be deleted, new steps may be added, or the order in which steps are performed may be changed, without departing from the spirit of the present invention.

[0068] In the information processing flow, a processor 10a of a server 10 constituting an information processing device 20 and a processor 12a of a user terminal 12 operated by each audience user each execute corresponding processes. Data communication is performed between the server 10 and the user terminal 12 via a network 14.

[0069] The information processing flow is triggered to start, for example, when the time in the real space reaches the virtual live performance start time set in the time schedule. In the information processing flow, first, the processor 10a of the server 10 distributes video of the virtual live performance to each audience user (S001). Specifically, the processor 10a generates video data showing at least a portion of the virtual live performance venue where a character C as a performer and audience objects (more specifically, audience avatars D and cheering goods E) corresponding to multiple audience users are arranged, and transmits the video data to the user terminal 12 of each audience user. The user terminal 12 of each audience user receives and expands the video data, thereby displaying the video on the display (S002). This allows each audience user to watch the video of the virtual live performance through the user terminal 12.

[0070] Meanwhile, while viewing the video, each audience user can operate the user terminal 12 to perform the above-mentioned registration operation (S003). In the registration operation, the audience user specifies the music to be registered and a predetermined effect object F, and performs a reaction action as a registration target action, and the detection device 16 detects the registration target action. When the processor 12a of the user terminal 12 accepts the registration operation (S004), it transmits registration data regarding the content of the operation (i.e., the specified music to be registered and the predetermined effect object F, and the detected registration target action) to the server 10 (S005).

[0071] When the server 10 receives the registration data (S006), the processor 10a of the server 10 associates and registers the specified song to be registered and the specified performance object F with the detected action to be registered based on the registration data (in other words, in response to the registration operation accepted by the user terminal 12) (S007). It should be noted that steps S006 and S007 are performed before the singing of the song to be registered begins in the virtual live performance.

[0072] Then, during the period when the song to be registered is being sung in the virtual live performance, if an audience user who has performed the registration operation performs an action to be registered while singing the song to be registered (S008), the action is detected by the detection device 16, and the audience user's user terminal 12 transmits detected action data indicating the detected action to be registered to the server 10 (S009).

[0073] When the server 10 receives the detected motion data (S010), the processor 10a of the server 10 displays, in a moving image, a predetermined effect object F associated with the motion to be registered indicated by the detected motion data. Specifically, the processor 10a distributes a moving image displaying the predetermined effect object F associated with the music to be registered and the motion to be registered (S011). On the user terminal 12 of each spectator user, as shown in FIG. 4, the predetermined effect object F is displayed in the moving image, specifically, at the seat position of the spectator user who performed the motion to be registered in the spectator seat area Ar2 (S012).

[0074] The above series of steps are repeated until the video distribution of the virtual live event is completed (S013). Furthermore, suppose that one audience user (specific audience user) performs a registration operation for each song, and registers the same reaction action (specific action) as a registration target action for each of a first song corresponding to a first performance and a second song corresponding to a second performance. In this case, when the specific audience user performs the specific action while singing the first song and the action is detected by detection device 16, processor 10a of server 10 displays a first performance object associated with the first song and the specific action within the video. Also, when the specific audience user performs the specific action while singing the second song and the action is detected by detection device 16, processor 10a displays a second performance object associated with the second song and the specific action within the video. In this case, the second performance object may be a performance object different from the first performance object or may be the same as the first performance object. Then, when the virtual live ends and the video distribution ends, the information processing flow ends.

[0075] <<Regarding other embodiments of the present invention>> The information processing device, information processing method, program, and information processing system of the present invention have been described above using one specific example. However, the above-described embodiment is merely an example given to facilitate understanding of the present invention, and does not limit the present invention. Embodiments (modifications) different from the above-described embodiment may also be considered. Modifications of the present invention will now be described.

[0076] (Modifications of the configuration of the information processing device) In the above embodiment, the server 10 functions as the information processing device 20, but some of the functions of the server 10 may be performed by the user terminal 12 of the audience user. In that case, the information processing device of the present invention is configured by the cooperation of the server 10 and the user terminal 12.

[0077] (Modifications for the display) In the above embodiment, multiple spectator users individually watch the video on the display of their own user terminal 12. However, this is not limited to this, and for example, spectator users may go to a specific venue or store in real space and watch the virtual live video displayed on a display device installed in the venue (for example, a large-screen monitor used for public viewing or the like). The present invention can also be applied to cases where a single audience user watches a video of a virtual live performance alone.

[0078] (Modifications regarding association between reaction actions and performance objects) In the above embodiment, one specified performance object F is associated with and registered as one registration target action, but the number of specified performance objects F that can be associated with and registered as one registration target action may be two or more. Furthermore, the association between the registration target action and a specified performance object F that each audience user can register for one performance (e.g., a song to be registered), i.e., the combination of the registered reaction action and performance object, is not limited to one, but may be two or more.

[0079] (Modification of units for registering reaction actions and performance objects) In the above embodiment, it is assumed that reaction actions and predetermined effect objects can be registered in association with each other for each song. However, this is not limited to this, and reaction actions and predetermined effect objects can also be registered in association with each other for each section that makes up a song. In other words, an audience member can, for example, specify a section in a song using the registration operation panel P shown in FIG. 16 and register a reaction action and a predetermined effect object F for that section in association with each other.

[0080] To explain more specifically, for example, suppose an "A-melody" is designated as a section that takes place in a first period in a song, and a "chorus" is designated as a section that takes place in a second period different from the first period in the same song. In this case, for the A-melody, the audience user can register the effect object F to be displayed during the A-melody in association with the reaction action (action to be registered) that will trigger the display of that effect object F. Also, for the chorus, the audience user can register the effect object F to be displayed during the chorus in association with the reaction action (action to be registered) that will trigger the display of that effect object F.

[0081] (Variations regarding the timing of registration) In the above embodiment, before the song to be registered, which is the performance to be registered, is sung, a registration operation is performed for a predetermined effect object F to be displayed while the song is being sung. That is, in the above embodiment, before the character C sings the song to be registered in a virtual live performance, the predetermined effect object F and a reaction action that triggers the display of the effect object F are associated and registered. However, this is not limited to this, and the registration operation for the effect object F to be displayed while the song to be registered may be performed while the song is being sung. In this case, while the character C is singing the song to be registered, the audience user's user terminal 12 accepts an operation to designate the effect object F, i.e., a registration operation. The user terminal 12 transmits registration data related to the content of the accepted registration operation to the server 10, which constitutes the information processing device 20. Then, the processor 10a of the server 10 (more specifically, the aforementioned registration unit 22) registers the specified performance object F specified in the registration operation in association with the song to be registered that is being sung at that time (i.e., the performance that is being performed) and the action to be registered of the audience user that is detected by the detection device 16 at the time of the registration operation.

[0082] (Variations regarding registration contents) In the above embodiment, a reaction action (registration target action) performed by an audience user and detected by the detection device 16 is registered in association with a predetermined effect object F. Also, in the above embodiment, when an audience user performs a registration target action while singing a song to be registered, the predetermined effect object F is displayed in the video. However, the content associated with the predetermined effect object F is not limited to the above reaction action, and may further include, for example, the voice of the audience user. In this case, the audience user performs the registration target action and utters the registered voice in order to display the predetermined effect object F in the video while singing a song to be registered.

[0083] More specifically, when a registration operation by an audience user is accepted by the user terminal 12, the processor 10a of the server 10 (more specifically, the aforementioned registration unit 22) may, in accordance with the registration operation, associate and register a predetermined effect object F, the audience user's reaction action detected by the detection device 16, and the audience user's voice. In this case, the audience user's voice may be collected by the microphone 12f mounted on the user terminal 12, where the voice is produced by the audience user when the registration operation is performed. The collected audience user's voice may be converted into data as part of the details of the registration operation and transmitted to the server 10 as registration data.

[0084] In the above modification, suppose that while character C is performing (specifically, while singing the song to be registered), an audience member performs a registration target action, and the action is detected by detection device 16. The audience member also produces a sound that is the same as the registered sound, and the sound is collected by microphone 12f. In this case, processor 10a of server 10 (more specifically, the aforementioned video control unit 22) may display a predetermined effect object F associated with the registration target action and sound of the audience member within the video.

[0085] (Modifications regarding registered users) In the above embodiment, the audience user who performed the registration operation can use the registered content. Specifically, while singing the song to be registered, the audience user performs the action to be registered, and when the action is detected by the detection device 16, a predetermined effect object F associated with the action to be registered is displayed in the video. However, this is not limited to this, and audience users other than the audience user who performed the registration operation (hereinafter referred to as the first user) may also be able to use the registered content.

[0086] Specifically, suppose that a plurality of audience users watching the same virtual live video includes a second user who is a friend of the first user, and a third user who is not a friend of the first user. In response to a registration operation received by the user terminal 12 of the first user, the processor 10a of the server 10 (more specifically, the aforementioned registration unit 22) associates and registers the registration target song and predetermined effect object F designated by the first user with the registration target action (hereinafter also referred to as specific action Z) of the first user detected by the detection device 16. Meanwhile, the second and third users have user terminals 12 and detection devices 16 with the same configurations as those used by the first user.

[0087] In the above configuration, when a second user performs a specific motion Z while singing a song to be registered that was designated by a registration operation by a first user and the specific motion Z is detected by the detection device 16, a certification process is executed by the certification unit 24. As shown in FIG. 17 , the certification unit 24 is one of the functions of the information processing device 20 that the server 10 implements, and like the other functions, is realized by cooperation between various hardware devices that configure the server 10 and programs installed on the server 10. The certification unit 24 reads out and refers to the user information of the second user, whose specific motion Z was detected by the detection device 16, from the storage 10d, and certifies (determines) whether or not the second user is in a friend relationship with the first user.

[0088] Then, when the second user is recognized as being in a friend relationship with the first user, the processor 10a of the server 10 (more specifically, the video control unit 23 described above) displays in the video a predetermined effect object F associated with the specific action Z. More specifically, as shown in Fig. 18, the predetermined effect object F designated by the first user in the registration operation is displayed in the spectator seat area Ar2 at the position of the spectator seat of the second user who performed the specific action Z.

[0089] In addition, in order to share the registration operation of the first user and utilize the association between the specific action Z and the specified performance object F registered based on that registration operation, the second user may copy the registration content of the first user and perform a registration operation to make the copied registration content his or her own registration content.

[0090] Specifically, suppose that a first user performs a registration operation, and the detection device 16 detects the first user's registration target action while singing a song to be registered, causing a predetermined effect object F designated by the first user to be displayed in the video. At this time, when a second user performs a predetermined screen operation (e.g., a tap operation) on the predetermined effect object F, an operation window W shown in FIG. 19 is displayed on the display of the second user's user terminal 12, superimposed on the video, for example, near the display position of the predetermined effect object F. The operation window W displays the specific action Z and the contents of the predetermined effect object F registered based on the first user's registration operation, as well as a message inquiring the second user about whether or not to use (copy) the registered contents. The second user can then copy and use the registered contents of the first user by performing a predetermined operation (e.g., tapping a predetermined button) on the operation window W. A second user who copies the registration details of the first user can, by performing a specific action Z while singing the song to be registered, display in the video a performance object that is the same as a predetermined performance object F that the first user specified in the registration operation. Incidentally, when copying the registered contents of the first user, only a part of the registered contents may be copied, for example, only the predetermined effect object F designated by the first user may be copied and used. In this case, the second user may separately perform a registration operation for the reaction action to display the predetermined effect object F, and the reaction action may be newly registered based on that operation.

[0091] On the other hand, if a third user performs a specific action Z while singing the song to be registered designated by the first user and the specific action Z is detected by the detection device 16, the certification unit 24 certifies that the third user is not in a friend relationship with the first user. In this case, the processor 10a of the server 10 (more specifically, the above-mentioned video control unit 23) does not display a predetermined effect object F associated with the specific action Z in the video, and strictly speaking, does not display the effect object F at the position of the third user's audience seat in the audience seat area Ar2.

[0092] In the above-described modified example, whether or not the registered content can be shared (copied) based on the operation of the first user, i.e., whether or not the association between the registered specific action Z and the predetermined effect object F can be used, is determined depending on whether or not there is a friend relationship with the first user, but this is not limiting. For example, whether or not the registered content can be shared may be determined based on the physical distance to the first user in real space, or the distance to the first user's audience avatar D in the virtual live venue (more specifically, the distance between the audience seats in the audience seating area Ar2).

[0093] It should be noted that, for the multiple embodiments including the above-described modified examples, the elements and features of the respective embodiments may be combined as appropriate. In other words, the configuration of any of the multiple embodiments described above may be applicable to the other embodiments, and the effects brought about by the configuration may be similarly achieved.

[0094] [summary] One object of the present invention is to simplify the operation for a user watching a video to display a desired effect on the video. Another object of the present invention is to improve the ease of operation for users watching videos in a virtual space in which performing characters, audience members, and corresponding objects are placed to obtain a predetermined effect in the video. More specifically, the object is to solve the problems corresponding to [Appendix 1] to [Appendix 11] shown below.

[0095] [Appendix 1] One of the objectives of the present invention is to solve the problem corresponding to Appendix 1, specifically, to enable users watching a video to more easily display predetermined reactions to performances in the video within the video. In order to solve the above problem, [Appendix 1] has the following configuration. An information processing device comprising: a video control unit that causes a display to display at least a portion of a video of a virtual space in which a character performing a performance and an object corresponding to a user watching the performance are placed; and a registration unit that registers a predetermined performance object in association with a user's action detected by a detection device in accordance with a user's operation received by the user's terminal; and when the registered user's action is detected by the detection device while the character is performing a performance, the video control unit causes the predetermined performance object to be displayed within the video. According to the above configuration, when a user performs a pre-registered action while a character is performing a performance, the action is detected by the detection device, and a predetermined effect object associated with the action can be displayed within the video. This improves the ease with which a predetermined effect object can be displayed within the video as a reaction to the performance. To explain in more detail, an example of an operation to display a performance object during a performance may be to select a type of performance object while watching the screen of the user terminal 12 on which the video of the performance is being played back. This operation is performed while searching for a button or the like to be operated (for example, a button corresponding to the type of performance object that the user wants to select), which is time-consuming, and if the user becomes absorbed in the operation, they may not be able to watch the video of the performance and may miss important or exciting scenes. In contrast, according to the information processing device of the present invention, the actions for displaying a specified performance object are registered in advance, so that the above-mentioned screen operations are not required during the performance, and it is possible to avoid missing the performance due to such operations.

[0096] [Appendix 2] One of the objects of the present invention is to solve the problem corresponding to Supplementary Note 2, specifically, to more easily change the performance objects displayed in a video for each performance. In order to solve the above problem, [Appendix 2] has the following configuration. The performance includes a first performance performed in a first period and a second performance performed in a second period, and the registration unit registers a first performance object in association with the user's specific action detected by the detection device and the first performance in accordance with the user's operation received by the terminal, and registers a second performance object in association with the specific action and the second performance, and when the user's specific action is detected by the detection device while the character is performing the first performance, the video control unit displays the first performance object in the video, and when the user's specific action is detected by the detection device while the character is performing the second performance, the video control unit displays the second performance object in the video. According to the above configuration, the action for displaying the first effect object in the video during the first performance and the action for displaying the second effect object in the video during the second performance are the same, which makes it easier to change the effect object displayed in the video for each performance, and can save the user, for example, the trouble of having to memorize actions (reaction actions) for displaying the video for each performance.

[0097] [Appendix 3] One of the objects of the present invention is to solve the problem corresponding to Supplementary Note 3, specifically, to improve the ease with which a user can display a desired performance object within a video. In order to solve the above problem, [Appendix 3] has the following configuration. An information processing device described in [Appendix 1] or [Appendix 2], in which after the terminal accepts an operation by the user to specify a performance object, the registration unit registers the specified performance object in association with the user's action detected by the detection device. According to the above configuration, the performance object designated by the user is registered in association with the user's motion detected by the detection device. As a result, if the user performs the registered motion during a performance, the performance object desired by the user can be more easily displayed in the video.

[0098] [Appendix 4] One of the objects of the present invention is to solve the problem corresponding to Supplementary Note 4, specifically, to display a desired rendering object for a user in a video at an appropriate timing. In order to solve the above problem, [Appendix 4] has the following configuration. An information processing device described in any of [Appendix 1] to [Appendix 3], in which, after the terminal accepts an operation by the user to specify a performance and a performance object, the registration unit registers the specified performance object in association with the specified performance and the user's action detected by the detection device. According to the above configuration, a performance object designated by a user is registered in association with a performance designated by the user and the user's action detected by the detection device. As a result, if the user performs the registered action during a performance, the specified performance object can be displayed in a video of a performance suitable for displaying the performance object (for example, a performance with content that matches the performance object). In other words, the performance object desired by the user can be displayed in the video at the appropriate timing.

[0099] [Appendix 5] One of the objects of the present invention is to solve the problem corresponding to Appendix 5, specifically, to improve versatility when registering a specific performance object in association with an action for displaying that performance object. In order to solve the above problems, [Appendix 5] has the following configuration. An information processing device described in any of [Appendix 1] to [Appendix 4], in which, after the terminal accepts an operation by the user to specify a performance object while a character is performing a performance, the registration unit registers the specified performance object in association with the performance being performed and the user's actions detected by the detection device. According to the above configuration, a user's designation of a performance object is accepted during a performance, and the performance object is registered in association with the performance and the user's actions detected by the detection device. This allows a specific performance object to be registered in association with an action for displaying the performance object even during the performance, improving the versatility of the registration procedure. As a result, even if a user forgets to perform a registration operation before the performance begins, the performance object can be registered during the performance—specifically, before the time the specific performance object should be displayed—and the performance object can be displayed in the video at the appropriate timing during the performance.

[0100] [Appendix 6] One of the objects of the present invention is to solve the problem corresponding to Supplementary Note 6, specifically, to display a predetermined performance object at an appropriate timing during a performance. In order to solve the above problems, [Appendix 6] has the following configuration. An information processing device described in any of [Appendix 1] to [Appendix 5], wherein the registration unit associates and registers a predetermined performance object with the user's actions detected by the detection device and the user's voice collected by the sound collection device in accordance with the user's operations received by the terminal, and when the registered actions are detected by the detection device and the registered voice is collected by the sound collection device while the character is performing, the video control unit displays the predetermined performance object within the video. According to the above configuration, a predetermined effect object, a user's action, and a user's voice are registered in association with each other, and the predetermined effect object is displayed in the video on the condition that the registered action is detected and the registered voice is collected during the performance. This makes it possible to avoid a situation where, for example, a user unintentionally (by mistake) performs a registered action during the performance, which would cause the predetermined effect object to be displayed at a timing different from the original display timing.

[0101] [Appendix 7] One of the objectives of the present invention is to solve the problem corresponding to Appendix 7, specifically, to enable each of multiple users watching the same performance to more easily display a specified performance object within the video. In order to solve the above problem, [Appendix 7] has the following configuration. An information processing device as described in any of [Appendix 1] to [Appendix 6], in which objects corresponding to multiple users watching the character's performance are placed in the virtual space, and the multiple users include a first user and a second user who has a predetermined relationship with the first user, and in which the registration unit associates and registers a predetermined performance object with a specific action of the first user detected by a detection device in accordance with an operation received by the first user's terminal, and when a specific action of the second user is detected by the detection device while the character is performing a performance, the video control unit displays the predetermined performance object in the video. According to the above configuration, a predetermined effect object and a specific action of the first user are associated and registered based on the operation of the first user. After that, when a specific action by a second user who has a predetermined relationship with the first user is detected during the performance, the predetermined effect object is displayed in the video. In other words, any user (second user) who has a predetermined relationship with the first user can share (use) the registration details of the first user. This eliminates the need for the second user to specify the predetermined effect object again, saving the effort and making it easier to display the predetermined effect object in the video.

[0102] [Appendix 8] One of the objects of the present invention is to solve the problem corresponding to Supplementary Note 8, and specifically, to realize an appropriate configuration for utilizing registered content based on the operation of the first user. In order to solve the above problem, [Appendix 8] has the following configuration. An information processing device as described in [Appendix 7], in which, when the multiple users further include a third user who does not have a predetermined relationship with the first user, even if a specific action of the third user is detected by the detection device while a performance is being performed by a character, the video control unit does not display a predetermined performance object in the video. According to the above configuration, a third user who does not have a predetermined relationship with the first user cannot share (use) the first user's registered content, and a predetermined effect object is not displayed in the video even if a specific action is detected. This makes it possible to avoid a situation in which the first user suffers a disadvantage due to unlimited use (copying) of the registered content based on the first user's operation or careless use of the registered content.

[0103] [Appendix 9] One of the objectives of the present invention is to solve the problem corresponding to Appendix 9, specifically, to provide an information processing method that enables a user watching a video to display a predetermined reaction to a performance in the video within the video with simpler operations. In order to solve the above problems, the information processing method of [Supplementary Note 9] has the following configuration. An information processing method in which a processor displays on a display at least a portion of a video in a virtual space in which a character performing a performance and an object corresponding to a user watching the performance are placed, the processor associates and registers a predetermined performance object with a user's action detected by a detection device in accordance with a user's operation received by the user's terminal, and when the registered user action is detected by the detection device while the character is performing a performance, the processor displays the predetermined performance object in the video. According to the above method, while a character is performing a performance, the user performs a pre-registered action, and the action is detected by the detection device, allowing a specified performance object to be displayed within the video, thereby improving the ease of displaying a specified performance object within the video in response to the performance.

[0104] [Appendix 10] One of the objects of the present invention is to solve the problem corresponding to Appendix 10, specifically to provide a program that enables a user watching a video to display a predetermined reaction to a performance in the video within the video with simpler operations. In order to solve the above problem, the program of [Appendix 10] has the following configuration. A program that causes a processor to execute the following processes: a process of displaying on a display at least a portion of a video in a virtual space in which a character performing a performance and an object corresponding to a user watching the performance are placed; a process of registering a predetermined performance object in association with a user's action detected by a detection device in accordance with a user's operation received by the user's terminal; and a process of displaying the predetermined performance object in the video when the registered user action is detected by the detection device while the character is performing a performance. According to the above program, while a character is performing a performance, the user performs a pre-registered action, and when that action is detected by a detection device, a specified performance object can be displayed within the video, thereby improving the ease of displaying a specified performance object within the video in response to the performance.

[0105] [Appendix 11] One of the objects of the present invention is to solve the problem corresponding to Appendix 11, specifically to provide an information processing system that enables a user watching a video to display a predetermined reaction to a performance in the video within the video with simpler operations. In order to solve the above problem, the information processing system of [Supplementary Note 11] has the following configuration. An information processing system including a user terminal and a server capable of communicating with the terminal via a network, the information processing system comprising: a video control unit that causes a display to display at least a portion of a video of a virtual space in which characters performing a performance and objects corresponding to users watching the performance are located; and a registration unit that associates and registers a predetermined performance object with a user's action detected by a detection device in accordance with a user's operation received by the user's terminal; and when the registered user's action is detected by the detection device while the character is performing a performance, the video control unit displays the predetermined performance object within the video. According to the above information processing system, while a character is performing a performance, the user performs a pre-registered action, and the action is detected by a detection device, allowing a specified performance object to be displayed within the video, thereby improving the ease of displaying a specified performance object within the video in response to the performance. [Explanation of symbols]

[0106] 10 Servers 10a processor 10b memory 10c communication interface 10d Storage 12 User terminal 12a processor 12b memory 12c communication interface 12d Input Device 12e output device 12f Microphone 14 Network 16. Detection Device 20 Information processing equipment 21 Operation input section 22 Registration Department 23 Video control section 24 Certification Department 100 Information Processing Systems Ar1 Stage Area Ar2 Spectator Area C character D. Audience avatar E Support Goods F. Production Objects F1 First production object F2 Second production object P Registration operation panel S1 Device Selection Screen S2 Operation acceptance screen S3 Operation end screen W Operation window

Claims

1. a video control unit that displays on a display device at least a part of a video in a virtual space in which characters performing a performance and objects corresponding to users watching the performance are located; a registration unit that registers a predetermined effect object and a motion of the user detected by a detection device in association with each other in response to the user's operation received by the user's terminal, When the registered user's movement is detected by the detection device while the character is performing a performance, the video control unit displays the specified performance object within the video.

2. the performance includes a first performance performed in a first period and a second performance performed in a second period; The registration unit, in response to the user's operation accepted by the terminal, registering a first performance object in association with the specific action of the user detected by the detection device and the first performance; registering a second performance object in association with the specific action and the second performance; when the specific action of the user is detected by the detection device while the character is performing the first performance, the video control unit displays the first performance object within the video; The information processing device according to claim 1 , wherein when the specific action of the user is detected by the detection device while the character is performing the second performance, the video control unit displays the second performance object within the video.

3. The information processing device according to claim 1 , wherein after the terminal receives an operation by the user to specify a performance object, the registration unit registers the specified performance object in association with the user's action detected by the detection device.

4. The information processing device described in claim 1, wherein after the terminal accepts an operation by the user to specify a performance and a performance object, the registration unit registers the specified performance object in association with the specified performance and the user's action detected by the detection device.

5. The information processing device described in claim 1, wherein, after the terminal receives an operation by the user to specify a performance object while the character is performing a performance, the registration unit registers the specified performance object in association with the performance being performed and the user's actions detected by the detection device.

6. the registration unit registers the predetermined effect object, the user's action detected by the detection device, and the user's voice collected by the sound collection device in association with each other in response to the user's operation received by the terminal; The information processing device described in claim 1, wherein when the registered action is detected by the detection device and the registered sound is collected by the sound collection device during a performance by the character, the video control unit displays the specified performance object within the video.

7. The objects corresponding to a plurality of users who watch the performance of the character are placed in the virtual space; In a case where the plurality of users includes a first user and a second user who has a predetermined relationship with the first user, the registration unit registers a predetermined effect object in association with a specific action of the first user detected by the detection device in response to an operation received by the terminal of the first user; The information processing device according to claim 1 , wherein when the specific action of the second user is detected by the detection device while the character is performing a performance, the video control unit displays the specified performance object within the video.

8. In a case where the plurality of users further includes a third user who is not in the predetermined relationship with the first user, The information processing device according to claim 7, wherein even if the specific action of the third user is detected by the detection device while the character is performing a performance, the video control unit does not display the specified performance object within the video.

9. a processor causing a display device to display a moving image of at least a part of a virtual space in which a character performing a performance and an object corresponding to a user watching the performance are located; a processor registering a predetermined effect object and a motion of the user detected by a detection device in association with each other in response to the user's operation received by the user's terminal; An information processing method in which, when the registered user's movement is detected by the detection device while the character is performing a performance, the processor displays the specified performance object within the video.

10. A process of displaying on a display device at least a part of a moving image in a virtual space in which characters performing a performance and objects corresponding to users watching the performance are placed; a process by a processor of associating a predetermined effect object with the user's action detected by a detection device in accordance with the user's operation received by the user's terminal, and registering the associated effect object; a process of displaying the predetermined performance object within the video when the registered user's movement is detected by the detection device while the character is performing a performance; A program that causes a processor to execute the following.

11. An information processing system including a user terminal and a server capable of communicating with the terminal via a network, a video control unit that displays on a display device at least a part of a video in a virtual space in which characters performing a performance and objects corresponding to users watching the performance are located; a registration unit that registers a predetermined effect object and a motion of the user detected by a detection device in association with each other in response to the user's operation received by the user's terminal, An information processing system in which, when the detection device detects the registered user's movement while the character is performing a performance, the video control unit displays the specified performance object within the video.

Images