Distribution system, information processing device, control method and program

Abstract

To provide a distribution system, an information processing device, a control method, and a program that distribute highly interesting contents while reducing a production burden.SOLUTION: A distribution system includes a distribution device for distributing a content to be presented by drawing a space in which characters are arranged, and an information processing device for presenting the content. The distribution device sequentially transmits progress information necessary for presenting the content according to the progress of the content, determines whether to transmit status information indicating a presentation state of the content, and if it determines to transmit, transmits the status information while it is contained in the progress information. The information processing device acquires in advance drawing data used to draw the space and performance data related to the performance, constructs a space according to the received progress information, generates and presents a screen using the drawing data, and if the received progress information includes status information, generates and presents a screen drawn by applying the performance based on the performance data to the space.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a distribution system, an information processing device, a control method, and a program, and in particular to a distribution technology for binocular stereoscopic content. [Background technology]

[0002] Devices that enable binocular stereoscopic viewing, such as television sets and head-mounted displays (HMDs), are now widely available, and viewers can easily enjoy binocular stereoscopic content by using such devices (client terminals).

[0003] One type of binocular stereoscopic content that can be viewed on such a client terminal is a virtual reality live (hereinafter referred to as a VR live), which places characters (hereinafter referred to as performer characters) linked to the behavior of performers on a three-dimensional stage that imitates a venue such as a music hall or live music club, and provides the experience of watching performances such as singing and dancing performed through the performer characters from a specific position on the floor of the space. Patent Document 1 discloses a content distribution system that provides a VR live that can be viewed by multiple viewers simultaneously.

[0004] In a VR live performance that involves the playback of music such as a singing performance, not only the performers' performance but also various effects that are played in accordance with the progress of the music and performance increase the interest and immersion of the viewing experience. In other words, presenting effects that change appropriately as the VR live performance progresses leads to an improved user experience when watching the VR live performance. Such effects can include, for example, switching stages, changing lighting effects, changing spatial backgrounds, etc.

[0005] Meanwhile, performers' performances and the music they use in their performances vary widely. For example, the atmosphere and impression a song gives to its audience can change depending on its tempo, melody, lyrics, and other factors. Furthermore, even for the same song, performers may interpret the song differently, and the desired performance content may also differ from performer to performer. For this reason, in previous VR live viewing experiences, the performances performed during a live performance were individually planned in advance based on the set list of songs to be performed, the lyrics and melody progression of each song, and the performance content planned by the performers, and were realized based on performance data that specified the performance state for each frame. The performance data specifies the state of various objects involved in the performance for each frame (progression position) of the VR live performance or song, and allows the three-dimensional space in which the stage is located to be displayed in each frame when the VR live performance is viewed.

[0006] Therefore, previous content distribution for VR live performances required various data for constructing a three-dimensional space, music audio data, and performance data to be downloaded to the client device in advance. This pre-downloaded data was expanded into memory and used primarily to render the three-dimensional space during the VR live broadcast (the performance during which data related to the performers' performances was broadcast). The client device then acquired the data related to the performers' performances broadcast in real time (audio data emitted by the performers and motion data showing the performers' movements) and information identifying their progress. This information was then reflected in the performer characters, generating left-eye and right-eye screens and presenting them to the viewer along with the audio data, thereby providing a VR live viewing experience. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6688378 Summary of the Invention [Problem to be solved by the invention]

[0008] However, with the recent rapid spread of devices capable of viewing binocular stereoscopic content, the demand for binocular stereoscopic content is increasing, and the number of opportunities for VR live performances is also increasing. However, creating individual performance data as described above is time-consuming, and because the performance content varies depending on the performers' performances and songs, it is difficult to reuse performance data created for other VR live performances, making it difficult to keep up with the increasing number of VR content performances using previous methods.

[0009] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a distribution system, an information processing device, a control method, and a program that distribute highly interesting content while reducing the production burden. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, the distribution system of the present invention is a distribution system comprising a distribution device that distributes content and an information processing device that presents the content distributed by the distribution device, wherein the presentation of the content is accompanied by the display of a screen generated by drawing a space in which characters are placed, the distribution device has a transmission means that sequentially transmits progress information necessary for presenting the content according to the progress of the content, and a first determination means that determines whether to transmit status information indicating the presentation status of the content according to the progress of the content, and the information processing device has an acquisition means that, prior to presenting the content, acquires drawing data used to draw the space and performance data related to performance applicable to the space, a construction means that constructs the space based on the drawing data acquired by the acquisition means, a reception means that receives the progress information from the distribution device, a control means that controls the state of the space constructed by the construction means according to the reception of the progress information by the reception means, and a presentation means that generates and presents a screen that draws the space, a management means for managing the state of the performance being applied to the space; and A plurality of effect objects involved in the effect are arranged in the space, and each of the plurality of effect objects is classified into a plurality of effect groups based on the effect items and managed; the effect data is effect element data corresponding to each of the plurality of types of effect, and includes a plurality of effect element data each defining a change in the effect state for one of the effect groups; the state information includes information specifying the effect element data to be applied to each of the effect groups defining the effect state;The transmitting means, when it is determined by the first determining means that the status information is to be transmitted, transmits the status information by including it in the progress information, and the control means, when the status information is included in the received progress information, On the condition that the presentation state indicated by the state information is different from the presentation state managed by the management means, A performance based on performance data corresponding to the state information is applied to the space, and the presentation means, when the performance is applied by the control means, generates and presents a screen depicting the space after the application. [Effects of the Invention]

[0011] With this configuration, the present invention makes it possible to distribute highly entertaining content while reducing the production burden. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an example of a configuration of a distribution system according to an embodiment and a modification of the present invention. [Figure 2] FIG. 1 is a block diagram illustrating the hardware configuration of a performer terminal 100 according to an embodiment and a modification of the present invention. [Figure 3] FIG. 1 is a block diagram illustrating a hardware configuration of a distribution server 200 according to an embodiment and a modification of the present invention. [Figure 4] FIG. 1 is a block diagram illustrating a hardware configuration of a client terminal 300 according to an embodiment and a modification of the present invention. [Figure 5] FIG. 10 is a diagram showing an example of the data structure of various information used in the distribution system according to the embodiment and the modified example of the present invention. [Figure 6] 1 is a flowchart illustrating a transmission process executed by the performer terminal 100 according to an embodiment of the present invention. [Figure 7] 10 is a flowchart illustrating a performance presentation process executed by the client terminal 300 according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] [Embodiment] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0014] In the embodiment described below, an example of the present invention is applied to a distribution system capable of providing a viewing experience of binocular stereoscopic content (hereinafter simply referred to as VR content) featuring a performer character whose behavior is controlled based on the performer's physical movements. However, the present invention can be applied to any distribution system capable of providing a viewing experience of content that involves displaying a screen generated by drawing a space in which a character is placed.

[0015] In addition, in this specification, the term "performer character" will be described as referring to a viewed character that appears in VR content. In the following embodiment, the performer character will be described as having its behavior controlled based on the physical movements (including movements of body parts and vocalizations) actually performed by a human performer. In this embodiment, the VR content will be described as live content that provides the viewing experience of a live performance by performer characters. In other words, the VR content is content in which performer characters perform a live performance on a stage in a virtual space that simulates a live venue, and viewers watch the performance through virtual characters (hereinafter referred to as audience characters) that are placed on the floor in the space corresponding to the viewers.

[0016] In this embodiment, the VR content is configured so that a viewing experience can be provided for a predetermined period (distribution time period), and during that period, an application related to the distribution of the VR content is executed on a communication terminal used by the performer, and an application related to viewing the VR content is executed on a communication terminal used by the viewer, thereby providing the viewing experience. That is, in the distribution system of this embodiment, during the distribution time period, the physical exercises and singing performed by the performer are distributed as a live performance of the performer character, and a viewing experience is provided to viewers using the distribution system during that time period.

[0017] <<Configuration of the distribution system>> Hereinafter, a configuration of a distribution system for a viewing service that provides a viewing experience of VR content according to an embodiment of the present invention will be described with reference to FIG.

[0018] As shown in the figure, the distribution system includes a distribution server 200 that distributes VR content, a performer terminal 100 used by the performer to control the behavior (movements and vocalizations) of the performer character, and a client terminal 300 used by the viewer of the VR content. The performer terminal 100 and the client terminal 300 may both be information processing devices used by users of the service, but their roles in the distribution system (performer terminal or viewer terminal) are distinguished by, for example, the applications executed on the devices related to the use of the service. In other words, the information processing device that executes the application for live distribution (distribution application) is the performer terminal 100, and the information processing device that executes the application for live viewing of the distributed VR content (viewing application) is the client terminal 300.

[0019] 1, the performer terminal 100 used by the performer is shown as a single terminal, and the distribution server 200 involved in the distribution of VR content is shown as a single server, but the configuration of each is not limited to the example shown. That is, the device that acquires information related to the live performance and the device used for input may include multiple devices, and the server that performs various processes related to distribution may include multiple servers, for example, for each type of processing purpose or to achieve parallel distribution.

[0020] In the distribution system of this embodiment, in order to efficiently distribute VR content, data necessary for drawing the three-dimensional space corresponding to the live venue of the VR content (hereinafter referred to as drawing data), including three-dimensional models of performer characters performing the live performance and three-dimensional models of background objects such as the stage of the live venue, is downloaded in advance to client terminal 300 and performer terminal 100 prior to distribution. The information downloaded to each device (hereinafter sometimes referred to as advance information) also includes information about the song that the performer will sing and data (hereinafter referred to as production data) for productions that can be applied to the three-dimensional space corresponding to the live venue (such as changes to the stage, changes and movements of lighting, and changes to the spatial background) described below.

[0021] Therefore, while the VR content is being distributed, information for controlling the behavior of the performer character and information on the vocals and MC sounds emitted by the performer character are basically distributed via the distribution server 200, thereby enabling the presentation of the VR content on the client terminal 300. In other words, the information for controlling the behavior of these performer characters is information necessary for the presentation of the VR content on the client terminal 300 while the VR content is being distributed, and is hereinafter referred to as "progress information."

[0022] The distribution server 200 acquires progress information and stores it for distribution. In the distribution system of this embodiment, the behavior of the performer character is controlled by the performer's physical movements and vocalizations, so the distribution server 200 acquires progress information from the performer terminal 100. The performer terminal 100 is configured to be able to detect and acquire the performer's physical movements and acquire the performer's vocalizations, and composes progress information based on the acquired information and sequentially sends it to the distribution server 200 with which it is connected. Here, the progress information includes at least one of movement information indicating the performer's physical movements and audio information related to the performer's vocalizations. The progress information stored for distribution is acquired by the client terminal 300 connected to the distribution server 200 in connection with the viewing service. In this way, in the distribution system of this embodiment, the progress information sent from the performer terminal 100 is transmitted to the client terminal 300 connected to the distribution server 200, thereby realizing the provision of a VR content viewing experience on the client terminal 300.

[0023] The performer terminal 100 used by the performer is connected to a motion capture system 120 and a microphone (not shown in Fig. 1) to support body movement control and audio distribution of the performer character in the VR content. The motion capture system 120 is, for example, a group of sensors attached to the performer, and can detect body movements made by the performer and generate motion data (movement information) to be applied to the performer character. Alternatively, the motion capture system 120 can be a system that uses an imaging device to detect markers attached to the performer.

[0024] A display device 320 that displays different screens for the left and right eyes, such as an HMD, is connected to a client terminal 300 used by a viewer so that the viewer can view binocular stereoscopic VR content. In the distribution system of this embodiment, a screen related to the VR content is rendered in the client terminal 300 according to the position and posture of the display device 320 using progress information, so as to provide a viewing experience of the VR content from a viewpoint according to the head position of the viewer. In the example shown in the figure, the display device 320 is mainly shown as being separate from the client terminal 300, but the implementation of the present invention is not limited to this, and the display device 320 may be a so-called glasses-type device configured integrally with the client terminal 300.

[0025] These devices are configured to be communicatively connected via a network 400. In the example of Fig. 1, to facilitate understanding of the invention, the devices are described as being communicatively connected via a single network 400, but it should be understood that the implementation of the present invention is not limited to this. Separate networks may be provided between the communicatively connected devices.

[0026] <Hardware configuration of the performer terminal> The hardware configuration of the performer terminal 100 in the distribution system of this embodiment will be described below with reference to the block diagram of FIG.

[0027] The control unit 101 is, for example, a CPU, and controls the operation of each block of the performer terminal 100. Specifically, the control unit 101 controls the operation of each block by reading out the operation program of each block or the distribution application program recorded in the storage device 102, for example, and expanding and executing them in the memory 103.

[0028] The storage device 102 is a recording device capable of permanently retaining data, such as a non-volatile memory or HDD. The storage device 102 stores the operation programs of each block of the performer terminal 100 and the distribution application programs, as well as information such as parameters required for the operation of each block. The memory 103 is a storage device used for temporary data storage, such as a volatile memory. The memory 103 is used not only as an area for expanding the operation programs of each block, but also as a storage area for temporarily storing data output during the operation of each block.

[0029] The rendering device 104 is, for example, a rendering device such as a GPU or a graphics chip. The rendering device 104 generates a screen for the performer to check the state of the performer character and the state of the virtual space (the three-dimensional space of the live venue) while the VR content is being distributed. Basically, the rendering device 104 generates the confirmation screen based on advance information and motion information detected and acquired by the motion capture system 120. More specifically, the rendering device 104 generates the confirmation screen by arranging objects and production effects, including models of the performer characters and audience characters, in the three-dimensional space and drawing the space from a predetermined viewpoint. The confirmation screen generated by the rendering device 104 is displayed on the display device 140 via the output I / F 106.

[0030] The input I / F 105 is an interface provided in the performer terminal 100 that accepts various inputs. In the performer terminal 100 of this embodiment, the input I / F 105 acquires not only motion information indicating the performer's body movements detected and acquired by the motion capture system 120, but also information regarding operation inputs made on the input device 110, such as a keyboard or mouse. In addition, the input I / F 105 also acquires audio signals input via the microphone 130. The input I / F 105 stores information regarding the accepted input in, for example, the memory 103.

[0031] Output I / F 106 is an interface that outputs various types of information for presenting information provided by performer terminal 100. In performer terminal 100 of this embodiment, information is presented to the performer via output I / F 106 by displaying a screen showing the three-dimensional space related to the live venue on display device 140 and outputting audio related to the VR content from speaker 150.

[0032] The communication I / F 107 is a communication interface that the performer terminal 100 has with external devices. The communication I / F 107 can connect to external devices via a network 400 (whether wired or wireless) to enable data transmission and reception. The communication I / F 107 converts, for example, information input as a transmission target into data in a predetermined format and transmits it to an external device such as the distribution server 200 via the network 400. Furthermore, when the communication I / F 107 receives information from an external device via the network 400, for example, it decodes the information as necessary and stores it in the memory 103.

[0033] <Distribution server hardware configuration> Next, the hardware configuration of the distribution server 200 in the distribution system of this embodiment will be described with reference to the block diagram of FIG.

[0034] The control unit 201 is, for example, a CPU, and controls the operation of each block of the distribution server 200. Specifically, the control unit 201 controls the operation of each block by reading out an operation program for each block recorded in, for example, the storage device 202, expanding the program into the memory 203, and executing the program.

[0035] The storage device 202 is a recording device capable of permanently retaining data, such as a non-volatile memory or HDD. The storage device 202 stores information such as parameters required for the operation of each block, in addition to the operation programs of each block possessed by the distribution server 200. The memory 203 is a storage device used for temporary data storage, such as a volatile memory. The memory 203 is used not only as an area for expanding the operation programs of each block, but also as a storage area for temporarily storing data output in the operation of each block. The memory 203 includes an area for storing information for distribution (progress information). The information stored in this area is sequentially updated, for example, for each frame or each sequence of the VR content.

[0036] The communication I / F 204 is a communication interface with an external device that the distribution server 200 has. The communication I / F 204 can connect to the external device via the network 400 (whether wired or wireless) and transmit and receive data. The communication I / F 204 converts, for example, information input as a transmission target into data in a predetermined format and transmits the data to an external device, such as the connected client terminal 300, via the network 400. Furthermore, when the communication I / F 204 receives information from an external device via the network 400, for example, it decodes the information as necessary and stores the information in the memory 203.

[0037] <Hardware Configuration of Client Device> Next, the hardware configuration of each client terminal 300 in the distribution system of this embodiment will be described with reference to the block diagram of FIG.

[0038] The control unit 301 is, for example, a CPU, and controls the operation of each block of the client terminal 300. Specifically, the control unit 301 controls the operation of each block by reading out, for example, an operation program for each block or a viewing application program recorded in the storage device 302, and expanding and executing the program in the memory 303.

[0039] The storage device 302 is a recording device capable of permanently retaining data, such as a non-volatile memory or a HDD. The storage device 302 stores the operation programs of each block of the client terminal 300, the program of the viewing application, and advance information, as well as information such as parameters required for the operation of each block. The memory 303 is a storage device used for temporary data storage, such as a volatile memory. The memory 303 is used not only as an area for expanding the operation programs of each block, but also as a storage area for temporarily storing data output during the operation of each block.

[0040] The rendering device 304 is, for example, a GPU or a graphics chip. The rendering device 304 generates a screen for providing a viewing experience of the VR content during use of the service. As described above, the VR content of this embodiment provides a viewing experience of a live performance based on the viewpoint of an audience character positioned in a three-dimensional space related to the live venue corresponding to the viewer. Furthermore, by changing the viewpoint parameters according to the viewer's head movement, a viewing experience that makes the viewer feel as if they are actually at the live venue can be provided. Here, the viewer's head movement may be determined based on the sensor output of an orientation sensor 321 built into the HMD, which is the display device 320. The orientation sensor 321 may be, for example, an acceleration sensor or a gyro sensor, and the sensor output of the orientation sensor 321 is acquired via the input I / F 305 described below. Alternatively, the orientation sensor 321 may be a module that optically detects the orientation of the HMD by analyzing the distribution of feature points in images sequentially captured by an imaging device. Therefore, the rendering device 304 generates a screen by arranging objects and performance effects, including models of performer characters and audience characters, in a three-dimensional space based on progress information acquired from the connected distribution server 200, and rendering the space from a viewpoint determined based on operation inputs related to the viewer's head movement and viewpoint operation. The screen generated by the rendering device 304 is displayed on the display device 320 via the output I / F 306.

[0041] The input I / F 305 is an interface that accepts various inputs provided to the client terminal 300. In the client terminal 300 of this embodiment, the input I / F 305 not only acquires sensor output from the attitude sensor 321 provided in the display device 320, but also acquires information related to operation inputs made on the input device 310, such as a controller or keyboard. The input I / F 305 stores information related to the accepted input in the memory 303, for example.

[0042] The output I / F 306 is an interface that outputs various information for presenting information provided in the client terminal 300. In the client terminal 300 of this embodiment, a screen showing a three-dimensional space related to a live concert venue is displayed on the display device 320, and audio related to the VR content is output from the speaker 322 via the output I / F 306, to provide the viewer with a viewing experience of the VR content.

[0043] The communication I / F 307 is a communication interface with an external device that the client terminal 300 has. The communication I / F 307 can connect to the external device via a network 400 (whether wired or wireless) and enable transmission and reception of data. For example, in a mode in which the VR content has an interactive element, the communication I / F 307 can convert information input as a transmission target into data in a predetermined format and transmit it to the external device via the network 400. Furthermore, when the communication I / F 307 receives information from an external device via the network 400, it decodes the information as necessary and stores it in the memory 303.

[0044] Content distribution overview An overview of the distribution of VR content in the distribution system of this embodiment will be described below.

[0045] In one embodiment, VR content distribution is carried out according to a schedule predetermined by a distribution provider or the like. For example, a performer executes a distribution application on the performer terminal 100 prior to the distribution start time planned in the schedule, thereby communicatively connecting the performer terminal 100 to the distribution server 200. More specifically, the distribution application sends a session establishment request to the distribution server 200 to establish a session between the distribution server 200 and the performer terminal 100. During the session, the distribution application sequentially acquires progress information related to the live performance and transmits it to the distribution server 200. More specifically, while the distribution application is transitioning to a state where information transmission is possible, the distribution application generates progress information including at least one of motion information indicating the performer's body movements acquired by the motion capture system 120 and audio information of the performer acquired by the microphone 130, and transmits the progress information to the distribution server 200. The progress information transmitted to the distribution server 200 is stored in a distribution area in the memory 203.

[0046] The distribution application also causes the rendering device 104 to perform a rendering process based on the configured progress information and advance information, and presents the generated VR content to the performer by displaying it on the display device 140. The rendering process renders a three-dimensional space related to the live venue based on a viewpoint specified by the performer, for example. The performer can check the performer character during the performance, the state of the live venue, etc. on the screen displayed on the display device 140.

[0047] A viewer executes a viewing application on the client terminal 300 they use, causing the client terminal 300 to obtain information on available VR content from a management device (not shown). The viewing application presents available VR content, and when the VR content to be viewed is selected, it transmits a viewing request to the management device. Furthermore, when the viewing application receives information on the distribution server 200 to which the viewer is connected in response to the viewing request from the management device, it transmits a session establishment request to the distribution server 200, thereby establishing a session between the distribution server 200 and the client terminal 300. During the session, the viewing application sequentially obtains information stored in a distribution area in the memory 203 of the currently connected distribution server 200.

[0048] The viewing application also causes the rendering device 304 to perform a drawing process based on the acquired distribution information and advance information, and presents the generated screen (for the left eye and the right eye) of the VR content to the viewer by displaying it on the display device 320. More specifically, the rendering device 304 generates a screen of the VR content by arranging necessary objects and the like in the three-dimensional space of the live venue based on the progress information, and drawing the space for the viewpoints (for the left eye and the right eye) set based on the sensor output of the orientation sensor 321 and operation input related to a viewpoint change. The viewer can view the VR content being distributed on the screen.

[0049] To efficiently distribute such VR content, data (drawing data) of 3D models of performer characters performing live and 3D models of background objects such as a stage at a live venue can be downloaded to client terminal 300 and performer terminal 100 before distribution begins. The information downloaded to each device also includes information about the songs that the performers will sing and performance data that can be generated during content distribution.

[0050] In summary, prior to the distribution / viewing of VR content, the performer terminal 100 and the client terminal 300 download, for example, from the distribution server 200, the drawing data necessary for drawing the three-dimensional space related to the VR content, information about the music, and production data related to the production applicable to the three-dimensional space. The drawing device of each terminal uses the drawing data to construct the three-dimensional space related to the VR content, and then, based on the progress information acquired thereafter (including information for controlling the behavior of the performer character), changes the state of the corresponding object in the three-dimensional space as needed, thereby realizing an expression corresponding to the VR content within the three-dimensional space. The drawing device generates the screen displayed on the display device of each terminal by drawing the three-dimensional space constructed in this way from a specific viewpoint.

[0051] Although details are omitted, in the distribution system of this embodiment, the information downloaded in advance as information regarding the song that the performer plans to perform a singing performance is different between the performer terminal 100 and the client terminal 300. A singing performance is a performance that is premised on the combined playback of the music audio and the performer's singing audio (hereinafter referred to as vocal audio). If there is a time lag in the playback of these audio, the viewing experience will be unsatisfactory. On the other hand, in a mode where a performer's singing performance is distributed in real time as VR content, as in this embodiment, the vocal audio must be recorded and transmitted sequentially by the performer terminal 100. However, if the client terminal 300 controls the playback of the music audio and vocal audio, which are acquired at different times, individually, there is a possibility that a desirable viewing experience will not be provided. To avoid such a viewing experience, in the distribution system of this embodiment, the music audio and vocal audio are synthesized in the performer terminal 100, and the client terminal 300 is configured to stream the synthesized audio. Therefore, music data including music audio for the performer to sing is downloaded in advance to the performer terminal 100 and used by the performer to sing, while information about other music is downloaded in advance to the client terminal 300, although it does not include music data.

[0052] 〈Production presentation〉 As described above, applying 3D rendering to a VR content viewing experience is effective in enhancing the interest and immersion of the VR content viewing experience. Conventional distribution methods, in which the state of each frame of objects involved in the rendering (hereinafter referred to as rendering objects) is specified in advance and then downloaded in advance along with rendering data as rendering data and used to generate screens related to the VR content, can place an excessive burden on the VR content provider (performers and directors) in preparing for distribution. Furthermore, the data size of the rendering data specifying the state of rendering objects in each frame increases as the length of the planned performance of the VR content increases, increasing the total amount of data that must be downloaded in advance to the client terminal 300. For example, if a communication problem occurs during viewing of VR content, causing a disconnection between the client terminal 300 and the delivery server 200, various data must be re-downloaded, potentially resulting in a prolonged wait time for the viewer to resume viewing the VR content on the client terminal 300.

[0053] The delivery system of this embodiment employs a method different from conventional methods in order to easily present effects in VR content and flexibly, independent of music, etc. More specifically, the method employed in the delivery system is similar to conventional methods in that the client terminal 300 downloads performance data in advance, but differs in that the performance data is not configured for the entire VR content or the entire specific music, and therefore the data size is light.

[0054] In the distribution system of this embodiment, the production data downloaded in advance to the client terminal 300, etc., does not specify the states of all production objects in all frames to realize the production of the entire VR content, but simply specifies whether certain state transitions are to be made to some of the production objects placed in three-dimensional space. Furthermore, the production data does not specify fixed state transitions that are suited to the music used in the VR content, but specifies general state transitions that are independent of the music, and is configured to be widely usable / reusable for, for example, multiple songs, sections in a single song where the same progression appears, or singing performances by various performers.

[0055] Using the thus configured production data, the production presentation of VR content in the client terminal 300 is performed based on information (hereinafter referred to as state information) specifying the production state to be applied to the three-dimensional space that is sequentially delivered as the VR content progresses. Receipt of the state information triggers the client terminal 300 to reference the production data corresponding to the specified production state and control the state of the corresponding production object. In other words, with the method of this embodiment, the production presentation in the VR content can be controlled in response to the transmission of state information from the performer terminal 100, and this can be done without having to configure data that specifies the state transitions of all production objects in advance. In other words, the production state of each production object in the three-dimensional space related to the VR content is determined by the delivery application executed on the performer terminal 100.

[0056] In short, in the distribution system of this embodiment, instead of pre-downloading information specifying the sequential presentation status of each presentation object to the client terminal 300, status information specifying the presentation data to be applied to the three-dimensional space can be included as necessary as progress information required for presenting the VR content distributed via the distribution server 200.

[0057] In this embodiment, a plurality of effect objects are provided to be arranged in a three-dimensional space. The plurality of effect objects are, for example, classified into a plurality of effect groups according to the effect items presented by the effect objects, and the effect data is configured so that the effect state can be changed for each effect group. In the VR content of this embodiment, effect items include switching the stage, changing the lighting mode of normal lighting, changing the lighting mode of spotlights, and changing the celestial background (the background as a distant view in three-dimensional space), and one effect group is assigned to each of these. Therefore, for example, a performance group related to switching the stage includes multiple types of stage models arranged as stages in three-dimensional space. Also, for example, a performance group related to changing the lighting mode of spotlights includes spotlight light sources arranged in three-dimensional space. Also, for example, a change in the celestial background includes multiple types of textures applied to the inner surface of the celestial model arranged in three-dimensional space.

[0058] In the following description, data for transitioning a rendering object (or a group of rendering objects) belonging to one rendering group to a specific rendering state will be referred to as "rendering element data." In other words, one rendering element data specifies a change in the rendering state of a rendering object belonging to one of the rendering groups. One rendering group may have one or more possible rendering states, and one or more rendering element data may be prepared for each rendering group. Therefore, the rendering data downloaded in advance includes one or more rendering element data for each of the multiple rendering groups.

[0059] (Processing on the performer's terminal) During the distribution of VR content, the performer (or the performer's staff, etc.) can specify which production state to use for each production group. The distribution application is provided with a user interface that allows the performer to specify which production state to use for each production group. Via this user interface, the distribution application can accept changes to the production state for each production group from the performer at any time.

[0060] In the distribution system of this embodiment, the designation of the performance state of each performance group is described as being accepted based on user input via the distribution application, but the implementation of the present invention is not limited to this. For example, the designation of the performance state can be set in advance by the performer, etc., so that a predetermined state is automatically designated upon detection of a performer performing a specific physical exercise or reaching a specific time point in a song.

[0061] The distribution application manages the rendering state applied to each rendering group for the VR content being distributed. That is, the distribution application manages the rendering state of each rendering group, for example, after the distribution application is launched, and sequentially changes the state in response to user input specifying the rendering state. That is, the rendering state is also managed on the distribution application side so that the rendering applied to the rendering objects of each rendering group can be understood for the three-dimensional space constructed in the viewing application. Hereinafter, the rendering state information of each rendering group managed by the distribution application in the performer terminal 100, which is changed based on user input specifying the rendering state, will be referred to as "distribution-side state management information." In contrast, the rendering state information of each rendering group managed by the viewing application in each client terminal 300 will be referred to as "viewer-side state management information" and will be described below.

[0062] Such management of the distribution-side state management information is effective in reducing the amount of progress information to be distributed. As described above, in the distribution system of this embodiment, state information must be distributed from the performer terminal 100 to the client terminal 300 when applying a production to a three-dimensional space. Production presentation is generally not completed in a single frame but is carried out over a fixed period of time. For a production presented over such a fixed period of time, it is not necessary to continuously transmit state information throughout that period. Once the state information is transmitted once, such as when a start command is issued and received by the client terminal 300, the client terminal 300 can subsequently read the corresponding production element data and control the state of the production object. Therefore, in the distribution system of this embodiment, when there is a change in the distribution-side state management information for any production group, the distribution application is configured to transmit state information included in the progress information in order to transition the three-dimensional space of the client terminal 300 to the changed production state.

[0063] For this reason, when a user input specifying a rendering state is made, the delivery application needs to determine whether or not to change the delivery-side status management information managed for the corresponding rendering group to the rendering state specified by the user input. That is, the delivery application does not update the delivery-side status management information for the corresponding rendering group every time a user input specifying a rendering state is made, but first determines whether an update is necessary. Then, only when the delivery application determines that an update is necessary, does it change the delivery-side status management information for the corresponding rendering group based on the user input.

[0064] Here, the determination of whether or not the distribution side state management information needs to be updated may be made based on, for example, the following two types of conditions. One condition is that the corresponding effect group has already transitioned / is currently transitioning to the effect state specified by user input. This condition is met when a user input specifying the same effect state for the corresponding effect group has already been accepted, and the distribution side state management information indicates that effect state. In this case, the distribution application can determine that there is no need to update the distribution side state management information based on the new user input.

[0065] Another condition is that the corresponding effect group cannot transition to the effect state specified by user input. For example, if the effect element data specifies an effect sequence in which the state of an effect object changes sequentially over time (e.g., rotating a spotlight), transmitting state information to transition to another effect state while applying an effect sequence of effect element data related to a previously specified effect state may not provide a suitable effect. Furthermore, depending on the effect states of other effect groups, applying the specified effect state to a specific effect group may not work properly (e.g., causing display inconsistencies or unintended effect, or the effect object of the effect group being excluded from the three-dimensional space and therefore unable to be applied at all). (Hereinafter, such a condition may be referred to as "interference with other effects"). Therefore, this condition is met when the corresponding effect group is currently applying a previously specified effect state or when interference occurs with effects applied to other effect groups. In this case as well, the distribution application may determine that it is not necessary to update the distribution-side state management information based on new user input.

[0066] On the other hand, in an aspect in which state information is transmitted only when a change occurs in the distribution-side state management information for any of the rendering groups, for example, if some client terminals 300 stop receiving progress information normally and then later recover to a state where they can receive progress information normally, viewers using those client terminals 300 may not be provided with a desirable viewing experience. In other words, in the above aspect, state information is not retransmitted during a section in which the same rendering state is maintained, and a client terminal 300 that stops receiving progress information normally before that section and then becomes able to receive progress information normally during that section is provided with a viewing experience that differs from that of other viewers during that section. In other words, in such a situation, differences in the VR content viewing experiences occur among viewers, and fairness is not guaranteed.

[0067] Therefore, in order to prevent such a viewing experience from being provided for a long period of time without guaranteeing fairness, the distribution system of this embodiment controls the transmission of status information so that it is transmitted not only when a change occurs in the distribution-side status management information for any of the production groups, but also periodically at predetermined time intervals. By transmitting the status information periodically, the frequency of receiving status information is guaranteed even when, for example, a communication failure or a device failure causes a temporary inability to obtain progress information from the distribution server 200, or when viewing of VR content begins midway through distribution, etc., and therefore it becomes possible to provide the client terminal 300 with the same viewing experience as other viewers.

[0068] Furthermore, in order to provide the same VR content viewing experience as other viewers by receiving the state information once, the state information is configured to specify the information on the rendering state applied to each of all rendering groups defined for the VR content. In other words, the state information includes rendering state information indicated by the distribution-side state management information managed for all rendering groups at the time of transmission, regardless of the rendering state of each rendering group. In other words, the state information specifies information indicating which rendering element data is applied to the rendering objects belonging to each rendering group.

[0069] (Processing on the client terminal side) If the received progress information includes state information, the viewing application executed on the client terminal 300 executes processing to apply effects to the three-dimensional space based on the state information. As described above, in the distribution system of this embodiment, state information is transmitted when the performer terminal 100 determines that a change has occurred in the effect state applied to any of the effect groups, or when it determines that the periodically set transmission timing has arrived. Therefore, the state information continuously acquired by the client terminal 300 may indicate that all effect groups indicate the same effect state. Furthermore, the state information continuously acquired by the client terminal 300 may indicate that some effect groups indicate different effect states, while other effect groups indicate the same effect state. Therefore, the viewing application does not control the effect of all effect groups based on the received state information each time it receives state information. Instead, it identifies the effect states indicated by the state information that need to be applied and applies them to the corresponding effect groups. In other words, the viewing application identifies the effect states defined in the state information that have not yet been applied to the effect objects belonging to the corresponding effect group in the three-dimensional space, and performs effect control to apply them.

[0070] To achieve this type of rendering control, the viewing application, like the distribution application, manages the rendering states applied to the rendering objects of each rendering group for the three-dimensional space currently constructed in the viewing application as viewing-side state management information. That is, the rendering states managed as viewing-side state management information for each rendering group in the viewing application indicate the rendering states already applied to each rendering group based on initialization at the start of viewing or previously acquired state information, and a rendering group for which subsequently acquired state information indicates a different rendering state can be identified as a target for applying the rendering identified by the subsequently acquired state information. That is, if the acquired state information includes a rendering group associated with a rendering state different from the viewing-side state management information, the viewing application identifies that rendering group as a rendering group for which new rendering control should be performed.

[0071] It should be readily apparent that the viewing-side state management information managed by the viewing application and the distribution-side state management information managed by the distribution application do not necessarily indicate the same rendering state at the same time. The rendering-side state management information is basically updated based on user input relating to the specification of the rendering state accepted by the distribution application. On the other hand, the viewing-side state management information is updated based on state information acquired by the client terminal 300 executing the viewing application. In other words, the two information are updated based on different inputs, and there can naturally be a time lag in the timing of acquisition of that input, so they do not necessarily indicate the same rendering state for each rendering group at the same time.

[0072] When the viewing application identifies an effect group indicating an unapplied effect state in the acquired state information, it reads the effect element data corresponding to that effect state and controls the state of the effect objects in that effect group. In this way, the three-dimensional space constructed by the viewing application transitions to a state in which the effect specified by the state information is applied. The distribution application generates a display screen by causing the rendering device 304 to render the three-dimensional space after the effect has been applied, and as a result, the viewer sees VR content with the effect applied.

[0073] (Control related to production sequence) In content that progresses by including a music playback section in which music is played, such as the VR content distributed by the distribution system of this embodiment, the content provider (performer) can select various music pieces to play. However, when creating production element data for general use to reduce the data size of the pre-downloaded data to the client terminal 300, depending on the music piece selected, the changes in the production state of the production objects belonging to the production group defined in the production element data may not match the music piece. That is, when production element data defines a production sequence that transitions over time, the production sequence may match music pieces with a specific tempo (e.g., BPM), but inconsistencies may occur in music pieces with other tempos, such as the timing of the bar changes and the end of the production not aligning.

[0074] To avoid such inconsistencies, in the viewing application of this embodiment, each piece of effect element data is associated with reference tempo information indicating the tempo of the music to which the effect element data is compatible. The viewing application can also acquire tempo information for the music to be played in the music playback section from pre-downloaded information. When applying effect element data to an effect group based on the acquired state information, the viewing application determines a transition speed for the effect based on the effect element data that is compatible with the tempo of the music, based on the reference tempo information associated with the effect element data and the tempo information for the music to be played, and performs effect control based on the determined transition speed.

[0075] For example, if the reference tempo of the production element data is 100 BPM and the tempo of the music is 150 BPM, the viewing application will determine the transition speed of the production element data to be 1.5 times faster. For example, if the transition speed is 1.5 times faster, the states applied in each frame as a production sequence to the production objects belonging to the production group can be changed to the average of the states of the first and second frames of the production sequence defined by the production element data, the state of the third frame, the average of the states of the fourth and fifth frames, the state of the sixth frame, and so on.

[0076] Furthermore, when applying such effect element data adapted to the tempo of a piece of music, it is preferable that the timing at which an effect based on the effect element data starts corresponds to a beat or measure in the piece of music. Generally, effect presentation accompanying music playback can be started or switched in synchronization with the arrival of a beat or measure in the piece of music, thereby providing the viewer with a suitable effect that matches the flow of the music. For this reason, in the distribution system of this embodiment, the state information transmitted by the distribution application includes timing information for specifying the timing at which the application of effect element data based on the state information to the effect objects belonging to the corresponding effect group starts.

[0077] Timing information is information that can uniquely identify a specific time position in the VR content to be distributed, and in the distribution system of this embodiment, the elapsed time (number of frames elapsed) from the start of the distribution application is used. Timing information is included not only in the status information, but also in the motion information and audio information included in the progress information transmitted from the performer terminal 100. In this way, by including timing information in various types of information distributed in the distribution system, when each piece of information is played back on the client terminal 300 (application of motion information to the performer character, playback of audio information, application of performance element data), it becomes possible to control the playback timing of each piece of information in the manner expressed on the performer terminal 100, i.e., in the manner intended by the performer.

[0078] (Timing of effect application) Incidentally, audio information and motion information can be played back on the client terminal 300 according to the time position at which they were acquired on the performer terminal 100. For this reason, by associating the time position at which the audio information and motion information were acquired as timing information and playing back the information on the client terminal 300 at the time position in the viewing application specified by the timing information, it is possible to present information in the manner intended by the performer.

[0079] On the other hand, in an embodiment in which the time position at which application of production element data to a production group in the client terminal 300 begins based on state information is set to the time position at which a user input specifying the corresponding production state is made in the performer terminal 100, there is a possibility that the production presentation intended by the performer will not be realized. This is because, in this embodiment, in order to apply production element data in synchronization with the timing of the arrival of beats and measures of the music as described above, the user must input the production state specification in time with the beats and measures of the music in the performer terminal 100. In other words, an embodiment in which user input in time with the beats and measures of the music in the performer terminal 100 is essential to realize a suitable production presentation may impose an excessive input burden on the performer during distribution.

[0080] Therefore, unlike audio information and movement information, the viewing application of this embodiment specifies a beat or measure in the music piece determined based on the time position of the timing information included in the state information, rather than a time position specified by the timing information, and controls playback (application of corresponding performance element data) to begin at the time position corresponding to the beat or measure. In the following embodiment, the viewing application determines the time position at which to begin applying performance element data to the performance group in question based on the state information to be the time position corresponding to the beat that arrives immediately after the time position specified by the timing information included in the state information for the performance group in question.

[0081] While the time position of a beat or bar in a song can be determined by analyzing the song data, the viewing application of this embodiment derives the position based on information about the song being played in the VR content, which is acquired in advance. The information about the song includes the tempo (BPM) of the song and the time elapsed from the beginning of the song until the first bar of the song arrives (the bar offset). For example, if the distribution system of this embodiment is configured to only allow songs with simple time signatures, such as 4 / 4, the viewing application can identify the positions of all beats and bars in the song based on this information. That is, the time positions in the song of beats arriving after the time position of the first bar can be derived by sequentially adding the duration of one beat, which can be determined from the BPM of the song, to the time position of the first bar. Furthermore, the time positions of each bar in the song can be derived based on the time signature used in the song. To convert such a time position within a song into a time position during the distribution of VR content, i.e., the elapsed time from the start of the distribution application, such as timing information associated with audio information or movement information, it is sufficient to add the time position specified by the timing information associated with the start position of the music playback section for the song in question in the VR content included in the progress information.

[0082] Therefore, the effects to be applied to the three-dimensional space can be configured to be presented in synchronization with the arrival of beats in the music in the viewing application. Note that this control related to adjusting the timing of effect start is also performed when displaying the three-dimensional space in the distribution application. That is, for example, when a performer performs a user input related to specifying a performance state on the performer terminal 100, it can be confirmed that the specified performance is presented in synchronization with the arrival of beats in the music on the screen of the three-dimensional space constructed by the distribution application displayed on the display device 140.

[0083] Data Structure In the distribution system of this embodiment as described above, the presentation of effects related to the VR content being distributed is realized by using information having the following data configuration, for example.

[0084] <Basic Progress Information> The audio information and motion information that are basically distributed as progression information include acquired data 501, which is data acquired from the microphone 130 or motion capture system 120 at the performer terminal 100, and timing information 502 that indicates the time position at which the acquired data was acquired (the timing at which the acquired data should be played back), as shown in Figure 5(a). When the audio information and motion information are distributed to the client terminal 300 as progression information, they are applied to the playback / performance characters at the time position based on the timing information 502 associated with each, and are presented as VR content.

[0085] <Information about the presentation in the distribution application> In the distribution application, the distribution side status management information managed for each performance group of the VR content being distributed is composed of, as shown in Figure 5(b), a group ID 511 that uniquely identifies the performance group, an applied performance ID 512 that uniquely identifies the performance state currently applied to the performance group, and timing information 513 that indicates the time position when the user input specifying the currently applied performance state was made.

[0086] As described above, the presentation of effects related to VR content is realized by applying corresponding effect element data from the effect data pre-downloaded to the performer terminal 100 and the client terminal 300 to the effect object of the corresponding effect group. Because the process of reading and applying such effect element data only needs to be performed on each terminal, information that can uniquely identify the effect element data is sufficient when managing the applied effect state and distributing information. Therefore, in the distribution-side state management information, the effect state currently applied to the effect group is managed by the applied effect ID 512 assigned to the effect element data. The applied effect ID 512 can be simply an integer value (number), and the integer value assigned to each effect element data for the distributed VR content is predetermined and shared between the distribution application and the viewing application.

[0087] When a user inputs a designation of a designation group's designation state on the performer terminal 100, the distribution application acquires input information as shown in Fig. 5(c). As shown in the figure, the input information is associated with a group ID 521 that uniquely identifies the designation target designation group, and includes an applied designation ID 522 that indicates the designated designation state, and timing information 523 that indicates the time position at which the user input was made.

[0088] Therefore, when a user input specifying a rendering state is accepted, the delivery application references the delivery-side state management information associated with the group ID 511 that is the same as the group ID 521 of the input information to determine whether to change the delivery-side state management information. This determination may be based on whether the applied rendering ID 512 of the delivery-side state management information differs from the applied rendering ID 522 of the input information, whether the timing information 513 of the delivery-side state management information differs from the timing information 523 of the input information, or whether both differ. In addition, this determination also includes determining whether rendering interference will occur or whether the previously specified rendering state is currently in transition. When the delivery application determines to change the delivery-side state management information, it changes it by replacing each piece of information in the delivery-side state management information with the input information.

[0089] The distribution-side status management information configured in this manner is included in the status information when the status information is transmitted as progress information. The status information transmitted includes the distribution-side status management information for all production groups at the transmission timing, so the data configuration is not shown in the figure.

[0090] <Information about the presentation in the viewing application> Next, in the viewing application, the viewing side state management information managed for each performance group of the VR content being distributed is composed of, as shown in Figure 5(d), an applied performance ID 532 that uniquely identifies the performance state currently applied to the performance group, associated with a group ID 531 that uniquely identifies the performance group, and timing information 533 that indicates the time position when the user input specifying the currently applied performance state was made.

[0091] When state information is received as described above, the viewing application compares the information contained in the state information with the viewer-side state management information for each effect group, and determines whether to change the viewer-side state management information, i.e., whether to apply a new effect, based on the effect state information contained in the state information for that effect group. If the viewing application determines to change the viewer-side state management information, it changes it by replacing the applied effect ID 532 and timing information 533 in the viewer-side state management information for the corresponding effect group with the applied effect ID and timing information of the state information for that group.

[0092] For an effect group whose viewing-side state management information has been changed in this way, a process is performed to apply effects to the effect objects belonging to that effect group. At this time, the effect element data to be applied to that effect object is identified based on the applied effect ID 532 (the effect element data related to the effect group to which the same integer value is assigned). The application of the identified effect element data is controlled so that it begins at a time position identified based on timing information 533 and information about the currently playing music. Here, the information about the music includes, for example, a music ID 541 that uniquely identifies the music, a tempo 542 of the music, and a bar offset 543 indicating the time position of the first bar in the music, as shown in FIG. 5(e). When applying the effect element data, the viewing application controls the transition speed based on the reference tempo information associated with the effect element data and the tempo 542 of the information about the music.

[0093] <<Send process>> The transmission process executed in relation to the transmission of status information in the performer terminal 100 of the distribution system of this embodiment will be specifically described below using the flowchart in Figure 6. The process corresponding to this flowchart can be realized by the control unit 101 of the performer terminal 100 reading out a program related to a distribution application stored in, for example, the storage device 102, and expanding and executing it in memory 103. Note that this transmission process will be described as being executed for each frame of the VR content to be distributed.

[0094] In S601, the control unit 101 determines whether a user input has been made to specify the rendering state of any of the rendering groups. The determination in this step is made depending on whether input information related to the specification of the rendering state has been acquired. If the control unit 101 determines that a user input has been made to specify the rendering state of any of the rendering groups, it proceeds to S602; if it determines that no user input has been made, it proceeds to S606. In the explanation of this transmission process, the user input will be referred to as the "target input," the rendering state specified in the user input as the "specified rendering state," and the rendering group whose rendering state is specified by the user input as the "target group."

[0095] In S602, the control unit 101 determines whether the currently applied effect state for the target group is a designated effect state. That is, in this step, the control unit 101 determines whether the designated effect state has already been applied to the target group. If the control unit 101 determines that the currently applied effect state for the target group is the designated effect state, the control unit 101 proceeds to S606, and if it determines that the currently applied effect state is not the designated effect state, the control unit 101 proceeds to S603.

[0096] In S603, the control unit 101 determines whether the designated effect state can be applied to the target group. That is, in this step, the control unit 101 determines whether the target group can be transitioned to the designated effect state. If the control unit 101 determines that the designated effect state can be applied to the target group, the control unit 101 proceeds to S604, and if the control unit 101 determines that the designated effect state cannot be applied, the control unit 101 proceeds to S606.

[0097] In S604, the control unit 101 changes the distribution side status management information related to the target group based on the input information related to the target input.

[0098] In S605, the control unit 101 configures status information based on the distribution-side status management information for all performance groups, including the distribution-side status management information for the target group changed in S604, and controls the status information to be included in the progress information and transmitted, thereby completing the transmission process. In this way, a performance presentation corresponding to the target input can be distributed to each client terminal 300.

[0099] On the other hand, if it is determined in S601 that no target input has been made, if it is determined in S602 that the effect state currently applied to the target group is the designated effect state, or if it is determined in S603 that the designated effect state cannot be applied to the elephant group, the control unit 101 determines in S606 whether or not to transmit state information. The determination made in this step is made based on whether or not the periodically set timing for transmitting state information has arrived. If the control unit 101 determines that the state information should be transmitted, it proceeds to S607, and if it determines that the state information should not be transmitted, it completes this transmission process.

[0100] In S607, the control unit 101 configures state information based on the distribution side state management information related to all the production groups, and performs control so that the state information is included in the progress information and transmitted, thereby completing this transmission process.

[0101] "Performance presentation processing" Next, the specific processing of the effect presentation process executed in the client terminal 300 of this embodiment for presenting the effect of VR content will be described using the flowchart in Fig. 7. The processing corresponding to this flowchart can be realized by the control unit 301 of the client terminal 300 reading out, for example, a program of an appreciation application stored in the storage device 302, expanding it in the memory 303, and executing it. Note that this effect presentation process will be described as being started when progress information about the VR content to be distributed is acquired.

[0102] In S701, the control unit 301 determines whether the acquired progress information includes state information. If the control unit 301 determines that the acquired progress information includes state information, it proceeds to S702, and if it determines that the acquired progress information does not include state information, it completes this effect presentation process. In the description of this effect presentation process, the state information included in the acquired progress information will be referred to as "target state information."

[0103] In S702, the control unit 301 determines whether or not there is a rendering group for which a rendering state different from the rendering state currently applied to the three-dimensional space is specified in the target state information. The determination in this step is made by comparing the information included in the target state information for each rendering group with the viewer-side state management information for that rendering group. If the control unit 301 determines that there is a rendering group for which a rendering state different from the rendering state currently applied to the three-dimensional space is specified in the target state information, it proceeds to S703; if it determines that there is no such group, it completes this rendering presentation process. In the description of this rendering presentation process, a rendering group for which a rendering state different from the current one is specified in the target state information will be referred to as the "target group."

[0104] In S703, the control unit 301 changes the viewer-side state management information related to the target group based on the state information.

[0105] In S704, the control unit 301 starts processing to present effects related to the target group. The processing in this step is performed based on the viewer-side state management information related to the target group changed in S703. The control unit 301 reads the effect element data identified by the applied effect ID 532 in the viewer-side state management information and performs state control of the effect objects belonging to the target group. The state control is started in synchronization with the timing of playback of audio information, etc. separately acquired in the progression information, for the beat arriving immediately after the time position specified in the timing information 533 in the viewer-side state management information. Furthermore, if the applied effect element data defines an effect sequence, the control unit 301 also controls the transition speed of the effect element data based on the tempo of the currently played music and the reference tempo associated with the effect element data.

[0106] In cases where the viewing application is once again able to normally obtain progress information from the distribution server 200 after a temporary period of being unable to normally obtain progress information due to a communication failure, equipment failure, or the like as described above, the beat that arrives immediately after the time position specified in the timing information 533 of the viewer-side state management information for the target group may have already passed by the time the state information is received. In such cases, the control unit 301 may perform control so that the application of the effect element data begins at the time position corresponding to the beat, and apply the effect from the frame in the effect element data that corresponds to the time position of the currently played music.

[0107] As described above, the distribution system of this embodiment can distribute highly interesting content while reducing the production burden.

[0108] In the distribution system of this embodiment, the performance data is configured to be universally usable regardless of the song, and can be adapted to various songs by, for example, changing the transition speed. However, the performance data does not need to be limited to data that can be used to present performances for other songs, and may be, for example, data that is dedicated to a specific song. When performance data is provided for a specific song, each of the performance element data included in the performance data does not need to have a reference tempo associated with it, because the transition speed does not need to be changed during playback of the song.

[0109] In the present embodiment, the tempo 542 and the bar offset 543 are provided as information about the music, and the time position at which the application of the effect should start is determined based on this information, but the present invention is not limited to this. The time position at which the application of the effect should start may be determined, for example, by analyzing the audio information that is distributed, or by referring to information indicating a beat that is added to the audio information, etc.

[0110] In addition, in the distribution system of this embodiment, status information indicating the production status of the production group is described as being included in the progress information and transmitted, but the manner in which the information is transmitted is not limited to this. In other words, the information distributed in the distribution system does not need to be transmitted in a specific package configured as progress information; for example, audio information, movement information, and status information may each be transmitted independently. In this specification, progress information refers to all information necessary for presenting VR content on the client terminal 300, and status information refers to one form of the necessary information.

[0111] In addition, in the distribution system of this embodiment, we have described a mode in which the elapsed time from the start of the distribution application is used as timing information to synchronize various pieces of information contained in the progress information, but it should be easily understood that the time used as the basis for such synchronization control is not limited to this.

[0112] [Variation 1] In the above-described embodiment, the performer terminal 100 executes a distribution application and the client terminal 300 executes a viewing application to provide a viewing experience of VR content, but the implementation of the present invention is not limited to this. An application may be configured so that the user of the terminal on which it is installed can select whether to act as a performer or a viewer. In other words, by installing one application, the user of the terminal may be able to distribute VR content as a performer and also be able to view VR content distributed by other users as a viewer. Switching between these modes may be easily performed, for example, by selecting an operating mode (distribution mode or viewing mode) provided in the application.

[0113] [Variation 2] In the above-described embodiment, the VR content is distributed in a so-called live broadcast format, in which the performer terminal 100 acquires information in real time about the performance of the performer, composes progress information, and transmits it, but the present invention is not limited to this. The present invention transmits status information indicating the performance status of the VR content from the performer terminal 100 as progress information, and the acquisition of the status information triggers the client terminal 300 to apply the performance data to a three-dimensional space, thereby realizing a performance presentation to the viewer, and does not impose any time restrictions on the generation of the status information.

[0114] That is, the generation of a series of progress information related to VR content does not necessarily require real-time content distribution, and may be performed, for example, by pre-recording a performer's performance. Alternatively, for example, by recording a series of progress information sent during real-time content distribution in the distribution system as redistribution data, the distributed VR content can be redistributed so that viewers can view it at any time. In the latter case, for example, by recording a user input specifying a performance state in association with timing information at which the user input was made (recording the input information in FIG. 5(c)), the distribution system according to the present invention can reproduce the distribution of status information (progress information) similar to that during real-time distribution. Here, in order to reduce the data size of the redistribution data for VR content, the input information specifying the performance state can be configured to be recorded only when it is reflected in the distribution-side status management information in S604 of the transmission process.

[0115] [Variation 3] In the above-described embodiment and modified example, the content distributed by the distribution system is described as VR content that presents a three-dimensional space in which characters are arranged in a binocular stereoscopic manner, but the implementation of the present invention is not limited to this. The content that controls the presentation of the performance is not limited to that presented in a binocular stereoscopic manner or that depicting characters arranged in a three-dimensional space, and the present invention is also applicable to, for example, a mode in which characters arranged in a two-dimensional space are presented as two-dimensional content.

[0116] [Other embodiments] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0117] 100: Performer terminal, 101: Control unit, 102: Storage device, 103: Memory, 104: Drawing device, 105: Input I / F, 106: Output I / F, 107: Communication I / F, 110: Input device, 120: Motion capture system, 130: Microphone, 140: Display device, 150: Speaker, 200: Distribution server, 201: Control unit, 202: Storage device, 203: Memory, 204: Communication I / F, 300: Client terminal, 301: Control unit, 302: Storage device, 303: Memory, 304: Drawing device, 305: Input I / F, 306: Output I / F, 307: Communication I / F, 310: Input device, 320: Display device, 321: Posture sensor, 322: Speaker, 400: Network

Claims

1. A distribution system including a distribution device that distributes content and an information processing device that presents the content distributed by the distribution device, the presentation of the content is performed by displaying a screen generated by drawing a space in which the character is placed, The distribution device a transmitting means for sequentially transmitting progress information necessary for presenting the content in accordance with the progress of the content; a first determination means for determining whether or not to transmit state information indicating a presentation state of the content in accordance with the progress of the content; and The information processing device includes: an acquisition means for acquiring, prior to presentation of the content, drawing data used to draw the space and performance data related to performance applicable to the space; a construction means for constructing the space based on the drawing data acquired by the acquisition means; a receiving means for receiving the progress information from the distribution device; a control means for controlling a state of the space constructed by the construction means in response to reception of the progress information by the receiving means; a presentation means for generating and presenting a screen on which the space is drawn; a management means for managing the state of the performance being applied to the space; and A plurality of performance objects involved in the performance are arranged in the space, Each of the plurality of effect objects is managed by being classified into a plurality of effect groups based on the effect items, The performance data includes a plurality of performance element data corresponding to each of a plurality of types of performances, each of which defines a change in a performance state for one of the performance groups; The state information includes, for each of the effect groups defining the effect state, information specifying the effect element data to be applied to the effect group; the transmitting means, when the first determining means determines that the status information should be transmitted, transmitting the status information together with the progress information; when the received progress information includes the state information, the control means applies to the space a performance based on the performance data corresponding to the state information, on the condition that the performance state indicated by the state information differs from the performance state managed by the management means; The presentation means, when the application of the effect by the control means is performed, generates and presents a screen depicting the space after the application. A distribution system characterized by:

2. The distribution device further includes a determination unit for determining a rendering state of the plurality of rendering groups for the content being distributed, The first determination means determines that the state information should be transmitted when the effect group for which the change to a different effect state has been determined by the determination means exists.

2. The distribution system according to claim 1.

3. the distribution device further includes an input unit for receiving a user input specifying a performance state of the performance group; The determination means determines the presentation state of the presentation group in response to a user input specifying the presentation state of the presentation group being accepted by the input means.

3. The distribution system according to claim 2.

4. the distribution device further includes a second determination means for determining whether or not a transition to a designated rendering state is possible for the rendering group in response to a user input specifying the rendering state of the rendering group being accepted by the input means; The determining means changes the presentation state of the presentation group for which a user input specifying a presentation state has been accepted to the designated presentation state when the second determining means determines that the presentation state can be transitioned to the designated presentation state.

4. The distribution system according to claim 3.

5. the distribution device further includes a recording means for recording the user input specifying the rendering state of the rendering group received by the input means as data for re-distribution of the content in association with information indicating a time position at which the user input was made; The recording means records, as the redistribution data, only the user input determined by the second determination means to be capable of transitioning to the designated presentation state.

5. The distribution system according to claim 4.

6. The distribution system described in claim 2, characterized in that the first judgment means further judges that the status information will be sent at a predetermined time interval even if there is no performance group for which the decision means has decided to change to a different performance state.

7. A distribution system comprising: a distribution device that distributes content; and an information processing device that presents the content distributed by the distribution device, the presentation of the content is performed by displaying a screen generated by drawing a space in which the character is placed, The distribution device a transmitting means for sequentially transmitting progress information necessary for presenting the content in accordance with the progress of the content; a first determination means for determining whether or not to transmit state information indicating a presentation state of the content in accordance with the progress of the content; and The information processing device includes: an acquisition means for acquiring, prior to presentation of the content, drawing data used to draw the space and performance data related to performance applicable to the space; a construction means for constructing the space based on the drawing data acquired by the acquisition means; a receiving means for receiving the progress information from the distribution device; a control means for controlling a state of the space constructed by the construction means in response to reception of the progress information by the receiving means; a presentation means for generating and presenting a screen on which the space is drawn; and the content includes a music playback section in which music is played, the performance data defines performances that transition over time; the state information includes timing information for specifying a timing to start applying an effect based on the effect data corresponding to the state information; the transmitting means, when the first determining means determines that the status information should be transmitted, transmitting the status information together with the progress information; when the received progress information includes the state information, the control means applies to the space a performance based on the performance data corresponding to the state information; the presentation means, when the application of the effect by the control means is performed, generates and presents a screen depicting the space after the application; When the control means receives the state information for the music playback section, the control means specifies, based on the timing information included in the state information, the time position of a beat or a bar in the music played in the music playback section as the timing to start applying the effect based on the effect data corresponding to the state information. A distribution system characterized by:

8. the content includes a music playback section in which music is played, When the control means receives the state information for the music playback section, the control means determines a transition speed of the effect based on the effect data corresponding to the state information in accordance with the tempo of the music played in the music playback section.

8. The distribution system according to claim 7.

9. the timing information indicates a time position; The control means specifies, in the progression of the music played in the music playback section, the time position of the beat or measure that comes immediately after the time position indicated by the timing information as the timing to start applying the effect.

8. The distribution system according to claim 7.

10. An information processing device that presents content distributed by a distribution device, the content including a screen display generated by drawing a space in which a character is placed, an acquisition means for acquiring, prior to presentation of the content, drawing data used to draw the space and performance data related to performance applicable to the space; a construction means for constructing the space based on the drawing data acquired by the acquisition means; a receiving means for sequentially receiving progress information necessary for presenting the content from the distribution device in accordance with the progress of the content; a control means for controlling a state of the space constructed by the construction means in response to reception of the progress information by the receiving means; a presentation means for generating and presenting a screen on which the space is drawn; and the content includes a music playback section in which music is played, the performance data defines performances that transition over time; when the received progress information includes status information indicating a rendering status of the content, the control means applies to the space a rendering based on the rendering data corresponding to the status information; the presentation means, when the application of the effect by the control means is performed, generates and presents a screen depicting the space after the application; the state information includes timing information for specifying a timing to start applying an effect based on the effect data corresponding to the state information; When the control means receives the state information for the music playback section, the control means specifies, based on the timing information included in the state information, the time position of a beat or a bar in the music played in the music playback section as the timing to start applying the effect based on the effect data corresponding to the state information.

1. An information processing device comprising:

11. A control method for an information processing device that presents content distributed by a distribution device, the content including a screen display generated by drawing a space in which a character is placed, comprising: an acquisition step of acquiring, prior to presentation of the content, drawing data used to draw the space and performance data related to performance applicable to the space; a construction step of constructing the space based on the drawing data acquired in the acquisition step; a receiving step of sequentially receiving progress information necessary for presenting the content from the distribution device in accordance with the progress of the content; a control step of controlling a state of the space constructed in the construction step in response to reception of the progress information in the reception step; a presentation step of generating and presenting a screen depicting the space; and the content includes a music playback section in which music is played, the performance data defines performances that transition over time; In the control step, when the received progress information includes status information indicating a rendering status of the content, a rendering based on the rendering data corresponding to the status information is applied to the space; When the effect is applied in the control step, a screen depicting the space after the effect is applied is generated and presented in the presentation step, the state information includes timing information for specifying a timing to start applying an effect based on the effect data corresponding to the state information; In the control step, when the state information is received for the music playback section, the time position of a beat or a bar in the music played in the music playback section is identified as the timing to start applying the effect based on the effect data corresponding to the state information, based on the timing information included in the state information. A control method comprising:

12. A program that presents content distributed by a distribution device, the content including a screen display generated by drawing a space in which a character is placed, On the computer, an acquisition process for acquiring, prior to presentation of the content, drawing data used for drawing the space and performance data related to performance applicable to the space; a construction process for constructing the space based on the drawing data acquired in the acquisition process; a receiving process of sequentially receiving progress information necessary for presenting the content from the distribution device according to the progress of the content; a control process for controlling a state of the space constructed in the construction process in response to reception of the progress information in the reception process; a presentation process for generating and presenting a screen on which the space is drawn; A program for executing the content includes a music playback section in which music is played, the performance data defines performances that transition over time; In the control process, when the received progress information includes state information indicating a rendering state of the content, a rendering based on the rendering data corresponding to the state information is applied to the space; In the presentation process, when the effect is applied by the control process, a screen depicting the space after the effect is applied is generated and presented; the state information includes timing information for specifying a timing to start applying an effect based on the effect data corresponding to the state information; In the control process, when the state information is received for the music playback section, the time position of a beat or a bar in the music played in the music playback section is identified as the timing to start applying the effect based on the effect data corresponding to the state information, based on the timing information included in the state information. A program characterized by:

13. A program configured to be switchable between a distribution mode for distributing content and a viewing mode for presenting the content distributed in the distribution mode, the presentation of the content is performed by displaying a screen generated by drawing a space in which the character is placed, In the distribution mode, the computer a transmission process of sequentially transmitting progress information necessary for presenting the content in accordance with the progress of the content; a first determination process for determining whether or not to transmit state information indicating a presentation state of the content in accordance with the progress of the content; Execute In the viewing mode, the computer an acquisition process for acquiring, prior to presentation of the content, drawing data used for drawing the space and performance data related to performance applicable to the space; a construction process for constructing the space based on the drawing data acquired in the acquisition process; a receiving process for receiving the progress information; a control process for controlling a state of the space constructed in the construction process in response to reception of the progress information in the reception process; a presentation process for generating and presenting a screen on which the space is drawn; a management process for managing the performance status being applied to the space; Execute A plurality of performance objects involved in the performance are arranged in the space, Each of the plurality of effect objects is managed by being classified into a plurality of effect groups based on the effect items, The performance data includes a plurality of performance element data corresponding to each of a plurality of types of performances, each of which defines a change in a performance state for one of the performance groups; The state information includes, for each of the effect groups defining the effect state, information specifying the effect element data to be applied to the effect group; In the transmission process, when it is determined in the first determination process that the status information is to be transmitted, the status information is included in the progress information and transmitted; In the control process, when the received progress information includes the state information, a performance based on the performance data corresponding to the state information is applied to the space on the condition that the performance state indicated by the state information is different from the performance state managed in the management process, In the presentation process, when a performance is applied in the control process, a screen depicting the space after the performance is applied is generated and presented. A program characterized by:

14. A program configured to be switchable between a distribution mode for distributing content and a viewing mode for presenting the content distributed in the distribution mode, the presentation of the content is performed by displaying a screen generated by drawing a space in which the character is placed, In the distribution mode, the computer a transmission process of sequentially transmitting progress information necessary for presenting the content in accordance with the progress of the content; a first determination process for determining whether or not to transmit state information indicating a presentation state of the content in accordance with the progress of the content; Execute In the viewing mode, the computer an acquisition process for acquiring, prior to presentation of the content, drawing data used for drawing the space and performance data related to performance applicable to the space; a construction process for constructing the space based on the drawing data acquired in the acquisition process; a receiving process for receiving the progress information; a control process for controlling a state of the space constructed in the construction process in response to reception of the progress information in the reception process; a presentation process for generating and presenting a screen on which the space is drawn; Execute the content includes a music playback section in which music is played, the performance data defines performances that transition over time; the state information includes timing information for specifying a timing to start applying an effect based on the effect data corresponding to the state information; In the transmission process, when it is determined in the first determination process that the status information is to be transmitted, the status information is included in the progress information and transmitted; In the control process, when the received progress information includes the state information, a performance based on the performance data corresponding to the state information is applied to the space; In the presentation process, when a performance is applied in the control process, a screen depicting the space after the performance is applied is generated and presented, In the control process, when the state information is received for the music playback section, the time position of a beat or a bar in the music played in the music playback section is identified as the timing to start applying the effect based on the effect data corresponding to the state information, based on the timing information included in the state information. A program characterized by:

Images