Distribution system, information processing device, control method and program

Abstract

To provide a distribution system, an information processing device, a control method, and a program that distribute highly interesting contents while reducing a production burden.SOLUTION: A distribution system includes a distribution device for distributing a content to be presented by drawing a space in which characters are arranged, and an information processing device for presenting the content. The distribution device sequentially transmits progress information necessary for presenting the content according to the progress of the content, determines whether to transmit status information indicating a presentation state of the content, and if it determines to transmit, transmits the status information while it is contained in the progress information. The information processing device acquires in advance drawing data used to draw the space and performance data related to the performance, constructs a space according to the received progress information, generates and presents a screen using the drawing data, and if the received progress information includes status information, generates and presents a screen drawn by applying the performance based on the performance data to the space.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a distribution system, an information processing device, a control method, and a program, and in particular to a technology for distributing binocular stereoscopic content. [Background technology]

[0002] Devices that enable binocular stereoscopic vision, such as television sets and head-mounted displays (HMDs), are now widely available, and viewers can easily enjoy binocular stereoscopic content by using such devices (client terminals).

[0003] One of the binocular stereoscopic contents that can be viewed on such a client terminal is a virtual reality live (hereinafter referred to as a VR live), which provides an experience of viewing a performance such as singing or dancing performed through a performer character from a specific position on the floor of the three-dimensional space on a stage that imitates a venue such as a music hall or a live music venue, by placing a character (hereinafter referred to as a performer character) that is linked to the behavior of the performer. Patent Document 1 discloses a content distribution system that provides a VR live that can be viewed by multiple viewers at the same time.

[0004] In a VR live performance accompanied by the playback of music such as singing performance, not only the performance by the performers but also various effects that match the progress of the music and performance are played back, which increases the interest and immersion of the viewing experience. In other words, presenting effects that change appropriately according to the progress of the VR live performance leads to an improvement of the user experience when watching the VR live performance. Such effects can include, for example, switching of the stage, changing of lighting effects, changing of spatial background, etc.

[0005] On the other hand, there are many different types of performances by performers and songs used in performances. For example, the atmosphere and the impression given to the audience by a song can change depending on the tempo, melody, lyrics, etc. Also, even if it is the same song, the interpretation of the song may differ depending on the performer, and the desired performance content during the performance may also differ from performer to performer. For this reason, in providing the viewing experience of a previous VR live performance, the performance during the live performance was individually planned in advance based on the set list of the songs to be performed, the progression of the lyrics and melody in each song, the performance content planned by the performer, etc., and was realized based on the performance data that specified the performance state in each frame. The performance data specifies the state of various objects involved in the performance for each frame (progression position) of the VR live performance or song, and the three-dimensional space in which the stage is located can be in a state in which the corresponding performance is applied in each frame when the VR live performance is viewed.

[0006] Therefore, in the past, content distribution related to VR live performances was configured to have the client terminal download in advance various data required for constructing a three-dimensional space, music sound source data, and performance data. This data downloaded in advance is expanded in memory and is mainly used to draw the three-dimensional space during the distribution of the VR live performance (during the performance when data related to the performers' performance is distributed). The client terminal then obtains the data related to the performers' performance distributed in real time (voice data emitted by the performers and motion data showing the performers' behavior) and information specifying their progress position, and reflects this information on the performer characters to generate screens for the left and right eyes and present them to the audience together with the voice data, thereby providing a VR live viewing experience. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6688378 Summary of the Invention [Problem to be solved by the invention]

[0008] However, with the recent rapid spread of devices capable of viewing binocular stereoscopic content, the demand for binocular stereoscopic content is increasing, and the number of opportunities for VR live performances is also increasing. However, the configuration of individual performance data as described above is time-consuming, and since the content of the performance varies depending on the performance of the performers and the music, it is difficult to reuse the performance data configured for other VR live performances, so it has been difficult to respond to the increase in the number of VR content performances using the previous method.

[0009] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a distribution system, an information processing device, a control method, and a program that distribute highly interesting content while reducing the production burden. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, a distribution system of the present invention is a distribution system including a distribution device that distributes content and an information processing device that presents the content distributed by the distribution device, and the presentation of the content is accompanied by display of a screen generated by drawing a space in which a character is arranged, the distribution device has a transmission means that sequentially transmits progress information necessary for the presentation of the content according to the progress of the content, and a first determination means that determines whether to transmit status information indicating a presentation state of the content according to the progress of the content, and the information processing device transmits drawing data used for drawing the space and production data related to production applicable to the space, prior to the presentation of the content, the control means, when the first determination means determines that the status information is to be transmitted, transmits the status information together with the progress information; the control means, when the received progress information includes the status information, applies to the space a performance based on the performance data corresponding to the status information; and the presentation means, when the control means has applied the performance, generates and presents a screen depicting the space after the performance has been applied. Effect of the Invention

[0011] With this configuration, the present invention makes it possible to distribute highly entertaining content while reducing the production burden. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an example of a configuration of a distribution system according to an embodiment and a modification of the present invention. [Diagram 2] FIG. 1 is a block diagram illustrating the hardware configuration of a performer terminal 100 according to an embodiment and a modification of the present invention. [Diagram 3]FIG. 2 is a block diagram illustrating a hardware configuration of a distribution server 200 according to an embodiment and a modification of the present invention. [Figure 4] FIG. 1 is a block diagram illustrating a hardware configuration of a client terminal 300 according to an embodiment and a modification of the present invention. [Diagram 5] FIG. 2 is a diagram showing an example of the data structure of various information used in the distribution system according to the embodiment and the modified example of the present invention. [Figure 6] A flowchart illustrating a transmission process executed by the performer terminal 100 according to an embodiment of the present invention. [Figure 7] A flowchart illustrating a performance presentation process executed by the client terminal 300 according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] [Embodiment] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] In the embodiment described below, an example of the present invention is applied to a distribution system capable of providing a viewing experience of binocular stereoscopic content (hereinafter simply referred to as VR content) in which a performer character appears, whose behavior is controlled based on the body movement of the performer, as an example of a distribution system. However, the present invention is applicable to any distribution system capable of providing a viewing experience of content accompanied by the display of a screen generated by drawing a space in which a character is placed.

[0015] In addition, in this specification, a "performer character" is described as referring to a viewed character that appears in VR content. In the following embodiment, the performer character is described as having its behavior controlled based on the physical movements (including movements of body parts and vocalizations) actually performed by a human performer. In this embodiment, the VR content is described as live content that provides a viewing experience of a live performance by a performer character. That is, the VR content is content in which a performer character performs a live performance on a stage in a virtual space that mimics a live venue, and viewers view the performance through virtual characters (hereinafter referred to as audience characters) that are placed on the floor in the space corresponding to the viewers.

[0016] In this embodiment, the VR content is configured such that a period (distribution time period) during which a viewing experience can be provided is predetermined, and during that period, an application related to the distribution of the VR content is executed on a communication terminal used by the performer, and an application related to viewing the VR content is executed on a communication terminal used by the viewer, thereby providing a viewing experience. That is, in the distribution system of this embodiment, during the distribution time period, physical exercise and singing performed by the performer are distributed as a live performance of the performer character, and a viewing experience is provided to the viewer using the distribution system during that time period.

[0017] Distribution system configuration Hereinafter, a configuration of a distribution system for a viewing service that provides a viewing experience of VR content according to an embodiment of the present invention will be described with reference to FIG.

[0018] As shown in the figure, the distribution system includes a distribution server 200 that distributes VR content, a performer terminal 100 used by the performer side to control the behavior (movement and vocalization) of the performer character, and a client terminal 300 used by the viewer of the VR content. The performer terminal 100 and the client terminal 300 may both be information processing devices used by users of the service, but the roles in the distribution system (a terminal for the performer or a terminal for the viewer) are distinguished by, for example, different applications executed on the devices related to the use of the service. In other words, the information processing device on which the application for live distribution (distribution application) is executed is the performer terminal 100, and the information processing device on which the application for live viewing of the distributed VR content (viewing application) is executed is the client terminal 300.

[0019] In the example of Fig. 1, the performer terminal 100 used by the performer is shown as one terminal, and the distribution server 200 involved in the distribution of VR content is shown as one server, but the configuration of each is not limited to the example shown in the figure. That is, the device for acquiring information related to the live performance and the device used for input may include multiple devices, and the server for performing various processes related to distribution may include multiple servers, for example, for each processing purpose or to realize parallel distribution.

[0020] In the distribution system of this embodiment, in order to efficiently distribute VR content, data required for drawing a three-dimensional space corresponding to a live venue related to the VR content (hereinafter referred to as drawing data), including a three-dimensional model of a performer character performing a live performance and a three-dimensional model of a background object such as a stage related to the live venue, is downloaded in advance to the client terminal 300 and the performer terminal 100 prior to distribution. The information downloaded to each device (hereinafter sometimes referred to as advance information) also includes information on the song that the performer plans to sing, and data on production (such as changes to the stage, changes and movements of lighting, and changes to the spatial background) that can be applied to the three-dimensional space corresponding to the live venue described later (hereinafter referred to as production data).

[0021] Therefore, during the distribution of VR content, information for controlling the behavior of the performer character and information on the vocals and MC sounds emitted by the performer character are basically distributed via the distribution server 200, thereby enabling the presentation of the VR content on the client terminal 300. In other words, the information for controlling the behavior of these performer characters is information necessary for the presentation of the VR content on the client terminal 300 during the distribution of the VR content, and is hereinafter referred to as "progress information."

[0022] The distribution server 200 acquires the progress information and stores it for distribution. In the distribution system of this embodiment, the behavior of the performer character is controlled by the performer's physical movements and vocalizations, so the distribution server 200 acquires the progress information from the performer terminal 100. The performer terminal 100 is configured to be able to detect and acquire the physical movements performed by the performer and acquire the voice uttered by the performer, and configures the progress information based on the acquired information and sequentially sends it to the distribution server 200 in communication connection. Here, the progress information includes at least one of motion information indicating the physical movements performed by the performer and voice information related to the voice uttered by the performer. The progress information stored for distribution is acquired by the client terminal 300 connected to the distribution server 200 in relation to the use of the viewing service. In this way, in the distribution system of this embodiment, the progress information sent from the performer terminal 100 is transmitted to the client terminal 300 connected to the distribution server 200, and the provision of a viewing experience of VR content in the client terminal 300 is realized.

[0023] The performer terminal 100 used by the performer is connected to a motion capture system 120 and a microphone (not shown in Fig. 1) in order to support the body movement control and voice distribution of the performer character in the VR content. The motion capture system 120 is, for example, a group of sensors attached to the performer, and can detect the body movements made by the performer and generate motion data (movement information) to be applied to the performer character. Alternatively, the motion capture system 120 can be a system that detects markers attached to the performer by an imaging device.

[0024] A display device 320 that displays different screens for the left and right eyes, such as an HMD, is connected to the client terminal 300 used by the viewer so that the viewer can view binocular stereoscopic VR content. In the distribution system of this embodiment, a screen related to the VR content according to the position and attitude of the display device 320 is drawn in the client terminal 300 using progress information in order to provide a viewing experience of the VR content from a viewpoint according to the head position of the viewer. In the example of the figure, the display device 320 is mainly shown as being separated from the client terminal 300, but the embodiment of the present invention is not limited to this, and the display device 320 may be a so-called glasses-type device configured integrally with the client terminal 300.

[0025] These devices are configured to be communicatively connected via a network 400. In the example of Fig. 1, in order to facilitate understanding of the invention, the devices are described as being communicatively connected via one network 400, but it should be understood that the embodiment of the present invention is not limited to this. A network may be provided separately for each device that is communicatively connected.

[0026] <Hardware configuration of the performer terminal> The hardware configuration of the performer terminal 100 in the distribution system of this embodiment will be described below with reference to the block diagram of FIG.

[0027] The control unit 101 is, for example, a CPU, and controls the operation of each block of the performer terminal 100. Specifically, the control unit 101 reads out the operation programs of each block and the distribution application programs recorded in, for example, the storage device 102, and loads them into the memory 103 and executes them to control the operation of each block.

[0028] The storage device 102 is a recording device capable of permanently retaining data, such as a non-volatile memory or HDD. The storage device 102 stores the operation programs of each block of the performer terminal 100 and the distribution application programs, as well as information such as parameters required for the operation of each block. The memory 103 is a storage device used for temporary data storage, such as a volatile memory. The memory 103 is used not only as an area for expanding the operation programs of each block, but also as a storage area for temporarily storing data output during the operation of each block.

[0029] The rendering device 104 is, for example, a rendering device such as a GPU or a graphic chip. The rendering device 104 generates a screen for the performer to check the state of the performer character and the state of the virtual space (the three-dimensional space of the live venue) during the distribution of VR content. Basically, the rendering device 104 generates the confirmation screen based on advance information and motion information detected and acquired by the motion capture system 120. More specifically, the rendering device 104 generates the confirmation screen by arranging objects and production effects including models of the performer character and the audience character in a three-dimensional space and drawing the space from a predetermined viewpoint. The confirmation screen generated by the rendering device 104 is displayed on the display device 140 via the output I / F 106.

[0030] The input I / F 105 is an interface that accepts various inputs provided to the performer terminal 100. In the performer terminal 100 of this embodiment, the input I / F 105 acquires not only motion information indicating the performer's body motion detected and acquired by the motion capture system 120, but also information regarding operation inputs made on the input device 110 such as a keyboard or mouse. In addition, the input I / F 105 acquires audio signals input via the microphone 130. The input I / F 105 stores information regarding the accepted inputs in, for example, the memory 103.

[0031] The output I / F 106 is an interface that outputs various information for presenting information provided in the performer terminal 100. In the performer terminal 100 of this embodiment, as information presentation to the performer, a screen showing a three-dimensional space related to a live performance venue is displayed on the display device 140, and audio related to the VR content is output from the speaker 150 via the output I / F 106.

[0032] The communication I / F 107 is a communication interface with an external device that the performer terminal 100 has. The communication I / F 107 can connect to an external device via a network 400 (whether wired or wireless) to enable sending and receiving of data. For example, the communication I / F 107 converts information input as a transmission target into data of a predetermined format, and transmits it to an external device such as the distribution server 200 via the network 400. Furthermore, when the communication I / F 107 receives information from an external device via the network 400, it decodes the information as necessary and stores it in the memory 103.

[0033] <Hardware configuration of distribution server> Next, the hardware configuration of the distribution server 200 in the distribution system of this embodiment will be described with reference to the block diagram of FIG.

[0034] The control unit 201 is, for example, a CPU, and controls the operation of each block of the distribution server 200. Specifically, the control unit 201 reads out an operation program for each block recorded in, for example, the storage device 202, expands the program in the memory 203, and executes the program to control the operation of each block.

[0035] The storage device 202 is a recording device capable of permanently retaining data, such as a non-volatile memory or HDD. The storage device 202 stores information such as parameters required for the operation of each block in addition to the operation programs of each block possessed by the distribution server 200. The memory 203 is a storage device used for temporary data storage, such as a volatile memory. The memory 203 is used not only as an area for developing the operation programs of each block, but also as a storage area for temporarily storing data output in the operation of each block. The memory 203 includes an area for storing information for distribution (progress information). The information stored in this area is updated sequentially, for example, in relation to each frame or each sequence of the VR content.

[0036] The communication I / F 204 is a communication interface with an external device that the distribution server 200 has. The communication I / F 204 can connect to an external device via a network 400 (whether wired or wireless) to enable data transmission and reception. For example, the communication I / F 204 converts information input as a transmission target into data of a predetermined format, and transmits the information to an external device such as the connected client terminal 300 via the network 400. Furthermore, when the communication I / F 204 receives information from an external device via the network 400, it decodes the information as necessary and stores the information in the memory 203.

[0037] Hardware Configuration of Client Device Next, the hardware configuration of each client terminal 300 in the distribution system of this embodiment will be described with reference to the block diagram of FIG.

[0038] The control unit 301 is, for example, a CPU, and controls the operation of each block of the client terminal 300. Specifically, the control unit 301 reads out, for example, an operation program for each block recorded in the storage device 302 and a program for an appreciation application, and loads the program into the memory 303 and executes the program to control the operation of each block.

[0039] The storage device 302 is a recording device capable of permanently retaining data, such as a non-volatile memory or a HDD. The storage device 302 stores information such as parameters required for the operation of each block, in addition to the operation programs of each block possessed by the client terminal 300, the program of the viewing application, and advance information. The memory 303 is a storage device used for temporary data storage, such as a volatile memory. The memory 303 is used not only as an area for expanding the operation programs of each block, but also as a storage area for temporarily storing data output in the operation of each block.

[0040] The drawing device 304 is, for example, a drawing device such as a GPU or a graphic chip. The drawing device 304 generates a screen for providing a viewing experience of the VR content during the use of the service. As described above, the VR content of this embodiment provides a viewing experience of a live performance based on the viewpoint of an audience character arranged in a three-dimensional space related to a live venue corresponding to the viewer. In addition, by changing the parameter of the viewpoint according to the movement of the viewer's head, it is possible to provide a viewing experience as if the viewer were at the live venue. Here, the movement of the viewer's head may be specified based on the sensor output of a posture sensor 321 built into the HMD, which is the display device 320. The posture sensor 321 may be, for example, an acceleration sensor or a gyro sensor, and the sensor output of the posture sensor 321 is acquired via an input I / F 305 described later. Alternatively, as the posture sensor 321, a module that analyzes the distribution of feature points in images sequentially captured by an imaging device and optically detects the posture of the HMD may be used. Therefore, the rendering device 304 generates a screen by arranging objects and production effects, including models of performer characters and audience characters, in a three-dimensional space and rendering the space from a viewpoint determined based on operation inputs related to the head movements and viewpoint operations of the audience. The screen generated by the rendering device 304 is displayed on the display device 320 via the output I / F 306.

[0041] The input I / F 305 is an interface that accepts various inputs provided to the client terminal 300. In the client terminal 300 of this embodiment, the input I / F 305 not only acquires sensor output from the attitude sensor 321 provided in the display device 320, but also acquires information regarding an operation input performed on an input device 310 such as a controller or a keyboard. The input I / F 305 stores the information regarding the accepted input in, for example, the memory 303.

[0042] The output I / F 306 is an interface that outputs various information for presenting information provided in the client terminal 300. In the client terminal 300 of this embodiment, a screen showing a three-dimensional space related to a live venue is displayed on the display device 320, and audio related to the VR content is output from the speaker 322 via the output I / F 306, in order to provide the viewer with a viewing experience of the VR content.

[0043] The communication I / F 307 is a communication interface with an external device that the client terminal 300 has. The communication I / F 307 can connect to an external device via a network 400 (whether wired or wireless) and enable data transmission and reception. For example, in a mode in which the VR content has an interactive element, the communication I / F 307 can convert information input as a transmission target into data of a predetermined format and transmit it to an external device via the network 400. Furthermore, when the communication I / F 307 receives information from an external device via the network 400, it decodes the information as necessary and stores it in the memory 303.

[0044] Content distribution overview Below, an overview of VR content distribution in the distribution system of this embodiment will be described.

[0045] In one embodiment, the distribution of VR content is carried out according to a schedule determined in advance by a distribution company or the like. For example, a performer executes a distribution application on the performer terminal 100 prior to the distribution start time planned in the schedule, thereby communicatively connecting the performer terminal 100 to the distribution server 200. More specifically, the distribution application transmits a session establishment request to the distribution server 200 to establish a session between the distribution server 200 and the performer terminal 100. In the session, the distribution application sequentially acquires progress information related to the live performance and transmits it to the distribution server 200. More specifically, during a period in which the distribution application is in a transition state in which information transmission is possible, the distribution application configures progress information including at least one of motion information indicating the performer's body motion acquired by the motion capture system 120 and audio information of the performer acquired by the microphone 130, and transmits the progress information to the distribution server 200. The progress information transmitted to the distribution server 200 is stored in an area for distribution in the memory 203.

[0046] The delivery application also causes the rendering device 104 to perform a rendering process based on the configured progress information and advance information, and presents the generated VR content to the performer by displaying the screen on the display device 140. The rendering process renders a three-dimensional space related to a live venue based on a viewpoint specified by the performer, for example. The performer can check the performer character during the performance, the state of the live venue, and the like, on the screen displayed on the display device 140.

[0047] A viewer executes a viewing application on the client terminal 300 to be used, causing the client terminal 300 to acquire information on viewable VR content from a management device (not shown). The viewing application presents viewable VR content, and when a VR content to be viewed is selected, transmits a viewing request to the management device. Furthermore, when the viewing application receives information on the distribution server 200 to which the viewer is to connect in response to the viewing request from the management device, the viewing application transmits a session establishment request to the distribution server 200, and establishes a session between the distribution server 200 and the client terminal 300. During the session, the viewing application sequentially acquires information stored in an area for distribution in the memory 203 of the distribution server 200 to which the viewer is connected.

[0048] The viewing application also causes the rendering device 304 to perform a drawing process based on the acquired information for distribution and advance information, and presents the generated screen (for the left eye and the right eye) related to the VR content to the viewer by displaying it on the display device 320. More specifically, the rendering device 304 generates a screen related to the VR content by arranging necessary objects, etc. in the three-dimensional space related to the live venue based on the progress information, and drawing the space for the viewpoints (for the left eye and the right eye) set based on the sensor output of the orientation sensor 321 and operation input related to a change of viewpoint. The viewer can view the VR content being distributed on the screen.

[0049] In order to efficiently distribute such VR content, data (drawing data) of 3D models of performer characters performing live and 3D models of background objects such as a stage at a live venue can be downloaded to the client terminal 300 and the performer terminal 100 before distribution. The information downloaded to each device also includes information about the songs that the other performers will be singing and performance data that can be generated during content distribution.

[0050] In summary, before the delivery / viewing of VR content, the performer terminal 100 and the client terminal 300 download, for example, from the delivery server 200, drawing data required for drawing the three-dimensional space related to the VR content, information about the music, and performance data related to performance applicable to the three-dimensional space. The drawing device of each terminal uses the drawing data to construct the three-dimensional space related to the VR content, and further realizes an expression corresponding to the VR content in the three-dimensional space by changing the state of the corresponding object in the three-dimensional space as needed based on the progress information (including information for controlling the behavior of the performer character) acquired thereafter. The drawing device generates a screen to be displayed on the display device of each terminal by drawing the three-dimensional space constructed in this way from a specific viewpoint.

[0051] Although details are omitted, in the delivery system of this embodiment, the information downloaded in advance as information on the song that the performer is scheduled to perform a singing performance is different between the performer terminal 100 and the client terminal 300. A singing performance is a performance that is premised on the reproduction of a combination of the music sound and the performer's singing sound (hereinafter referred to as vocal sound), and if there is a time lag in the reproduction of these sounds, the viewing experience is not favorable. On the other hand, in a mode in which a singing performance by a performer is distributed in real time as VR content as in this embodiment, the vocal sound needs to be recorded and transmitted sequentially by the performer terminal 100. However, in a mode in which the client terminal 300 controls the reproduction of the music sound and the vocal sound, which are acquired at different times, individually, there is a possibility that a favorable viewing experience is not provided. In order to avoid such a viewing experience, in the delivery system of this embodiment, the music sound and the vocal sound are synthesized in the performer terminal 100, and the client terminal 300 is configured to stream the synthesized sound. Therefore, music data including music audio for the performer to sing is pre-downloaded to the performer terminal 100 and used by the performer to sing, while the client terminal 300 does not include music data but has information about other songs pre-downloaded.

[0052] 〈Production presentation〉 As described above, in order to increase the interest and immersion of the viewing experience of VR content, it is effective to apply the production to a three-dimensional space. In the case of a conventional distribution method in which all the states of objects involved in the production (hereinafter referred to as production objects) in each frame are specified in advance, and the state is downloaded in advance together with the drawing data as production data and used for generating a screen related to the VR content, the burden on the VR content provider (performer or director) in preparing for distribution may become excessive. In addition, the data size of the production data specifying the state of the production objects in each frame increases as the length of the performance planned for the VR content increases, so the total amount of data to be downloaded in advance at the client terminal 300 increases. For example, when communication between the client terminal 300 and the distribution server 200 is cut off due to a communication trouble while viewing the VR content, various data must be re-downloaded, and the viewer's waiting time until viewing of the VR content at the client terminal 300 is resumed may be extended.

[0053] The delivery system of this embodiment employs a method different from the conventional one in order to easily present the performance in the VR content and flexibly, regardless of the music, etc. More specifically, the method employed in the delivery system has the client terminal 300 download the performance data in advance, as in the conventional method, but differs in that the performance data is not configured for the entire VR content or the entire specific music, and therefore the data size is light.

[0054] In the delivery system of this embodiment, the production data downloaded in advance to the client terminal 300, etc., does not prescribe the state of all production objects in all frames in order to realize the production of the entire VR content, but simply prescribes whether a specific state transition is to be made to some of the production objects placed in the three-dimensional space. Furthermore, the production data does not prescribe fixed state transitions that are suited to the music used in the VR content, but prescribes general state transitions that are independent of the music, and is configured to be widely usable / reusable, for example, for multiple songs, sections in one song where the same progression appears, or singing performances by various performers.

[0055] Using the thus configured production data, the presentation of the VR content in the client terminal 300 is based on information (hereinafter referred to as state information) that specifies the production state to be applied to the three-dimensional space that is sequentially delivered in accordance with the progress of the VR content. In the client terminal 300, the reception of the state information is used as a trigger to refer to the production data corresponding to the specified production state, and the state of the corresponding production object is controlled. In other words, in the method of this embodiment, the presentation of the production in the VR content can be controlled in response to the transmission of state information from the performer terminal 100, and this can be done without configuring data that specifies the state transition of all the production objects in advance. In other words, the production state of each production object in the three-dimensional space related to the VR content is determined by the delivery application executed in the performer terminal 100.

[0056] In short, in the distribution system of this embodiment, information sequentially specifying the presentation state of each presentation object is not pre-downloaded to the client terminal 300, and instead, status information specifying the presentation data to be applied to the three-dimensional space may be included as necessary as progress information required for the presentation of VR content distributed via the distribution server 200.

[0057] In this embodiment, a plurality of production objects are provided to be arranged in a three-dimensional space. The plurality of production objects are classified into a plurality of production groups according to the production items presented by the production objects, for example, and the production data is configured so that the production state can be changed on a production group basis. In the VR content of this embodiment, the production items include stage switching, changing the lighting mode of normal lighting, changing the lighting mode of spotlights, and changing the celestial sphere background (background as a distant view in a three-dimensional space), and one production group is assigned to each of them. Therefore, for example, a production group related to stage switching includes a plurality of types of stage models arranged as stages in a three-dimensional space. Also, for example, a production group related to a change in the lighting mode of spotlights includes a spotlight light source arranged in a three-dimensional space. Also, for example, a change in the celestial sphere background includes a plurality of types of textures applied to the inner surface of a celestial sphere model arranged in a three-dimensional space.

[0058] In the following description, data for transitioning a rendering object (or a group of rendering objects) belonging to one rendering group to a specific rendering state will be referred to as "rendering element data." That is, one rendering element data specifies a change in the rendering state of a rendering object belonging to one of the rendering groups. One rendering group may have one or more possible rendering states, and one or more rendering element data may be prepared for each rendering group. Thus, the rendering data downloaded in advance includes one or more rendering element data for each of the multiple rendering groups.

[0059] (Processing on the performer's terminal) During the distribution of VR content, the performer (or the performer's staff, etc.) can specify which production state each production group should be in. The distribution application is provided with a user interface that allows the performer to specify which production state each production group should be in. Through the user interface, the distribution application can accept changes to the production state for each production group from the performer at any time.

[0060] In the delivery system of this embodiment, the designation of the performance state of each performance group is described as being accepted based on user input via the delivery application, but the implementation of the present invention is not limited to this. For example, the designation of the performance state can be set in advance by the performer, etc., so that a predetermined state is automatically designated by detecting that the performer has performed a specific physical movement or reached a specific time position in the music.

[0061] The distribution application manages the rendering state applied to each rendering group for the VR content being distributed. That is, the distribution application manages the rendering state of each rendering group, for example, after starting the distribution application, and sequentially changes the state in response to a user input related to the designation of the rendering state. That is, the rendering state is also managed on the distribution application side so that the rendering applied to the rendering object of each rendering group can be grasped for the three-dimensional space constructed in the viewing application. Hereinafter, the information on the rendering state of each rendering group managed by the distribution application in the performer terminal 100, which is changed based on a user input related to the designation of the rendering state, is referred to as "distribution side state management information". On the other hand, the information on the rendering state of each rendering group managed by the viewing application in each client terminal 300 is referred to as "viewing side state management information" and will be described later.

[0062] Such management of the distribution side state management information is effective in terms of reducing the amount of progress information to be distributed. As described above, in the distribution system of this embodiment, state information needs to be distributed from the performer terminal 100 to the client terminal 300 when applying a production to a three-dimensional space. The presentation of a production is not basically completed in a single frame, but is performed over a certain period of time. For a production presented using such a certain period of time, it is not necessary to keep transmitting state information all the time during that period. If the state information is transmitted once at the timing of instructing the start of the production and the client terminal 300 receives it, the client terminal 300 can read the corresponding production element data thereafter to control the state of the production object. Therefore, in the distribution system of this embodiment, the distribution application is configured to transmit state information included in the progress information in order to transition the three-dimensional space of the client terminal 300 to the changed production state when there is a change in the distribution side state management information of any production group.

[0063] For this reason, when a user input is made to specify a rendering state, the delivery application needs to determine whether or not to change the delivery side status management information managed for the corresponding rendering group to the rendering state specified by the user input. That is, the delivery application does not update the delivery side status management information of the corresponding rendering group every time a user input is made to specify a rendering state, but first determines whether an update is necessary. Then, the delivery application changes the delivery side status management information of the corresponding rendering group based on the user input only when it determines that an update is necessary.

[0064] Here, the determination of whether or not the distribution side status management information needs to be updated may be made based on, for example, the following two types of conditions. One condition is that the corresponding production group has already transitioned / is in transition to a production state specified by user input. This condition is met when a user input specifying the same production state for the corresponding production group has already been accepted, and the distribution side status management information indicates that production state. In this case, the distribution application can determine that it is not necessary to update the distribution side status management information based on the new user input.

[0065] Another condition is that the corresponding production group cannot transition to the production state specified by the user input. For example, when the production element data specifies a production sequence in which the state of the production object changes sequentially over time (such as turning a spotlight), there is a possibility that a suitable production presentation will not be made even if state information is sent to transition to another production state while the production sequence of the production element data related to the previously specified production state is being applied. Also, for example, depending on the production state of other production groups, there is a possibility that the production will not work properly when the specified production state is applied to a specific production group (causing display inconsistency or unintended production effects, or the production object of the production group is excluded from the three-dimensional space and cannot be applied in the first place, etc.) (hereinafter, such a state may be described as "interference occurs with other productions"). Therefore, this condition is met when the corresponding production group is in the process of applying another previously specified production state, or when interference occurs with a production applied to another production group. In this case as well, the distribution application can determine that it is not necessary to update the distribution side state management information based on the new user input.

[0066] On the other hand, in a mode in which state information is transmitted only when a change occurs in the distribution side state management information related to any of the production groups, for example, in a case where some of the client terminals 300 are unable to receive the progress information normally and then are restored to a state where they can receive it normally again, a suitable viewing experience may not be provided to the viewer using the client terminal 300. That is, in the above mode, since state information is not retransmitted during a section in which the same production state is maintained, a client terminal 300 that is unable to receive the progress information normally before the section and is then able to receive the progress information normally during the section is provided with a viewing experience different from that of other viewers during the section. That is, in such a situation, differences arise in the viewing experience of the VR content between viewers, and fairness is not guaranteed.

[0067] For this reason, in the distribution system of this embodiment, in order to prevent such a viewing experience that does not guarantee fairness from being provided for a long period of time, the transmission of the status information is controlled so that it is transmitted periodically at a predetermined time interval, not only when a change occurs in the distribution side status management information related to any of the production groups. By transmitting the status information periodically, the frequency of receiving the status information is guaranteed even in cases where, for example, a communication failure or a device failure has temporarily prevented the progress information from the distribution server 200 from being acquired, or when viewing of VR content is started halfway through distribution, etc., so that the client terminal 300 can be provided with the same viewing experience as other viewers.

[0068] In addition, in order to provide the same VR content viewing experience as other viewers by receiving the state information once, the state information is configured to specify information on the rendering state applied to each of all the rendering groups defined for the VR content. In other words, the state information includes information on the rendering state indicated by the delivery side state management information managed for all the rendering groups at the time of transmission, regardless of the rendering state of each rendering group. In other words, the state information specifies information indicating which rendering element data is applied to the rendering objects belonging to each rendering group.

[0069] (Processing on the client terminal side) In the viewing application executed on the client terminal 300, when the received progress information includes state information, the viewing application executes a process of applying a performance to the three-dimensional space based on the state information. As described above, in the distribution system of this embodiment, state information is transmitted when the performer terminal 100 determines that a change has occurred in the performance state applied to any of the performance groups, and when it determines that a periodically set transmission timing has been reached. For this reason, the state information continuously acquired by the client terminal 300 may indicate that all performance groups show the same performance state. Also, the state information continuously acquired by the client terminal 300 may indicate that some performance groups show different performance states, and other performance groups show the same performance state. Therefore, the viewing application does not control the performance of all performance groups based on the received state information each time the state information is received, but identifies the performance state that needs to be applied among the performance states indicated by the state information, and performs performance control to apply it to the relevant performance group. In other words, the viewing application identifies the performance state that has not yet been applied to the performance object belonging to the relevant performance group in the three-dimensional space among the performance states defined in the state information, and performs performance control to apply it.

[0070] To realize such rendering control, the viewing application, like the distribution application, manages the rendering state applied to the rendering object of each rendering group for the three-dimensional space currently constructed in the viewing application as viewing side state management information. That is, the rendering state managed as viewing side state management information for each rendering group in the viewing application indicates the rendering state already applied to each rendering group by the initialization at the start of viewing or by previously acquired state information, and a rendering group for which subsequently acquired state information indicates a different rendering state can be specified as a target for applying the rendering specified by the subsequently acquired state information. That is, when the acquired state information includes a rendering group associated with a rendering state different from the viewing side state management information, the viewing application specifies the rendering group as a rendering group for which new rendering control should be performed.

[0071] It should be easily understood that the viewing side state management information managed by the viewing application and the distribution side state management information managed by the distribution application do not necessarily indicate the same presentation state at the same time. The distribution side state management information is basically updated based on a user input relating to the designation of the presentation state accepted by the distribution application. On the other hand, the viewing side state management information is updated based on state information acquired by the client terminal 300 executing the viewing application. In other words, the two are based on different inputs that serve as the basis for updates, and naturally there can be a time lag in the timing of acquiring the input, so they do not necessarily indicate that the presentation states are the same at the same time for each presentation group.

[0072] When the viewing application identifies that the acquired state information contains a performance group indicating an unapplied performance state, it reads out the performance element data corresponding to that performance state and performs state control of the performance object of the corresponding performance group. In this way, the three-dimensional space constructed in the viewing application transitions to a state in which the performance specified by the state information is applied. The distribution application generates a display screen by causing the rendering device 304 to render the three-dimensional space after the performance is applied in this way, and as a result, the VR content to which the performance is applied is displayed to the viewer.

[0073] (Control related to production sequence) Incidentally, in content that progresses including a music playback section in which music is played, such as the VR content distributed by the distribution system of this embodiment, the content provider (performer) can select various music to be played. On the other hand, when the production element data is configured to be generally usable in order to reduce the data size of the pre-download to the client terminal 300, depending on the selected music, the change in the production state of the production object belonging to the production group defined in the production element data may not match the music. In other words, when the production element data defines a production sequence that transitions over time, the production sequence is suitable for music with a specific tempo (e.g., BPM), but for music with other tempos, inconsistencies may occur, such as the timing of the bar change and the end of the production not being aligned.

[0074] In order to avoid such inconsistencies, in the viewing application of this embodiment, each of the production element data is associated with reference tempo information indicating the tempo of the music to which the production element data is adapted. The viewing application can also obtain tempo information of the music to be played in the music playback section from pre-downloaded information. When applying production element data to a production group based on the obtained state information, the viewing application determines a transition speed of the production based on the production element data adapted to the tempo of the music based on the reference tempo information associated with the production element data and the tempo information of the music to be played, and performs production control based on the determined transition speed.

[0075] For example, if the reference tempo of the production element data is BPM 100 and the tempo of the music is BPM 150, the viewing application determines the transition speed of the production element data to be 1.5 times. For example, if the transition speed is 1.5 times, the states applied in each frame as a production sequence to the production objects belonging to the production group can be changed to the average of the states of the first and second frames of the production sequence defined by the production element data, the state of the third frame, the average of the states of the fourth and fifth frames, the state of the sixth frame, etc.

[0076] In addition, it is preferable that the application of the production element data adapted to the tempo of the music corresponds to the timing at which the production based on the production element data starts and the beats and bars in the music. In general, the presentation of the production that accompanies the playback of the music can be started or switched in synchronization with the timing at which the beats and bars of the music arrive, thereby providing the viewer with a suitable effect that matches the flow of the music. For this reason, in the distribution system of this embodiment, the state information transmitted by the distribution application includes timing information for specifying the timing at which the application of the production element data based on the state information starts to the production objects that belong to the corresponding production group.

[0077] Timing information is information that can uniquely identify a specific time position for the VR content to be distributed, and in the distribution system of this embodiment, the elapsed time (number of frames that have passed) from the start of the distribution application is used. Timing information is included not only in the state information, but also in the motion information and audio information included in the progress information transmitted from the performer terminal 100. In this way, by including timing information in various information distributed in the distribution system, when each piece of information is played back on the client terminal 300 (application of motion information to the performer character, playback of audio information, application of performance element data), it becomes possible to control the playback timing of each piece of information to the form expressed on the performer terminal 100, i.e., the form intended by the performer.

[0078] (Timing of application of effects) Meanwhile, audio information and motion information may be reproduced on the client terminal 300 according to the time position at which they were acquired on the performer terminal 100. For this reason, by associating the time position at which the audio information and motion information were acquired as timing information and reproducing the information on the client terminal 300 at the time position in the viewing application specified by the timing information, it is possible to present information in a manner intended by the performer.

[0079] On the other hand, in an aspect in which the time position at which the application of directing element data to a directing group in the client terminal 300 starts based on state information is set to the time position at which a user input is made at the performer terminal 100 to specify the corresponding directing state, the directing presentation in the aspect intended by the performer may not be realized. This is because, in this aspect, in order to apply the directing element data in synchronization with the timing of the arrival of beats and bars of the music as described above, the user input for specifying the directing state needs to be made at the performer terminal 100 in time with the beats and bars of the music. In other words, an aspect in which user input in time with the beats and bars of the music is essential in order to realize a suitable directing presentation at the performer terminal 100 may impose an excessive input burden on the performer during distribution.

[0080] Therefore, in the appreciation application of this embodiment, unlike audio information and motion information, the timing information included in the state information is not a time position specified by the timing information, but rather identifies one beat or bar in the music determined based on the time position, and is controlled to start playback (application of corresponding performance element data) at the time position corresponding to the beat or bar. In the following embodiment, the appreciation application determines the time position at which to start applying the performance element data to the corresponding performance group based on the state information to be the time position corresponding to the beat that arrives immediately after the time position specified by the timing information included in the state information for the corresponding performance group.

[0081] Where in the music there is a beat or a bar can be obtained by analyzing music data, but in the viewing application of this embodiment, the information is derived based on information about the music to be played in the VR content, which is acquired as advance information. The information about the music includes information about the tempo (BPM) of the music and information about the time that elapses from the beginning of the music to the arrival of the first bar of the music (bar offset). For example, if the distribution system of this embodiment is configured to only allow music with a simple meter, such as 4 beats, to be used, the viewing application can identify the positions of all beats or bars in the music based on the information about the music. That is, the time positions in the music of beats that arrive after the time position of the first bar can be derived by sequentially adding the time length of one beat that can be identified from the BPM of the music to the time position of the first bar, and further, the time positions of each bar in the music can be derived based on the meter adopted in the music. To convert such a time position in a song into a time position during the distribution of VR content, i.e., the elapsed time from the launch of a distribution application such as timing information associated with audio information or movement information, it is sufficient to add the time position specified by the timing information associated with the start position of the music playback section for the song in question in the VR content included in the progress information, for example.

[0082] Therefore, the effects to be applied to the three-dimensional space can be configured to be presented in synchronization with the arrival of a beat in the music in the appreciation application. Note that such control related to adjustment of the timing of the start of the effects is also performed when displaying the three-dimensional space in the distribution application. That is, for example, when a performer performs a user input related to the designation of the performance state in the performer terminal 100, it can be confirmed that the designated performance is presented in synchronization with the arrival of a beat in the music on the screen of the three-dimensional space constructed by the distribution application displayed on the display device 140.

[0083] Data Structure In the distribution system of this embodiment as described above, presentation of effects related to VR content being distributed is realized by using information having the following data configuration, for example.

[0084] <Basic Progress Information> The audio information and motion information that are basically distributed as progression information include acquired data 501, which is data acquired from the microphone 130 or the motion capture system 120 in the performer terminal 100, and timing information 502 that indicates the time position at which the acquired data was acquired (the timing at which the acquired data should be played back), as shown in Fig. 5(a). When the audio information and motion information are distributed to the client terminal 300 as progression information, they are applied to the playback / performance characters at the time position based on the timing information 502 associated with each, and are presented as VR content.

[0085] <Information regarding the presentation of the distribution application> In the distribution application, the distribution side status management information managed for each performance group of the VR content being distributed is composed of, as shown in Figure 5 (b), a group ID 511 that uniquely identifies the performance group, an applied performance ID 512 that uniquely identifies the performance state currently applied to the performance group, and timing information 513 that indicates the time position when the user input specifying the currently applied performance state was made.

[0086] As described above, the presentation of effects related to VR content is realized by applying corresponding effect element data from among the effect data downloaded in advance to the performer terminal 100 and the client terminal 300 to the effect object of the corresponding effect group. Since the process of reading and applying such effect element data only needs to be performed on each terminal, when managing the applied effect state and distributing information, information that can uniquely identify the effect element data is sufficient. Therefore, in the distribution side state management information, the effect state currently applied to the effect group is managed by the applied effect ID 512 assigned to the effect element data. The applied effect ID 512 can be simply an integer value (number), and for the VR content to be distributed, which integer value is assigned to which effect element data is predetermined and shared between the distribution application and the viewing application.

[0087] When a user input is made to specify a production state of a production group on the performer terminal 100, the distribution application acquires input information as shown in Fig. 5(c). As shown in the figure, the input information is associated with a group ID 521 that uniquely identifies the production group to be specified, and includes an applied production ID 522 that indicates the specified production state, and timing information 523 that indicates the time position at which the user input was made.

[0088] Therefore, when a user input related to the designation of a performance state is accepted, the distribution side status management information related to the same group ID 511 as the group ID 521 of the input information is referenced, and it is determined whether or not the distribution side status management information is changed. The determination may be made depending on whether the applied performance ID 512 of the distribution side status management information is different from the applied performance ID 522 of the input information, or whether the timing information 513 of the distribution side status management information is different from the timing information 523 of the input information, or whether both are different. In addition to this, the determination also includes a determination of whether there is no interference of performances or whether the previously specified performance state is in transition. When the distribution application determines to change the distribution side status management information, it changes it by replacing each piece of information of the distribution side status management information with the input information.

[0089] The distribution side status management information configured in this manner is included in the status information when the status information is transmitted as progress information. Since the status information only needs to be transmitted including the distribution side status management information related to all production groups at the transmission timing, the data configuration is omitted from the illustration.

[0090] <Information about the presentation in viewing applications> Next, in the viewing application, the viewing side state management information managed for each performance group of the VR content being distributed is composed of, as shown in Figure 5 (d), an applied performance ID 532 that uniquely identifies the performance state currently applied to the performance group in association with a group ID 531 that uniquely identifies the performance group, and timing information 533 that indicates the time position when user input specifying the currently applied performance state was made.

[0091] When state information is received as described above, the viewing application compares the information included in the state information for each performance group with the viewing side state management information, and determines whether or not to change the viewing side state management information, i.e., whether or not to apply a new performance, based on the performance state information included in the state information for that performance group. When the viewing application determines to change the viewing side state management information, it changes it by replacing the applied performance ID 532 and timing information 533 of the viewing side state management information for the corresponding performance group with the applied performance ID and timing information of the state information for that group.

[0092] For the effect group for which the viewing side state management information has been changed in this way, a process is performed to apply effects to the effect objects belonging to the effect group. At this time, the effect element data to be applied to the effect object is specified based on the applied effect ID 532 (the effect element data related to the effect group to which the same integer value is assigned). The application of the specified effect element data is controlled so as to be performed from a time position specified based on timing information 533 and information about the currently played music. Here, the information about the music includes, for example, a music ID 541 that uniquely identifies the music, a tempo 542 of the music, and a bar offset 543 that indicates the time position of the first bar in the music, as shown in FIG. 5(e). In applying the effect element data, the viewing application controls the transition speed based on information about the reference tempo associated with the effect element data and the tempo 542 of the information about the music.

[0093] <<Transmission process>> The transmission process executed in relation to the transmission of status information in the performer terminal 100 of the distribution system of this embodiment will be described in detail below with reference to the flowchart in Fig. 6. The process corresponding to this flowchart can be realized by the control unit 101 of the performer terminal 100 reading out a program related to a distribution application stored in, for example, the storage device 102, and expanding and executing the program in the memory 103. Note that this transmission process will be described as being executed for each frame of the VR content to be distributed.

[0094] In S601, the control unit 101 determines whether or not a user input has been made to specify the presentation state of any of the presentation groups. The determination in this step is made depending on whether or not input information related to the specification of the presentation state has been acquired. If the control unit 101 determines that a user input has been made to specify the presentation state of any of the presentation groups, it moves the process to S602, and if it determines that no user input has been made, it moves the process to S606. In the explanation of this transmission process, the user input is referred to as the "target input," the presentation state specified in the user input is referred to as the "specified presentation state," and the presentation group whose presentation state is specified by the user input is referred to as the "target group."

[0095] In S602, the control unit 101 determines whether the currently applied presentation state for the target group is a designated presentation state. That is, in this step, the control unit 101 determines whether the designated presentation state has already been applied to the target group. If the control unit 101 determines that the currently applied presentation state for the target group is the designated presentation state, it proceeds to S606, and if it determines that it is not the designated presentation state, it proceeds to S603.

[0096] In S603, the control unit 101 determines whether or not the designated presentation state can be applied to the target group. That is, in this step, the control unit 101 determines whether or not the target group can be transitioned to the designated presentation state. If the control unit 101 determines that the designated presentation state can be applied to the target group, the process proceeds to S604, and if the control unit 101 determines that the designated presentation state cannot be applied, the process proceeds to S606.

[0097] In S604, the control unit 101 changes the distributor status management information related to the target group based on the input information related to the target input.

[0098] In S605, the control unit 101 configures state information based on the distribution side state management information related to all the performance groups, including the distribution side state management information related to the target group changed in S604, and controls the state information to be included in the progress information and transmitted, completing this transmission process. In this way, it is possible to transmit a performance presentation according to the target input to each client terminal 300.

[0099] On the other hand, if it is determined in S601 that no target input has been made, if it is determined in S602 that the presentation state currently applied to the target group is the designated presentation state, or if it is determined in S603 that the designated presentation state cannot be applied to the elephant group, the control unit 101 determines in S606 whether or not to transmit status information. The determination made in this step is based on whether or not the periodically set timing for transmitting status information has arrived. If the control unit 101 determines that the status information should be transmitted, it proceeds to S607, and if it determines that the status information should not be transmitted, it completes this transmission process.

[0100] In S607, the control unit 101 configures state information based on the distribution side state management information related to all production groups, and performs control so as to include the state information in the progress information and transmit it, thereby completing this transmission process.

[0101] <<Performance Presentation Processing>> Next, the specific process of the performance presentation process executed in the client terminal 300 of this embodiment for the performance presentation of VR content will be described with reference to the flowchart in Fig. 7. The process corresponding to the flowchart can be realized by the control unit 301 of the client terminal 300 reading, for example, a program of an appreciation application stored in the storage device 302, and expanding and executing the program in the memory 303. Note that this performance presentation process will be described as being started when progress information about the VR content to be distributed is acquired.

[0102] In S701, the control unit 301 judges whether or not the acquired progress information includes state information. If the control unit 301 judges that the acquired progress information includes state information, it moves the process to S702, and if it judges that the information does not include state information, it ends this performance presentation process. In the explanation of this performance presentation process, the state information included in the acquired progress information is referred to as "target state information."

[0103] In S702, control unit 301 determines whether or not there is a performance group for which a performance state different from the performance state currently applied to the three-dimensional space is specified in the target state information. The determination in this step is made by comparing the information included in the target state information for each performance group with the audience side state management information for that performance group. If control unit 301 determines that there is a performance group for which a performance state different from the performance state currently applied to the three-dimensional space is specified in the target state information, it proceeds to S703, and if it determines that there is no such group, it completes this performance presentation process. In the explanation of this performance presentation process, the performance group for which a performance state different from the current one is specified in the target state information will be referred to as the "target group."

[0104] In S703, the control unit 301 changes the viewer-side state management information related to the target group based on the state information.

[0105] In S704, the control unit 301 starts a process for presenting a performance for the target group. The process in this step is performed based on the audience-side state management information for the target group changed in S703. The control unit 301 reads out the performance element data specified by the applied performance ID 532 of the audience-side state management information, and performs state control of the performance object belonging to the target group. The state control is started in synchronization with the timing of playback of audio information, etc., separately acquired in the progress information, for the beat arriving immediately after the time position specified by the timing information 533 of the audience-side state management information. In addition, when the applied performance element data specifies a performance sequence, the control unit 301 also performs control to change the transition speed of the performance element data based on the tempo of the currently played music and the reference tempo associated with the performance element data.

[0106] In cases where the viewing application is again able to normally obtain progress information after a temporary state in which it is not possible to normally obtain progress information from distribution server 200 due to a communication failure, equipment failure, or the like as described above, the beat that arrives immediately after the time position specified in timing information 533 of the viewer-side state management information for the target group may have already passed by the time the state information is received. In such cases, control unit 301 may perform control so as to apply effects from the frame in the performance element data that corresponds to the time position of the currently played music, assuming that application of the performance element data has started at the time position corresponding to that beat.

[0107] As described above, the distribution system of this embodiment makes it possible to distribute highly interesting content while reducing the production burden.

[0108] In the delivery system of this embodiment, the performance data is configured to be generally usable regardless of the music, and it has been described that it is possible to accommodate various music by, for example, changing the transition speed. However, the performance data does not need to be limited to data that can be used for presentation of performances in other music, and may be data that is dedicated to a specific music, for example. When performance data is provided for a specific music, each of the performance element data included in the performance data does not need to have a reference tempo associated therewith, since it is not necessary to change the transition speed during playback of the music.

[0109] In the present embodiment, the tempo 542 and the bar offset 543 are provided as information about the music piece, and the time position at which the application of the effect should start is specified based on the information, but the present invention is not limited to this. The time position at which the application of the effect should start may be specified by, for example, analyzing the audio information to be distributed, or may be specified by referring to information indicating a beat that is added to the audio information, etc.

[0110] In addition, in the distribution system of this embodiment, the status information indicating the production status of the production group is described as being included in the progress information and transmitted, but the manner in which the information is transmitted is not limited to this. That is, the information distributed in the distribution system does not need to be included in a specific package configured as progress information and transmitted, and for example, the audio information, the motion information, and the status information may each be transmitted independently. In this specification, the progress information refers to all information necessary for the presentation of VR content in the client terminal 300, and the status information indicates one aspect of the necessary information.

[0111] In addition, in the distribution system of this embodiment, the elapsed time from the launch of the distribution application is used as timing information to synchronize various pieces of information contained in the progress information, but it will be easily understood that the time used as the reference for such synchronization control is not limited to this.

[0112] [Variation 1] In the above embodiment, the performer terminal 100 executes a distribution application and the client terminal 300 executes a viewing application to provide a viewing experience of VR content, but the implementation of the present invention is not limited to this. The application may be configured so that the user of the terminal on which it is installed can select whether to be a performer or a viewer. In other words, by installing one application, the user of the terminal may be able to distribute VR content as a performer and also be able to view VR content distributed by other users as a viewer. These switching may be easily performed by selecting, for example, an operation mode (distribution mode and viewing mode) provided in the application.

[0113] [Variation 2] In the above embodiment, the VR content is distributed in a so-called live broadcasting manner, in which the performer terminal 100 acquires information on the performance performed by the performer in real time, composes progress information, and transmits it, but the implementation of the present invention is not limited to this. The present invention realizes the presentation of the performance to the viewer by transmitting status information indicating the performance status of the VR content from the performer terminal 100 as progress information, and applying the performance to the three-dimensional space based on the performance data in the client terminal 300 in response to the acquisition of the status information, and does not impose any time constraints on the generation of the status information.

[0114] That is, the generation of a series of progress information related to VR content does not necessarily require real-time content distribution, and may be performed, for example, by recording the performer's performance in advance. Alternatively, for example, a series of progress information sent in real-time content distribution in the above-mentioned distribution system can be recorded as redistribution data, so that the distributed VR content can be redistributed so that the viewer can view it at any timing. In the latter case, for example, by recording a user input related to the designation of a production state in association with timing information at which the user input was made (recording the input information in FIG. 5(c)), it is possible to reproduce the distribution of state information (progress information) similar to that at the time of real-time distribution using the distribution system according to the present invention. Here, in order to reduce the data size of the redistribution data of the VR content, the input information related to the designation of the production state can be configured to be recorded only when it is reflected in the distribution side state management information in S604 of the transmission process.

[0115] [Variation 3] In the above-mentioned embodiment and modified example, the content distributed by the distribution system is described as VR content that presents a three-dimensional space in which characters are arranged in a binocular stereoscopic manner, but the implementation of the present invention is not limited to this. The content that controls the presentation of the performance is not limited to that presented in a binocular stereoscopic manner or that drawing characters arranged in a three-dimensional space, and the present invention is applicable even to a form in which, for example, a character arranged in a two-dimensional space is presented as two-dimensional content.

[0116] [Other embodiments] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0117] 100: performer terminal, 101: control unit, 102: storage device, 103: memory, 104: drawing device, 105: input I / F, 106: output I / F, 107: communication I / F, 110: input device, 120: motion capture system, 130: microphone, 140: display device, 150: speaker, 200: distribution server, 201: control unit, 202: storage device, 203: memory, 204: communication I / F, 300: client terminal, 301: control unit, 302: storage device, 303: memory, 304: drawing device, 305: input I / F, 306: output I / F, 307: communication I / F, 310: input device, 320: display device, 321: attitude sensor, 322: speaker, 400: network

Claims

1. A distribution system including a distribution device that distributes content and an information processing device that presents the content distributed by the distribution device, the presentation of the content is performed by displaying a screen generated by drawing a space in which the character is placed; The distribution device includes: a transmitting means for sequentially transmitting progress information necessary for presenting the content in accordance with the progress of the content; a first determination means for determining whether or not to transmit state information indicating a presentation state of the content in accordance with a progress of the content; having The information processing device includes: an acquisition means for acquiring, prior to the presentation of the content, drawing data used for drawing the space and performance data related to performance applicable to the space; a construction means for constructing the space based on the drawing data acquired by the acquisition means; a receiving means for receiving the progress information from the distribution device; a control means for controlling a state of the space constructed by the construction means in response to reception of the progress information by the receiving means; a presentation means for generating and presenting a screen on which the space is rendered; having the transmitting means, when it is determined by the first determining means that the status information is to be transmitted, transmitting the status information together with the progress information; when the received progress information includes the state information, the control means applies to the space a performance based on the performance data corresponding to the state information; The presentation means, when the application of the effect by the control means is performed, generates and presents a screen depicting the space after the application. A distribution system comprising:

2. The information processing device further includes a management unit for managing a performance state being applied to the space, The control means applies a rendering based on the rendering data corresponding to the received state information to the space on condition that the rendering state indicated by the received state information is different from the rendering state managed by the management means.

2. The distribution system according to claim 1.

3. A plurality of production objects involved in the production are arranged in the space; Each of the plurality of effect objects is classified into a plurality of effect groups based on the effect of the object, and the group is managed accordingly; The performance data includes a plurality of performance element data corresponding to a plurality of types of performances, each of which specifies a change in a performance state for one of the performance groups; The state information includes information for identifying the performance element data to be applied to each of the performance groups that define the performance state.

3. The distribution system according to claim 2.

4. The distribution device further includes a determination unit for determining a performance state of the plurality of performance groups for the content being distributed, The first determination means determines that the state information is to be transmitted when the performance group for which a change to a different performance state has been determined by the determination means exists.

4. The distribution system according to claim 3.

5. The distribution device further includes an input unit for receiving a user input for specifying a performance state of the performance group, The determination means determines the presentation state of the presentation group in response to a user input specifying the presentation state of the presentation group being accepted by the input means.

5. The distribution system according to claim 4.

6. the distribution device further includes a second determination means for determining whether or not a transition to a designated presentation state is possible for the presentation group in response to a user input for designating the presentation state of the presentation group being accepted by the input means; The determining means changes the presentation state of the presentation group for which a user input designating a presentation state has been accepted to the designated presentation state when the second determining means determines that the presentation state can be transitioned to the designated presentation state.

6. The distribution system according to claim 5.

7. the distribution device further includes a recording means for recording the user input specifying the rendering state of the rendering group received by the input means as data for re-distribution of the content in association with information indicating a time position at which the user input was made; The recording means records, as the re-distribution data, only the user input determined by the second determination means as a transition to a designated presentation state.

7. The distribution system according to claim 6.

8. The distribution system described in claim 4, characterized in that the first judgment means further judges that the status information should be transmitted at a predetermined time interval even if there is no performance group for which the decision means has decided to change to a different performance state.

9. The performance data defines a performance that transitions over time, The state information includes timing information for specifying a timing to start applying a performance based on the performance data corresponding to the state information.

2. The distribution system according to claim 1.

10. The content includes a music playback section in which a music piece is played, When the control means receives the state information for the music playback section, the control means determines a transition speed of the effect based on the effect data corresponding to the state information in accordance with a tempo of the music played in the music playback section.

10. The distribution system according to claim 9.

11. The content includes a music playback section in which a music piece is played, When the control means receives the state information for the music playback section, the control means specifies, based on the timing information included in the state information, a time position of a beat or a bar in the music played in the music playback section as a timing to start applying a performance based on the performance data corresponding to the state information.

10. The distribution system according to claim 9.

12. the timing information indicates a time position; The control means specifies, in the progression of the music played in the music playback section, a time position of a beat or a bar that comes immediately after the time position indicated by the timing information as a timing to start applying the effect.

12. The distribution system according to claim 11.

13. An information processing device that presents content distributed by a distribution device, the content including a display of a screen generated by drawing a space in which a character is arranged, an acquisition means for acquiring, prior to the presentation of the content, drawing data used for drawing the space and performance data related to performance applicable to the space; a construction means for constructing the space based on the drawing data acquired by the acquisition means; a receiving means for sequentially receiving progress information required for presenting the content from the distribution device in accordance with the progress of the content; a control means for controlling a state of the space constructed by the construction means in response to reception of the progress information by the receiving means; a presentation means for generating and presenting a screen on which the space is rendered; having when the received progress information includes state information indicating a rendering state of the content, the control means applies to the space a rendering based on the rendering data corresponding to the state information; The presentation means, when the application of the effect by the control means is performed, generates and presents a screen depicting the space after the application.

23. An information processing apparatus comprising:

14. A method for controlling an information processing device that presents content distributed by a distribution device, the content including display of a screen generated by drawing a space in which a character is placed, comprising: an acquisition step of acquiring, prior to the presentation of the content, drawing data used for drawing the space and performance data related to performance applicable to the space; a construction step of constructing the space based on the drawing data acquired in the acquisition step; a receiving step of sequentially receiving progress information required for presenting the content from the distribution device in accordance with the progress of the content; a control step of controlling a state of the space constructed in the construction step in response to reception of the progress information in the reception step; a presentation step of generating and presenting a screen depicting the space; having In the control step, when the received progress information includes status information indicating a rendering status of the content, a rendering based on the rendering data corresponding to the status information is applied to the space, When the application of the effect is performed in the control step, a screen depicting the space after the application is generated and presented in the presentation step. A control method comprising:

15. A program for presenting content distributed by a distribution device, the content including a display of a screen generated by drawing a space in which a character is placed, On the computer, an acquisition process for acquiring, prior to the presentation of the content, drawing data used for drawing the space and performance data related to performance applicable to the space; a construction process for constructing the space based on the drawing data acquired in the acquisition process; a receiving process of sequentially receiving progress information required for presenting the content from the distribution device in accordance with the progress of the content; a control process for controlling a state of the space constructed in the construction process in response to reception of the progress information in the reception process; A presentation process of generating and presenting a screen on which the space is drawn; A program for executing In the control process, when the received progress information includes state information indicating a rendering state of the content, a rendering based on the rendering data corresponding to the state information is applied to the space, In the presentation process, when the effect is applied by the control process, a screen depicting the space after the effect is applied is generated and presented. A program characterized by:

16. A program configured to be switchable between a distribution mode for distributing content and a viewing mode for presenting the content distributed in the distribution mode, the presentation of the content is performed by displaying a screen generated by drawing a space in which the character is placed; In the distribution mode, the computer a transmission process of sequentially transmitting progress information necessary for presenting the content according to the progress of the content; a first determination process for determining whether or not to transmit state information indicating a presentation state of the content according to a progress of the content; Run the command, in the viewing mode, the computer an acquisition process for acquiring, prior to the presentation of the content, drawing data used for drawing the space and performance data related to performance applicable to the space; a construction process for constructing the space based on the drawing data acquired in the acquisition process; a receiving process for receiving the progress information; a control process for controlling a state of the space constructed in the construction process in response to reception of the progress information in the reception process; A presentation process of generating and presenting a screen on which the space is drawn; Run the command, in the transmission process, when it is determined in the first determination process that the status information is to be transmitted, the status information is included in the progress information and transmitted; In the control process, when the received progress information includes the state information, a performance based on the performance data corresponding to the state information is applied to the space, In the presentation process, when a performance is applied in the control process, a screen depicting the space after the performance is applied is generated and presented. A program characterized by:

Images