Three-dimensional content distribution system, three-dimensional content distribution method, and computer program

Abstract

The live content is rendered based on the specification data of the virtual object, the data of the action of the virtual object, and the like, and is provided for viewing and playing on the viewer's terminal.

Description

Technical Field

[0001] The present disclosure relates to a three-dimensional content distribution system, a three-dimensional content distribution method, and a computer program. Background Art

[0002] In recent years, a service for distributing live content via the Internet has become popular as an Internet live broadcast. In such Internet live broadcast, a technology has been proposed in which a distributor appears as a virtual character (avatar) in a virtual space and its image is distributed as live content. In such live content, an image synthesized by using the distributor's movements as the movements of the avatar is distributed using motion capture technology.

[0003] Unlike a structure for distributing live content via a network, in a structure in which viewers want to generate their own dedicated broadcast images, for example, Patent Document 1 discloses a structure in which backgrounds, characters, virtual objects, etc. are sent from a broadcast station to audio-visual terminals through different dedicated channels as broadcast material data, and the audience selects the material they want from them and only produces their own dedicated three-dimensional image. The produced three-dimensional image is not shared with other viewers.

[0004] In addition, similarly, in Patent Document 2, a structure in which a viewer wants to generate a playback image dedicated to himself / herself is as follows: that is, virtual three-dimensional content data such as a football match is prepared in advance, which includes multiple players, referees, sports object data such as a ball, and background data such as a goal or a ground. When the viewer selects a viewpoint that he / she wants to see (such as a specific player or a camera outside the field), the server renders the image observed from the viewpoint and sends it to the viewer's terminal. Similar to Patent Document 1, a structure is disclosed in which only a three-dimensional image dedicated to himself / herself is produced, and the resulting rendered image is not shared with other viewers.

[0005] In addition, Patent Document 3 discloses the following structure: Regarding the flat dynamic image content that is not a 3D (three-dimensional) virtual space, a server is prepared in advance with two-dimensional animation content such as live performance, and the audience receives and plays the data from the server while adding their own virtual characters (two-dimensional images of the audience) to the playback image. After or during the playback, the data of the virtual character added by the audience is sent to the server and added to the previous two-dimensional animation content, and other audiences can see the virtual character during playback. As a result, each time each audience plays, the virtual character of each audience is added to the content, so that the image of the content becomes vivid.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Publication No. 2001-28765

[0009] Patent Document 2: Japanese Patent Application Publication No. 2008-117042

[0010] Patent Document 3: Japanese Patent No. 5231093

[0011] Patent Document 4: Japanese Patent Application Publication No. 2012-120098 Summary of the invention

[0012] In recent years, the number of viewers who enjoy live content through Internet live broadcasts has continued to increase, and new demands include the following: the desire to watch and enjoy live content that was not available during the live broadcast at a later time, and the desire to get a sense of presence and immersion as if watching the live broadcast even in subsequent viewings.

[0013] At first glance, the desire of "wanting to get the sense of presence and immersion as if watching and listening to a live broadcast" seems to be incompatible with other desires such as "wanting to watch and enjoy the live content that was not seen and heard during the live broadcast in the future". Therefore, the previously shown patent documents 1 and patent documents 2 only generate images that can be watched and listened to by oneself, and therefore do not meet these needs.

[0014] Furthermore, in Patent Document 3, a virtual character of a two-dimensional image that is one's own avatar is added to the image as a viewer of the live content of the two-dimensional image, and the image of one's own virtual character or the image of a virtual character added by other viewers can be visually confirmed by all viewers, but one's own two-dimensional virtual character image is merely superimposed on the content as the two-dimensional image. Therefore, since one cannot virtually participate in the live content, it is difficult to obtain a sufficient sense of solidarity, excitement, and immersion, and this problem is not solved at all.

[0015] Furthermore, other issues related to the conventional technology include reducing the burden on the communication network and each terminal hardware and the desire to enjoy viewing and listening to live content while sharing it with a plurality of viewers.

[0016] Patent Documents 1 and 2 shown above do not provide a structure for providing live content that can be shared and enjoyed with multiple viewers. Similarly, Patent Document 3 is content composed of two-dimensional flat dynamic images, which will incur a burden on the communication paths and terminals required for the broadcast distribution of general dynamic images.

[0017] The present invention is completed in view of the above-mentioned situation, and its purpose is to provide a 3D content distribution system, a three-dimensional content distribution method, and a computer program. The 3D content distribution system is used to solve the following problems, namely, to watch and enjoy the live content that could not be watched when it was broadcast live at a later time; even in the subsequent audio-visual playback, it is hoped to obtain the sense of presence and immersion as if watching the live broadcast; or to reduce the burden on the communication network and each terminal hardware, and enjoy the audio-visual of the live content while sharing it with multiple viewers.

[0018] In order to solve the above-mentioned problems, the present invention provides a three-dimensional content distribution system, a three-dimensional content distribution method, and a computer program as described in the following 1) to 11). 1)

[0020] A three-dimensional content distribution system, which distributes content using a virtual three-dimensional space to an audio-visual terminal for viewing, is characterized in that the three-dimensional content distribution system has:

[0021] a recording unit that records content data, the content data including at least (A) first data related to specifications of each virtual object arranged in the three-dimensional virtual space, and (B) second data related to actions of each virtual object; and

[0022] The generating unit generates image data for viewing using at least the first data and the second data in response to a request for viewing by the viewing terminal. 2)

[0024] The three-dimensional content distribution system as described in 1) is characterized in that:

[0025] The recording unit further records (C) third data related to the viewpoint arranged in the three-dimensional virtual space, and the generating unit is configured to perform generation using the third data. 3)

[0027] The three-dimensional content distribution system as described in 1) or 2) is characterized in that:

[0028] The first data and the second data recorded by the recording unit are data initially recorded when the content is first distributed over the network. 4)

[0030] The three-dimensional content distribution system according to any one of 1) to 3) is characterized by comprising:

[0031] The change accepting unit makes a change for adding or modifying at least one of the first data and the second data in response to a change request from a viewing terminal that is currently viewing, thereby generating new content data for subsequent viewing. 5)

[0033] The three-dimensional content distribution system as described in 2) is characterized by further comprising:

[0034] The change accepting unit performs a change for adding or modifying at least one of the first data and the second data in response to a change request from a viewing terminal that is currently viewing the content, thereby generating new content data for subsequent viewing.

[0035] The change accepting unit is further configured to make a change for adding or correcting the third data, thereby generating new content data for subsequent viewing. 6)

[0037] A three-dimensional content distribution method is provided, which distributes content using a virtual three-dimensional space to an audio-visual terminal for viewing, wherein the three-dimensional content distribution method comprises:

[0038] A recording step of recording content data, the content data including at least (A) first data related to the specifications of each virtual object arranged in the three-dimensional virtual space, and (B) second data related to the action of each virtual object; and

[0039] The generating step generates image data for viewing by using at least the first data and the second data in response to a request for viewing by the viewing terminal. 7)

[0041] The three-dimensional content distribution method as described in 6) is characterized in that:

[0042] The recording step further records (C) third data related to the viewpoint arranged in the three-dimensional virtual space, and the generating step is configured to also generate using the third data. 8)

[0044] The three-dimensional content distribution method as described in 6) or 7) is characterized in that:

[0045] The first data and the second data recorded in the recording step are data initially recorded in the first network distribution of the content. 9)

[0047] The three-dimensional content distribution method according to any one of items 6 to 8) is characterized by comprising:

[0048] The change acceptance step performs a change for adding or modifying at least one of the first data and the second data in response to a change request from a viewing terminal currently viewing the content, thereby generating new content data for subsequent viewing. 10)

[0050] The three-dimensional content distribution method as described in 7) is characterized by further comprising:

[0051] a change acceptance step of performing a change for adding or modifying at least one of the first data and the second data in response to a change request from a viewing terminal that is currently viewing the content, thereby generating new content data for subsequent viewing;

[0052] The change acceptance step is further configured to make a change for adding or correcting the third data, thereby generating new content data for subsequent viewing. 11)

[0054] A computer program causes a computer to execute the steps of the three-dimensional content distribution method described in any one of 6) to 10). BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a diagram schematically showing the structure of the content distribution and reception system of the present invention.

[0056] Figure 2 yes Figure 1 A block diagram of a content distribution server included in a content distribution receiving system.

[0057] Figure 3 yes Figure 1 A block diagram of a first distributor terminal of a content distribution and reception system.

[0058] Figure 4 yes Figure 1 A block diagram of a first viewer terminal of a content distribution and receiving system.

[0059] Figure 5 yes Figure 4 A schematic diagram of the appearance of a first viewer terminal.

[0060] Figure 6 It is a signal flow chart regarding the viewing and listening of content in live broadcast executed by the content distribution and receiving system of the present invention.

[0061] Figure 7 It is a schematic diagram for explaining data related to a virtual character used in the content distribution and reception system of the present invention.

[0062] Figure 8 It is a schematic diagram of the virtual space in the live broadcast in the content distribution and receiving system of the present invention.

[0063] Fig. 9 yes Figure 8 Schematic diagram of rendering observed from various viewpoints in the virtual space shown.

[0064] Fig.10It is a signaling flow chart related to content playback after live broadcast in the content distribution and receiving system of the present invention.

[0065] Fig.11 It is a signaling flow chart executed by the content distribution and receiving system of the present invention regarding adding virtual items to the content played after live broadcast.

[0066] Fig.12 It is a schematic diagram of a virtual space to which virtual objects are added in the content distribution and reception system of the present invention.

[0067] Fig.13 yes Fig.12 Schematic diagram of rendering observed from various viewpoints in the virtual space shown.

[0068] Fig.14 It is a signal flow chart related to the action of adding virtual characters to the broadcast content after live broadcast and distributing it by other distributors, which is executed by the content distribution and reception system of the present invention.

[0069] Fig.15 It is a schematic diagram of a virtual space distributed by other distributors by adding virtual characters to the broadcast content after live broadcast in the content distribution and receiving system of the present invention.

[0070] Fig.16 yes Fig.15 Schematic diagram of rendering observed from various viewpoints in the virtual space shown.

[0071] Fig.17 This is a schematic diagram showing an example of adding a virtual object to a virtual space in the content distribution and reception system of the present invention.

[0072] Fig.18 This is a schematic diagram showing an example of adding a virtual object to a virtual space in the content distribution and reception system of the present invention. DETAILED DESCRIPTION

[0073] Hereinafter, embodiments of the present disclosure will be described. In addition, the embodiments described below are merely examples, and various modifications can be made without departing from the spirit of the present invention, and these modifications are also included in the present invention.

[0074] [Structure of Content Distribution Reception System 1]

[0075] Figure 1 It is a diagram schematically showing the structure of the content distribution and reception system 1 according to the present invention.

[0076] The content distribution receiving system 1 is configured to include a content distribution server 2, a first distributor terminal 11, a first viewer terminal 12, a second distributor terminal 21, and a second viewer terminal 22, and these components are connected via an Internet communication network 3 or other communication networks in a manner that allows data to be exchanged with each other. The content distribution server 2 is a general-purpose computer configured to operate through a computer program dedicated to the system 1.

[0077] The first distributor terminal 11 is used for distribution of live content, i.e., Internet live broadcast, and is used by the distributor who hosts the live content, such as a personal computer, and the live content uses a virtual space 30 to be described later. The first audience terminal 12 is used by an audience who watches the live content as a broadcast content after the live broadcast ends, such as a personal computer, or can also be implemented using various information terminals such as a smart phone and a PDA (personal digital assistant).

[0078] The second distributor terminal 21 is used for distributing the following live content, that is, to broadcast the live content once distributed live again on the Internet as described later, and is a personal computer used by the distributor of the live content. The second viewer terminal 22 is used by viewers who watch the live content in the form of broadcast content or Internet live broadcast, such as a personal computer, or can also be implemented using various information terminals such as smart phones and PDAs (personal digital assistants).

[0079] in addition, Figure 1 The illustrated structure and the above description limit the number of terminals, etc. to the range mentioned in the description for the sake of simplicity and clarity. However, it can be configured to include a larger number of publisher terminals and a larger number of viewer terminals during implementation.

[0080] [Structure of Content Distribution Server 2]

[0081] like Figure 2 As shown, the content distribution server 2 includes an input / output interface 2a as a data transmission end inside and outside the server, and a control unit 2b including a CPU (central processing unit) for controlling and managing the operation of each component of the server 2.

[0082] Similarly, the content information storage unit 2c of the server 2 stores information about the content (program) that was initially broadcast live on the Internet, including information about the background of the virtual space, virtual objects, virtual cameras (with information about the position and direction of the viewpoint, and capable of moving in the virtual space), and virtual characters (virtual agents) that can move in the virtual space, including information about the specifications of the appearance, activities (motions), etc., information about the specifications of the virtual objects, and information about the motion of the virtual objects, for the period from the beginning to the end of the content. The characteristic point of the present system 1 is that the content data is not stored as rendered data (drawn data. Data of so-called dynamic images that are directly displayed on the display screen), but is stored as motion data of the virtual objects (data about actions, directions, position coordinates, etc.) in the same manner as the data about the specifications of the virtual objects (appearance, voice characters, sound effect data, music data, etc.). Then, when the content is played by the viewer terminal, etc., it is configured so that, instead of merely displaying the rendered drawing data on the screen as in the previous structure, the drawing data is generated by rendering “during playback” according to the specification data of the virtual object and the motion data of the virtual data shown above. In this way, “rendering and generating drawing data according to the specification data of the virtual object and the motion data of the virtual data” is actually the same as the action when the content is broadcast live via the Internet, etc. That is, by utilizing the structure of this system 1, the content can be played with substantially the same action, structure, and quality as when the content is broadcast live, which is the biggest difference compared to the prior art structure. Furthermore, by utilizing the point that “rendering and generating drawing data according to the specification data of the virtual object and the motion data of the virtual data” during playback, it becomes easy to modify, add to, and change these data during playback, and generate a revised version of the content.

[0083] The content storage unit of the server 2 stores the new version of the content that has been changed from the original live broadcast, and transmits it according to the subsequent playback request from each viewer terminal. As a storage method, when a new version of the content is generated, the old version can be discarded and only the new version can be stored, or the new version of the content can be stored while retaining the old version.

[0084] The content distribution unit 2d transmits each content stored in the content information storage unit 2c to each terminal that requests viewing. The program storage unit 2f stores a program for realizing each execution operation of the server 2.

[0085] The user management unit 2g stores the information of the distributor terminal used by the distributor of each live content previously described, the audience terminal used by the audience who watched, listened to, changed or added the live content, etc. as data such as IP (Internet Protocol) addresses and account information of the motion picture distribution service, thereby using the information stored in the user management unit 2g to store which terminal a certain live content is sent to, which terminal changes or adds which live content as data. Therefore, the live content that was originally broadcasted over the Internet is changed or added in what process, and the system diagram information of the live content is also retained.

[0086] The bus 2i connects the components of these servers 2 to each other so that data can be exchanged.

[0087] The first distributor terminal 11 and the second distributor terminal 21 have a common structure, and the following description will be made taking the first distributor terminal 11 as an example.

[0088] [Structure of the First Distributor Terminal 11]

[0089] like Figure 3 As shown, the first distributor terminal 11 first includes an input / output interface 11a as an input / output terminal for information inside and outside the terminal, and a control unit 11b including a CPU (central processing unit) for overall control of various components of the terminal 11.

[0090] In order for the distributor to make a virtual character, which is his / her virtual avatar, appear in the virtual space and perform the desired actions, the virtual character operating unit 11c first stores the virtual character's appearance data, the virtual character's voice character data, the virtual character's sound effect data, the virtual character's music data, etc. in the data storage unit 11i of the first distributor terminal 11 as the specification data of the virtual character, which is one of the specification data of the virtual object described previously. The virtual character operating unit 11c uses an acceleration sensor worn on the body of the distributor as an operator, data of the distributor's own activities captured by a 3D camera, or data of the activities of various parts of the distributor's body sensed by a Doppler sensor to obtain these motion data, and sends them to the server 2 as the motion data of the virtual character, which is one of the motion data of the virtual object described previously, or stores them in the data storage unit 11i. Specifically, Figure 7 As shown, the virtual character operation unit 11c determines by preparing specification data of one of the specification data of the virtual object, including the appearance data 7-1 of the virtual character, or selecting from the selections prepared by the system, or specifying the URL (Uniform Resource Locator) of the appearance data or specification data configured on the network. Then, as a result of sensing the movement of various parts of the distributor's body, that is, the movement data 7-2, as shown in FIG. Figure 7As shown in (B), each part of the body is regarded as a node 7-2a to obtain its position information and activity information. It can also be configured to further generate information about the branch 7-2b that connects the node 7-2a, so that the relationship between the nodes 7-2a is clarified. In addition, the specification data of the virtual character, which is one of the specification data of the virtual object, refers to, for example, when a man operates a virtual character with the appearance of a young woman, the frequency characteristic data of the sound (sound character data) used to transform the sound emitted by the male into the sound of a young woman, the unique sound effect when the virtual character is active, the data of the background music that flows representatively when the virtual character appears, etc., and of course, other data can also be included. Similarly, for virtual objects other than virtual characters (also called Virtual items, virtual items, virtual objects, etc.), not only their appearance data, but also sound character data can be attached as specification data when an item is generated, and sound effect data and music data can also be attached as specification data to each virtual object. The above points are common to various virtual objects including the virtual characters used in this system 1, and there is a case where the description is omitted in the future. In addition, Table 1 shows a summary of data of virtual objects used in the present system 1 .

[0091]

Mathematical formula 1

[0092] Table 1. Related data of virtual objects

[0093]

[0094] The drawing unit 11e uses the specification data of the virtual objects sent from the outside of the terminal 11 or stored in the terminal 11, and the motion data of the virtual objects to render (draw) the structure of these virtual objects, and receives the data related to the virtual objects when playing the content recorded in the past and the live content being distributed from the current terminal 11 from the server 2, and is used for real-time confirmation of the specific content of the content. Therefore, the rendering unit 11e uses the specification data and motion data 7-2 of the virtual character including the data 7-1 of the appearance of the virtual character as shown above, and uses the specification data and motion data of the virtual objects projected into the three-dimensional virtual space as described later, the background image of the three-dimensional virtual space, and the specification data and motion data of the virtual objects placed in the virtual space to perform drawing (rendering) for visualizing the virtual space, and the obtained image of the virtual space containing the virtual objects is displayed on the display unit 11f of the first distributor terminal 11. As described above, for example, with regard to the rendering of the virtual character, Figure 7As shown in (C), for the virtual character appearance data 7-1, which is one of the specification data of the virtual object, the position and movement of each part are obtained from the motion data 7-2, and the virtual character rendering data 7-3 indicating the current position and movement of the virtual character is generated. In addition, since the rendering can be performed using the virtual character appearance data 7-1, which is not animation data but static image data, and the data of the section 7-2a or branch 7-2b constituting the motion data 7-2 of the virtual character, the data capacity is small because the appearance data 7-1 is static image data, and the data of the section 7-2a or branch 7-2b is not image data but coordinate data of a limited number of points or motion vector data, so its capacity is also small, as a result, the transmission path and the burden of hardware required for data transmission, reception, storage, and rendering can be extremely small. Furthermore, if the virtual character appearance data 7-1 or specification data is not the original data (specific original data) itself, but is specified as the data storage destination using a URL (Uniform Resource Locator), it can be transmitted and stored with smaller character string data. The advantage of these is that no matter which structure of the system is used to generate, transmit, store, and render these appearance data, specification data, and motion data, the same effect is achieved, not limited to human virtual characters, and it becomes a common effect when you want to render virtual objects. In addition, in the drawings of this application, the schematic diagrams of virtual characters recorded in various places use simple displays, without writing in the activities of the limbs, etc., but these diagrams refer to Figure 7(C) Rendering virtual character data 7-3 shown. In addition, when displaying not only appearance but also sound (in this specification, "display" is used to mean displaying not only images but also sounds using speakers, vibrators, etc., and sound effects, etc.), the information of the sound emitted by the virtual character and the character information of the sound can also be included in the object of data sent, received, or stored by the system 1. For example, as described above, when the virtual character has the appearance and role of a young woman and the distributor who operates the virtual character is a man, the original voice of the male is first stored, and the stored male voice is converted into the voice of a young woman for display of the virtual character. It is effective that the character information of the voice used in the voice conversion (the spectrum of the voice of the young female character, etc.) is included in the virtual character appearance information or other information together with the voice information of the distributor who operates the virtual character. Similarly, when adding a virtual object to live content, sound effects (explosion sounds, collision sounds, flying sounds, imitation sounds, etc.) and music can also be added, and these sound effect data and music data can also be included as data of objects sent, received, or stored by the system 1. They are also added to the avatar specification data or other data. In addition, the specification data may also include data related to vibration, in which case, for example, a controller (an operating tool with buttons, etc.) for operating the avatar may also be vibrated, and this is also the same in all places and aspects of the system 1.

[0095] The display unit 11f is a structure for the distributor to watch live content, monitor rendered images to confirm the specific content of the live content currently being distributed by the distributor, or display images for various operations. It is implemented by a display panel of a personal computer, a goggle-type HMD (head-mounted display) worn on the face, etc. The operating unit 11g is used for the distributor to perform various operations. For example, it can be a keyboard, mouse, touch panel of a personal computer, or it can also be a device that is operated by the operator's motion data. The program storage unit 11h stores computer programs required for the distribution and viewing of live content, and may also include a general-purpose computer operating system (OS), an Internet browsing program (browser), etc. The data storage unit 11i stores data on the background of the virtual space used for live content, specification data of virtual data, motion data, etc.

[0096] The comment posting unit 11k is a structure for posting a comment as character information to the server 2, the comment being displayed as an overlay on the display screen of the live content, or displayed on the surface of a virtual object in the virtual space 30, or displayed as an independent virtual object such as an object in the shape of a large character. The bus 11m connects the various structures of the distributor terminal 11 so that data can be exchanged with each other.

[0097] The first viewer terminal 12 and the second audience terminal 22 have a common structure, and the following description will be made taking the first audience terminal 12 as an example.

[0098] [Structure of the First Viewer Terminal 12]

[0099] like Figure 4 As shown, the first viewer terminal 12 includes an input / output interface 12 a as a data transmission / reception terminal inside and outside the terminal 12 , and a control unit 12 b including a CPU (central processing unit) for overall control of various parts of the terminal 12 .

[0100] First, the virtual character operating unit 12c stores the virtual character's appearance data, the virtual character's voice character data, the virtual character's sound effect data, the virtual character's music data, etc., as one of the virtual object's specification data described previously, in the data storage unit 12i. The virtual character operating unit 12c makes the virtual character, which is the viewer's own virtual avatar, appear in the virtual space, and performs the desired action by using the acceleration sensor attached to the viewer's body, the data of the viewer's own movement captured by the 3D camera, or the data of the movement of various parts of the viewer's body sensed by the Doppler sensor, and obtains these motion data, and sends them directly or as drawn image information to the server 2. As described previously, if the reference Figure 7 , the avatar operation unit 12c determines by preparing the avatar specification data including the avatar appearance data 7-1 as one of the specification data of the avatar data, or by selecting from the avatar specification data prepared by the system and available for selection, or by specifying the URL (Uniform Resource Locator) including the specification data including the appearance data configured on the network. Then, as a result of the movement of various parts of the body being sensed, that is, the motion data 7-2, such as Figure 7As shown in (B), each part of the body is regarded as a node 7-2a to obtain its position information and activity information. It can also be configured to further generate information about the branch 7-2b that connects the node 7-2a, so that the relationship between the nodes 7-2a is clarified. In addition, in the case of displaying not only the appearance but also the sound (in this specification, "display" is used to mean not only the display of images but also the display of sounds, sound effects, etc. using speakers, vibrators, etc.), the information of the sound emitted by the virtual character and the role information of the sound can also be included in the appearance data of the virtual character, and included in the object of data sent, received or stored by the system 1 as the specification data of the virtual character. For example, in the case where the virtual character has the appearance and role of a young woman and the distributor operating the virtual character is a man, the original voice of the man is first stored, and the stored male voice is converted into a voice and used as the voice of a young woman for the display of the virtual character. It is effective that the character information of the voice used in the voice conversion (the spectrum of the voice of the young female character, etc.) is included in the virtual character specification information combined with the appearance information of the virtual character together with the voice information of the audience and other people who operate the virtual character. Similarly, when adding a virtual object to the live content, sound effects (explosion sound, collision sound, flying sound, imitation sound, etc.) and music can also be added, and these sound effect data and music data can also be used as data of objects sent, received or stored by the system 1. They are also added to the specification data of the virtual character together with the appearance information or other data.

[0101] In order for viewers to make their own virtual characters appear in the live content they are watching, the project submission unit 12e is used to submit the specification data of the virtual object including the appearance data of the virtual character and the motion data of the virtual object described above to the server 2, or to submit a new virtual object to the live content they are currently watching.

[0102] The display unit 12f is configured to display a screen for the viewer to view live content or perform various operations, and is implemented by a display panel of a personal computer, a goggle-type HMD (head mounted display) worn on the face, or the like.

[0103] The operation unit 12g is used by the audience to perform various operations, and may be a keyboard, mouse, or touch panel of a personal computer, or may be a device operated by the operator's motion data. The program storage unit 12h stores computer programs required for viewing live content, and may also include a general-purpose computer operating system (OS), an Internet browsing program (browser), etc. The data storage unit 12i stores the data described above and various other data.

[0104] The comment posting unit 12k is a structure for posting comments to the server 2, and the comments are overlaid on the display screen of the live content, or displayed on the surface of the virtual object in the virtual space 30, or are character information displayed as its own independent virtual object such as a large character-shaped virtual object. The bus 12m connects the various structures of the viewer terminal 12 in a manner that can exchange data with each other.

[0105] Figure 5 This figure schematically shows the appearance of the first viewer terminal 12 implemented by a personal computer, and includes a display panel 12-1 that displays various images, static images, dynamic images, and character information. When the viewer enters a URL (Uniform Resource Locator) that has been notified in advance in order to utilize the services of this system 1 and connects to the server 2, the necessary data and script (JavaScript (registered trademark)) program are received from the server 2, thereby displaying a portal screen (not shown) for the animation distribution service.

[0106] Figure 5 The diagram shows a situation in which it is assumed that the viewer selects live content that he or she wants to watch as an Internet live broadcast or as broadcast content from a display list (not shown) in advance, and the live content is displayed.

[0107] exist Figure 5 In the video, there is an animation display area 12-1a for displaying live content animation in the display panel 12-1, and a virtual character 12-1b of the first distributor who is the distributor of the live content, and a tree 12-1n and a house 12-1o which are virtual objects arranged in the virtual space using the live content are displayed here. In addition, a comment 12-1r is further displayed in the display panel 12-1, and the comment 12-1r with the content such as "Turn up! Thank you!!" appears from the right end of the display panel 12-1, scrolls to the left end, and disappears. The range in which the comment 12-1r is displayed is set to be larger than the animation display area 12-1a for displaying the live content animation, and extends from there, so that the audience can clearly understand that these comments 12-r are not the original content of the content but are contributed by other contributors.

[0108] Similarly, the display panel 12-1 displays a home button display 12-1e for returning to the home (entrance) page (not shown) of the animation distribution service, a stop button display 12-1f used when the viewing content is not live but broadcast content, a pause button display 12-1g, and a play button display 12-1h.

[0109] The comment posting button display 12 - 1 i in the display panel 12 - 1 is a button for posting a previously explained comment on the content that the viewer is viewing.

[0110] When the content is a playback content, the search bar display 12-1k makes the playback start time correspond to the left end and the playback end time correspond to the right end. The position of the search button display 12-1m indicates the current playback time. The position of the search button display 12-1m can be moved by the mouse pointer 12-3 to move it to the desired playback time. As the operation unit 12g described above, there are a mouse pointer 12-3 and a mouse cursor 12-2, and the viewer terminal 12 also has a keyboard 12-4.

[0111] [Audio and video of live broadcast]

[0112] Next, use Figure 6 Signal flow diagram of Figure 8 Schematic diagram of the first virtual space, Fig. 9 The diagram of the rendering results observed from each viewpoint explains the operation until the live content of the Internet broadcast posted and distributed from the first distributor terminal 11 is viewed and listened to by the first viewer terminal 12.

[0113] The distributor who intends to host live content for distribution uses the first distributor terminal 11 to apply for content distribution to the server 2 (step S101). When the server 2 permits the distribution, the first distributor terminal 11 is notified of the permission from the server 2 (step S102), and the server 2 publicizes the live content as a list on an appropriate page of the distribution service website as content that can be viewed and listened to, or distributes the list to the logged-in viewer terminal (step S103).

[0114] Since the distribution is permitted, the first distributor terminal 11 will Figure 8 Each virtual object within the virtual space 30 shown, i.e., the virtual character display 12-1b of the first distributor, specification data related to other virtual objects, and motion data are sent to the server 2 as live content for Internet live broadcast (S104), and the specification data and motion data related to these virtual objects are stored in the server 2.

[0115] As described above, the distributor generates motion data and controls the actions of the virtual character 12-1b, so the virtual character 12-1b can move, change direction, change posture, etc. in the virtual space 30. Similarly, when using "Table 1 Virtual Object Association Data", as previously explained, not only the appearance of virtual objects including virtual characters but also sounds are displayed (in this specification, "display" is used to mean not only the display of images but also the display of sounds, sound effects, etc., such as speakers and vibrators.), the information on the sounds emitted by virtual objects including virtual characters and the character information of the sounds are included in the objects of data sent, received, or stored by the system 1 as specification information of the virtual objects.

[0116] The virtual camera 32 configured in the virtual space 30 does not need to be set in the center as shown in the figure, and the setting location can be arbitrary, movable, or multiple. In addition, the audience who is playing can also add or move these virtual cameras from the middle of the live content as part of changing or adding actions to the live content. The use of these virtual cameras is also common in other places in this manual, but in order to avoid complexity in the subsequent description, the virtual camera is limited to only one example in each virtual space.

[0117] Through the list, the audience who knows the existence of the live content in the broadcast applies to the server 2 from the first audience terminal 12 for viewing and listening to the live content as an Internet live broadcast (step S105). As a result, the server 2 sends the first audience terminal 12 the specification data and motion data of the virtual object located in the virtual space 30 including the motion data 7-2 of the virtual character 12-1b of the first distributor or the appearance data 7-1 of the virtual character (step S106), and renders it at the first audience terminal 12. The rendering result is exemplified as Fig. 9 (A) (image from the viewpoint of the first distributor avatar) and Fig. 9 (B) (Image viewed from the viewpoint of the first virtual camera).

[0118] Furthermore, the live content distributed by the first distributor terminal 11 is not limited to the first viewer terminal 12 , and other users of the present moving image distribution system including the second viewer terminal 22 can also apply for and view it.

[0119] [Playback of content after live broadcast]

[0120] Next, refer to Fig.10 , the following actions are explained: the first distributor terminal 11 performs Internet live broadcast, and the live content stored as specification data and motion data of the virtual object in the server 2 is played from the first viewer terminal 12 by application after the live broadcast time.

[0121] First, a viewer who wants to play live content stored in the server 2 selects the desired live content from a list representing the entire live content prepared in advance by the server 2, and applies to the server 2 for playing. In the server 2, the specification data and motion data of the virtual object as the designated live content are sent to the first viewer terminal 12, so that the viewer uses the first viewer terminal 12 to render (draw). Fig. 9 The images shown in (A) and (B) are displayed on the display panel 12-1 of the first viewer terminal 12, and as a result, the viewer can view and listen to the live content used in the virtual space. Fig. 9 (A) Fig. 9As shown in (B), the audience can also watch and listen to the rendered image observed from the viewpoint of the first publisher's virtual character 12-1b, and can also watch and listen to the rendered image observed from the first virtual camera 32 according to the switching operation. Therefore, unlike the situation of simply passively watching the TV broadcast screen or the movie screen, it is possible to actively switch multiple different viewpoints at will while watching the specific content of the content. Here, the live content uses a 3D (three-dimensional) virtual space. Therefore, while switching the viewpoint of viewing, the audience can watch and listen to the virtual space 30 from multiple directions. The audience can get a sense of presence, immersion, and connection that cannot be obtained by watching 2D (two-dimensional) content. Such a sense of presence, etc., can convey to the audience the feeling of watching live content being broadcasted even when watching the broadcast content. In addition, if it is explained from another aspect, it is the point explained before, but as a representative point of this system 1, the data of the content is not maintained as rendered data (already drawn data. So-called dynamic image data directly displayed on the display screen), but the specification data of the virtual object as a virtual object (appearance, voice character, sound effect data, music data, etc.) and the motion data of the same virtual object (action data, direction data, position coordinate data, etc.) are maintained. And when the content is played on the audience terminal, etc., it is not like the existing structure that only the rendered drawing data is displayed on the screen, but it is rendered at "play time" according to the specification data of the virtual object and the motion data of the virtual data shown above to generate drawing data. In this way, "rendering and generating drawing data according to the specification data of the virtual object and the motion data of the virtual data" is the same as the action when the content is actually broadcast live via the Internet. That is, by using the structure of this system 1, the content can be played with the same action, structure, and quality as when the content is broadcast live, which is a major difference from the existing technical structure. Furthermore, by utilizing the fact that "rendering is performed based on the specification data of the virtual object and the motion data of the virtual data to generate the drawing data" during playback, it is easy to modify, add to, and change these data during playback to become a modified version of the content. This is explained below.

[0122] [Addition of virtual items to the content played after the live broadcast]

[0123] Next, use Figures 11 to 14 , indicating that in use Figures 6 to 9 The description is about the operation in which the first viewer terminal 12 plays and views the live content previously broadcast over the Internet by the first distributor terminal 11, and the viewer using the first viewer terminal 12 changes or adds the live content to the content he or she desires.

[0124] Similar to the live content playback process described previously, an application for live content playback is made from the first viewer terminal 12 to the server 2 (step S301).

[0125] The server 2 that has accepted the application transmits the live content as a target to the first viewer terminal 12 as data of virtual object specifications and motion data for rendering and content viewing as described above (step S302 ).

[0126] As described above, the audience can simply watch the live content being transmitted, but if it is a dance scene, for example, they themselves participate in the live broadcast as a virtual character, and sometimes they may want to dance with the operators of the virtual characters who are already participating in the live broadcast to gain a sense of solidarity. They may also want to immerse their virtual character in the live content and watch the live content again from their own desired viewpoint.

[0127] Therefore, in order to add his own virtual character 12-1c to the live content while playing the live content, the viewer requests the server 2 to send the specification data or motion data of the virtual object including the appearance information of the virtual character as necessary data to the server 2 as additional item data (step S303). Figure 7 As described above, data 7-3 rendered using specification data including appearance data 7-1 of a virtual character, motion data 7-2 of a virtual character, etc. is added. The same applies when other virtual objects are added from the audience terminal 12. Therefore, other viewers who play and watch the live content later watch the viewer's virtual character 12-1c and other added virtual objects with the desired actions and postures operated by the viewer. The viewer who initially added the virtual objects using the audience terminal 12 can get the feeling as if he or she is participating in the live content.

[0128] In response to this, the server 2 newly adds the audience's virtual character 12-1c or other virtual objects in the virtual space 30, and sends the specification data and motion data of all virtual objects of the content including the additional part to the first audience terminal 12 (step S304), and also adds the data of the virtual character 12-1c and other virtual objects added to the data of the live content stored in the server 2, so that other audience terminals that play the live content in the future can watch and hear the live content including the audience's virtual character 12-1c and the virtual objects added. In addition, for the newly added virtual objects such as virtual characters, it is very effective to set the added time information, the added terminal or user account information, etc. as the data of the object to be sent, received, and stored by the system 1. Such information about the time, terminal, and user can be stored by the server 2, and can also be written into the information of the specifications of the virtual objects added in parallel or separately. Setting the added time information, the added terminal or user account information, etc. as the data to be sent, received, and stored by the system 1 is also effective in other aspects of the present system 1, and this point is the same and may be omitted in the future.

[0129] exist Fig.12 , Fig.13 In addition to the virtual character 12 - 1 c of the first viewer, a state in which virtual objects such as a traffic light 33 and a second virtual camera 34 are newly added is shown.

[0130] In addition, the change of the live content may require the permission of the original sponsor of the live content, or the server 2 may continue to store both the changed data and the data before the change when the live content is changed, and each viewer may play any live content according to the operation. If so constituted, the viewer can get satisfaction in that the live content changed by himself can be played to other multiple viewers, and the viewer who does not like such a change can also screen the content before the change, and it is possible to take into account the original difficult-to-take-to-take-to-take issues such as the raising of the sense of solidarity and the appreciation of one's own preferences.

[0131] In addition, in addition to the organizers, performers, etc. of the live content that you watch or respect, you can also specifically select and watch the live content that has been changed by the audience you watch or respect. As described above, the server 2 can also prepare system tree data of changes to a certain live content, so that by tracking the changes to the live content that is preferred to you, the audience can also enjoy the fun of achieving a version of the content that is more suitable for them.

[0132] [Add virtual characters to the content played after the live broadcast, and distribute it by other distributors]

[0133] In the above description, the example is that each viewer can later change or add to the live content that was originally broadcast over the Internet according to their own preferences, but the broadcast live content (which can be the same as the live broadcast, or can be changed or added to later) can also be broadcast again through the Internet by the same distributor or a different distributor, and the audience who watch and listen to the broadcast can similarly change or add to the live content.

[0134] exist Fig.14 , Fig.15 , Fig.16 , but in each figure, a second distributor's avatar 35 is added. The details of each action are the same as those described above.

[0135] [Variation 1: Arbitrariness of Arrangement of Each Structure]

[0136] In the above description of the present embodiment, the viewer terminal used by the viewer renders the viewpoint images of the virtual space that has been changed or added by the viewer. However, such a configuration is not necessary or essential in the implementation of the present invention. That is, the rendering may also be performed by a server or other various terminals. In order to perform such rendering, it is sufficient to send the specification data and motion data of the virtual object to the server, terminal or both to be rendered in advance. The detailed structure or process is clear from the structure of the present embodiment described above, so repetition is avoided.

[0137] [Variation 2: Throwing a virtual object into a virtual space]

[0138] In the foregoing description, examples of adding virtual characters 12-b, 12-1c as one's own avatars and adding a traffic light 33 were described as examples of viewers of live content adding new virtual objects to the virtual space according to their own preferences.

[0139] However, in live performances and dramas in the real world, performers can be more and more excited by sending blessings (also called "ohineri") or flowers to the performers. Correspondingly, in content distribution, there are also proposals to display virtual objects (virtual objects) equivalent to such blessings in the virtual space to express approval and applause for the performers.

[0140] Japanese Patent Gazette No. 2012-120098, which is a well-known patent document, discloses that virtual items such as flowers, characters, logos, etc., purchased as "decorations" by viewers of the content are pasted on the display screen of a live broadcast program and arranged at free positions on the display screen to decorate the display screen.

[0141] In the embodiment system 1 of the present invention, as part of the invention, the audience playing the live content using the virtual space will invest virtual objects to express applause and approval to the performer. Later, other viewers playing the live content can visually confirm the invested virtual objects, and as a result, the audience's approval or praise intention of investing the virtual objects can be widely recognized, or the audience who watch and listen to the live content in the future will also feel the approval or praise. Unlike the situation of investing virtual objects in live content during a live broadcast, although the investment in the broadcast content is not real-time investment (approval or praise), if it is broadcast, it can be watched and listened to multiple times, the level or number of the audience is also expanded, and fame brings fame, and a large amount of investment can be made. Since they are also accumulated, both the audience side and the performer side who receives the gift have advantages.

[0142] Fig.17 , Fig.18 The schematic diagram shown in the figure shows a mode in which a strip-shaped virtual object 12 - 1 d as such a virtual object (virtual item) is thrown into the virtual space 30 by a viewer who plays back the content using a viewer terminal.

[0143] The ribbon-type virtual object 12-1d includes specification data including image data of appearance, motion data including position data and orientation data, and can be added to the virtual space 30 by the audience using the audience terminal, etc. in the same way as the distributor or audience's virtual objects 12-1b and 12-1c are added to the virtual space 30 as described previously.

[0144] In more detail, the system 1 of this embodiment is constructed as follows: the strip-shaped virtual object 12-1d sends and stores the specification data and motion data of the virtual object in the same steps and structure as the specification information including the appearance information of the virtual character (agent) of the distributor or the audience, and the motion data (action information).

[0145] Therefore, the strip-shaped virtual object 12-1d used as a virtual object in the investment can have specification information and action information (position information, movement direction / speed information, posture information, etc.) including appearance information, voice character information, sound effects, and music information, just like the virtual object.

[0146] By providing a virtual object for investment with specification information and action information including such information of appearance, the strip-shaped virtual object 12-1d can perform various performances that increase the user's interest and attention. In addition, as a virtual object for investment, it is possible to implement it as a virtual object having data of various appearances, etc. different from the strip-shaped virtual object 12-1d.

[0147] As an example using the strip-shaped virtual object 12-1d, Fig.17 (A) The schematic diagram shown in the figure means a performance in which a strip-shaped virtual object 12-1d, which is a virtual item thrown into the virtual space 30, falls from top to bottom and lands on the ground of the virtual space. The throwing of virtual objects is similar to the feeling of throwing a "wedding seal" onto the stage in a theater or throwing a bouquet of flowers at a wedding. Such a performance of falling from top to bottom and landing on the ground of the virtual space 30 is a natural performance that easily resonates with the audience.

[0148] Likewise, Fig.17 The schematic diagram shown in (B) is a performance as follows: if the target person who is to inject the virtual object is determined to be, for example, the first distributor 12-1b and a predetermined operation is performed using the viewer terminal, the strip-shaped virtual object 12-1d injected into the virtual space 30 moves in the virtual space toward the first distributor 12-1b as the target of injection. The example of "Hi Seal" described above can also be a natural performance.

[0149] Likewise, Fig.18 (A) is a performance in which the strip-shaped virtual object 12-1d inserted into the virtual space 30 automatically rotates and moves (flies around) in the virtual space 30, or moves back and forth in the virtual space according to the operation of the inserted person. Compared with the case where the inserted virtual object remains stationary in the virtual space, this is a very noticeable performance and contributes to the expression of engagement (appeal).

[0150] Fig.18 (B) The schematic diagram shown in the figure is a performance in which the strip-shaped virtual objects 12-1d thrown by many viewers increase in number or the same person throws multiple virtual objects continuously, and these thrown objects are temporarily stacked and then collapse. For popular live content, popular scenes, and popular performers, it is predicted that the throwing of virtual objects will be concentrated, and such a performance is highly feasible and has a great effect on the performance.

[0151] Including the points already explained, if the preferred method for the performance when inserting a virtual object is listed, first of all, in addition to setting the object to be inserted as the entire live content, it is also possible to determine a specific performer or other object to operate and then insert it. The virtual object to be inserted can also be placed next to the subject or marked with the subject's name.

[0152] In addition, it is desirable to place the thrown virtual object within the field of view of the virtual camera.

[0153] (Description of Effects of the Invention)

[0154] The purpose of the present invention is to solve various problems, namely, to want to watch and enjoy live content that was not available when the live broadcast was performed, and to obtain the sense of presence and immersion as if watching the live broadcast even in the future audio-visual broadcast, or to reduce the burden on the communication network and the hardware of each terminal, and to enjoy the audio-visual of the live content while sharing it with multiple viewers.

[0155] The present invention can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope or spirit of the invention, and are also included in the invention described in the claims and their equivalents.

Claims

1. A three-dimensional content distribution system that distributes content using a virtual three-dimensional space to viewing terminals for viewing, characterized in that: The three-dimensional content distribution system comprises: a recording unit that records content data, the content data including at least (A) first data related to specifications of each virtual object arranged in the three-dimensional virtual space, and (B) second data related to actions of each virtual object; as well as a first transmitting unit that transmits data including at least the first data and the second data in response to a request for the viewing terminal to view the audio; as well as The second sending unit receives the data sent from the first sending unit, and sends data including at least the second data to the recording unit so that the viewer can immerse a virtual character as himself in the content while viewing the content. The second data is obtained by the operating unit as motion data including movements of various parts of the viewer's body in order to make the virtual character perform desired actions.

2. The three-dimensional content distribution system according to claim 1, characterized in that: The recording unit further records (C) third data related to a viewpoint arranged in the three-dimensional virtual space.

3. The three-dimensional content distribution system according to claim 1 or 2, characterized in that: The first data and the second data recorded by the recording unit are data initially recorded when the content is first distributed over the network.

4. The three-dimensional content distribution system according to any one of claims 1 to 3, characterized in that: have: The change accepting unit makes a change for adding or modifying at least one of the first data and the second data in response to a change request from the viewing terminal that is currently viewing the content, thereby generating new content data for subsequent viewing.

5. The three-dimensional content distribution system according to claim 2, characterized in that: Also features: a change accepting unit that performs a change for adding or modifying at least one of the first data and the second data in response to a change request from the viewing terminal that is currently viewing the video, thereby generating new content data for subsequent viewing; The change accepting unit is further configured to make a change for adding or correcting the third data, thereby generating new content data for subsequent viewing.

6. A three-dimensional content distribution method, which distributes content using a virtual three-dimensional virtual space to an audio-visual terminal for viewing, characterized in that: The three-dimensional content distribution method comprises: A recording step of recording content data, the content data including at least (A) first data related to specifications of each virtual object arranged in the three-dimensional virtual space, and (B) second data related to actions of each virtual object; A first sending step of sending data including at least the first data and the second data in response to a request for the viewing terminal to view the audio; as well as The second sending step receives the data sent in the first sending step, and sends data including at least the second data to a recording unit so that the viewer can immerse a virtual character as himself in the content while viewing the content, wherein the second data is obtained by the operating unit as motion data including the movement of various parts of the viewer's body in order to make the virtual character perform a desired action.

7. The three-dimensional content distribution method according to claim 6, characterized in that: The recording step further records (C) third data related to a viewpoint arranged in the three-dimensional virtual space.

8. The three-dimensional content distribution method according to claim 6 or 7, characterized in that: The first data and the second data recorded in the recording step are data initially recorded in the first network distribution of the content.

9. The three-dimensional content distribution method according to any one of claims 6 to 8, characterized in that: have: The change acceptance step performs a change for adding or modifying at least one of the first data and the second data in response to a change request from the viewing terminal performing the viewing, thereby generating new content data for subsequent viewing.

10. The three-dimensional content distribution method according to claim 7, characterized in that: Also features: a change acceptance step of performing a change for adding or modifying at least one of the first data and the second data in response to a change request from the viewing terminal that is currently viewing the video, thereby generating new content data for subsequent viewing; The change acceptance step is further configured to perform a change for adding or correcting the third data, thereby generating new content data for subsequent viewing.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a computer, the steps of the three-dimensional content distribution method according to any one of claims 6 to 10 are implemented.

Images