VIRTUAL SPACE CONTENT DELIVERY SYSTEM, VIRTUAL SPACE CONTENT DELIVERY PROGRAM, AND VIRTUAL SPACE CONTENT DELIVERY METHOD

Abstract

To provide a virtual space content delivery system capable of preventing an excessive increase in processing load. [Solution] When a specific condition corresponding to the processing load related to virtual space content in a virtual space content distribution system increasing to a predetermined specific level is met, a second viewer avatar which requires a smaller processing load for display is used as the viewer avatar, whereas when the specific condition is not met, a first viewer avatar is used as the viewer avatar.

Description

[Technical field]

[0001] The present invention relates to a virtual space content distribution system capable of distributing virtual space content in a virtual space, a virtual space content distribution program, and a virtual space content distribution method. [Background technology]

[0002] Conventionally, a distribution system has been known that distributes content of a character object (avatar) 10 operating in a virtual space, and also enables viewer users to virtually participate as spectators in the virtual space in which the distributor's (performer's) character object (avatar) 10 is located, in response to a request from the viewer user (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-186797 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the system of Patent Document 1, in addition to processing to sequentially update the virtual space data, processing is required to draw the virtual space and each character object (avatar) of the distributor (performer) and viewer user to generate an image of the virtual space, but no consideration is given to an increase in the number of character objects of the distributor (performer) in the virtual space, or an increase in character objects of viewer users who virtually participate in the virtual space as spectators. As a result, an increase in the number of character objects 10 participating in these virtual spaces causes an excessive increase in the processing load on the content distribution server 500 and viewer terminal 300 that make up the distribution system, resulting in inconveniences such as viewers being unable to view the content properly.

[0005] The present invention has been made in response to such problems, and aims to provide a virtual space content distribution system, a virtual space content distribution program, and a virtual space content distribution method that can prevent an excessive increase in processing load. [Means for solving the problem]

[0006] The virtual space content delivery system of claim 1 comprises: Avatars can participate A virtual space content distribution system capable of distributing virtual space content (e.g., a virtual live performance) in a virtual space, The image of the virtual space content output a viewer user terminal (e.g., viewer terminal 300) that is a terminal capable of viewing the video and allows a viewer user to view the video; A server computer (e.g., a delivery server computer 100) that is connected to the viewer user terminal via a communication network and is capable of executing at least processes related to the virtual space and processes related to the delivery of the virtual space content; At least the server computer stores timeline data capable of identifying a specific timing corresponding to a specific control for an avatar participating in the virtual space content in the virtual space content; When a specific condition corresponding to the specific control is satisfied at the specific timing specified based on the timeline data, the specific control is executed. It is characterized by the following. According to this feature, when certain conditions are met, Specific control is performed Therefore, it is possible to prevent an excessive increase in the processing load caused by the specific condition being satisfied.

[0020] The virtual space content delivery program according to claim 15, Avatars can participate A virtual space content distribution program for distributing virtual space content (e.g., a virtual live performance) in a virtual space, The virtual space content includes: a server computer that is capable of executing at least processes related to the virtual space and processes related to the distribution of the virtual space content, Viewer User To Content including video that can be viewed by a viewer user at an assigned viewer user terminal (e.g., viewer terminal 300), a specific timing identification process for identifying a specific timing based on timeline data that is data stored in the server computer and that can identify a specific timing corresponding to a specific control related to an avatar participating in the virtual space content in the virtual space content; A specific control that executes the specific control when a specific condition corresponding to the specific control is satisfied at the specific timing specified by the specific timing specification process. and A virtual space content delivery program comprising: According to this feature, when certain conditions are met, Specific control is performed Therefore, it is possible to prevent an excessive increase in the processing load caused by the specific condition being satisfied.

[0021] The virtual space content delivery method according to claim 16, Avatars can participate A virtual space content distribution method for distributing virtual space content (e.g., a virtual live performance) in a virtual space, comprising: The virtual space content includes: a server computer that is capable of executing at least processes related to the virtual space and processes related to the distribution of the virtual space content, Viewer User To Content including video that can be viewed by a viewer user at an assigned viewer user terminal (e.g., viewer terminal 300), a specific timing identifying step of identifying a specific timing based on timeline data that is data stored in the server computer and that can identify a specific timing corresponding to a specific control related to an avatar participating in the virtual space content in the virtual space content; A specific control that executes the specific control when a specific condition corresponding to the specific control is satisfied at the specific timing specified in the specific timing specifying procedure. and A virtual space content distribution method comprising: According to this feature, when certain conditions are met, Specific control is performed Therefore, it is possible to prevent an excessive increase in the processing load caused by the specific condition being satisfied.

[0022] Furthermore, the present invention may have only the invention-specific matters described in the claims of the present invention, or may have the invention-specific matters described in the claims of the present invention as well as configurations other than the invention-specific matters. [Brief description of the drawings]

[0023] [Figure 1] 1 is a block diagram showing an example of a system configuration of a virtual space content delivery system according to a first embodiment of the present invention. [Diagram 2] 2 is a diagram illustrating an example of the configuration of a delivery server computer in the virtual space content delivery system according to the first embodiment of the present invention. FIG. [Diagram 3]2 is a diagram showing various data stored in a storage device of a delivery server computer according to the first embodiment of the present invention. FIG. [Figure 4] 2 is a diagram illustrating an example of the configuration of a performer terminal in the virtual space content delivery system according to the first embodiment of the present invention. FIG. [Diagram 5] 2 is a diagram illustrating an example of the configuration of a viewer terminal in the virtual space content delivery system according to the first embodiment of the present invention. FIG. [Figure 6] 2 is a diagram illustrating an example of the configuration of an administrator terminal in the virtual space content delivery system according to the first embodiment of the present invention. FIG. [Figure 7] 3 is a diagram showing an example of the configuration of event data in the virtual space content delivery system according to the first embodiment of the present invention; FIG. [Figure 8] 2 is a diagram showing an example of the configuration of performer user data in the virtual space content delivery system according to the first embodiment of the present invention; FIG. [Figure 9] 2 is a diagram showing an example of the configuration of viewer user data in the virtual space content delivery system according to the first embodiment of the present invention. FIG. [Figure 10] 2 is an explanatory diagram showing data exchanged between each terminal and a server computer constituting the virtual space content delivery system according to the first embodiment of the present invention; FIG. [Figure 11] FIG. 1 is a schematic diagram showing the facilities of a studio used by a performer user in a virtual space content delivery system according to a first embodiment of the present invention. [Figure 12] FIG. 1 is a diagram showing a virtual live concert venue, which is a virtual space used in a virtual space content delivery system according to a first embodiment of the present invention. [Figure 13] 3 is a diagram showing a display example on a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention. FIG. [Figure 14] 3 is a diagram showing a display example on a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention. FIG. [Figure 15] 3A to 3C are diagrams illustrating examples of displays on a performer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention. [Figure 16]1A to 1C are diagrams illustrating an example of a display by a zoom operation on a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention. [Figure 17] 1A to 1C are diagrams illustrating an example of a display by a zoom operation on a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention. [Figure 18] 3 is a diagram showing a display example on a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention. FIG. [Figure 19] FIG. 2 is a diagram showing an example of the display of a flying performance period on a viewer terminal constituting the virtual space content delivery system of the first embodiment of the present invention. [Figure 20] FIG. 2 is a flow diagram showing an example of a viewer viewpoint video control process executed in the viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention. [Figure 21] FIG. 11 is an explanatory diagram showing the contents of a virtual live performance delivered by the virtual space content delivery system according to the second embodiment of the present invention. [Figure 22] 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a second embodiment of the present invention. FIG. [Figure 23] FIG. 11 is a diagram showing a participant user table used in a delivery server computer constituting a virtual space content delivery system according to a second embodiment of the present invention. [Figure 24] 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a second embodiment of the present invention. FIG. [Diagram 25] 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a second embodiment of the present invention. FIG. [Figure 26] 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a second embodiment of the present invention. FIG. [Figure 27]FIG. 11 is an explanatory diagram showing the relationship between the period type of a virtual live show and the entity that generates the virtual camera viewpoint video in the virtual space content delivery system according to the second embodiment of the present invention. [Figure 28] FIG. 11 is an explanatory diagram showing the generation status of a virtual camera viewpoint video and a viewer viewpoint video in a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a second embodiment of the present invention. [Figure 29] FIG. 11 is a flowchart showing a viewer viewpoint video generation process in a modified example of the second embodiment of the present invention. [Diagram 30] FIG. 13 is a diagram showing the contents of virtual space update data transmitted from a delivery server computer in a modified example of the second embodiment of the present invention. [Diagram 31] FIG. 11 is an explanatory diagram showing the contents of a virtual live performance delivered by a virtual space content delivery system according to a third embodiment of the present invention. [Diagram 32] FIG. 11 is a diagram showing a virtual live concert venue, which is a virtual space used in a virtual space content distribution system according to a third embodiment of the present invention. [Diagram 33] FIG. 11 is a diagram showing a virtual live concert venue, which is a virtual space used in a virtual space content distribution system according to a third embodiment of the present invention. [Diagram 34] FIG. 11 is a diagram showing a virtual live concert venue, which is a virtual space used in a virtual space content distribution system according to a third embodiment of the present invention. [Diagram 35] FIG. 11 is a diagram showing a virtual live concert venue, which is a virtual space used in a virtual space content distribution system according to a third embodiment of the present invention. [Diagram 36] FIG. 11 is a diagram showing an example of the configuration of time schedule data (TS data) stored in a delivery server computer constituting a virtual space content delivery system according to a third embodiment of the present invention. [Figure 37] FIG. 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a third embodiment of the present invention. [Figure 38]FIG. 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a third embodiment of the present invention. [Figure 39] FIG. 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a third embodiment of the present invention. [Diagram 40] FIG. 11 is an explanatory diagram showing a friend user data table stored in a viewer terminal constituting a virtual space content delivery system according to a third embodiment of the present invention. [Diagram 41] FIG. 11 is an explanatory diagram showing an example of an avatar simplification determination process in the virtual space content delivery system according to the third embodiment of the present invention. [Diagram 42] FIG. 11 is an explanatory diagram showing an example of a viewer's viewpoint video generation process in the virtual space content delivery system according to the third embodiment of the present invention. [Diagram 43] FIG. 11 is an explanatory diagram showing an example of a virtual camera viewpoint video generation process (viewer terminal) in the virtual space content delivery system according to the third embodiment of the present invention. [Diagram 44] 10(A) is an explanatory diagram showing the appearance of each avatar generated in the viewer viewpoint video generation process in the virtual space content delivery system of Example 3 of the present invention, and FIG. 10(B) is an explanatory diagram showing the appearance of each avatar generated in the virtual camera line of sight video generation process (viewer terminal) in the virtual space content delivery system of Example 3 of the present invention. [Diagram 45] 10(A) is an explanatory diagram showing the appearance of a performer avatar in a virtual space content distribution system according to a third embodiment of the present invention, and FIG. 10(B) is an explanatory diagram showing the appearance of a viewer avatar in a virtual space content distribution system according to a third embodiment of the present invention. [Figure 46] FIG. 11 is an explanatory diagram showing changes in the appearance of an avatar in a virtual space content delivery system according to a third embodiment of the present invention. [Figure 47] FIG. 11 is an explanatory diagram showing changes in the appearance of an avatar in a virtual space content delivery system according to a third embodiment of the present invention. [Figure 48]FIG. 11 is an explanatory diagram showing changes in the appearance of an avatar in a virtual space content delivery system according to a third embodiment of the present invention. [Figure 49] FIG. 11 is an explanatory diagram showing changes in the appearance of a viewer avatar in a virtual space content delivery system according to a third embodiment of the present invention. [Figure 50] FIG. 13 is an explanatory diagram showing changes in the appearance of a performer avatar in a modified example of the third embodiment of the present invention. [Figure 51] FIG. 13 is an explanatory diagram of a simplified avatar as a viewer avatar in a modified example of the third embodiment of the present invention. [Figure 52] FIG. 13 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system in a modified example of the third embodiment of the present invention. [Diagram 53] FIG. 13 is an explanatory diagram illustrating an example of a pre-data transmission process in a modified example of the third embodiment of the present invention. [Figure 54] FIG. 13 is an explanatory diagram of a simplified avatar as a viewer avatar in a modified example of the third embodiment of the present invention. [Figure 55] FIG. 13 is an explanatory diagram of a simplified avatar as a viewer avatar in a modified example of the third embodiment of the present invention. [Figure 56] FIG. 13 is an explanatory diagram of a simplified avatar as a viewer avatar in a modified example of the third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] The following describes the form for implementing the virtual space content distribution system of the present invention based on the embodiments and with reference to the drawings. Note that the same or similar components in the drawings are given the same reference numerals, and duplicated explanations are omitted. The virtual space content distribution program according to the present invention may be a program used in the entire system consisting of the server computer and each terminal described later that constitute the virtual space content distribution system, or may be a program that operates only on a part of the devices that constitute the virtual space content distribution system, such as the server computer and each terminal, and each process described below is executed by a program stored in the device that performs the process. EXAMPLES

[0025] Fig. 1 is a block diagram showing an example of the system configuration of a virtual space content distribution system in Example 1 which is an embodiment of the present invention, and in the system of Example 1 shown in Fig. 1, a virtual live performance in which a performer avatar 1 corresponding to a performer user performs a live event in a virtual live venue shown in Fig. 12 is distributed as virtual space content. Note that in the following, distribution means transmission to multiple people and is synonymous with transmission.

[0026] As shown in FIG. 1, the virtual space content distribution system of this embodiment 1 is mainly composed of a distribution server computer 100 capable of executing various processes related to the virtual live venue shown in FIG. 11, which is a virtual space, and various processes related to the distribution of the virtual live event, an administrator terminal 150 used by an event manager M of an event management organization that operates and manages the virtual live event to manage the distribution of the virtual live event, a performer terminal 200 used by a performer user, and a viewer terminal 300 that can be used by a viewer user who virtually participates in and watches the virtual live event, and the distribution server computer 100, the administrator terminal 150, the performer terminal 200, and the viewer terminal 300 are connected to each other so that they can communicate data with each other via the Internet N, which is an open computer network.

[0027] In this embodiment 1, the administrator terminal 150 and the performer terminal 200 are also shown as being connected to the distribution server computer 100 via the Internet network N, but the present invention is not limited to this. For example, if the studio shown in FIG. 11 is a studio operated by an event management organization which provides the studio to a performer user, and the distribution server computer 100 is installed in this studio, the administrator terminal 150 and the performer terminal 200 may be locally connected to the distribution server computer 100 via data communication, or may be connected via a local area network LAN within the studio facility.

[0028] 1, there are multiple viewer users, such as viewer user A, viewer user B, viewer user C, etc., and viewer terminals 300 include viewer terminals 300a, 300b, 300c, etc. that can be used by each of viewer users A, B, C, etc. Note that there are usually multiple viewer users, but there may be one or four or more, and the number may be appropriately determined depending on the virtual space, such as a virtual live venue. Hereinafter, viewer user A, viewer user B, viewer user C, etc. may be collectively referred to as viewer users, and viewer terminals 300a, 300b, 300c, etc. may be collectively referred to as viewer terminals 300.

[0029] Furthermore, in this embodiment 1, an example is given of a form in which there is one performer user, but the present invention is not limited to this, and the virtual live performance may be a collaborative event in which multiple performer users working individually virtually participate, or a group event in which a group of multiple people working together virtually participate. In this way, when multiple performers appear, performer terminal 200a, performer terminal 200b, performer terminal 200c... may be provided for each performer user, or if a group of multiple people appears, one performer terminal 200 may be used by multiple people.

[0030] In addition, although FIG. 1 illustrates the distribution server computer 100 as a single unit, the present invention is not limited to this. These distribution server computers 100 may be configured with multiple server computers, such as a first server computer that mainly performs virtual space processing and a second server computer that mainly performs distribution processing, or may be configured as a cloud server, etc., so that the number of server computers can be changed appropriately depending on the scale of the event, the number of participating listener users, etc.

[0031] Similarly, although the administrator terminal 150 is illustrated as a single unit in FIG. 1, the present invention is not limited to this, and it goes without saying that there may be multiple administrator terminals 150 depending on the number of performer users and viewer users virtually participating in the live performance.

[0032] <Distribution server computer> Fig. 2 is a diagram showing the configuration of the distribution server computer 100 used in this embodiment 1. As described above, the distribution server computer 100 used in this embodiment 1 is a normal server computer with relatively excellent processing power installed in a studio operated by an event management organization, and as shown in Fig. 2, it has a processor (CPU) 101 connected to a data bus 108, a memory (RAM) 102, a storage 103 such as a hard disk, a communication interface (I / F) 104 capable of two-way communication by the TCP / IP protocol via the Internet, which is an open computer network, and an input / output interface (I / F) 105 to which an input device such as a keyboard can be connected.

[0033] In this embodiment 1, an input / output interface (I / F) 105 is provided to enable local operation, but if remote operation is possible via a communication interface (I / F) 104, the input / output interface (I / F) 105 may not be provided.

[0034] In addition to an operation system (server OS) for providing server functions (not shown), storage 103 of distribution server computer 100 stores various data and programs as shown in Fig. 3. Specifically, mainly stored are an event management program 110 for providing an event management function for managing live events, a virtual space generation program 120 for providing a virtual space of a virtual live venue, which is a virtual space, a performer user management program 130 for providing a performer user management function for managing performer users, and a viewer user management program 132 for providing a viewer user management function for managing viewer users.

[0035] The event management program 110 is a program for managing a virtual live event, which is an event held at a virtual live venue (see FIG. 12), in cooperation with the administrator terminal 150, and is capable of managing the progress of the virtual live event by using event data 111 (see FIG. 7), time schedule data (TS data) 112, and music data 113 stored in the storage 103 together with the event management program 110.

[0036] An example of the event data 111 used in the present embodiment 1 is shown in Fig. 7. The event data 111 stores information such as an event name, a file name of time schedule (TS) data, a performer ID uniquely assigned to a performer user who is a performer, a file name of a music list, a file name of a participating viewer list in which information of special viewer users who are permitted to participate virtually is registered in a third area of ​​the virtual audience area described later, and a file name of a participating viewer list in which information of viewer users who have reserved virtual participation is registered in a fourth area of ​​the virtual audience area described later, in association with an event ID uniquely assigned to the event.

[0037] The time schedule data (TS data) 112 is data in which the order of progress of the virtual live event to be held and the programs such as songs are described in chronological order, and the virtual live event is managed so as to progress based on the time schedule described in the time schedule data (TS data) 112. Moreover, the time schedule data (TS data) 112, event data 111, etc. can be changed, updated, or added by the event manager M operating the manager terminal 150.

[0038] Furthermore, the song data 113 is data for the performance parts (so-called karaoke) of each song sung by the performer user, and is played by the audio control program 124 (described later) based on the time schedule data (TS data) 112 by the event management program 110, and transmitted to the performer terminal 200 and the viewer terminal 300 (see FIG. 11).

[0039] The virtual space generation program 120 is a program for providing a virtual live venue, which is the virtual space shown in Figure 12, and has the function of providing a virtual live venue in which performer avatars and viewer avatars are virtually participating, using virtual space data 121 describing the structure of the virtual live venue, which is stored in storage 103 together with the virtual space generation program 120, object data 122 for placing various objects to be placed in the virtual live venue (including virtual placement object 43, which is a star decoration on the stage, and virtual aerial objects 50-52, which are star objects in the sky), etc., within the virtual space, and performer avatar data contained in performer user data 131 described later and viewer avatar data contained in viewer user data 133.

[0040] 12, viewpoint image control program 123 is a program for providing a function to generate viewpoint images from virtual cameras C1-C4 virtually set in the virtual live venue and an airborne virtual camera (not shown), and also to provide a function to display images from the viewpoints of viewer avatars 11-14 virtually participating in the third area of ​​the virtual live venue and viewer avatars 21-32 virtually participating in the fourth area on viewer terminal 300. Virtual camera C2 is a virtual camera set to provide an image from the viewpoint of performer avatar 1 on the stage of the virtual live venue, and the viewpoint image from virtual camera C2 is projected onto a screen S installed in the studio by projector 211 (described later) connected to performer terminal 200 in cooperation with performer terminal 200.

[0041] The audio control program 124 is a program for controlling the audio (including music) distributed during the virtual live event, and specifically provides a function for playing music data 113 and a function for synthesizing the audio produced by the performer user with the played music and distributing it to the administrator terminal 150 and the viewer terminal 300.

[0042] The comment control program 125, in cooperation with a viewer comment control program stored in the viewer terminal 300 described below, provides a function for displaying various comments entered by each viewer user at the viewer terminal 300 during a virtual live event in chronological order on the performer terminal 200 and the viewer terminal 300.

[0043] The gift control program 126 is a program for controlling gifts given from viewer users to performer users in cooperation with a viewer program stored in the viewer terminal 300 described below, and specifically provides a function for the viewer user to purchase gifts based on information on the unit purchase price set for various gifts stored in the gift data 134, a processing function for the viewer user to give a gift purchased using the gift object data included in the gift data 134 to the performer user (including processing for the presentation when giving the gift), and a processing function for changing the owner of a gift given from the viewer user to the performer user.

[0044] The performer user management program 130 provides a management function for information relating to performer users, an authentication function, and the like, based on performer user data 131 shown in FIG. 8, which is stored in the storage 103 together with the performer user management program 130 .

[0045] The performer user data 131 used in the present embodiment 1 is illustrated in Fig. 8. The performer user data 131 stores various information, such as the performer user's account (email address), name, authentication information, the file name of the avatar data of the avatar used in the virtual space, and the file name of the item list in which the items owned are registered, in association with a performer user ID uniquely assigned to the performer user. Although not illustrated in Fig. 8, information on virtual value such as points owned by each performer user may also be stored as the performer user data 131.

[0046] Performer users can be authenticated by matching their performer user ID, account, and authentication information, and each performer user can identify from an item list the items (gifts) that they own as a result of being given to them by viewers.

[0047] The viewer / user management program 132 provides functions such as management of information relating to viewer / users and authentication based on viewer / user data 133 shown in FIG. 9 stored in the storage 103 together with the program.

[0048] The viewer user data 133 used in the present embodiment 1 is illustrated in Fig. 9. The viewer user data 133 stores various information, such as the viewer user's account (email address), authentication information, the file name of the avatar data of the avatar to be used in the virtual space, the owned points which are the number of points that can be used in the virtual space, the file name of the item list in which the items (gifts) purchased using the points are registered, and personal information such as the name, date of birth, and telephone number, in association with the viewer user ID uniquely assigned to the viewer user. Although not shown in Fig. 9, a nickname (comment name) to be displayed together with the comment is also registered in the viewer user data 133, and the nickname (comment name) is displayed together with the comment. In addition, points can be increased by, for example, purchasing points from a predetermined operating company.

[0049] <Performer terminal> Fig. 4 is a diagram showing an example of the configuration of a performer terminal 200 in the virtual space content delivery system of this embodiment 1. In this embodiment 1, as shown in Fig. 11, the performer terminal 200 is installed in a control room adjacent to a studio where a performer user performs his / her acting motion, and uses a normal computer with relatively excellent processing power, and as shown in Fig. 4, has a processor (CPU) 201 connected to a data bus 208, a memory (RAM) 202, a storage 203 such as a hard disk, a communication interface (I / F) 204 capable of two-way communication by the TCP / IP protocol via the Internet, which is an open computer network, an image processing unit 206 including a graphics processing unit (GPU) to which display devices A210 to C212 are connected, and an input / output interface (I / F) 205 to which various input / output devices are connected.

[0050] The input / output interface (I / F) 205 is connected with input devices such as a motion sensor 220 including multiple wearable sensors 220C1-220C5 (see FIG. 11) worn by the performer user on the left and right limbs and waist, a facial expression input device 221 consisting of an imaging camera for inputting the performer's facial expressions, a voice input device 222 consisting of a sound collecting microphone worn on the performer user's head, and an operation input device 224 such as a keyboard or touch panel operable by an assistant operator O who assists the performer user. Note that by providing a controller that can be held by the performer user to perform various operations as these input devices, the performer himself may be able to perform various operations without the assistance of the assistant operator O.

[0051] In this embodiment 1, in order for the performer user to perform performance actions in accordance with the music (including musical actions such as singing and dancing), an assistant operator O assists with the operations during the performance. However, it is also possible to have multiple assistant operators O, or to set the operation content in advance in the performer terminal 200, for example, in a sequence program, thereby eliminating the need for an assistant operator O.

[0052] An audio output device 223 including high-performance earphones (in-ear monitors) worn by the performer user and speakers placed in the control room is connected to the input / output interface (I / F) 205 as an output device, so that the sound of the played music transmitted from the distribution server computer 100 is output to the performer user via the high-performance earphones (in-ear monitors), and the music audio is output from the speakers along with the performer's voice, allowing an assistant operator O, etc. to check the status of the music, including the voice produced by the performer user, in the control room.

[0053] In this embodiment 1, high-performance earphones (in-ear monitors) are used to avoid the inconvenience caused by the played music audio being input from the sound collection microphone worn by the performer user, but the present invention is not limited to this, and the music audio may be output from speakers in the studio, etc., as long as the inconvenience caused by the input of the music audio can be avoided.

[0054] In the first embodiment, the sound collecting microphones are worn by the performers, but these sound collecting microphones may be installed on the floor, wall, or ceiling of the studio.

[0055] Here, the motion sensor 220 used in the performer terminal 200 of the present embodiment 1 will be described. Any motion sensor can be used as long as it can properly detect (measure) the body movements (movements) of the performer user. In the present embodiment 1, in order to enable more accurate and short-period detection of the performer's movements, a number of wearable sensors 220C1 to 220C5 are used that are worn by the performer user on the body. Note that the motion sensor 220 may be of a type that does not require the performer to wear any equipment, such as LiDAR that uses laser light or the like.

[0056] In this embodiment 1, in order to reduce the burden of movement, particularly the head movement, caused by wearing sensors etc. when the performer user performs a performance movement, the head movement of the performer user is detected by image recognition using images captured by the imaging camera constituting the facial expression input device 221 as described below, and five wearable sensors 220C1 to 220C5 are used to detect movements other than the head. However, for example, wearable sensors may also be provided on the performer user's head, etc., or more wearable sensors (7 or more) may be worn to detect more detailed movements.

[0057] As shown in FIG. 11, the mounting sensors 220C1 to 220C5 detect their own positions and orientations in cooperation with base stations 220a and 220b installed in a studio room separated from the adjacent control room by a glass window.

[0058] Base station 220a and base station 220b may use, for example, a multi-axis laser emitter. Base station 220a may emit a flashing light for synchronization, and then scan the laser light around, for example, a vertical axis. Base station 220b may scan the laser light around, for example, a horizontal axis.

[0059] Each of the mounting sensors 220C1 to 220C5 may include a plurality of optical sensors that detect the incidence of blinking light and laser light from the base station 220a and the base station 220b.

[0060] Each of the wearing sensors 220C1 to 220C5 is capable of detecting its own position and orientation based on the time difference between the timing of the incidence of the flashing light and the timing of the incidence of the laser light, the light reception time at each optical sensor, the incidence angle of the laser light detected by each optical sensor, and other information as necessary. For example, the ViveTracker or base station provided by HTC CORPORATION can be suitably used.

[0061] Base station 220a and base station 220b emit blinking light and perform laser light scanning at regular intervals, so that the detection information of each of worn sensors 220C1-220C5 is updated at each interval. Detection information indicating the position and orientation of each motion sensor calculated in each of these worn sensors 220C1-220C5 is input to performer terminal 200 by short-range wireless communication and transmitted to distribution server computer 100 as performer avatar information together with facial motion information (see FIG. 10).

[0062] In this embodiment 1, an example is shown in which two base stations, base station 220a and base station 220b, but the present invention is not limited to this, and the number of base stations may be three or more.

[0063] In addition, the motion sensor 220 may be configured such that each of multiple motion sensors worn by the performer user is equipped with a number of infrared LEDs or visible light LEDs, and the position and orientation of each of the motion sensors is detected by detecting the light from these LEDs with an infrared camera installed on the floor or wall of the studio.

[0064] Also, in order to reduce the burden on the performer user, the motion sensor 220 may be replaced with a motion sensor using a lighter reflective marker instead of the wearable sensors 220C1-220C5. That is, the motion sensor may detect the motion of the performer user by capturing images of reflective markers attached to various parts of the performer user's body with adhesive tape or the like to generate image data, and then processing the captured image data to detect the positions and orientations of the reflective markers. In this case, a cameraman who captures the performer user may be placed in the studio to capture the performer user's motion from a direction suitable for detection, so that the motion of the performer user can be detected well.

[0065] In addition, the motion sensor 220 may be a suit with a built-in inertial sensor, such as a suit with a built-in inertial sensor for the MVN motion capture system marketed by Xsens.com, and the movements of the performer user may be detected by analyzing the sensor signal output from the inertial sensor.

[0066] Any facial expression input device 221 can be used as long as it can appropriately detect the head movement including the face of the performer user. In this embodiment 1, specifically, it is configured with an imaging camera (digital camera) arranged so as to be able to continuously capture head images including the face of the performer user, and the head movement and facial movement (facial expression) of the performer user captured by the imaging camera are detected by image recognition, and the facial motion information, which is the detected head movement and facial movement (facial expression), is transmitted to the delivery server computer 100 as performer avatar information together with body motion information, as shown in Fig. 10.

[0067] The facial expression input device 221 is not limited to the imaging camera (digital camera) used in the present embodiment 1, and may be, for example, a 3D camera capable of detecting the depth of a person's face, or may be a mobile terminal such as a smartphone equipped with a LiDER device. In this case, the performer user may wear such a mobile terminal.

[0068] As described above, the performer terminal 200 of this Example 1 has three display devices A210 to C212, where display device A210 is an LCD monitor or the like installed in a control room as shown in FIG. 11, display device B211 is a projector that projects an image onto a screen S installed in a studio, and display device C212 is a large vertical display installed adjacent to the screen S in the studio.

[0069] Display device A210, which is placed in the adjustment room of the studio, displays images from each viewpoint of virtual cameras C1 to C4, viewpoint images from a virtual camera moving in the air during the flying performance period described below, the contents of comments, and a message input window into which assistant operator O inputs any message he or she wishes to convey to the performer user.

[0070] Meanwhile, a screen S projected from a projector, which is a display device B211 installed in the studio, displays an image from the viewpoint of a virtual camera C2 (see FIG. 12), which is the viewpoint of the performer, for example, as shown in FIG. 15, including avatars of viewers who are virtually participating in the live performance in the third and fourth areas, thereby enabling the performer user to check the status of the viewer users who are virtually participating in the virtual live performance through the image.

[0071] In addition, comments from viewer users, as well as comments and messages entered by the assistant operator O, are displayed on a display device C212 installed in the studio, allowing the performer user to check the comments from viewer users and messages from the assistant operator O even during the performance.

[0072] In addition to an operating system (OS) for operating the computer that is the performer terminal 200, the storage 203 also stores performer programs that, when executed by the processor (CPU) 201 etc., work in cooperation with the distribution server computer 100 to provide various functions, including the function of allowing the performer user to control the movement of the performer avatar 1 in a virtual live performance, which is virtual space content.

[0073] As shown in FIG. 4, the performer programs include a performer avatar control program, a performer viewpoint image control program, a performer viewpoint image control program, a performer voice control program, a performer comment control program, performer authentication data, etc.

[0074] The performer avatar control program is a program that mainly provides the function of scanning the body motion of the performer user using the motion sensor 220 and scanning the facial motion (facial expression) of the performer user using the facial expression input device 221, generating performer avatar information for operating the performer avatar, and transmitting it to the distribution server computer 100.

[0075] The performer's viewpoint image control program is a program that provides a function for generating and outputting a viewpoint image of virtual camera C2, which is the performer's viewpoint, based on the virtual space update data distributed from distribution server computer 100.

[0076] The performer's audio control program is a program that provides the function of outputting the playback sound of a song based on the playback music data distributed from the distribution server computer 100 via the audio output device 223, converting the audio input from the audio input device 222 into data and transmitting it to the distribution server computer 100 as performer audio data, and generating and outputting monitor audio consisting of the playback sound of the song and the performer's audio.

[0077] The performer's comment control program is a program that provides a function for displaying each comment by an audience user that is distributed from the distribution server computer 100 .

[0078] The performer authentication data is data for authenticating the identity of the performer user in a communication connection with the distribution server computer 100 .

[0079] In addition, the performer avatar control program includes a motion parameter generation program capable of generating parameter information for the positions and rotation angles of the bones that make up the rig data (sometimes called skeleton data) that indicates the skeleton of the performer avatar contained in the performer avatar data, as well as an image recognition processing program for image-recognizing the position of the performer user's head and facial expression from images captured by an imaging camera.

[0080] In addition, in this embodiment 1, the performer terminal 200 is exemplified as a form in which a computer serving as the performer terminal 200 installed in a studio operated by an event management organization is jointly used by multiple performer users to hold a live event, and each performer user can use it as their own performer terminal 200 by inputting performer authentication data. However, the present invention is not limited to this, and for example, when a performer user appears in a virtual live performance from their home, etc., the performer terminal 200 may be a computer installed in the performer user's home and used exclusively by the performer user, and in such cases, storage of performer authentication data is not necessarily required.

[0081] In this way, by using the performer terminal 200 constituting the virtual space content delivery system of the present embodiment 1, the performer user or the like can move the performer avatar 1 placed on the virtual stage G in the virtual live venue in conjunction with the performer user's own movements, and can reflect the performer user's own facial expression in the facial expression of the performer avatar 1. In addition, the performer user can deliver his or her own voice as the voice of the performer avatar 1 to the viewers via the delivery server computer 100.

[0082] <Viewer device> Fig. 5 is a diagram showing a configuration example of a viewer terminal 300 in the virtual space content delivery system of the present embodiment 1. In the present embodiment 1, the viewer terminal 300 is a smartphone P carried by a viewer user or a regular desktop computer (PC) installed at a viewer user's home or the like, and as shown in Fig. 5, has a processor (CPU) 301 connected to a data bus 308, a memory (RAM) 302, a storage 303 such as a hard disk or non-volatile memory, a communication interface (I / F) 304 capable of two-way communication by the TCP / IP protocol via the Internet, which is an open computer network, an image processing unit 306 including a graphics processing unit (GPU) to which a display device 310 is connected, and an input / output interface (I / F) 305 to which various input / output devices are connected.

[0083] The display device 310 may display each viewpoint video generated based on the virtual space data of the virtual live venue, which is a virtual space updated by the virtual space update data (see FIG. 10) distributed from the distribution server computer 100, by the viewer viewpoint video control program included in the viewer program stored in the storage 303, as described later, and the flying viewpoint video distributed from the distribution server computer 100 during the flying production period described later. If the viewer terminal 300 is a smartphone P, the display device 310 corresponds to the display device of the smartphone P, and if the viewer terminal 300 is a normal computer (PC), the display device 310 corresponds to the display device of the stationary type constituting the computer (PC). Note that these displays are not limited to those that display planar images (2D images), and may be, for example, those that can display stereoscopic images (3D images) by displaying right-eye images and left-eye images, such as a head-mounted display (HMD) that can be worn by the viewer.

[0084] The input / output interface (I / F) 305 is connected to an operation input device 321 consisting of one or more devices that allow the viewer to perform various operations as input devices, and an audio output device 322 such as earphones or speakers that can output live audio, etc., distributed from the distribution server computer 100.

[0085] It goes without saying that the audio output device 322 need not be an independent device, but may be integrated with the display device 310 like the above-mentioned head mounted display (HMD).

[0086] The operation input device 321 may be any device that allows the viewer to perform various operations such as motion operations including moving the viewer avatar, zoom operations, operations related to comments, operations related to gifts, etc., and may be composed of at least one of, for example, a transparent touch panel formed on the display surface of a smartphone, a keyboard constituting a computer, a game controller, etc.

[0087] In this embodiment 1, as described below, the virtual audience area in the virtual live venue is divided into a third area near the virtual stage G in which only viewer users who have been specially permitted by the performer user can virtually participate, and a fourth area around the third area in which general viewer users can virtually participate, as shown in Figure 12. As described above, a viewer user who virtually participates in the fourth area may have an operation input device 321 such as a touch panel, keyboard, game controller, etc., but a viewer user who virtually participates in the third area owns an operation input device 321 such as an MVN motion capture suit with the above-mentioned inertial sensor built in, and is a viewer user who can precisely move the viewer avatar with their own movements, just like a performer user.

[0088] However, even viewer users who are permitted to virtually participate in the third area cannot change the facial expression of the viewer avatar, but the present invention is not limited to this, and the facial expressions of these viewer avatars may be changed, for example, as described above, by the viewer user using the camera function of a smartphone or a camera connected to a computer to recognize the facial expression of the viewer user, even if it is to a different level than the level of change made by the performer user.

[0089] The various input / output devices connected to the input / output interface (I / F) 305 may be built-in or external, and in the case of external devices, the connection form may be either wired or wireless.

[0090] In addition to an operating system (OS) for operating the viewer terminal 300, which is a smartphone or computer, the storage 303 also stores a viewer program as a viewing application that, when executed by a processor (CPU) 301 or the like, works in cooperation with the distribution server computer 100 to provide various functions, including a live viewing function for viewer users in a virtual live, which is virtual space content.

[0091] As shown in FIG. 5, the viewer program includes a viewer viewpoint image control program which provides the function of generating each viewpoint image of the virtual live to be displayed on the display device 310, a viewer audio control program which provides the function of outputting live audio from the audio output device 322 based on the live audio (data) of the virtual live distributed from the distribution server computer 100, a viewer comment control program which displays comments based on comment data distributed from the distribution server computer 100 and provides functions related to comments such as comments entered by the viewer and sending them to the distribution server computer 100, and viewer authentication data for authenticating that the viewer user is the correct person in a communication connection with the distribution server computer 100.

[0092] In addition, although not shown in Figure 5, in addition to the above-mentioned programs, there are included a gift control program that performs control related to gifts, and an area designation program for reserving an area (position) in which a viewer avatar will virtually participate in the virtual live venue in advance before the start of a live event, as shown in variant example 3-1 (see Figures 42 and 43) described below, but other programs other than these may also be included.

[0093] In the first embodiment, the comments input by the viewer are short messages in text format, but are not limited to these short messages in text format, and may be, for example, still image messages, video messages, or any other electronic message format. The comments are displayed on the display device 310, superimposed on each viewpoint video.

[0094] In this way, by using the viewer terminal 300 that constitutes the virtual space content distribution system of this Example 1, a viewer user can move and operate the viewer avatar that is virtually participating in the virtual live venue, and by changing the viewpoint of the viewer avatar, the viewpoint image displayed on the display device 310 can be changed, and the zoom state (viewpoint state) can also be changed, allowing the viewer user to operate the viewer avatar to cheer on the performer avatar 1.

[0095] It should be noted that a viewer avatar who has virtually participated in the third area is only allowed to move within the third area, and cannot move to the fourth area. On the other hand, a viewer avatar who has virtually participated in the fourth area is only allowed to move within the fourth area, and cannot move to the third area. However, the present invention is not limited to this, and a viewer avatar who has virtually participated in three areas may be allowed to move to the fourth area. It should be noted that a viewer avatar cannot naturally move to the virtual stage G, etc., so the visibility of the performer avatar 1 is not obstructed by the viewer avatar moving onto the virtual stage G.

[0096] In addition, by displaying viewpoint images on these display devices 310 and outputting live audio, the user can enjoy the virtual live performance with the feeling that he or she is actually present at the virtual live performance venue.

[0097] Although detailed explanation is omitted, the comment function provided by the viewer comment control program allows viewers to enjoy the virtual live performance while checking comments entered by themselves and other viewer users, and the gift function provided by the gift control program allows viewers to liven up the virtual live performance by gifting items they own to the performer avatar 1.

[0098] <Administrator terminal> Fig. 6 is a diagram showing an example of the configuration of an administrator terminal 150 in the virtual space content delivery system of this embodiment 1. In this embodiment 1, the administrator terminal 150 uses a normal computer installed in a control room adjacent to the studio together with the performer terminal 200, and as shown in Fig. 6, has a processor (CPU) 151 connected to a data bus 158, a memory (RAM) 152, a storage 153 such as a hard disk, a communication interface (I / F) 154 capable of two-way communication by the TCP / IP protocol via the Internet, which is an open computer network, an image processing unit 156 including a graphics processing unit (GPU) to which a display device 160 is connected, and an input / output interface (I / F) 155 to which various input / output devices are connected.

[0099] As described below, the display device 160 is capable of individually displaying each viewpoint image of each of the virtual cameras C1 to C4 described below and each viewpoint image of the virtual camera moving in the air during the flying performance period, which are generated based on virtual space data of the virtual live venue, which is a virtual space updated by virtual space update data (see Figure 10) distributed from the distribution server computer 100, by an administrator viewpoint image control program included in the administrator program stored in storage 153, and is also capable of displaying comments based on comment data distributed from the distribution server computer 100, and is composed of one or more stationary displays.

[0100] The input / output interface (I / F) 155 is connected to an operation input device 161 consisting of one or more devices that enable the event manager M to perform various operations as input devices, and an audio output device 162 such as earphones, headphones, or speakers that can output live audio, etc., distributed from the distribution server computer 100.

[0101] The operation input device 161 can be any device that allows the event manager M to perform various operations such as input and settings related to the virtual live event, and may be composed of at least one of, for example, a keyboard that constitutes a computer, a transparent touch panel formed on the surface of a display, a viewpoint switching controller, etc.

[0102] The various input / output devices connected to the input / output interface (I / F) 155 may be connected in either a wired or wireless manner.

[0103] In addition to an operating system (OS) for operating the computer that is the administrator terminal 150, the storage 153 also stores an administrator program that is executed by a processor (CPU) 151, etc. to provide various functions in cooperation with the distribution server computer 100, including distribution management functions for the distribution of virtual live shows, which are virtual space content.

[0104] As shown in FIG. 6, the administrator programs include an administrator viewpoint video control program, an administrator audio control program, an administrator comment control program, an audience management program, a performer management program, and the like.

[0105] The viewpoint video control program for administrator is a program that provides a function for generating viewpoint videos from each virtual camera to be displayed on the display device 160, as well as a function for changing and switching each viewpoint.

[0106] The administrator audio control program is a program that provides a function for outputting live audio from the audio output device 162 based on the audio data of the virtual live event distributed from the distribution server computer 100 .

[0107] The administrator's comment control program is a program that displays comments based on comment data distributed from the distribution server computer 100, and provides comment-related functions such as prohibiting comments from being distributed and selecting viewers to prevent them from being distributed.

[0108] The viewer management program is a program for managing viewer users who virtually participate in the virtual live performance, and the performer management program is a program for managing performer users who virtually participate in the virtual live performance.

[0109] Although not shown in FIG. 6, the system includes an authentication program for authenticating whether the administrator who is the operator is the actual person, and a schedule program for editing the virtual live show time schedule and song order, etc., but may also include other programs.

[0110] In this way, by the event manager M operating the manager terminal 150 that constitutes the virtual space content distribution system of this embodiment 1, various settings for executing the event can be implemented, such as settings related to the program, order of songs, and performance, as well as the time schedule and the trajectory of the virtual camera moving in the air during the flying performance period, and the virtual live performance can be managed based on the settings made in this way.

[0111] <Virtual live venue> The virtual live venue, which is the virtual space used in Example 1, is shown in Fig. 12. As shown in Fig. 12, the virtual live venue, like a live venue in real space, is a virtual space having a virtual stage G where performer users appear as performer avatars 1 and a virtual audience area where viewer users who will be in the audience virtually participate as viewer avatars.

[0112] 12, virtual stage G has a relatively large trapezoidal floor surface, and performer avatar 1 can move on virtual stage G in the same way as in a live venue in real space. On the side of virtual stage G opposite the virtual audience seats, as shown in Fig. 12, a virtual stage wall is formed that is divided into three parts, a center part, a right part, and a left part, and virtual displays 40, 41, and 42 are provided in front of these virtual stage walls, and images and videos for performance are virtually displayed on these virtual displays 40, 41, and 42.

[0113] A star virtual arrangement object 43 related to the performer user is placed between the virtual displays 40, 41, 42, and large star virtual aerial objects 50-52 are placed in a second area, which is an area on the virtual stage G and is above a first area surrounded by the virtual stage walls, as shown in Fig. 12. The virtual aerial objects 50-52 are movable within the second area.

[0114] A virtual audience area where viewer users can virtually participate in the virtual live venue as viewer avatars is provided in front of the virtual stage G. As shown in Fig. 12, this virtual audience area is made up of a third area, which is a special area close to the center of the virtual stage G where the performer avatar 1 appears, and a fourth area, which is formed to surround the third area and is farther away from the center of the virtual stage G than the third area.

[0115] These third areas are available for virtual participation by special viewer users who are permitted to participate virtually, up to a predetermined upper limit. In this embodiment 1, as shown in FIG. 12, for example, viewer avatars 11 to 14 of four viewer users who have met a predetermined virtual participation record and have been permitted to participate virtually by the performer user are arranged.

[0116] In addition, the fourth area includes viewer avatars 21-32 of general viewer users who have reserved virtual participation before the start of the virtual live. Note that, in Fig. 12, the viewer avatars 21-32 are shown in a simplified manner for the sake of convenience, but are the same avatars as the viewer avatars 11-14.

[0117] Furthermore, each viewer avatar 11-14 placed in the third area can be moved within the third area by the viewer user operating the viewer terminal, and each viewer avatar 21-32 placed in the fourth area can be moved within the fourth area by the viewer user operating the viewer terminal.

[0118] Four virtual cameras C1 to C4 are virtually arranged (set) in the virtual live performance venue, as shown in Fig. 12. Virtual camera C1 is virtually arranged in front of performer avatar 1 so as to face the performer avatar 1, and the viewpoint image captured by virtual camera C1 is a viewpoint image viewed from a position close to the front of the performer avatar 1, as shown in Fig. 13(a).

[0119] Virtual camera C2 is a camera virtually placed (set) above the head of performer avatar 1, and the viewpoint image captured by virtual camera C2 is a performer viewpoint image captured by viewing the virtual audience area from performer avatar 1, as shown in FIG.

[0120] Virtual camera C3 is a virtual camera virtually placed on the virtual display 41 diagonally behind the performer avatar 1, and the viewpoint image from virtual camera C3 is a viewpoint image looking at the virtual audience area from diagonally behind the performer avatar 1, as shown in Figure 13(c).

[0121] Virtual camera C4 is a virtual camera virtually placed on the virtual display 42 diagonally behind the performer avatar 1, and the viewpoint image from virtual camera C4 is a viewpoint image looking at the virtual audience area from diagonally behind the performer avatar 1, as shown in Figure 13(b).

[0122] In addition to the fixedly placed (set) virtual cameras C1 to C4, in this embodiment 1, an air-moving virtual camera (not shown) is virtually placed so as to be movable in the air within the virtual live venue space, which is a virtual space. The air trajectories along which these air-moving virtual cameras move and the viewpoint directions (angles) at each position on the air trajectory are preset by the administrator terminal 150, and during the flying performance period described later, the performer avatar 1 virtually flies so as to follow the air trajectory of the air-moving virtual camera, thereby generating a viewpoint video of the virtually flying performer avatar 1 captured by the air-moving virtual camera.

[0123] In this embodiment 1, as shown in FIG. 12, an example is given in which four virtual cameras C1 to C4 are virtually arranged as fixedly arranged (set) virtual cameras, but the present invention is not limited to this, and the number of these fixedly arranged (set) virtual cameras may be five or more, or conversely, without providing fixedly arranged (set) virtual cameras, only the viewpoint images seen from the viewpoints corresponding to each avatar virtually participating in the virtual live venue may be displayed on the performer terminal 200 and the viewer terminal 300.

[0124] In addition, in this embodiment 1, an example is given of a configuration in which there is one virtual camera moving through the air, but the present invention is not limited to this, and a configuration in which there are multiple virtual cameras moving through the air may also be used, or conversely, a configuration in which no virtual camera moving through the air is virtually placed may also be used.

[0125] <Studio> 11 is a diagram showing the studio used in Example 1, and as described above, the studio is adjacent to the control room separated by a glass window, and the performer user performs each movement corresponding to singing in the studio, while the assistant operator O assists with the operation of the performer terminal 200 installed in the control room. Around the performer user wearing the wearable sensors 220C1 to 220C5, base stations 220a and 220b are mounted on a stand, and an imaging camera constituting the facial expression input device 221 is mounted on a stand in a position almost in front of the performer user.

[0126] A screen S is provided on the wall facing the performer user in the studio, and a projector (display device B211) installed on the ceiling of the studio projects the viewpoint image of virtual camera C2 onto screen S, so that the viewpoint image seen from performer avatar 1 as looking at the virtual audience area is displayed on screen S, as shown in Figure 15. This allows the performer user to proceed with the live performance while constantly keeping track of the behavior of the viewer avatars virtually participating in the virtual audience area (the viewer's reaction to the live performance).

[0127] A vertical large display is arranged to the side of the screen S as the display device C212, and comments from the audience and messages that the assistant operator O wants to convey to the performer users are displayed on the large display. The display device C212 may be configured to display information (performer support information) such as the lyrics of the song sung by the performer during the live performance and lines.

[0128] <Operation of virtual space content delivery system> FIG. 10 is an explanatory diagram for explaining the operation of the virtual space content delivery system of the first embodiment, specifically, the flow of delivery of video and audio of a virtual live performance.

[0129] First, the video system will be described. As described above, the distribution server computer 100 is capable of providing a virtual space of a virtual live venue based on the virtual space generation program 120 and the avatar data of the performer users and the viewer users, and the actions of the avatars of the performer users and the viewer users are reflected in the actions of the performer avatar 1 and the viewer avatar 1 actions (operations) of the viewer users are reflected in the actions of the viewer avatar by updating the virtual space data in which the state of the virtual space is described based on the performer avatar information transmitted from the performer terminal 200 and the viewer avatar information transmitted from the viewer terminal 300.

[0130] Although not shown in FIG. 10, when the performer terminal 200 and the viewer terminal 300 communicate with the distribution server computer 100 in order to virtually participate in the virtual live performance, virtual space data of the virtual live performance venue is delivered in advance from the distribution server computer 100 and stored in the performer terminal 200 and the viewer terminal 300. As described below, the virtual space data stored in advance in this manner is successively updated by virtual space update data delivered successively from the distribution server computer 100, allowing the performer terminal 200 and the viewer terminal 300 to identify the latest state of the virtual live performance venue.

[0131] Specifically, the movements and facial expressions of the performer user in the studio are scanned at predetermined time intervals by the body motion scan and face motion scan functions of the performer avatar control program described above, and performer avatar information is generated and transmitted to the distribution server computer 100.

[0132] On the other hand, viewer users who are permitted to virtually participate in the third area use motion sensors, just like the performers, and are scanned at specified intervals using the body motion scan and face motion scan functions, and viewer avatar information is generated and transmitted to the distribution server computer 100.

[0133] Furthermore, a viewer user permitted to virtually participate in the fourth area can use the touch panel, keyboard, or controller of the viewer terminal 300 to perform various operations such as movement, changing the viewpoint direction, changing the viewpoint state (zoom), raising hands, clapping hands, jumping, etc., to make the viewer avatar perform the corresponding action. Then, of these operations, viewer avatar information is generated based on the operation related to the viewer avatar's action and transmitted to the delivery server computer 100.

[0134] Furthermore, viewer users virtually participating in the third area can also use a controller or the like to perform various operations such as movement, changing the viewpoint (zoom), raising their hands, clapping their hands, jumping, etc. In this case, a menu item corresponding to each action is displayed, and the menu item corresponding to each action can be selected using a controller or the like.

[0135] In this manner, an avatar action operation process is executed in the viewer terminal 300 to receive operations corresponding to each action to be performed by the viewer avatar. The avatar action operation process also includes processes such as displaying and erasing menu items corresponding to each of the above-mentioned actions.

[0136] In this way, based on the performer avatar information transmitted from the performer terminal 200 and the viewer avatar information transmitted from the viewer terminal 300, the distribution server computer 100 executes a virtual space update process that updates the virtual space data, thereby reflecting the actions (operations) of the performer user and the viewer user in each avatar virtually participating in the virtual live venue.

[0137] Then, after executing a virtual space update data generation process for generating virtual space update data based on the virtual space data before the update and the virtual space data after the update, the generated virtual space update data is distributed to the performer terminal 200 and the viewer terminal 300.

[0138] At the performer terminal 200 and viewer terminal 300 to which the virtual space update data has been delivered, as shown in FIG. 10, the pre-stored virtual space data is updated based on the delivered virtual space update data, and a performer perspective video generation process and a viewer perspective video generation process are executed using the updated virtual space data, thereby generating performer perspective video and viewer perspective video based on the updated virtual space data, and the generated performer perspective video and viewer perspective video are displayed on the performer terminal 200 and the viewer terminal 300 (viewer perspective video output process).

[0139] In this way, in this embodiment 1, the viewpoint video of each viewer user who is virtually participating in the virtual live venue through a viewer avatar is generated and displayed on the viewer terminal 300 of each viewer user, and by generating the viewer avatar viewpoint video of each viewer user on the distribution server computer 100, it is possible to prevent the processing load on the distribution server computer 100 from becoming significantly large, which would result in many viewer users being unable to participate in the virtual live, and it is also possible to prevent distribution from becoming difficult due to an increased processing load.

[0140] In this embodiment 1, a viewpoint image is generated and displayed on the display device 310 of the viewer terminal 300 of the viewer user of the viewer avatar 13 who is virtually participating in the third area close to the performer avatar 1 on the virtual stage G, in which a viewpoint image of almost the front of the performer avatar 1 is displayed large, as shown in Figure 13 (d), which is a viewpoint image seen from the viewer avatar 13 close to the performer avatar 1, while a viewpoint image is generated and displayed on the display device 310 of the viewer terminal 300 of the viewer user of the viewer avatar 28 who is virtually participating in the fourth area far from the performer avatar 1 on the virtual stage G, for example, at a position behind the viewer avatar 13, in which a viewpoint image of almost the front of the performer avatar 1 appears relatively small through the viewer avatar 13 and viewer avatar 14 who are virtually participating in the third area, as shown in Figure 13 (e).

[0141] In addition, the display device 310 of the viewer terminal 300 of the viewer user of the viewer avatar 32 who is virtually participating at the edge position of the virtual stage G in the fourth area, which is set up as a U-shape when viewed from above, displays a viewpoint image seen from the viewer avatar 32 positioned diagonally forward of the performer avatar 1, as shown in Figure 13 (f), in which the performer avatar 1 appears relatively small through the viewer avatar 14 who is virtually participating in the third area.

[0142] Note that all of the display examples shown in Figure 13 are examples in which the viewer terminal 300 is a stationary computer (PC) and the display device 310 is a stationary display. However, for example, if the viewer terminal 300 is a smartphone P, tablet, etc., the viewpoint image of the virtual camera C1 is displayed as shown in Figure 14(a), and if the viewer terminal 300 is the viewer terminal 300 of the viewer avatar 13, it is displayed as shown in Figure 14(b).

[0143] Next, the audio system will be described. In the distribution server computer 100, music data 113 is played back by audio control program 124 based on time schedule data (TS data) 112, and transmitted to the performer terminal 200 as played music (data) (music playback process).

[0144] In the performer terminal 200, the music audio from the played music (data) transmitted from the distribution server computer 100 is output to the performer user through high-performance earphones (in-ear monitors) worn by the performer user (played music output process), and when the performer user sings along with the output played music, the performer voice input from the sound collection microphone (audio input device 222) is digitized and transmitted to the distribution server computer 100 as the performer voice (data).

[0145] The distribution server computer 100 generates distribution audio (data) from the performer audio (data) received from the performer terminal 200 and the played music played in the music playback process described above (distribution audio generation process), and distributes the generated distribution audio (data) to the viewer terminal 300 (generated audio distribution process).

[0146] In the viewer terminal 300, the distribution audio (data) distributed from the distribution server computer 100 is output from an audio output device 322 such as earphones or speakers, allowing the viewer to hear the live audio as if the performer avatar 1 were singing in the virtual live venue.

[0147] Furthermore, in the performer terminal 200, as described above, the performer's voice (data) is transmitted to the distribution server computer 100, and as shown in Figure 10, a monitor audio generation process and a monitor audio output process are executed, so that the music audio of the played music and the performer's voice are output from speakers placed in the control room, allowing an assistant operator O, etc. to check the singing status of the performer user in the control room.

[0148] <Viewer's viewpoint video control processing> Next, a viewer's viewpoint video control process executed in the viewer terminal 300 based on a viewer's viewpoint video control program included in the viewer's program will be described with reference to FIG.

[0149] In the viewer viewpoint video control process, first, it is determined whether or not the performer avatar 1 is in a flying performance period in which the performer avatar 1 is virtually flying, as shown in Fig. 19 (step S1). Whether or not the performer avatar 1 is in a flying performance period can be identified by performance status data indicating that the performer avatar 1 is in a flying performance period, which is sent from the delivery server computer 100. The delivery server computer 100 transmits performance status data indicating that the performer avatar 1 is in a flying performance period to the viewer terminal 300 when the performer avatar 1 is in a flying performance period based on the time schedule data.

[0150] If it is during the flying performance period (Y in step S1), proceed to step S21, whereas if it is not during the flying performance period (N in step S1), it is determined whether viewpoint-related operations including viewpoint movement operations (including changing the viewpoint by avatar movement operations and motion operations) and zoom operations are disabled (step S2).

[0151] If the viewpoint-related operation is invalid (Y in step S2), the process proceeds to step S7. If the viewpoint-related operation is not invalid (N in step S2), the process determines whether or not there is an operation that changes the viewpoint, such as an operation of moving an avatar or an operation of moving an avatar (step S3), and if there is an operation (Y in step S3), the viewer viewpoint is changed according to the operation, and if there is no operation (N in step S3), the process proceeds to step S5, and determines whether or not there is a zoom operation (step S5).

[0152] If a zoom operation is performed (Y in step S5), the image area displayed on the display device 310 is changed to an image area corresponding to the operation, and if a zoom operation is not performed (N in step S5), it is determined whether or not a motion tracking setting is present (step S7).

[0153] This motion tracking setting can be arbitrarily set by the viewer user on the viewer terminal 300, for example, before the start of a live performance or during a live performance, depending on the type of device used by the viewer user and his / her level of proficiency in the operation. Specifically, the motion tracking setting is set in cases where, for example, the viewer user is not familiar with operating the operation input device 321 such as a touch panel, keyboard, or game controller, or is not familiar with changing the viewpoint by operating the avatar to move or move, or, as will be described later, the virtual participation position of the viewer user's avatar is far from the performer avatar 1 and the viewer user is zoomed in and displayed in an enlarged manner, so that even a slight movement of the performer avatar 1 prevents the performer avatar 1 from being displayed appropriately, and so on, and the viewer user is unable to change the viewpoint in response to the movement or movement of the performer avatar 1.

[0154] If there is no motion following setting (N in step S7), the process proceeds to step S11. If there is a motion following setting (Y in step S7), it is further determined whether or not the motion following condition is satisfied (step S8).

[0155] In this embodiment 1, the motion tracking condition is a condition under which it is almost certain that the performer avatar 1 will no longer be displayed clearly on the display device 310. Specifically, when the head (or face) of the performer avatar 1 is within a specified range around the display area of ​​the display device 310, it is highly likely that the movement or motion of the performer avatar 1 will cause the head (face) to move outside the display area, causing it to no longer be displayed clearly, and therefore it is determined that the motion tracking condition is met.

[0156] In this way, in the first embodiment, by determining that the motion tracking condition is met before the head (face) of the performer avatar 1 disappears from the display area, it is possible to significantly reduce the occurrence of the head (face) of the performer avatar 1 disappearing from the display area, but the present invention is not limited to this, and these motion tracking conditions may be a condition that the head (face) moves out of the display area, or other conditions may be used as the motion tracking conditions. In other words, any conditions may be used as the motion tracking conditions as long as they can prevent the performer avatar 1 from disappearing from the display area in a good manner.

[0157] If the action tracking condition is met (Y in step S8), after disabling the viewpoint-related operations (step S12), the viewpoint image displayed in the display area of ​​display device 310 is automatically changed to a motion tracking viewpoint in which the head (face) of performer avatar 1 is within a non-metable range surrounded by a predetermined range around the display area, and the viewpoint image seen from the motion tracking viewpoint is displayed in the display area of ​​display device 310 (step S13).

[0158] On the other hand, if the action tracking condition is not met (N in step S8), it is further determined whether or not the viewpoint-related operation is disabled (step S9). If the viewpoint-related operation is disabled (Y in step S9), the disabled viewpoint-related operation is enabled (step S10) and then the process proceeds to step S11. If the viewpoint-related operation is not disabled (N in step S9), the process proceeds to step S11 without going through step S10.

[0159] In step 11, since the motion tracking setting is not set or the motion tracking condition is not satisfied, the viewpoint video of the viewer avatar is continuously displayed in the display area of ​​display device 310 (step S11).

[0160] Also, as mentioned above, if the flying performance period in which the performer avatar 1 virtually flies during a live performance has begun, the answer in step S1 is determined to be Y, and the process proceeds to step S21, where it is determined whether or not it is the start timing of the flying performance period (step S21).

[0161] If it is the start timing of the flying performance period (Y in step S21), the flying viewpoint image distributed from the distribution server computer 100 is displayed (step S22) in place of the viewer avatar's viewpoint image or action-tracking viewpoint image that was displayed in the display area of ​​the display device 310 before the start of the flying performance period, and then viewpoint-related operations are disabled (step S23) and the process returns to step S1.

[0162] In this embodiment 1, during the flying performance period, the same flying viewpoint video is displayed on the viewer terminals 300 of all viewer users. Instead of generating the flying viewpoint video on each viewer terminal 300 side, as in the case of viewer avatar viewpoint videos which are different for each viewer user, the flying viewpoint video is generated on the distribution server computer 100 side and distributed to the viewer terminals 300. This allows the flying viewpoint video, which involves movement of the viewpoint in the air and therefore imposes a large rendering load on the viewer terminals 300, to be displayed well even on viewer terminals 300 with low processing capabilities. However, the present invention is not limited to this, and these flying viewpoint videos may be generated on the individual viewer terminals 300 side, in the same way as viewer avatar viewpoint videos.

[0163] On the other hand, if it is not the start timing of the flying effect period (N in step S21), it is further determined whether it is the end timing of the flying effect period (step S24).

[0164] Whether or not it is the end timing of the flying performance period may be determined based on data indicating the end contained in the flying viewpoint video data transmitted from the distribution server computer 100, or may be determined based on the aforementioned performance status data that is different from the flying viewpoint video.

[0165] If it is not the end of the flying performance period (N in step S24), that is, if it is during the flying performance period, proceed to step S26, update the flying viewpoint video, and then return to step 1; on the other hand, if it is the end of the flying performance period (Y in step S24), enable viewpoint-related operations (step S25), proceed to step S11, display the viewer avatar's viewpoint video in the display area of ​​display device 310 in place of the flying viewpoint video that was displayed in the display area of ​​display device 310 during the flying performance period, and then return to step S1.

[0166] Thus, each viewpoint video displayed on the viewer terminal 300 by executing the viewer viewpoint video control process shown in FIG. 20 in the viewer terminal 300 of the first embodiment will be described with reference to FIGS. 16 to 19. FIG.

[0167] Fig. 16 is a diagram showing the display contents of the display device 310 when a zoom operation is performed on the viewer terminal 300 of the present embodiment 1. In the default state where no zoom operation is performed, as shown in Fig. 16(a), a viewpoint video of the video area centered on the upper body of the performer avatar 1 is displayed. In this default state, when the viewer user performs a zoom-up operation, for example, when the viewer terminal 300 is a smartphone P, by touching the "+" display (not shown), or when the viewer terminal 300 is a computer (PC), by operating the "+" key on the keyboard, as shown in Fig. 16(b), for example, a viewpoint video of the video area centered on the head (face) of the performer avatar 1 is displayed.

[0168] On the other hand, if the viewer user performs a zoom-down operation, for example, by touching the "- " display (not shown) if the viewer terminal 300 is a smartphone P, or by operating the "- " key on the keyboard if the viewer terminal 300 is a computer (PC), a viewpoint image of the image area including the entire body of the performer avatar 1 will be displayed, as shown in Figure 16 (c), for example.

[0169] In Figure 16, in order to make the zoom function easier to understand, the explanation was given using the viewpoint image of virtual camera C1 rather than the viewpoint of the viewer avatar, but as shown in Figure 17, zooming up and down can also be performed in the same way on the viewpoint of the viewer avatar.

[0170] Also, although not described in detail in the viewer viewpoint image control process of Figure 20, for example, when performer avatar 1 appears on virtual stage G for the first time when a live performance begins, the viewpoint of the viewer avatar will not be directed toward performer avatar 1, and there is a possibility that the appearing performer avatar 1 will not be displayed on the viewer terminal 300 for a long period of time. Therefore, the period when performer avatar 1 first appears may be designated as an appearance performance period, and during this performance period, the viewpoint image displayed on the viewer terminal 300 may be automatically switched to the viewpoint image of virtual camera C1, etc., instead of the viewpoint image of the viewer avatar, and individual viewer users may be allowed to zoom in and perform other operations on the viewpoint images of virtual camera C1, etc.

[0171] 17 shows, as an example of a viewpoint image of a viewer avatar, the viewpoint image of viewer avatar 28 who is virtually participating in the fourth area. As shown in Fig. 17, the viewpoint image of viewer avatar 28 includes the heads of viewer avatar 13 and viewer avatar 14 who are virtually participating in the third area, and the image shows performer avatar 1 seen through the heads of viewer avatar 13 and viewer avatar 14.

[0172] Therefore, the viewer user of viewer avatar 28 can enjoy the live performance by, for example, performing a zoom-in operation to change the image area so that the upper body of performer avatar 1 is displayed primarily, as shown in Figure 17(b), thereby preventing viewer avatar 13 and viewer avatar 14 from being displayed.

[0173] However, in the zoomed-in state as shown in Figure 17(b), for example, as the live performance progresses and the performer avatar 1 begins to move and make large movements, if the viewer user is unable to properly change the viewpoint of the viewer avatar 28 to match the movements of the performer avatar 1, the performer avatar 1 may not be displayed properly, as shown in Figures 18(d)-(f).If the viewer user has to pay attention to adjusting the viewpoint, it may become difficult to concentrate on the live performance and they may not be able to enjoy it well.

[0174] However, in this Example 1, when the viewer user sets the motion tracking setting as described above, as shown in Figures 18(a) to (c), the head of the performer avatar 1 enters within a specified peripheral area of ​​the display area, the motion tracking condition is met, the viewpoint is automatically changed to the motion tracking viewpoint, and the viewpoint image of the motion tracking viewpoint is displayed. Therefore, the viewer user does not need to pay attention to operating the viewpoint, and can concentrate on the live performance and enjoy it to the fullest.

[0175] Then, as the live performance progresses and the flying performance period begins, as shown in Figure 19, the viewpoint images of the individual viewer avatars are automatically changed to flying viewpoint images seen from a virtual camera moving in the air, without any operation by the viewer user.

[0176] Specifically, before the performer avatar 1 takes off in virtual flight, an image is displayed in which a wing object grows on the back of the performer avatar 1. Then, the performer avatar 1 looks up at a virtual aerial star object 50 virtually placed in the second area of ​​the virtual live venue, and the viewpoint of the viewer avatar moves in the same way. At this time, light effects 61 appear around the performer avatar 1.

[0177] Thereafter, when the performer avatar 1 enters virtual flight, the viewpoint image displayed on the display device 310 automatically switches from the viewpoint image of the viewer avatar in FIG. 19(b) to a flying viewpoint image seen from a virtual camera moving in the air, as shown in FIG. 19(c).

[0178] During the flying performance period, the aerial moving virtual cameras move within the second area based on a pre-set aerial trajectory as described above, and the flying viewpoint images seen from these aerial moving virtual cameras moving in the air are displayed on the display device 310 as shown in Figure 19(d).

[0179] <Modification of Example 1> In the above-mentioned first embodiment, the general action (first action) is an action of the performer avatar 1 standing without moving to the virtual stage G, the special action (second action) is an action of the performer avatar 1 flying virtually, and the specific action (third action) is an action of the performer avatar 1 moving rapidly in the lateral direction. However, the present invention is not limited to this, and the actions may be actions according to the content to be distributed. For example, the general action (first action) may be an action (normal action) that the performer avatar performs for the longest period in the distributed content, the special action (second action) may be an action that is performed in a special period that is the shortest period in the distributed content, or an action that is performed least frequently, and the specific action (third action) may be an action that is performed for a shorter period than the general action (first action) but for a longer period than the special action, or an operation that is performed less frequently than the general action (first action) but more frequently than the special action. The special action (second action) may include the specific action (third action).

[0180] In addition, in the above-mentioned Example 1, an example was given of a form having both a function of switching the viewpoint corresponding to a special action (second action) and a function of following a specific action (third action), but the present invention is not limited to this, and the virtual space content distribution system may have only one of these functions.

[0181] In the above-mentioned Example 1, an example was given of a form in which the viewpoint of the viewer user is changed and followed in accordance with the movements of the performer avatar, but the present invention is not limited to this, and for example, the change and tracking of the viewpoint may be performed in accordance with specific effects such as smoke or fireworks (pillars of fire) devices, special lighting, deformation of the virtual stage G, etc., which are effects on the virtual stage G. EXAMPLES

[0182] Next, a virtual space content delivery system according to a second embodiment will be described below with reference to Fig. 21 to Fig. 28. Note that the same components as those in the first embodiment will be denoted by the same reference numerals and description thereof will be omitted.

[0183] The configuration of the virtual space content distribution system of the second embodiment is the same as that of the first embodiment shown in FIG. 1. However, the progress of the virtual live performance is different, and therefore the period during which the viewpoint images from the virtual cameras C1 to C4 and the airborne moving virtual camera (hereinafter collectively referred to as virtual camera viewpoint images) are distributed and the entity that generates these virtual camera viewpoint images are different from those of the first embodiment, which is a characteristic feature of the second embodiment.

[0184] 21 is a diagram showing the progress of a virtual live performance delivered in a virtual space content delivery system of Example 2. In the virtual live performance of Example 2, a "live virtual space participation period" and a "pre-performance speech" period are set as periods before the start of the virtual live performance, and during the live virtual space participation period, viewer users can virtually participate in the virtual live performance venue, which is the target virtual space of the virtual live performance.

[0185] The period up to the timing for checking whether the pre-show download has been completed, which is set just before the end of the "pre-show speech" period, is the pre-show data download (DL) period, during which the virtual space data (basic data) of the virtual live venue, avatar data of performer avatar 1, and avatar data of other viewer users who have participated virtually are downloaded (DL) as pre-show data from the distribution server computer 100 to the viewer terminals 300 of viewer users who have participated virtually during the period during which they are able to participate in the live virtual space.

[0186] At the advance data DL completion check timing, which is the end timing of the advance data download (DL) period, it is checked whether the advance data download (DL) has been completed in each viewer terminal 300 or not.

[0187] If the download (DL) of the advance data is not completed by the time of the advance data DL completion check timing due to differences in the model, processing power, communication environment, etc. of the viewer terminal 300, the download (DL) of the advance data will continue. However, in this case, data that is highly important for the distribution of the virtual live, such as the virtual space data (basic data) of the virtual live venue required for the opening performance and the first MC, and the avatar data of the performer avatar 1, will be downloaded (DL) with priority.

[0188] Furthermore, in the second embodiment, as described above, an example is given of the data required to generate viewpoint video at the viewer terminal 300 during the distribution of a virtual live performance being downloaded (DL) as advance data, but the present invention is not limited to this, and these data, in particular special data required individually for each performance and each MC described below, may be downloaded (DL) sequentially each time at a timing corresponding to the timing at which each performance and each MC is distributed.

[0189] During the "pre-show speech" period, explanations about the virtual live show are given by video footage, etc. Therefore, by making the "pre-show speech" period a period during which advance data can be downloaded (DL), each viewer terminal 300 can also download (DL) from the distribution server computer 100 the avatar data of viewer avatars who virtually participated in the live virtual space just before the end of the participation period.

[0190] As shown in FIG. 21, the virtual live performance of Example 2 progresses as follows: "Opening performance" → "First MC" → "First performance" → "Second MC" → "Second performance" ... "Opening performance" is a performance in which the performer avatar 1 appears on the virtual stage G. In the "First MC", the performer avatar 1 appears on the virtual stage G, greets the audience through a talk, and introduces the first performance that will be performed first. The "First performance" is a performance that includes singing and dancing by the performer avatar 1 (performer), and also includes a flying performance in the middle of the performance, in which the performer avatar 1 virtually flies through the virtual live venue and is virtually captured by a virtual camera moving in the air. In the "Second MC", the performer avatar 1 (performer) mainly introduces the second performance through a talk. The "second performance" includes singing and dancing by the performer avatar 1 (performer), and also includes a costume change performance using footage virtually captured by the virtual camera C1 of the performer avatar 1 changing clothes, and a clone performance using footage virtually captured by the virtual camera C1 of the performer avatar 1 cloned into multiple people.

[0191] The processing load on the delivery server computer 100 during each period in which these effects and programs are performed is the lowest at "low" during the period in which participation in the live virtual space is possible, while it is the highest at "high" during the period in which the special effects "flying effect", "costume change effect", and "double-ego effect" are performed. The processing load on the delivery server computer 100 during the period of the "first performance" from the "introduction", "opening effect", "first MC", and "flying effect" is "medium", and the processing load is also "medium" during the period of the "first performance" after the "flying effect", the period of the "second performance" until the "second MC" and "costume change effect", the period of the "second performance" from the "costume change effect" to the "double-ego effect" is also "medium", and the period of the "second performance" after the "double-ego effect" is also "medium".

[0192] Next, regarding each period in the virtual live in Example 2, we will explain the viewpoint images that can be viewed on the viewer terminal 300. During the period during which participation in the live virtual space is possible, from the time of virtual participation until the start of the "pre-show speech," only the viewer avatar viewpoint images can be displayed.

[0193] Then, when the "pre-show speech" starts, only the virtual camera viewpoint video, specifically, the viewpoint video of virtual camera C1, can be displayed. In this way, the viewpoint video of the virtual camera is the only one used in the "pre-show speech" even though it is before the start of the virtual live performance. Since the pre-show speech includes an explanation of precautions regarding the virtual live performance, the purpose is to prevent a viewer user from not viewing the pre-show speech being performed on virtual stage G due to the viewer avatar viewpoint video being displayed on viewer terminal 300.

[0194] As described below, these virtual camera viewpoint images of the "introduction" are generated by the distribution server computer 100 and distributed to each viewer terminal 300, so that they can be displayed even on viewer terminals 300 that have not yet completed downloading the advance data.

[0195] For viewer terminals 300 for which the pre-data download has been completed at the above-mentioned pre-data download completion check timing, the viewer avatar viewpoint image can be displayed in addition to the virtual camera viewpoint image during the period of the "first performance" from the "first MC" to the "flying performance", as well as during the period after the "flying performance".

[0196] On the other hand, for viewer terminals 300 that have not completed downloading the advance data at the above-mentioned advance data download completion check timing, the viewer avatar viewpoint video cannot be generated accurately, so the virtual camera viewpoint video generated in the distribution server computer 100 is distributed, making it possible to display only the virtual camera viewpoint video during the entire virtual live broadcast, and it is not possible to display the viewer avatar viewpoint video.

[0197] Furthermore, among the images of the "Flying Performance," "Costume Change Performance," and "Double Ego Performance," which have a "high" processing load on the distribution server computer 100, the virtual camera viewpoint images of virtual camera C1 for the "Costume Change Performance" and "Double Ego Performance," as well as the virtual camera viewpoint images during the second performance period other than these "Costume Change Performance" and "Double Ego Performance," are generated on each viewer terminal 300 rather than on the distribution server computer 100, thereby making it possible to reduce the processing load on the distribution server computer 100 for generating the virtual camera viewpoint images of virtual camera C1 for these "Costume Change Performance" and "Double Ego Performance."

[0198] Furthermore, since the processing load for the "flying performance" period is high on the distribution server computer 100, it is possible to generate the "flying performance" at the viewer terminal 300, similar to the "costume change performance" and "doppelganger performance". In this case, detailed trajectory data of the movement trajectory of the virtual camera moving through the virtual live venue needs to be distributed in advance to each viewer terminal 300 to generate the video. Therefore, in Example 2, the video for the "flying performance" is generated and distributed at the distribution server computer 100, but the present invention is not limited to this, and the video for the "flying performance" may also be generated at each viewer terminal 300 in order to reduce the processing load on the distribution server computer 100.

[0199] Here, the participant user table used in the distribution server computer 100 of the second embodiment will be briefly described with reference to FIG.

[0200] The participant user table is a data table used to identify the participation status of viewer users in a virtual live event and the download status of pre-data for each virtual live event registered in the event data shown in Figure 7, and as shown in Figure 23, is a table that can store data such as participation identification data (participation identification flag) that can identify the participation status of the participant (viewer user) in correspondence with a participant ID (viewer user ID), and pre-data DL identification data (pre-data DL identification flag) that can identify the participation area of ​​the participant (viewer user) and the download status of the pre-data.

[0201] In addition, the participant ID field contains all the viewer user IDs of the prospective participants described in the file of each participant group stored in the event data corresponding to the virtual live.

[0202] The participation identification data (participation identification flag) is registered as "0" corresponding to a non-participating state when a participant ID (viewer user ID) is registered, and is updated to "1" indicating participation when the participation of each participant (viewer user) is permitted by authentication. Therefore, the participation identification data (participation identification flag) makes it possible to identify, among the viewer users who are scheduled to participate, those who are participating and those who could not participate for some reason.

[0203] The participation area data is data that can identify whether the participation area of ​​each participant (viewer user) is the third area or the fourth area, and is registered when the participation of each participant (viewer user) is permitted by authentication.

[0204] The pre-data DL identification data (pre-data DL identification flag) is updated from "0" indicating incomplete to "1" indicating complete if the download (DL) of the pre-data is completed based on a check as to whether or not the download (DL) of the pre-data is completed at the pre-data DL completion check timing described above. Therefore, by using this pre-data DL identification data (pre-data DL identification flag), it is possible to easily identify all participants (viewer users) who have not completed the download (DL) of the pre-data among the participants (viewer users) virtually participating.

[0205] Next, a flow of downloading advance data when a viewer user virtually participates in a virtual live concert venue in the second embodiment will be described with reference to FIG.

[0206] First, in a viewer program (viewing application) running on the viewer terminal 300, the viewer user performs an operation to participate in a virtual live event in which the viewer plans to virtually participate, for example, by selecting a participation menu or operating a specific key on the keyboard to which the participation operation is assigned.

[0207] By executing this participation operation, a participation request including the viewer user ID and authentication data for authenticating the viewer user is transmitted from the viewer terminal 300 to the distribution server computer 100.

[0208] The delivery server computer 100 that has received the participation request executes a participant authentication process on the condition that it is a period during which participation in the live virtual space is possible. In the participant authentication process, it is determined whether the viewer user is a prospective participant of the corresponding virtual live depending on whether the viewer user ID included in the received participation request is registered as a participant ID in the participant user database corresponding to the virtual live to be participated in, and the viewer user is authenticated by comparing the authentication data included in the participation request with the authentication data described in the authentication information file identified from the authentication information file data stored in association with the viewer user ID in the viewer user data shown in Fig. 9.

[0209] If the viewer user is a prospective participant and has been authenticated using the authentication data, the participant identification data (participation identification flag) in the participant user table is updated to "1" corresponding to "participation."

[0210] The delivery server computer 100 then transmits (responds to) advance data DL command information to the viewer terminal 300 that sent the participation request, to cause the viewer terminal 300 to download advance data corresponding to the virtual live event to be participated in. By receiving the advance data DL command information, the viewer terminal 300 can confirm that participation has been permitted.

[0211] In response to receiving the advance data DL instruction information, the viewer terminal 300 identifies advance data to be downloaded, and transmits to the delivery server computer 100 an advance data DL request requesting the download of the identified advance data.

[0212] In response to receiving this advance data DL request, the delivery server computer 100 executes an advance data transmission process.

[0213] In this advance data transmission process, the requested advance data is transmitted to the viewer terminal 300 that sent the advance data DL request. As described above, the advance data includes the virtual space data (basic data) of the virtual live venue, the avatar data of the performer avatar 1, and the avatar data of other viewer users who are already participating virtually.

[0214] In the advance data transmission process, the avatar data of the viewer user who has sent the advance data DL request is additionally distributed as advance data to the viewer terminals 300 other than the viewer terminal 300 that sent the advance data DL request and which are viewer terminals 300 of viewer users whose participation has already been confirmed by the participant data table. As a result, the avatar data of all viewer users who are virtually participating in the virtual live show are distributed in advance to and stored in the viewer terminals 300 of each viewer user who is virtually participating in the virtual live show, together with the virtual space data (basic data) and the avatar data of the performer avatars.

[0215] Then, at the pre-data download completion check timing described above (see FIG. 21), the pre-data stored in the viewer terminal 300 is checked to see whether all the pre-data has been downloaded, and if the pre-data download has been completed, the pre-data DL identification data (pre-data DL identification flag) in the participant user table is updated from "0", indicating incomplete, to "1", indicating complete.

[0216] Furthermore, for participants (viewer users) whose pre-data download identification data (pre-data DL identification flag) remains at "0", indicating incomplete download, because the pre-data download has not been completed, only the virtual camera viewpoint image will be displayed, as described above.

[0217] Next, a situation in which the subject from which various viewpoint videos displayed on the viewer terminal 300 are generated changes during each period of the virtual live will be described with reference to Figs. 24 to 27.

[0218] First, during the live virtual space participation period when only the viewer avatar viewpoint video can be displayed on the viewer terminal 300, the virtual space data already stored in the viewer terminal 300 is sequentially updated by receiving the advance data based on the virtual space update data including the performer avatar action data for generating the action of the performer avatar 1 that performs the action linked to the action of the performer user, which is transmitted from the distribution server computer 100, as shown in Fig. 24, and the viewer viewpoint video generation process is executed in the viewer terminal 300 based on the updated virtual space data to generate the viewer avatar viewpoint video, which is output (displayed) by the video output process. Note that, although the performer terminal 200 is omitted in Fig. 24, the virtual space update data is also transmitted to the performer terminal 200 as shown in Fig. 10 in the first embodiment.

[0219] Furthermore, during each of the periods of "introduction," "opening performance," and "flying performance" in which only virtual camera viewpoint video can be displayed on viewer terminals 300, as shown in Fig. 25, in the virtual space update data distribution process, distribution server computer 100 transmits virtual space update data only to performer terminals 200. Then, following the virtual space update data distribution process, a virtual camera viewpoint video generation process is executed in which a virtual camera viewpoint video is generated, and a virtual camera viewpoint video data distribution process is executed in which the video data of the virtual camera viewpoint video generated in the virtual camera viewpoint video generation process is distributed to all viewer terminals 300 participating in the virtual live.

[0220] The virtual camera viewpoint video data distributed from distribution server computer 100 in the virtual camera viewpoint video data distribution process is displayed by executing a video output process in viewer terminal 300 that receives the virtual camera viewpoint video data.

[0221] In other words, as shown in FIG. 27, for each period of “introduction”, “opening performance”, and “flying performance” when only virtual camera viewpoint images can be displayed, the virtual camera viewpoint images are generated in distribution server computer 100.

[0222] In addition, during the performance period excluding the MC period and the flying performance period, when either the viewer avatar viewpoint image or the virtual camera viewpoint image can be selected and displayed on the viewer terminal 300, as shown in FIG. 26, a virtual space update data distribution process is executed in the distribution server computer 100, and virtual space update data is distributed to the viewer terminal 300 of each viewer user participating in the virtual live, thereby executing the virtual space update process in the viewer terminal 300.

[0223] Then, in the viewer terminal 300, it is determined whether or not a viewer avatar viewpoint video has been selected as the video to be displayed. If a viewer avatar viewpoint video has been selected, a viewer avatar viewpoint video generation process is executed to generate a viewer avatar viewpoint video, similar to the flow shown in FIG. 24, and the generated viewer avatar viewpoint video is displayed by a video output process.

[0224] On the other hand, when the viewer avatar viewpoint video is not selected as the image to be displayed, i.e., when the virtual camera viewpoint video is selected as the image to be displayed, a virtual camera viewpoint video generation process is executed in the viewer terminal 300 to generate a virtual camera viewpoint video, and the generated virtual camera viewpoint video is displayed by a video output process.

[0225] Furthermore, the viewer user may be able to select which of the virtual cameras C1, C3, or C4 the virtual camera viewpoint image generated in this virtual camera viewpoint image should be, or the distribution server computer 100 may instruct all viewer terminals 300.

[0226] 27, during the performance period excluding the MC period and the flying performance period when either the virtual camera viewpoint video or the viewer avatar viewpoint video can be displayed on the viewer terminal 300, the virtual camera viewpoint video is generated on the viewer terminal 300. In this way, during the period when either the virtual camera viewpoint video or the viewer avatar viewpoint video can be selected and displayed on the viewer terminal 300, switching between the virtual camera viewpoint video and the viewer avatar viewpoint video is performed at the viewer user's selection, so that switching between these videos can be performed without problems such as video interruption and while preventing an increase in the processing load on the distribution server computer 100 that accompanies switching between these videos, as compared to when the virtual camera viewpoint video data is displayed by distribution from the distribution server computer 100.

[0227] On the other hand, during each of the periods of "introduction to the performance", "opening performance", and "flying performance", in which only the virtual camera viewpoint video can be displayed on the viewer terminal 300, the virtual camera viewpoint video is generated only on the distribution server computer 100 and distributed to each viewer terminal 300. In this way, during the period in which only the virtual camera viewpoint video can be displayed on the viewer terminal 300, the virtual camera viewpoint video is generated only on the distribution server computer 100, so that it is possible to prevent the process of generating the virtual camera viewpoint video from being executed redundantly on the viewer terminal 300, thereby preventing a needless increase in the processing load of the entire system.

[0228] Here, the situation when the viewpoint video is switched on the viewer terminal 300 will be described with reference to FIG. 28, taking as an example a period of "flying effect" during which only the virtual camera viewpoint video can be displayed.

[0229] For example, as shown in FIG. 28, in a viewer terminal 300 on which a viewer avatar viewpoint video is displayed during the "first MC" period and the first performance period up to the "flying performance", if the distribution server computer 100 starts distributing the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) from the start of the "flying performance" period in which only the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) can be displayed, the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) will not be displayed on the viewer terminal 300 during the period from the start of these distributions until the virtual camera viewpoint video is displayed, which could result in a problem in which the viewer is unable to view the first video of the "flying performance".

[0230] For this reason, at the start of a "flying performance" in which the viewpoint image is forcibly changed from the viewer avatar viewpoint image to the virtual camera viewpoint image (viewpoint image from a virtual camera moving in the air), as shown in FIG. 28, the distribution server computer 100 of the second embodiment sends a BF advance notification to the viewer terminal 300 at a specific timing prior to the start of the "flying performance", thereby notifying the viewer terminal 300 of the distribution start (buffering start) timing for starting advance distribution of the virtual camera viewpoint image (viewpoint image from a virtual camera moving in the air) before the start of the flying performance period.

[0231] In response to receiving the BF advance notification, the viewer terminal 300 identifies the timing at which distribution of the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) will begin from the distribution server computer 100, and begins buffering the received virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) from that timing.

[0232] The timing at which distribution of these virtual camera viewpoint images (viewpoint images from virtual cameras moving in the air) begins, that is, the timing at which buffering begins, can be determined appropriately taking into account differences in the image storage functions of the viewer terminals 300, but can be set to a period corresponding to several tens of frames of image, for example.

[0233] In this way, the distribution server computer 100 of Example 2 begins generating and distributing virtual camera viewpoint video of the flying performance (viewpoint video from a virtual camera moving in the air) before the start of the flying performance period, and the viewer terminal 300 buffers the distributed virtual camera viewpoint video. Therefore, when the flying performance period begins, the video can be instantly changed from the viewer avatar viewpoint video that was displayed before the start of the flying performance period to the virtual camera viewpoint video, thereby preventing the occurrence of a problem in which the viewer is unable to view the initial video of the "flying performance".

[0234] Furthermore, when the period of the ``flying performance'' ends, because the viewer avatar viewpoint video was selected before the ``flying performance'' began, the viewpoint video switches from the flying performance's virtual camera viewpoint video (viewpoint video from the virtual camera moving in the air) to the viewer avatar viewpoint video upon the end of the ``flying performance'' period. However, even in this case, if the viewer avatar viewpoint video is generated and displayed from the end of the ``flying performance'' period, there is a risk that a malfunction will occur in which the viewer avatar viewpoint video is not displayed on viewer terminal 300 during the period until the viewer avatar viewpoint video is generated and displayed, just as at the start of the ``flying performance''.

[0235] For this reason, even when the period of these "flying effects" ends, the distribution server computer 100 of the second embodiment sends an end TM (timing) advance notification to the viewer terminal 300, as shown in FIG. 28, thereby notifying the viewer terminal 300 in advance of the end timing of the "flying effects" period when the distribution of the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) will end.

[0236] In response to receiving the end TM (timing) advance notification, the viewer terminal 300 determines in advance the timing at which the distribution of the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) from the distribution server computer 100 will be stopped, and starts the viewer viewpoint video generation process at a specified timing before that timing, thereby pre-generating the viewer avatar viewpoint video.

[0237] Furthermore, the distribution server computer 100 starts distributing the virtual space update data from the time of sending the end TM (timing) advance notification, and the viewer terminal 300 executes virtual space processing to update the virtual space data using the received virtual space update data, so that at the specified timing mentioned above, a viewer avatar viewpoint image that reflects the situation of the virtual live venue at that time can be generated.

[0238] In this way, the distribution server computer 100 of Example 2 sends an end TM (timing) advance notification to the viewer terminal 300 before the end of the flying performance period, enabling the viewer terminal 300 to generate the viewer avatar viewpoint video in advance before the end of the flying performance period, thereby preventing a problem from occurring in which the viewer avatar viewpoint video is not displayed on the viewer terminal 300.

[0239] In addition, in Figure 28, the "flying performance" is used as an example, but similar control is also performed during the "pre-show speech" and "opening performance" periods, when the viewpoint image is forcibly changed from the viewer avatar viewpoint image to the virtual camera viewpoint image.

[0240] As described above, in the virtual space content distribution system of Example 2, there are multiple viewer users, and each viewer user is assigned a viewer terminal 300. When the viewer terminal 300 displays the viewer avatar viewpoint video, which is a viewpoint video of the virtual live venue (virtual space) corresponding to the viewer avatar, the viewer avatar viewpoint video is generated based on the virtual space update data (virtual space data) transmitted from the distribution server computer 100. Since the generation of these viewer avatar viewpoint videos, which have a large processing load, is performed by the viewer terminal 300, more viewer users can participate in the virtual live.

[0241] Furthermore, the virtual space content of Example 2 is content that allows a performer user to participate in a virtual live venue as a performer avatar 1, and is content in which the performer avatar 1 performs musical actions (performances) such as singing and dancing in the virtual live venue, thereby increasing the interest of the virtual space content.

[0242] Furthermore, since the virtual space update data (virtual space data) of Example 2 includes movement data for the performer avatar for generating the movements of the performer avatar 1, the movements of the performer avatar 1 can be accurately reflected in the viewer avatar viewpoint image generated in the viewer terminal 300.

[0243] In addition, the common viewpoint video displayed in common on each viewer terminal 300 in Example 2, such as the video of the "pre-show speech," "opening performance," and "early performance," is generated in the distribution server computer 100 and distributed to each viewer terminal 300, thereby preventing unnecessary increase in the processing load of the entire system due to duplicate processing being executed in each viewer terminal 300.

[0244] Furthermore, in the viewer terminal 300 of Example 2, during the performance period (specific period) other than the "MC period" and "flying performance" periods, during which either the viewer avatar view point video or the virtual camera view point video can be selected and displayed, the virtual camera view point video is generated and displayed in the viewer terminal 300, thereby preventing problems such as video interruptions when the view point video is switched.

[0245] In addition, during special periods such as "costume change performance" and "doppelganger performance" when the processing load on the distribution server computer 100 in Example 2 is high, each viewer terminal 300 also generates and displays the virtual camera viewpoint image (common viewpoint image), thereby preventing the processing load on the distribution server computer 100 from becoming temporarily extremely high.

[0246] Furthermore, the distribution server computer 100 of Example 2 transmits advance data including basic data of the virtual live venue (virtual space) and basic data of the performer avatar 1 and viewer avatars who will virtually participate in the virtual live venue to each viewer terminal 300 before the virtual live starts, thereby preventing the occurrence of a problem in which the viewer avatar's viewpoint image is not displayed properly at the start of the virtual live.

[0247] In addition, the distribution server computer 100 of Example 2 transmits virtual camera viewpoint video (common viewpoint video) to viewer terminals 300 for which it cannot confirm that they have completed receiving the advance data. In this way, it is possible to prevent the processing load on the distribution server computer 100 from increasing excessively due to the transmission of basic data, etc. after the start of the virtual live broadcast, while also preventing viewer users from being unable to watch the virtual live broadcast.

[0248] <Modification of Example 2> FIG. 29 is a flowchart showing the process of generating a viewer's viewpoint video in the modification 2-1 of the second embodiment.

[0249] In the above-mentioned Example 2, a form has been shown in which viewer avatar viewpoint images including the movements of the performer avatar 1 are always generated uniformly, but the present invention is not limited to this, and these viewer avatar viewpoint images may be generated in a number of ways with different processing loads in image generation, depending on the selection of the viewer user at the viewer terminal 300.

[0250] In other words, as the viewer terminal 300 used by the viewer user, as exemplified in Example 1, it is expected that terminals with different processing capabilities will be used, such as a desktop personal computer (PC) with high processing capabilities, or a smartphone terminal or tablet terminal with relatively lower processing capabilities than a personal computer (PC). If the viewer avatar viewpoint video is always generated uniformly on terminals with such different processing capabilities, the viewer avatar viewpoint video will not be generated properly on terminals with low processing capabilities, and there is a risk that a mismatch will occur between the viewer avatar viewpoint video and audio, or the viewer avatar viewpoint video will become a discontinuous, unnatural video. Therefore, for example, when generating viewer avatar viewpoint video including performer movements, it may be possible to allow the viewer to select appropriately depending on the processing capabilities of the terminal he or she is using, whether to generate viewer avatar viewpoint video including high-definition performer movements, which requires a high processing load for generating the video, or to generate viewer avatar viewpoint video including low-definition performer movements, which requires a low processing load for generating the video.

[0251] Specifically, the viewer viewpoint video generation process executed in the viewer terminal 300 may be as shown in the flow diagram of Fig. 29, and in the viewer viewpoint video generation process, first, the performer movement generation mode selected by the viewer user in the viewer terminal 300 is identified (step S101). Then, it is determined whether the identified performer movement generation mode is a high processing load mode (step S102), and if the high processing load mode is selected (Y in step S102), a viewer avatar viewpoint video including high-definition performer movements is generated (step S103). On the other hand, if the high processing load mode is not selected (N in step S102), a viewer avatar viewpoint video including low-definition performer movements is generated (step S104).

[0252] Although FIG. 29 shows an example of two modes, a high processing load mode and a low processing load mode, the present invention is not limited to this, and the processing load modes may be three or more modes.

[0253] In addition, in the above-mentioned Example 2 (which is also the same as Example 1), an example is given in which all actions, including the movement of the viewer avatar of the viewer user, are reflected in each viewer avatar by distributing virtual space update data, but the present invention is not limited to this. For example, in the case where the actions of these viewer avatars are actions that cannot normally be performed but become possible when a special fee is paid, for example, an operation to perform a special action that allows the viewer avatar to appeal to the performer avatar 1, the special action is directed to the performer user and not to other viewer users, so it is possible to prevent the virtual space update data including these special actions from being distributed to the viewer terminals 300 of other viewer users, thereby preventing an excessive increase in the processing load of the distribution server computer 100 by making these special actions executable.

[0254] In this way, if virtual space update data including special actions is not distributed to viewer terminals 300 of other viewer users, as shown in variant example 2-2 of Figure 30, when viewer user A performs an operation corresponding to the special action, causing viewer user A's viewer avatar to perform a special action, virtual space update data A corresponding to viewer user A's viewer avatar performing the special action is transmitted from the distribution server computer 100 to viewer terminal 300 of viewer user A, but virtual space update data B in which viewer user A's viewer avatar is not performing the special action is distributed to viewer terminal 300 of viewer user B, who is different from viewer user A.

[0255] Therefore, different virtual space update data is delivered from the distribution server computer 100 to the viewer terminal 300 of viewer user A and the viewer terminal 300 of viewer user B, thereby increasing the flexibility of the virtual space update data and making it possible to generate different viewer avatar viewpoint images on the viewer terminal 300 of viewer user A and the viewer terminal 300 of viewer user B.

[0256] Furthermore, viewer terminal 300 of viewer user A can generate and display an image of viewer avatar of viewer user A performing a special action different from the actions that viewer avatar of viewer user B is capable of performing based on virtual space update data A that is different from virtual space update data B delivered to viewer terminal 300 of viewer user B, thereby increasing viewer user A's interest in the virtual live performance, which is the virtual space content.

[0257] In addition, in the above-mentioned Example 2 (which is also the same as Example 1), the viewer avatars of viewer users virtually participating in the fourth area are illustrated as normal avatars in which the avatar appearance differs for each viewer user, but the present invention is not limited to this. For example, in order to reduce the processing load when generating various viewpoint images in a virtual live performance, the viewer avatars of viewer users virtually participating in the fourth area may be simplified avatars in which the avatar appearance is simplified.

[0258] Note that Figure 30 shows an example of executing a special action that becomes possible when a special fee is paid. However, for example, in the case of using a simplified avatar as described above, normal actions other than the movement of the simplified avatar, such as waving, clapping, and changing the direction of the viewer avatar, may be prevented from being included in the virtual space update data distributed to other viewer users, just like special actions, thereby further reducing the processing load on the distribution server computer 100. EXAMPLES

[0259] Next, a virtual space content delivery system according to a third embodiment will be described below with reference to Fig. 31 to Fig. 56. Note that the same components as those in the first and second embodiments will be denoted by the same reference numerals and description thereof will be omitted.

[0260] As shown in Fig. 31, the virtual space content delivery system of the third embodiment is provided with the same live performance content as that of the second embodiment. In particular, in the third embodiment, a performance is provided by the avatars of two performers, performer A and performer B, during the live performance.

[0261] Immediately after the start of the first MC period, performer avatar 1A of performer A, appears, and the first performance is a solo performance by performer avatar 1A of performer A. During the second MC period, performer avatar 1B of performer B, the second performer, appears (the number of performer avatars increases from one to two), and the second performance is a joint performance by performer avatar 1A of performer A and performer avatar 1B of performer B.

[0262] In addition, in the third embodiment, it is possible to execute a light emission effect in which the virtual aerial objects 50 to 52 are illuminated as part of the virtual live performance. In particular, the light emission effect is not executed during the virtual live performance venue participation period and the pre-performance opening period (light emission intensity of the light emission effect: none), but the virtual aerial objects 50 to 52 are illuminated with a relatively weak light emission intensity during the opening performance period, the first MC period, and the second MC period (light emission intensity of the light emission effect: weak), and the virtual aerial objects 50 to 52 are illuminated with a stronger light emission intensity during the first performance period and the second performance period than during the opening performance period, the first MC period, and the second MC period (light emission intensity of the light emission effect: strong).

[0263] Also, as shown in FIG. 31, in the period immediately up until the start of the first MC period, performer A's performer avatar 1A does not appear, so there is no movement of performer A's performer avatar 1A (performer A's performer avatar movement: none); in the first MC period and second MC period after performer A's performer avatar 1A appears, the movement of performer A's performer avatar 1A is relatively small (performer A's performer avatar movement: small); and in the first performance period and second performance period, performer A's performer avatar 1A dances and sings, so the movement of performer A's performer avatar is larger (more intense) than in other periods (performer A's performer avatar movement: large).

[0264] Similarly, the period until performer B's performer avatar 1B appears during the second MC period is a period during which performer B's performer avatar 1B makes no movements (performer B's performer avatar movements: none), the second MC period after performer B's performer avatar 1B appears is a period during which performer B's performer avatar 1B makes relatively small movements (performer B's performer avatar movements: small), and the second performance period is a period during which performer B's performer avatar 1B's movements become larger (more intense) than in other periods because performer B's performer avatar dances and sings together with performer A's performer avatar 1B (performer B's performer avatar movements: large).

[0265] In addition, the period during which participation in the virtual live venue is possible is a period during which multiple viewer terminals 300 sequentially connect to the distribution server computer 100, and therefore is a period during which viewer avatars who virtually participate in the virtual live venue through such connections are gradually added and placed within the virtual space over time.

[0266] 31 and 32(A), immediately after the start of the virtual live venue participation period, no light-emitting effects are being performed, performer avatars 1A, 1B of the performers are not being displayed, and the number of viewer avatars virtually participating in the virtual live venue is small, so that there are few viewer avatars requiring updates of location information, etc. in distribution server computer 100, and the update processing load for updating the virtual space is relatively small. Also, for each viewer terminal 300, because the number of viewer avatars virtually participating in the virtual live venue is small as described above, the drawing processing load for drawing the viewer avatars is relatively low.

[0267] 32(B), the amount of viewer avatar information that is the subject of updating the virtual space data in the distribution server computer 100 increases, increasing the update processing load for updating the virtual space in the distribution server computer 100. Also, as the number of viewer avatars virtually participating in the virtual live venue increases, the number of viewer avatars that are the subject of drawing in each viewer terminal 300 also increases, increasing the drawing processing load for drawing the viewer avatars.

[0268] Furthermore, as the number of viewer avatars virtually participating in the virtual live venue increases, the number of viewers performing movement operations from the viewer terminal 300 also increases, as shown in Figures 31 and 33, which may result in an increase in the number of viewer avatars moving around the virtual live venue.

[0269] Since a moving viewer avatar changes information such as its current location in the virtual live venue more frequently than a viewer avatar that is stationary in a specific location, an increase in the number of viewer avatars moving within the virtual live venue as shown in Figures 31 and 33 increases the update processing load for updating the virtual space by the distribution server computer 100. At this time, if the number of moving viewer avatars increases in each viewer terminal 300, the amount of drawing due to the movement of each viewer avatar also increases, and so the drawing processing load in the viewer terminal 300 also increases.

[0270] When performing light-emitting effects during each performance period, as shown in Fig. 31 and Fig. 34(A), the delivery server computer 100 must perform calculations of how light hits each viewer avatar and performer avatar and the shadows they generate based on the current position of each avatar and the distance and angle from the light source (virtual aerial objects 50-52 in this embodiment 3), increasing the update load of the virtual space update by the delivery server computer 100. At this time, the viewer terminals 300 must also perform drawing to make the virtual aerial objects 50-52 glow in association with the light-emitting effects, drawing to make the light hit each viewer avatar differently, and drawing shadows for each viewer avatar, increasing the drawing processing load.

[0271] In addition, in this embodiment 3, the light emission intensity of the virtual aerial objects 50 to 52 may be increased as a light emission effect (see Figure 31). When the light emission intensity is increased in this manner, the processing load associated with the calculation of the way the light hits each viewer avatar and performer avatar and the shadows generated in the distribution server computer 100 described above increases further, which results in a further increase in the update load of updating the virtual space by the distribution server computer 100, and similarly, the drawing processing load in each viewer terminal 300 also increases further.

[0272] In addition, when the performer avatar 1A appears, as shown in FIG. 31 and FIG. 34(B), the performer avatar information is added to the information to be updated in the virtual space data in the delivery server computer 100, so the update processing load of the virtual space update by the delivery server computer 100 increases. In addition, when the performer avatar 1B appears, as shown in FIG. 31 and FIG. 34(B), the performer avatar information of the performer avatar 1B is added to the information to be updated in the virtual space data in the delivery server computer 100 in addition to the performer avatar information of the performer avatar 1A, so the update processing load of the virtual space update by the delivery server computer 100 increases further. In the viewer terminal 300, the process of drawing the performer avatar 1A of performer A starts from the timing when the performer avatar 1A appears, so the drawing processing load increases. In addition, from the timing when the performer avatar 1B appears, the process of drawing the performer avatar 1B of performer B starts in addition to the process of drawing the performer avatar 1A, so the drawing processing load increases further. In this embodiment 3, after performer avatar 1B of performer B appears (after the number of performer avatars becomes two), the number of performer avatars does not decrease, so the process of drawing performer avatar 1A and the process of drawing performer avatar 1B continue until the end of the virtual live performance.

[0273] During each performance period, as shown in Fig. 31 and Fig. 35, in addition to increasing the light emission intensity of the virtual aerial objects 50-52 as a light emission effect, each performer avatar 1A, 1B performs a performance such as singing or dancing. Therefore, as described above, the processing load of the delivery server computer 100 for calculating the way the light hits each viewer avatar and performer avatar and the shadows generated further increases, and the update processing load of the delivery server computer 100 for updating the virtual space further increases. Also, in each viewer terminal 300, the drawing processing load increases for drawing the way the light hits and the shadows of each viewer avatar generated by illuminating the virtual aerial objects 50-52 and drawing the performer avatars 1A, 1B.

[0274] In this embodiment 3, when the processing load on the distribution server computer 100 or the viewer terminal 300 increases due to an increase in the number of viewer avatars and performer avatars placed in the virtual live venue as described above, an increase in the light emission intensity of the light emission effects, an increase in the number of moving viewer avatars and performer avatars, an increase in the amount of movement of the performer avatars, etc., it is possible to suppress an excessive increase in the processing load on the entire equipment that constitutes the virtual space content distribution system, such as the distribution server computer 100 and the viewer terminal 300, by changing from a full avatar, which has a relatively large processing load for displaying the display mode of the viewer avatar, to a simplified avatar, which has a relatively small processing load for displaying.

[0275] <Time schedule data> The virtual live of Example 3 proceeding as described above is managed by the delivery server computer 100 based on time schedule data (TS data), as shown in Fig. 36. The time schedule data is stored in the storage 103 of the delivery server computer 100 in a form included in the event management program 110.

[0276] The time schedule data is data in a table format in which the scheduled time of each timing, the elapsed time from the start of the live, the content of the timing, and the file name of the associated data related to the timing are stored in correspondence with a timing ID assigned to each individual timing in the virtual live.

[0277] As shown in Fig. 36, these timings include the start and end timings of the opening performance and the performer appearance performance, the timings when the performer avatars 1A and 1B appear during the performer appearance performance, as well as the start and end timings of each transition period and the timings when each viewer avatar is simplified (viewer avatar first simplified timing T1, viewer avatar second simplified timing T2, viewer avatar third simplified timing T3). Hereinafter, in this embodiment, these viewer avatar first simplified timing T1, viewer avatar second simplified timing T2, and viewer avatar third simplified timing T3 may be simply referred to as timings T1, T2, and T3.

[0278] Timing T1 is the timing immediately before the start of the first performance period when the light intensity of the light-emitting effects increases and the movements of performer avatar 1A become more intense, thereby increasing the drawing processing load on viewer terminal 300. Timing T2 is the timing immediately before the start of the second MC period when the drawing processing load on viewer terminal 300 increases due to the addition of performer avatar 1B. Timing T3 is the timing at the start of the second performance period when the light intensity of the light-emitting effects increases and the movements of performer avatars 1A, 1B become more intense, thereby increasing the drawing processing load on viewer terminal 300. These timings T1 to T3 are provided in order to change the viewer avatar from a full avatar to a simple avatar in advance, at a stage before the processing load has increased, immediately before the drawing processing load on viewer terminal 300 increases, as described above.

[0279] <Video generation during the period when live virtual space participation is possible> Also, during the live virtual space participation period in which only the viewer avatar viewpoint video can be displayed on the viewer terminal 300 in the third embodiment, as shown in FIG. 37, similar to the second embodiment, after performing avatar movement operation processing on the viewer terminal 300, virtual space update processing, and virtual space update data generation processing on the distribution server computer 100, the distribution server computer 100 further performs avatar simplification determination processing for determining whether to change the display of the viewer avatar from a full avatar to a simple avatar.

[0280] Note that after performing the avatar simplification determination processing, the distribution server computer 100 performs virtual space update data distribution processing in the same manner as in the second embodiment, and transmits the simplification determination value data, which is the result of the avatar simplification determination processing, together with the virtual space update data to the viewer terminal 300.

[0281] On the viewer terminal 300, based on receiving these virtual space update data and simplification determination value data, virtual space update processing, viewer viewpoint video generation processing, and video output processing are executed. In particular, in the third embodiment, although details will be described later, by using the received simplification determination value data, processing for drawing (generating) the viewer avatar as a full avatar or a simple avatar is performed in the viewer viewpoint video generation processing.

[0282] <Video Generation During the Performance Period Excluding the MC Period and the Flying Effect Period> Also, during the performance period excluding the MC period and the flying effect period in which the viewer avatar viewpoint video and the virtual camera viewpoint video can be displayed on the viewer terminal 300 in the third embodiment, as shown in FIG. 38, similar to the second embodiment, after performing avatar movement operation processing on the viewer terminal 300, virtual space update processing, and virtual space update data generation processing on the distribution server computer 100, the distribution server computer 100 further performs avatar simplification determination processing for determining whether to change the display of the viewer avatar from a full avatar to a simple avatar. Note that this avatar simplification determination processing is the same as the avatar simplification determination processing shown in FIG. 38.

[0283] After executing the avatar simplification determination process, the distribution server computer 100 executes the virtual space update data distribution process as in Example 2, thereby transmitting the simplification determination value data, which is the result of the above-mentioned avatar simplification determination process, together with the virtual space update data to the viewer terminal 300.

[0284] Based on receiving the virtual space update data and simplification judgment value data, the viewer terminal 300 executes a virtual space update process, a viewer viewpoint video generation process if a viewer avatar viewpoint video is selected as the video to be displayed, and a virtual camera designated video generation process if a virtual camera viewpoint video is selected as the video to be displayed. In particular, in this embodiment 3, the received simplification judgment value data is used to perform a process of drawing (generating) the viewer avatar as a full avatar or a simplified avatar in the viewer viewpoint video generation process and the virtual camera designated video generation process, as will be described in detail later.

[0285] <About friend registration> Moreover, in the virtual space content delivery system of the present embodiment 3, a viewer can register other viewers as his / her friends by operating the viewer terminal 300. Note that the storage 303 of the viewer terminal 300 of the present embodiment 3 stores a friend user data table (see FIG. 40) that stores the user IDs and nicknames of viewer users who have been registered as friends in advance.

[0286] As shown in FIG. 39, the viewer terminal 300 executes a viewer avatar selection process to select a specific viewer avatar from among multiple viewer avatars by accepting operation of the viewer's operation input device 321 during the period in which participation in the live virtual space is possible, and then executes a viewer avatar query process to identify the file name of the selected viewer avatar (viewer avatar file name) and send viewer avatar query information including the identified viewer avatar file name to the distribution server computer 100.

[0287] When the delivery server computer 100 receives viewer avatar inquiry information from the viewer terminal 300, it executes a viewer avatar identification process to identify the viewer user ID stored in association with the viewer avatar file name included in the viewer avatar inquiry information from the viewer user data shown in Fig. 9. Then, the delivery server computer 100 transmits viewer user ID information including the viewer user ID to the viewer terminal 300 that has transmitted the viewer avatar inquiry information.

[0288] When the viewer terminal 300 receives the viewer user ID information, it executes a friend addition operation process on the display device 310 to prompt the viewer to register the viewer (viewer user ID) of the selected viewer avatar as a friend by associating it with the viewer avatar file name.

[0289] At this time, if the viewer performs an operation to register a friend, a friend addition process is executed to store the viewer user ID and the viewer avatar file name in the friend user data table in association with each other, and if the operation to register a friend is not performed, the viewer terminal 300 executes a friend addition cancellation process to display on the display device 310 an indication that the friend registration will not be performed.

[0290] <Avatar simplification judgment process> Fig. 41 is a flow chart showing an example of the avatar simplification determination process shown in Fig. 38. In the avatar simplification determination process, the distribution server computer 100 first determines whether or not it is during the virtual live venue participation period (see Fig. 31) by referring to the time schedule data or the like (step S401). If it is not during the virtual live venue participation period (step S401; No), the process proceeds to step S407, and if it is during the virtual live venue participation period (step S401; Yes), the process further determines whether or not the participant number threshold reached flag is set (step S402). The participant number threshold reached flag is a flag indicating that the number of viewer avatars placed in the virtual live venue (the number of viewers watching the virtual live) has already reached a predetermined threshold.

[0291] If the participant number threshold reached flag is not set (step S402; Yes), the distribution server computer 100 proceeds to step S407. If the participant number threshold reached flag is not set (step S402; No), the distribution server computer 100 further refers to the participant user table shown in FIG. 23 to identify the number of participants in the live performance (the number of viewer avatars virtually participating in the virtual live performance venue) (step S403), and determines whether the identified number of participants in the live performance has reached the threshold (step S404).

[0292] The threshold value for the number of participants is a value determined from the number of full viewer avatars that can be rendered at one time on each viewer terminal 300 and the number of full viewer avatars that can be arranged in a virtual live venue. For example, if the limit number for the number of full viewer avatars that can be rendered at one time on each viewer terminal 300 and the limit number for the number of full viewer avatars that can provisionally participate in a virtual live venue is 200, the threshold value for the number of participants may be set by an administrator of the distribution server computer 100 to 180, which is about 10% lower than the limit number. The threshold value for the number of participants is not simply the number of viewer avatars to be rendered, but is a value determined solely by the processing load corresponding to the number of viewer avatars that can be rendered on the viewer terminal 300.

[0293] Incidentally, the administrator of the delivery server computer 100 may set the threshold value for the number of participants to a value different from the above in the administrator terminal 150, taking into consideration the performance (drawing processing capability) of each viewer terminal 300, etc.

[0294] If the number of participants has not reached the threshold (step S404; No), the process proceeds to step S407, whereas if the number of participants in the reached live event has reached the threshold (step S404; Yes), the distribution server computer 100 adds 1 to the simplification determination value based on the fact that the rendering processing load of the viewer avatar in full avatar form in the viewer terminal 300 is approaching its limit (step S405), sets a number-of-participants threshold reached flag (step S406), and proceeds to step S407. The simplification determination value, which will be described in detail later, is a value for specifying the range in which each viewer avatar displayed in each viewer terminal 300 is displayed as a simplified avatar with a smaller number of polygons, and is set to "0" as the initial value.

[0295] In step S407, the distribution server computer 100 determines whether it is within a period during which virtual live venue participation is possible or within an MC period by referring to time schedule data or the like (step S407). If it is not within a period during which virtual live venue participation is possible or within an MC period (step S407; No), the process proceeds to step S413, and if it is within a period during which virtual live venue participation is possible or within an MC period (step S407; Yes), the process determines whether a movement number threshold reached flag has been set (step S408). The movement number threshold reached flag is a flag indicating that the number of viewer avatars moving within the virtual live venue as full avatars among the viewer avatars placed in the virtual live venue has already reached a predetermined threshold.

[0296] If the moving number threshold reached flag is set (step S408; Yes), proceed to step S413; if the moving number threshold reached flag is not set (step S408; No), the distribution server computer 100 determines the number of viewer avatars currently moving (the number of viewer terminals 300 being moved by each viewer) from the viewer avatar information received from each viewer terminal 300 (step S409).

[0297] In step S409, the distribution server computer 100, for example, stores position information of each viewer avatar within the virtual live venue for each unit time (e.g., 10 ms), compares the stored position information of each viewer avatar within the virtual live venue with previously stored position information of each viewer avatar within the virtual live venue, and identifies the number of viewer avatars currently moving by tallying up the number of viewer avatars for which position information different from that previously stored.

[0298] Then, the delivery server computer 100 judges whether the number of the identified moving viewer avatars (the number of viewer avatars currently moving) has reached a threshold value (step S410). The threshold value of the number of moving viewer avatars in step S410 is a value determined from the number of moving viewer avatars in full avatars that can be drawn at one time in each viewer terminal 300. For example, if the limit number of moving viewer avatars in full avatars that can be drawn at one time in each viewer terminal 300 is 200, the threshold value of the number of moving viewer avatars may be set by an administrator of the delivery server computer 100 to 150, which is 25% lower than the limit number. The threshold value of the number of moving viewer avatars is not simply the number of viewer avatars to be drawn, but is a value determined by the processing load corresponding to the number of viewer avatars that can be drawn at the viewer terminal 300.

[0299] The threshold value for the number of moving viewer avatars may be set to a value different from the above by an administrator of the delivery server computer 100, taking into consideration the performance (drawing processing capability) of each viewer terminal 300, etc.

[0300] If the number of moving viewer avatars has not reached the threshold value (step S410; No), proceed to S413; if the number of moving viewer avatars has reached the threshold value (step S410; Yes), the distribution server computer 100 adds 1 to the simplification judgment value based on the fact that the drawing processing load of the full avatar viewer avatars on the viewer terminal 300 is approaching its limit (step S411), and sets the number-of-moving-people threshold-reached flag and proceeds to step S413 (step S412).

[0301] In step S413, the delivery server computer 100 determines whether or not the current time is timing T1 immediately before the start of the first performance (step S413). If it is determined that the current time is timing T1 immediately before the start of the first performance (step S413; Yes), the process proceeds to step S416.

[0302] If it is determined that the current time is not timing T1 immediately before the start of the first performance (step S413; No), the distribution server computer 100 determines whether it is timing T2 at which performer B's performer avatar 1B will appear (step S414). If it is determined that the current time is timing T2 immediately before performer B's performer avatar 1B will appear (step S414; Yes), the process proceeds to step S416.

[0303] Furthermore, if it is determined that the time is not the timing T2 immediately before the performer avatar 1B of performer B appears (step S414; No), the distribution server computer 100 determines whether or not the time is the timing T3 immediately before the start of the second performance (step S415). If it is determined that the time is the timing T3 immediately before the start of the second performance (step S415; Yes), the process proceeds to step S416, and if it is determined that the time is not the timing T3 immediately before the start of the second performance (step S415; No), the process proceeds to step S417.

[0304] In step S416, the distribution server computer 100 adds 1 to the simplification determination value based on the fact that the drawing processing load of the viewer terminal 300 increases immediately after the above-mentioned times T1, T2, and T3 are reached, and proceeds to step S417 (step S416).

[0305] Furthermore, in step S417, the delivery server computer 100 executes a combined simplification judgment process. The combined simplification judgment process judges whether or not the total processing load of the viewer avatar drawing process load as the full avatar in the viewer terminal 300, the calculation of the shadow generation of each viewer avatar, the drawing process load of the performer avatar, and the like has reached the limit of the processing load of the viewer terminal 300, even if the threshold value of the number of participants has not been reached in step S404, the threshold value of the number of moving viewer avatars has not been reached in step S410, or the timing is not T1 to T3 in steps S413 to S415. If the total processing load of the viewer terminal 300 has reached the limit of the processing load of the viewer terminal 300, the process of adding 1 to the simplification judgment value may be performed in the same manner as in steps S405, S411, and S416.

[0306] In this embodiment 3, an example is shown in which the simplification judgment value is incremented by 1 at timing T2 because the number of performer avatars will be 2 immediately after (performer avatar 1B appears in addition to performer avatar 1A). In other words, the threshold value for the number of performer avatars is set to "2", but the threshold value for the number of performer avatars is not simply the number of performer avatars to be drawn, but is a value determined by the processing load corresponding to the number of performer avatars that can be drawn in the viewer terminal 300.

[0307] The threshold value for the number of performer avatars may be set to a value different from the above as appropriate by the administrator of the distribution server computer 100, taking into consideration the performance (drawing processing capability) of each viewer terminal 300. In particular, in this embodiment 3, when the threshold value for the number of performer avatars is set to "1", the simplification judgment value is incremented by 1 immediately before the appearance of performer avatar 1A of performer A during the first MC period.

[0308] In addition, in the composite simplification judgment process of this embodiment 3, an example is given of a form in which it is judged whether the combined processing load of the drawing processing load of the viewer avatar as a full avatar in the viewer terminal 300, the calculation of the shadow generation of each viewer avatar, the drawing processing load of the performer avatar, etc. has reached the limit of the processing load of the viewer terminal 300. However, the present invention is not limited to this, and the composite simplification judgment process may add 1 to the simplification judgment value on the condition that the processing load of the viewer terminal 300 has reached a processing load different from the processing load corresponding to the threshold value for the number of participants or the threshold value for the number of moving viewer avatars.

[0309] After executing the process of step S417, the distribution server computer 100 performs only the transmission setting of the simplification determination value, and ends the avatar simplification determination process (step S418).

[0310] <Viewer's viewpoint video generation process> Fig. 42 is a flow chart showing an example of the viewer viewpoint video generation process shown in Fig. 38. In the viewer viewpoint video generation process, the viewer terminal 300 first generates a video of the performer avatar in the full avatar shown in Fig. 45(A) (step S501). As shown in Fig. 45(A), the performer avatars 1A and 1B include full avatars created based on the preferences of each performer (performer A and performer B in this embodiment 3) and simplified avatars that are simplified versions of the full avatars. The simplified avatars are avatar models created with fewer polygons than the full avatars (number of polygons: performer's full avatar>performer's simplified avatar), and the rendering process load on each viewer terminal 300 is lighter than that of the full avatars.

[0311] Next, it is determined whether the simplification determination value is 0 (step S502). If the simplification determination value is 0 (step S502; Yes), images of both the viewer avatars of the viewers who are registered as friends and the viewer avatars of the viewers who are not registered as friends are generated as full avatars, and the process proceeds to step S509 (step S503). As shown in FIG. 45(B), full avatars created based on the preferences of each viewer are provided as viewer avatars. These full avatars as viewer avatars are avatar models created with the same number of polygons as the performer's simplified avatar (number of polygons: performer's simplified avatar ≒ viewer's full avatar), and the shape, color scheme, etc. differ for each viewer.

[0312] In this third embodiment, an example is shown in which the performer's full avatar and the viewer's full avatar are created with approximately the same number of polygons, but the present invention is not limited to this. As long as the number of polygons in the performer's full avatar and the viewer's full avatar is less than the number of polygons in the performer's full avatar, the number of polygons in the viewer's full avatar may be more than the number of polygons in the performer's simplified avatar, or the number of polygons in the performer's simplified avatar may be more than the number of polygons in the viewer's full avatar.

[0313] 45(B), simplified avatars created with fewer polygons than each viewer's full avatar are also provided as viewer avatars. The simplified avatars as viewer avatars have a common form regardless of the form of each viewer's full avatar and are created with fewer polygons than each viewer's full avatar (number of polygons: viewer's full avatar > viewer's simplified avatar), and therefore impose a lighter rendering processing load on each viewer terminal 300 than a full avatar.

[0314] Returning to FIG. 42, if the simplification judgment value is not 0 (step S502; No), the viewer terminal 300 further judges whether the simplification judgment value is 1 or not (step S504). If the simplification judgment value is 1 (step S504; Yes), it judges whether the friend viewer avatar specified flag is set or not (step S505a). The friend viewer avatar specified flag is a flag indicating that a viewer avatar file name existing in the friend user data table has been specified. If the friend viewer avatar specified flag is set (step S505a; Yes), the process proceeds to step S506. If the friend viewer avatar specified flag is not set (step S505a; No), the process refers to the friend user data table shown in FIG. 40, specifies a viewer avatar data file name existing in the friend user data table, and stores the specified viewer avatar data file name (step S505b). Furthermore, the viewer terminal 300 sets the friend viewer avatar specified flag and proceeds to step S506 (step S505c).

[0315] Then, in step S506, the viewer terminal 300 generates a full avatar image of the viewer avatar (friend's viewer avatar) corresponding to the viewer avatar data file name identified in step S505 (step S506), while generating simplified avatar images of the viewer avatars (other people's viewer avatars) other than the viewer avatar corresponding to the viewer avatar data file name identified in step S505b (step S507), and proceeds to step S509.

[0316] Also, if the simplification determination value is not 1 (step S504; No), the viewer terminal 300 generates images of all viewer avatars as simplified avatars, regardless of whether the viewer is registered as a friend, since the simplification determination value is 2 or more (step S508), and proceeds to step S509.

[0317] In step S509, the viewer terminal 300 generates viewer viewpoint videos of other viewer avatars other than the performer avatars 1A and 1B and the viewer avatar (step S509), and ends the viewer viewpoint video generation process.

[0318] As described above, in this embodiment 3, the performer avatars 1A and 1B generated in the viewer viewpoint video generation process are always generated as full avatars regardless of the value of the simplification determination process, as shown in Fig. 44(A). In other words, the performer avatars 1A and 1B are always displayed as full avatars from the viewer avatar viewpoint of the viewer operating the viewer terminal 300.

[0319] Furthermore, among the viewer avatars, for the viewer avatars of viewers who are registered as friends, a full avatar image is generated when the simplification determination value is 1 or less, whereas a simplified avatar image is generated when the simplification determination value is 2 or more. In other words, from the viewpoint of the viewer avatar of the viewer operating the viewer terminal 300, the viewer avatars who are registered as friends are displayed as full avatars when the simplification determination value is 1 or less, whereas a simplified avatar is displayed when the simplification determination value is 2 or more.

[0320] Furthermore, for viewer avatars of viewers who are not registered as friends, a full avatar image is generated when the simplification determination value is 0, whereas a simplified avatar image is generated when the simplification determination value is 1 or greater. In other words, from the viewpoint of the viewer avatar of the viewer operating the viewer terminal 300, a viewer avatar who is not registered as a friend is displayed as a full avatar only when the simplification determination value is 0, whereas a simplified avatar is displayed when the simplification determination value is 1 or greater.

[0321] <Virtual camera viewpoint video generation process (viewer terminal)> Fig. 43 is an example of a flowchart showing the virtual camera viewpoint video generation process (virtual camera viewpoint video generation process executed by the viewer terminal 300) shown in Fig. 38. In the virtual camera viewpoint video generation process, the viewer terminal 300 first generates full avatar videos of the performer avatars 1A and 1B (step S511), and determines whether the simplification determination value is 0 (step S512). If the simplification determination value is 0 (step S512; Yes), full avatar videos of the viewer avatar of the viewer operating the viewer terminal 300 and all other viewer avatars regardless of whether they are registered as friends are generated (step S513), and the process proceeds to step S523.

[0322] If the simplification determination value is not 0 (step S512; No), it is further determined whether or not the simplification determination value is 1 (step S514). If the simplification determination value is 1 (step S514; Yes), it is determined whether or not the friend viewer avatar specified flag is set (step S515a). If the friend viewer avatar specified flag is set (step S515a; Yes), the process proceeds to step S516. If the friend viewer avatar specified flag is not set (step S515a; No), the viewer terminal 300 refers to the friend user data shown in FIG. 40 and specifies the viewer avatar data file name present in the friend user data (step S515b). In addition, the viewer terminal 300 sets the friend viewer avatar specified flag and proceeds to step S516 (step S515c).

[0323] Then, the viewer terminal 300 generates full avatar images of the viewer's own viewer avatar operating the viewer terminal 300 and the viewer avatar corresponding to the viewer avatar data file name identified in step S515b (friend's viewer avatar) (step S516), while generating simplified avatar images of viewer avatars (other people's viewer avatars) other than the viewer's own viewer avatar operating the viewer terminal 300 and the viewer avatar corresponding to the viewer avatar data file name identified in step S515 (step S517), and proceeds to step S523.

[0324] If the simplification determination value is not 1 (step S514; No), it is further determined whether or not the simplification determination value is 2 (step S518). If the simplification determination value is 2 (step S518; No), the viewer terminal 300 generates an image of a viewer avatar of the viewer operating the viewer terminal 300 as a full avatar (step S520), while generating an image of a viewer avatar of a viewer other than the viewer's own viewer user as a simplified avatar (step S521), and proceeds to step S523.

[0325] If the simplification determination value is not 2 (step S518; No), the viewer terminal 300 determines that the simplification determination value is 3 or more, generates images of the viewer avatar of the viewer operating the viewer terminal 300 and all other viewer avatars, regardless of whether they are registered as friends, using simplified avatars (step S522), and proceeds to step S523. In step S523, the viewer terminal 300 generates other virtual camera viewpoint images and ends the virtual camera viewpoint image generation process (step S523).

[0326] As described above, in this embodiment 3, the performer avatars 1A and 1B generated by the viewer viewpoint video generation process of the viewer terminal 300 are always generated as full avatars regardless of the value of the simplification determination process, as shown in Fig. 44(B). In other words, from the viewer avatar viewpoint of the viewer operating the viewer terminal 300, the performer avatars are always displayed as full avatars.

[0327] Furthermore, among the viewer avatars, with regard to the viewer avatar of the viewer operating the viewer terminal 300, if the simplification judgment value is 2 or less, a full avatar image is generated, whereas if the simplification judgment value is 3 or more, a simplified avatar image is generated. In other words, from the virtual camera viewpoint, the viewer avatar of the viewer operating the viewer terminal 300 is displayed as a full avatar if the simplification judgment value is 2 or less, whereas if the simplification judgment value is 3 or more, a simplified avatar image is generated.

[0328] Furthermore, for viewer avatars of viewers who are registered as friends, if the simplification determination value is 1 or less, a full avatar image is generated, whereas if the simplification determination value is 2 or more, a simplified avatar image is generated. In other words, from the virtual camera viewpoint, the viewer avatars of viewers who are registered as friends are displayed as full avatars if the simplification determination value is 1 or less, whereas if the simplification determination value is 2 or more, a simplified avatar image is generated.

[0329] Furthermore, for viewer avatars of viewers who are not registered as friends, a full avatar image is generated when the simplification determination value is 0, whereas a simplified avatar image is generated when the simplification determination value is 1 or greater. In other words, from the virtual camera viewpoint, viewer avatars of viewers who are not registered as friends are displayed as full avatars only when the simplification determination value is 0, whereas a simplified avatar image is generated when the simplification determination value is 1 or greater.

[0330] In this embodiment 3, as shown in Figures 38, 42, and 43, an example has been given in which each viewer avatar in the viewer viewpoint video and virtual camera viewpoint video generated in the viewer terminal 300 is changed from a full avatar to a simplified avatar in response to an increase in the drawing processing load in the viewer terminal 300, but the present invention is not limited to this, and each viewer avatar in the virtual camera viewpoint video generated in the delivery server computer 100 may also be changed from a full avatar to a simplified avatar as shown in Figure 25. In this case, each viewer avatar may be changed from a full avatar to a simplified avatar in response to an increase in the drawing processing load in the delivery server computer 100.

[0331] Furthermore, since it is necessary to prevent an excessive increase in the processing load of the entire virtual space content distribution system of the present invention, the viewer avatar in the viewer perspective image and the virtual camera perspective image generated by the viewer terminal 300 may be changed from a full avatar to a simplified avatar in response to an increase in the drawing processing load in the distribution server computer 100, or the viewer avatar in the virtual camera perspective image generated by the distribution server computer 100 may be changed from a full avatar to a simplified avatar in response to an increase in the drawing processing load in the viewer terminal 300.

[0332] In this embodiment 3, as described above, an example has been given of a form in which the viewer avatar is changed from a full avatar to a simplified avatar in the viewer terminal 300 in accordance with the processing load corresponding to the number of avatars that can be drawn in the viewer terminal 300. However, the present invention is not limited to this, and the viewer avatar may be changed from a full avatar to a simplified avatar simply based on exceeding a threshold value for the number of participants, a threshold value for the number of moving viewer avatars, or a threshold value for the number of performer avatars.

[0333] Next, changes in the display of performer avatars 1A, 1B and viewer avatars in the virtual live venue will be described using as an example the display from the viewpoint of virtual camera C2 on viewer terminal 300 of the viewer operating viewer avatar 11. In the following description, viewer avatars 12, 13, and 14 will be described as avatars of viewers who have been registered as friends by the viewer operating viewer avatar 11, and the other viewer avatars will be described as avatars of viewers who have not been registered as friends by the viewer operating viewer avatar 11.

[0334] In this embodiment 3, we will explain the changes in the display of performer avatars 1A, 1B and viewer avatars from the viewpoint of virtual camera C2. However, we will not explain the display of performer avatars 1A, 1B and viewer avatars from the viewer viewpoint (the viewpoint from viewer avatar 11) because it is the same as from the viewpoint of virtual camera C2 except that viewer avatar 11 is not displayed.

[0335] In addition, in this embodiment 3, since a composite simplification judgment process (step S417) is executed in the avatar simplification judgment process shown in FIG. 41, the viewer avatar does not necessarily change from a full avatar to a simplified avatar at the timing when the number of participants reaches a threshold value, the timing when the number of moving viewer avatars reaches a threshold value, or at any of the timings T1 to T3. However, to avoid complicating the explanation, the following will use an example in which the viewer avatar changes from a full avatar to a simplified avatar at the timing when the number of participants reaches a threshold value, the timing when the number of moving viewer avatars reaches a threshold value, or at timings T1 to T3.

[0336] First, as shown in Fig. 46, in the case where neither the threshold value for the number of participants nor the threshold value for the number of moving viewer avatars is reached within the virtual live venue is determined, performer avatar 1A of performer A is displayed as a full avatar from the time when he appears immediately after the start of the first MC period, and performer avatar 2B of performer B is displayed as a full avatar from the time when he appears during the second MC period. Note that both performer avatar 1A of performer A and performer avatar 1B of performer B are maintained as full avatars without changing to simple avatars.

[0337] Meanwhile, from the start of display during the virtual live concert venue participation period until timing T1, immediately before the performance (first performance) by performer avatar 1A of performer A begins, all viewer avatars are displayed as full avatars (see FIG. 49(A)). From timing T1, the illuminating intensity of the lighting effects increases from the start of the first performance immediately following, and the movements of performer avatar 1A become more intense, increasing the load of the rendering process on viewer terminal 300. In response to this, the simplification determination value is updated from 0 to 1, and the viewer avatars of viewers who are not registered as friends by the viewer operating viewer avatar 11 begin to be displayed as simplified avatars (see FIG. 49(B)).

[0338] Furthermore, immediately after timing T2, performer avatar 1B appears, that is, as the number of performer avatars increases from 1 to 2, which increases the drawing processing load on the viewer terminal 300, the simplification judgment value is updated from 1 to 2, and the viewer avatar 11 begins to be displayed as a simplified avatar as the viewer avatar of the viewer who has been registered as a friend by the viewer operating the viewer avatar 11 (see Figure 49 (C)).

[0339] Then, from timing T3, the simplification judgment value is updated from 2 to 3 in response to the start of the second performance, in which the light intensity of the lighting effects increases and the movements of performer avatars 1A, 2B of performers A and B become more intense, and the viewer avatar 11 itself begins to be displayed as a simplified avatar (see Figure 49 (D)).

[0340] 47, when a determination is made as to whether the participant number threshold has been reached in a virtual live venue, performer avatar 1A of performer A is displayed as a full avatar from the time when he appears, which is immediately after the start of the first MC period, and performer avatar 1B of performer B is displayed as a full avatar from the time when he appears during the second MC period. Note that both performer avatar 1A of performer A and performer avatar 1B of performer B are maintained as full avatars without changing to simple avatars.

[0341] On the other hand, from the time display begins during the virtual live venue participation period until it is determined that the number of participants threshold has been reached during the virtual live venue participation period, all viewer avatars are displayed as full avatars (see Figure 49(A)), but if it is determined that the number of participants threshold has been reached during the virtual live venue participation period, the simplified judgment value is updated from 0 to 1 at the timing of the reaching judgment, and the viewer avatars of viewers who are not registered as friends by the viewer operating the viewer avatar 11 will begin to be displayed as simplified avatars (see Figure 49(B)).

[0342] From timing T1, the first performance begins, in which the light intensity of the lighting effects increases and the movements of the performer avatar 1A become more intense, causing the drawing processing load on the viewer terminal 300 to increase. In response to this, the simplification judgment value is updated from 1 to 2, and the viewer avatar 11 begins to be displayed as a simplified avatar as the viewer avatar of the viewer who has been registered as a friend by the viewer operating the viewer avatar 11 (see Figure 49 (C)).

[0343] Then, immediately after timing T2, performer avatar 1B appears, that is, as the number of performer avatars increases from 1 to 2, the simplification judgment value is updated from 2 to 3 in response to an increase in the drawing processing load on the viewer terminal 300, and the viewer avatar 11 itself begins to be displayed as a simplified avatar (see Figure 49 (D)).

[0344] 48, when the threshold value for the number of participants and the threshold value for the number of moving viewer avatars are reached are determined within the virtual live venue, performer avatar 1A of performer A is displayed as a full avatar from the time when he appears immediately after the start of the first MC period, and performer avatar 1B of performer B is displayed as a full avatar from the time when he appears during the second MC period. Note that both performer avatar 1A of performer A and performer avatar 1B of performer B are maintained as full avatars without changing to simple avatars.

[0345] On the other hand, from the time display begins during the virtual live venue participation period until it is determined that the number of participants threshold has been reached during the virtual live venue participation period, all viewer avatars are displayed as full avatars (see Figure 49(A)), but if it is determined that the number of participants threshold has been reached during the virtual live venue participation period, the simplified judgment value is updated from 0 to 1 at the timing of the reaching judgment, and the viewer avatars of viewers who are not registered as friends by the viewer operating the viewer avatar 11 will begin to be displayed as simplified avatars (see Figure 49(B)).

[0346] In addition, if it is determined that the number of moving viewer avatars during the period in which participants can participate in the virtual live venue has reached a threshold value, the simplification determination value is updated from 1 to 2 at the time of reaching the threshold value, and simplified avatars will begin to be displayed as viewer avatars for viewers who have been registered as friends by the viewer operating the viewer avatar 11 (see Figure 49 (C)).

[0347] Then, immediately after timing T1, the first performance begins, in which the light intensity of the light-emitting effects increases and the movements of the performer avatar 1A become more intense, causing the drawing processing load on the viewer terminal 300 to increase. In response, the simplification judgment value is updated from 2 to 3, and the viewer avatar 11 itself begins to be displayed as a simplified avatar (see Figure 49 (D)).

[0348] In addition, in this Example 3, even after the simplification judgment value is updated to 3 and all viewer avatars start to be displayed as simplified avatars, the simplification judgment value may be updated to 4, 5, etc., but even if the simplification judgment value is updated to 4 or more, all viewer avatars will not change further from the simplified avatars.

[0349] As described above, in this embodiment 3, as shown in Figs. 41 to 44 and Figs. 46 to 49, the number of displayed performer avatars increases to two, which increases the processing load for drawing the performer avatars; the movement of the performer avatars becomes larger (more intense) with the start of the first performance period or the second performance period, which increases the processing load for updating the virtual space and the processing load for drawing the performer avatars; the light emission intensity of the light emission performance increases based on the start of the first performance period or the second performance period, which increases the processing load for drawing the virtual space and each avatar; and the number of viewer avatars virtually participating in the virtual live venue (the number of participants in the live performance) exceeds a threshold number, which increases the processing load for updating the virtual space and the processing load for viewing and listening. By changing the viewer avatars in the virtual live venue from full avatars to simplified avatars based on either the occurrence of an increase in the drawing processing load of the viewer avatars, or the number of moving viewer avatars (number of moving people) among the viewer avatars virtually participating in the virtual live venue exceeding a threshold value, causing an increase in the update processing load of the virtual space and the drawing processing load of the viewer avatars (number of people moving), (the establishment of a specific condition of the present invention), the viewer avatars in the virtual live venue are changed from full avatars to simplified avatars, thereby reducing the processing load on the viewer terminal 300 for displaying (drawing) these viewer avatars, thereby preventing an excessive increase in the processing load of the entire virtual space content distribution system due to the above-mentioned specific condition being established.

[0350] Furthermore, in this embodiment 3, time schedule data for managing the progress of the virtual live is stored in the storage 103 of the distribution server computer 100, and the distribution server computer 100 is able to identify the timing immediately before the processing load increases from the time schedule data using the avatar simplification determination process shown in FIG. 41, so that the timing at which the processing load increases can be accurately identified as timings T1 to T3 before the processing load actually increases.

[0351] In particular, in this embodiment 3, the viewer avatar is changed from a full avatar to a simple avatar from times T1 to T3 before the timing when the number of displayed performer avatars increases to two, causing an increase in the drawing processing load of the performer avatars, the timing when the update processing load of the virtual space and the drawing processing load of the performer avatars increase as the movements of the performer avatars become larger (more intense) with the start of the first performance period or the second performance period, and the timing when the drawing processing load of the virtual space and each avatar increases as the light emission intensity of the light emission effect increases based on the start of the first performance period or the second performance period.This prevents the processing load required for these processes from overlapping and causing an excessive increase in the processing load of the entire virtual space content distribution system.

[0352] In this embodiment 3, an example is given of a form in which time schedule data is stored in storage 103 of the distribution server computer 100, and the distribution server computer 100 changes the viewer avatar in the viewer terminal 300 from a full avatar to a simplified avatar based on the timings T1 to T3 identified. However, the present invention is not limited to this, and the time schedule data may be stored in storage 303 of the viewer terminal 300, and the viewer terminal 300 itself may identify the timings T1 to T3 from the time schedule data and change the viewer avatar from a full avatar to a simplified avatar based on the identification.

[0353] Furthermore, in this embodiment 3, an example is given of a form in which the viewer avatar is changed from a full avatar to a simplified avatar from timing T1 to T3 just before the drawing processing load increases, but the present invention is not limited to this, and if the viewer terminal 300 has sufficient drawing processing capacity, the viewer avatar may be changed from a full avatar to a simplified avatar at the timing when the drawing processing load increases.

[0354] Furthermore, as shown in FIG. 49, the viewer avatar of the viewer operating the viewer terminal 300 (viewer avatar 11 in the example of FIG. 49) is eventually changed from a full avatar to a simplified avatar and displayed in the virtual camera-specified image, thereby further preventing excessive increases in processing load.

[0355] Furthermore, in this embodiment 3, the viewer avatar of a viewer operating another viewer terminal 300 is given priority over the viewer's own viewer avatar (viewer avatar 11 in the example of Figure 49) operating the viewer terminal 300, and changes from a full avatar to a simplified avatar, thereby effectively preventing an excessive increase in processing load.

[0356] In this embodiment 3, the viewer avatars are divided into the viewer's own viewer avatar operating the viewer terminal 300 (viewer avatar 11 in the example of Figure 49), viewer avatars of viewers who have been registered as friends by the viewer (viewer avatars 12, 13, and 14 in the example of Figure 49), and viewer avatars of viewers who have not been registered as friends by the viewer, and the priority of changing from a full avatar to a simplified avatar is set to be the highest for the viewer avatar of a viewer who has not been registered as a friend, and the lowest for the viewer's own viewer avatar operating the viewer terminal 300 (viewer avatar 11 in the example of Figure 49) (priority of changing from a full avatar to a simplified avatar: viewer avatar of a viewer who has not been registered as a friend > viewer avatar of a viewer who has been registered as a friend > viewer's own viewer avatar who imagines the viewer terminal 300), but the present invention is not limited to this, and the priority of changing from a full avatar to a simplified avatar may be other than the above, and the priority of changing from a full avatar to a simplified avatar may not be set between viewer avatars.

[0357] 45 to 48, full avatars and simple avatars are provided for both the performer avatars 1A, 1B and the viewer avatars, but while the viewer avatar changes from a full avatar to a simple avatar as the drawing processing load of the viewer terminal 300 increases, only the full avatar is displayed as the performer avatars 1A, 1B regardless of the increase in the drawing processing load of the viewer terminal 300. In other words, during the virtual live performance, a simple avatar that requires a small drawing processing load to display as the performer avatars 1A, 1B is not displayed, so that a decrease in interest in the live performance due to the display of simple avatars as the performer avatars 1A, 1B can be prevented.

[0358] In this embodiment 3, an example is given in which a full avatar and a simplified avatar are provided as the performer avatars 1A and 1B, but the present invention is not limited to this. Since there is no opportunity to display simplified avatars as the performer avatars 1A and 1B in the first place, it is also possible to provide only full avatars as the performer avatars 1A and 1B and not provide simplified avatars.

[0359] In the third embodiment, a performer terminal 200 operable by a performer is provided as a device constituting the virtual space content delivery system, and the movements of the performer avatars 1A and 1B that immediately reflect the operations of the performer can be delivered as the virtual space content. However, the present invention is not limited to this, and the movements of the performer avatars 1A and 1B that have been photographed or created in advance may be delivered as the virtual space content.

[0360] In addition, in this embodiment 3, as shown in Figure 41, based on the number of viewer avatars installed in the virtual live venue or the number of viewer avatars moving around reaching a predetermined threshold, the distribution server computer 100 univocally determines that the drawing processing load on the viewer terminal 300 is approaching its limit, and changes the viewer avatar on the viewer terminal 300 from a full avatar to a simplified avatar. However, the present invention is not limited to this, and in reality, it is extremely unlikely that each viewer uses the same viewer terminal 300. Therefore, even if the distribution server computer 100 determines that the drawing processing load on the viewer terminal 300 is approaching its limit, if the viewer terminal 300 still has sufficient drawing processing capacity remaining to draw viewer avatars, performer avatars, etc., the viewer terminal 300 may continue to display the viewer avatar as a full avatar.

[0361] In addition, in this embodiment 3, an example is given of a form in which the distribution server computer 100 determines that the drawing load of the viewer terminal 300 will increase and changes the viewer avatar from a full avatar to a simplified avatar for the viewer terminal 300 when the number of viewer avatars virtually participating in the virtual live venue exceeds a threshold number, when the number of moving viewer avatars among the viewer avatars virtually participating in the virtual live venue exceeds a threshold number, when the light intensity of the light-emitting effects increases, when the movements of the performer avatars become larger (more intense), or when the number of performer avatars increases.However, the present invention is not limited to this, and the viewer terminal 300 itself may determine that the drawing processing load of the viewer terminal 300 will increase and change the viewer avatar from a full avatar to a simplified avatar when the number of viewer avatars virtually participating in the virtual live venue exceeds a threshold number, when the number of moving viewer avatars among the viewer avatars virtually participating in the virtual live venue exceeds a threshold number, when the light intensity of the light-emitting effects increases, when the movements of the performer avatars become larger (more intense), or when the number of performer avatars increases. In this case, the viewer watching the virtual live using the viewer terminal 300 may operate the viewer terminal 300 to set a threshold number for the number of viewer avatars virtually participating in the virtual live venue, and a threshold number for the number of viewer avatars moving around.

[0362] In addition, in the present embodiment 3, a mode is illustrated in which the viewer avatars of viewers who are not friends of the viewer user using the viewer terminal 300 are preferentially displayed as simplified avatars over the viewer avatars of viewers who are friends and have a strong relationship with the viewer user using the viewer terminal 300, but the present invention is not limited to this, and blood relatives such as parents and siblings of the viewer user using the viewer terminal 300, and related persons who belong to the same organization (for example, school, company, club, etc.) as the viewer user using the viewer terminal 300, etc. may be registered in advance in a similar procedure to that shown in Fig. 39, and the viewer avatars of viewers who are not blood relatives or related persons may be preferentially displayed as simplified avatars. In other words, the concept of "other viewer users who are highly related to each viewer user" in the present embodiment 3 may include blood relatives of the viewer user himself, related persons who belong to the same organization, etc. in addition to friends.

[0363] In addition, in this embodiment 3, an example was given of a form in which the viewer avatar is changed from a full avatar to a simplified avatar based on an increase in the processing load of the viewer terminal 300, but the present invention is not limited to this. A plurality of simplified avatars with different drawing loads may be provided as the viewer avatar, and when the processing load of the viewer terminal 300 increases, a simplified avatar according to the drawing capability of each viewer terminal 300 may be displayed as the viewer avatar.

[0364] <Modification of the third embodiment> The present invention has been described above based on each embodiment and modified examples, but the specific configuration is not limited to the above-mentioned embodiments and modified examples, and the present invention also includes modifications and additions that do not deviate from the gist of the present invention.

[0365] For example, in the above-mentioned Example 3, although both full avatars and simplified avatars are prepared for performer avatars 1A and 1B, a form was exemplified in which the simplified avatars are not displayed as performer avatars 1A and 1B. However, the present invention is not limited to this, and as a variant example 3-1, a case may be provided in which performer avatars 1A and 1B are displayed as simplified avatars, or a period may be provided in which each performer can select whether to display performer avatars 1A and 1B as a full avatar or a simplified avatar by operating the performer terminal 200.

[0366] 50, during periods other than the performance period, it is possible to select whether performer avatar 1A of performer A and performer avatar 1B of performer B are displayed as full avatars or simplified avatars by operation from each performer terminal 200. Also, during the performance period, if only performer avatar 1A of performer A sings or dances, performer avatar 1A of performer A may be displayed as a full avatar while performer avatar 1B of performer B may be displayed as a simplified avatar, thereby reducing the processing load for displaying performer avatar 1B of performer B on viewer terminal 300, and if only performer avatar 1B of performer B sings or dances during the performance period, performer avatar 1B of performer B may be displayed as a full avatar while performer avatar 1A of performer A may be displayed as a simplified avatar, thereby reducing the processing load for displaying performer avatar 1A of performer A on viewer terminal 300.

[0367] Furthermore, if the performance period is a performance period in which both performer avatar 1A of performer A and performer avatar 1B of performer B sing or dance, both performer avatar 1A of performer A and performer avatar 1B of performer B may be displayed.

[0368] In the third embodiment, the simplified avatar of the viewer avatar is provided in one mode regardless of whether the viewer sings or dances. However, the present invention is not limited to this. As shown in FIG. 51 as a modified example 3-2, the mode of the viewer's simplified avatar may be different depending on the performer avatar performing the performance. In particular, during non-performance periods such as the virtual live venue participation period, the opening performance period, and the MC period, the viewer avatar may be displayed in the mode shown in the third embodiment, while during each performance period, the viewer avatar may be displayed in the mode of the simplified avatar that is set in advance for each performer depending on the performer avatar performing the singing or dancing. In this way, the mode of the simplified avatar as the viewer avatar changes depending on the performer avatar, thereby improving the interest of the virtual live.

[0369] Furthermore, when varying the appearance of the viewer avatar depending on the performer, as shown in Figures 52 and 53, in the advance data transmission process, the distribution server computer 100 identifies performers for whom simple avatars have already been set for the viewer (performers for whom simple avatar data for the viewer is already stored in the storage 103 within the distribution server computer 100) from the advance data download request received from the viewer terminal 300 (step S601), then sets up the transmission of the viewer's simple avatar data for each identified performer (step S602), and sets up the transmission of other advance data from the received advance data download request (step S603), and transmits this set up advance data to the viewer terminal 300 (step S604).

[0370] In this way, the viewer terminal 300 can receive data of a simplified avatar for the viewer corresponding to each performer (performer avatar 1A, 1B) as part of the advance data from the distribution server computer 100, and can display a simplified avatar of a form corresponding to each performer avatar 1A, 1B as a viewer avatar during the performance period.

[0371] In addition, in this variant example 3-2, we have exemplified a form in which a simplified avatar of a type corresponding to each performer avatar 1A, 1B is displayed as the viewer avatar, but the present invention is not limited to this, and during the performance period of a performer who does not have a simplified avatar for the viewer set as pre-data (a performer for whom data of a simplified avatar for the viewer is not stored in advance in storage 103 in the distribution server computer 100), a simplified avatar of the same type as that during non-performance periods may be displayed as the viewer avatar, as shown in Figure 51.

[0372] In addition, in the present modified example 3-2, a simplified avatar in a form corresponding to each performer avatar 1A, 1B is displayed as a viewer avatar, but the present invention is not limited to this, and the simplified avatar displayed as the viewer avatar may be changed over time. For example, as shown in Fig. 54 as modified example 3-3, a heart-shaped character may be displayed as the viewer's simplified avatar from the start of the performance until a predetermined timing S, while a star-shaped character may be displayed as the viewer's simplified avatar after timing S.

[0373] Furthermore, in this variant example 3-2, an example has been given of a form in which a simplified avatar of a type corresponding to each performer avatar 1A, 1B is displayed as a viewer avatar, but the present invention is not limited to this. In cases where multiple virtual live venues (virtual spaces) in which viewer avatars can participate are provided and the same content is delivered to each virtual live venue, for example, it may be possible to display a different simplified avatar as the viewer avatar for each virtual live venue, as shown in Figure 55 as variant example 3-4.

[0374] In addition, in the above-mentioned Example 3, an example was given in which only one aspect of the simplified avatar is provided as the simplified avatar of the viewer avatar, but the present invention is not limited to this, and as a modified example 3-5, the simplified avatar of a viewer who is registered as a friend may have different aspects from the simplified avatar of a viewer who is not registered as a friend, as shown in Fig. 56. In this way, even when simplified avatars are displayed as viewer avatars, it is possible to distinguish between the viewer avatar of a viewer who is registered as a friend and the viewer avatar of a viewer who is not registered as a friend based on the aspect of each simplified avatar.

[0375] In addition, in the above-mentioned Example 3, an example was given of a form in which two types of avatars are provided for the viewer avatar and the performer avatars 1A, 1B, namely, a full avatar and a simplified avatar with a smaller drawing processing load than the full avatar. However, the present invention is not limited to this, and as a variant example 3-6, a plurality of simplified avatars with different drawing processing loads (for example, a first simple avatar with a smaller drawing processing load than the full avatar and a second simple avatar with a smaller drawing processing load than the first simple avatar) may be provided as the viewer avatar and the performer avatars 1A, 1B.

[0376] In this way, when multiple simple avatars with different drawing processing loads are provided, the state of the viewer avatar may be changed in stages from a full avatar, to a first simple avatar, to a second simple avatar, etc. so that the drawing processing load of the viewer terminal 300 decreases, under conditions such as the number of viewer avatars virtually participating in the virtual live venue exceeding a threshold number, the number of moving viewer avatars among the viewer avatars virtually participating in the virtual live venue exceeding a threshold number, or one of times T1 to T3 is reached.

[0377] In the third embodiment, the viewer avatar is changed from a full avatar to a simplified avatar on the condition that the number of moving viewer avatars among the viewer avatars virtually participating in the virtual live venue exceeds a threshold number, but the present invention is not limited to this, and as a modified example 3-7, the viewer avatar may be changed from a full avatar to a simplified avatar on the condition that the number of viewer avatars not moving but performing an action such as dancing or jumping among the viewer avatars virtually participating in the virtual live venue exceeds a threshold number. In other words, the "performer avatar operating in a virtual space" in the third embodiment may include performer avatars performing actions other than moving within the virtual live venue, in addition to performer avatars moving within the virtual live venue.

[0378] In addition, in the above-mentioned Example 3, an example was given of a form in which each viewer avatar is first displayed as a full avatar and then changed to a simplified avatar based on the satisfaction of certain conditions, but the present invention is not limited to this.As a variant example 3-8, if it is certain that the number of viewer avatars virtually participating in the virtual live venue will exceed a predetermined threshold number, each viewer avatar may be displayed as a simplified avatar from the beginning.

[0379] In the third embodiment, the number of displayed performer avatars increases to two, which increases the processing load for drawing the performer avatars; the movement of the performer avatars becomes larger (more intense) with the start of the first performance period or the second performance period, which increases the processing load for updating the virtual space and the processing load for drawing the performer avatars; the light emission intensity of the light emission effects increases based on the start of the first performance period or the second performance period, which increases the processing load for drawing the virtual space and each avatar; the number of viewer avatars virtually participating in the virtual live venue (the number of participants in the live performance) exceeds a threshold number, which increases the processing load for updating the virtual space and the processing load for drawing the viewer avatars; The specific condition in this invention is defined as the number of moving viewer avatars (number of people moving) exceeding a threshold value, thereby increasing the update processing load of the virtual space and the drawing processing load of the viewer avatar, and an example has been given in which the viewer avatar is changed from a full avatar to a simplified avatar based on the satisfaction of any of these specific conditions, but the present invention is not limited to this. As a variant example 3-9, the specific conditions in this invention may be only some of the above, or may include conditions other than those listed above as long as they are events that increase the processing load on the distribution server computer 100 and viewer terminal 300 that constitute the virtual space content distribution system of the present invention.

[0380] In the third embodiment, as shown in Figs. 41 to 43, the viewer avatar is changed from a full avatar to a simplified avatar using the same threshold value (threshold value of the number of participants and the threshold value of the number of moving viewer avatars) when generating a viewer viewpoint video and when generating a virtual camera viewpoint video, but the present invention is not limited to this. As a modified example 3-10, the viewer avatar may be changed from a full avatar to a simplified avatar using different threshold values ​​when generating a viewer viewpoint video and when generating a virtual camera viewpoint video. For example, as shown in Fig. 13, when generating a virtual camera C1 video, unlike when generating a virtual camera C2 video, a virtual camera C3 video, a virtual camera C4 video, and each viewer viewpoint video, almost only the performer avatar is drawn, so the threshold value of the number of participants and the threshold value of the number of moving viewer avatars are set high, while in the virtual camera C2 video, the virtual camera C3 video, the virtual camera C4 video, and each viewer viewpoint video, each viewer avatar is actively displayed in addition to the performer avatar, so the threshold value of the number of participants and the threshold value of the number of moving viewer avatars may be set high.

[0381] Furthermore, even in the case of viewer-perspective images, if the viewer avatar's line of sight is not directed toward virtual stage G, it is more difficult to render a performer avatar as viewer-perspective image than when the viewer avatar's line of sight is directed toward virtual stage G, and therefore the rendering process load caused by drawing the performer avatar is less likely to be high, so the threshold for the number of participants and the threshold for the number of moving viewer avatars may be set higher than when the viewer avatar's line of sight is directed toward virtual stage G.

[0382] In addition, in the above-mentioned Example 3, an example was given of a form in which, as the drawing processing load of the viewer terminal 300 increases, the viewer avatars are changed to simplified avatars in the following order: viewer avatars of viewers who are not registered as friends, viewer avatars of viewers who are registered as friends, and the viewer's own avatar. However, the present invention is not limited to this, and as a variant example 3-11, the avatars may be changed to simplified avatars in the order of viewer avatars that are farthest from the viewer's own avatar each time the drawing processing load of the viewer terminal 300 increases.

[0383] In addition, in the above-mentioned embodiments and modifications, the viewer avatar viewpoint video is generated and displayed on the viewer terminal 300 side, but the present invention is not limited to this, and for example, in the case of a live event with few virtual participants, the viewer avatar viewpoint video may also be generated and distributed on the distribution server computer 100 side. Also, the viewer user may be allowed to select whether the viewer avatar viewpoint video is generated on the viewer terminal 300 side or on the distribution server computer 100 side according to the processing capacity of the terminal he / she owns and the available data communication environment, or the distribution server computer 100 may specify the processing capacity of the viewer terminal 300 with which it is connected for communication, and determine whether the viewer avatar viewpoint video is generated on the viewer terminal 300 side or on the distribution server computer 100 side based on the specified processing capacity.

[0384] In addition, in each of the above embodiments and modified examples, an example is given of the distribution server computer 100 being installed in a studio operated by the event management organization that hosts the virtual live performance, but the present invention is not limited to this, and as the distribution server computer 100, a server computer owned in a data center or the like by the company that rents out the server computer may be used, or as described above, a server computer provided by a cloud service may be used, and the installation form of these server computers may be any form that provides the functions of the distribution server computer 100.

[0385] Furthermore, in each of the above embodiments and modified examples, a performer terminal 200 is provided for each performer user in a one-to-one correspondence, but the present invention is not limited to this, and multiple people may share one performer terminal 200. If the performer user is a unit or group consisting of multiple members, for example, a viewpoint tracking target designation process may be executed in at least one of the distribution server computer 100 and the viewer terminal 300, allowing the viewer user to select a member to follow the viewpoint.

[0386] In addition, while the above embodiments and modified examples illustrate a form in which a performer user sings, dances, and other performance actions are performed by one person, the present invention is not limited to this, and the singer who sings and the actor who performs the dancing actions may be separate persons, in which case the singer's performer terminal and the actor's performer terminal may be separate, and the singer may sing in a location other than the studio where the actor performs the performance actions.

[0387] In addition, in each of the above embodiments and variant examples, the Internet network is used as an example of the computer communication network connecting the distribution server computer 100, the administrator terminal 150, the performer terminal 200, and the viewer terminal 300, but the present invention is not limited to this, and these communication networks may be composed of local area networks capable of data communication only within a specific area, or may include a local area network and local data communication as part of them.

[0388] In addition, in each of the above embodiments and variant examples, the viewer terminal 300 is exemplified as a smartphone P or a computer (PC), but the present invention is not limited to this, and these computers may be portable laptops, tablet terminals, e-book readers, wearable computers, game consoles, and various other information processing devices capable of outputting at least 2D video and audio.

[0389] In addition, in each of the above embodiments and variant examples, a performer user performs in a studio, including singing, but the present invention is not limited to this. For example, a performer user may use a performer terminal 200 installed in a home or the like to appear in a virtual live performance from a home or the like.

[0390] Furthermore, in each of the above embodiments and variant examples, a virtual live performance has been given as an example of virtual space content, but the present invention is not limited to this, and such virtual space content may be any content that is beneficial to viewer users, such as a fan meeting where performer users gather with their fans, or a talk session, and that is related to a virtual space in which performer avatar 1 and viewer avatars virtually participate.

[0391] In addition, in each of the above embodiments and modified examples, an example is given in which viewpoint switching of the four virtual cameras C1 to C4 as fixedly positioned (set) virtual cameras or the airborne moving virtual camera can be performed in advance by settings and operations on the administrator terminal 150, but the present invention is not limited to this, and for example, these viewpoint switching may be performed in real time by the event manager M on the administrator terminal 150 using a viewpoint switching controller or the like constituting the operation input device 161.

[0392] Furthermore, in each of the above embodiments and modified examples, an example is given of the use of virtual cameras C1 to C4 fixedly positioned (set) in the virtual space as the virtual cameras, but the present invention is not limited to this. For example, in addition to or instead of the virtual cameras C1 to C4, a movable virtual moving camera may be placed in the virtual live venue, and the position and imaging direction of the virtual moving camera in the virtual space may be controlled by the event manager M, a cameraman, or the like through operation of the manager terminal 150 to produce the viewpoint image of the virtual moving camera.

[0393] In addition, in the above embodiments and modified examples, a form is exemplified in which the viewpoint image of the virtual camera C2 is displayed on the screen S so that the facial expression of the performer user can be detected to be reflected in the facial expression of the performer avatar, but the present invention is not limited to this, and for example, if the performer user has a high-performance head-mounted display with a sensor function to detect the facial expression of the performer user, the performer user may wear such a high-performance head-mounted display to perform the performance, which may improve the performer user's sense of immersion in the virtual live performance. Also, in cases where the facial expression of the performer user does not need to be reflected in detail in the facial expression of the performer avatar, the performer user may wear a normal head-mounted display to perform the performance.

[0394] Furthermore, although examples 1 to 3 and each of the modified examples have been given as examples of the virtual space content distribution system of the present invention, the present invention is not limited to these, and the virtual space content distribution system of the present invention may be implemented in such a manner that only any of the embodiments and modified examples are implemented, or a combination of multiple embodiments and modified examples is implemented.

[0395] In addition, instead of the performer user's singing or singing actions (performance), the singing and singing actions (performance actions) of a virtual performer created using computer graphics, etc., can be recorded as performer avatar information on a distribution server computer, and a virtual live performance can be held by playing back the recorded singing voice and singing actions (performance actions) as the live performance progresses.

[0396] In this way, when singing or singing actions (performance actions) are performed in advance to record and play back performer avatar information on the distribution server computer 100 to distribute a virtual live performance, the configuration of the virtual space content distribution system at the time of distribution will be made up of the distribution server computer 100, an administrator terminal 150, and a viewer terminal 300, and will not include a performer terminal 200, which is a performer user terminal that can be used by a performer user to perform operations to move the performer avatar 1 in the virtual space.

[0397] In addition, in claim 14 of the present invention, "available for use by the performer" does not mean that the performer user operates the virtual space, but also includes use by the performer user simply to view virtual space content. [Explanation of symbols]

[0398] 1. Performer Avatar 100 Distribution server computer 150 Administrator terminal 200 Performer terminals 300 Viewer terminals

Claims

1. A virtual space content distribution system capable of distributing virtual space content in a virtual space in which an avatar can participate, comprising: a viewer user terminal capable of outputting an image of the virtual space content, the viewer user being able to view the image; a server computer connected to the viewer user terminal via a communication network and capable of executing at least processes related to the virtual space and processes related to the distribution of the virtual space content; At least the server computer stores timeline data capable of identifying a specific timing corresponding to a specific control for an avatar participating in the virtual space content in the virtual space content; When a specific condition corresponding to the specific control is satisfied at the specific timing specified based on the timeline data, the specific control is executed. A virtual space content delivery system comprising:

2. A virtual space content distribution program for distributing virtual space content in a virtual space in which an avatar can participate, comprising: The virtual space content is a content including a video that can be viewed by a viewer user on a viewer user terminal that is connected to a server computer via a communication network and that is capable of executing at least a process related to the virtual space and a process related to the distribution of the virtual space content and that is assigned to the viewer user, a specific timing identification process for identifying a specific timing based on timeline data that is data stored in the server computer and that can identify a specific timing corresponding to a specific control related to an avatar participating in the virtual space content in the virtual space content; and a specific control process for executing the specific control when a specific condition corresponding to the specific control is satisfied at the specific timing specified by the specific timing specification process. A virtual space content delivery program comprising:

3. A virtual space content distribution method for distributing virtual space content in a virtual space in which an avatar can participate, comprising: The virtual space content is a content including a video that can be viewed by a viewer user on a viewer user terminal that is connected to a server computer via a communication network and that is capable of executing at least a process related to the virtual space and a process related to the distribution of the virtual space content and that is assigned to the viewer user, a specific timing identifying step of identifying a specific timing based on timeline data that is data stored in the server computer and that can identify a specific timing corresponding to a specific control related to an avatar participating in the virtual space content in the virtual space content; and a specific control procedure for executing the specific control when a specific condition corresponding to the specific control is satisfied at the specific timing specified in the specific timing specifying procedure. A virtual space content distribution method comprising:

Images