Virtual space content delivery system, virtual space content delivery program, and virtual space content delivery method

Abstract

To provide a virtual space content delivery system allowing a viewpoint of a viewer user to respond to operation of a performer user.SOLUTION: A virtual space content delivery system for delivering a virtual space content by a performer avatar, includes a viewer user terminal capable of outputting, as the virtual space content, viewpoint videos from a plurality of viewer viewpoint positions including an avatar viewpoint corresponding to a viewer avatar in a virtual space. When a performer avatar performs a general operation, the viewer user terminal defines the viewer viewpoint position as a first viewer viewpoint position where the performer avatar performing the general operation can be viewed, and when the performer avatar performs a special operation, changes the viewer viewpoint position to follow the special operation.SELECTED DRAWING: Figure 20

Description

[Technical field]

[0001] The present invention relates to a virtual space content distribution system, a virtual space content distribution program, and a virtual space content distribution method capable of distributing virtual space content using a performer avatar. [Background technology]

[0002] Conventionally, a distribution system is known that generates an animation of a character object (avatar) of a distribution user (performer) that moves in a virtual space in conjunction with the movements of the distribution user (performer), live-distributes a video including the animation of the character object (avatar), and also allows viewing users to virtually participate as spectators in the virtual space in which the avatar of the distribution user (performer) is placed, upon request from the viewing users (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-120098 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, there is a problem that the viewpoint from which a viewing user who virtually participates in a virtual space views may not be able to correspond to the movement of a distribution user (performer).

[0005] The present invention has been made in response to such problems, and aims to provide a virtual space content distribution system, a virtual space content distribution program, and a virtual space content distribution method that can match the viewing viewpoint of the viewing user to the actions of the distribution user (performer). [Means for solving the problem]

[0006] The virtual space content delivery system of claim 1 comprises: A virtual space content delivery system capable of delivering virtual space content (e.g., virtual space live) using a performer avatar, A viewer user terminal (e.g., the viewer terminal 300) that can be used by a viewer user who performs operations to move a viewer avatar in a virtual space (e.g., a virtual live performance venue) that is a target of the virtual space content; A server computer (e.g., a delivery server computer 100) that is connected to the viewer user terminal via a network and is capable of executing at least processes related to the virtual space and processes related to the delivery of the virtual space content; Including, the viewer user terminal is capable of outputting, as the virtual space content, viewpoint images from a plurality of viewer viewpoint positions including an avatar viewpoint corresponding to the viewer avatar in the virtual space (for example, a portion where the viewer terminal 300 is capable of displaying a viewpoint image of a viewer avatar 28 as shown in FIG. 13(e)); When the performer avatar is performing a special action (for example, a virtual flying action as shown in FIG. 19 or a quick lateral movement as shown in FIG. 17), a viewer viewpoint position changing means (for example, a part of the viewer terminal that executes the viewer viewpoint video control process as shown in FIG. 20) is provided for changing the viewer viewpoint position so as to follow the special action. It is characterized by the following. According to this feature, the viewpoint from which the viewer watches can be made to correspond to the movements of the performer.

[0007] The virtual space content delivery system of claim 2 is the virtual space content delivery system of claim 1, The viewer user terminal (e.g., viewer terminal 300) is capable of receiving a viewpoint state change operation (e.g., a viewpoint-related operation including a zoom operation, etc.) for changing the state of the viewer viewpoint from the viewer user, and is capable of outputting a viewpoint video corresponding to the received viewpoint state change operation (e.g., as shown in FIG. 16 , a portion capable of outputting a viewpoint video corresponding to a zoom operation); The viewer viewpoint position changing means is capable of changing the viewer viewpoint position without receiving the viewpoint state change operation (for example, as shown in FIG. 20, in step S13, the viewer user changes to a visiting tracking viewpoint image without any operation). It is characterized by the following. According to this feature, the viewer user can successively change the state of the viewer's viewpoint, which significantly improves the viewer's sense that he or she is participating in the virtual space.

[0008] The virtual space content delivery system of claim 3 is the virtual space content delivery system according to claim 2, The viewer viewpoint position changing means invalidates the viewpoint state changing operation when the viewer viewpoint position is made to follow the special action (for example, as shown in FIG. 20, a part for invalidating viewpoint-related operations in step S12). It is characterized by the following. According to this feature, the viewer's viewpoint position can be made to accurately follow the special action.

[0009] The virtual space content delivery system of claim 4 is the virtual space content delivery system according to any one of claims 1 to 3, The special action is a movement action (e.g., a virtual flight) of moving within the virtual space for a predetermined period (e.g., a flying performance period), The viewer viewpoint position moves in the virtual space in conjunction with the continuous movement action (for example, a portion where an airborne moving virtual camera moves on an airborne trajectory that is set in advance in a second area during a flying performance period). It is characterized by the following. According to this feature, viewpoint images corresponding to continuous movement actions can be distributed as virtual space content.

[0010] The virtual space content delivery system of claim 5 is the virtual space content delivery system according to claim 1 or 2, The viewer avatar is movable within the virtual space based on the operation of the viewer user (for example, the viewer avatar can move within the third area and the fourth area by a movement operation on the viewer terminal 300). It is characterized by the following. This feature can improve the sense of reality of the virtual space.

[0011] The virtual space content delivery system of claim 6 is the virtual space content delivery system according to claim 1 or 2, The viewer users are multiple (for example, as shown in FIG. 1, the viewer users include viewer users A to C), The viewer users can place their viewer avatars at specific positions in the virtual space that are assigned to each viewer user in advance (for example, as shown in the modified examples of FIG. 21 and FIG. 22, a viewer user can virtually participate in a live broadcast by placing a viewer avatar in an area selected before the live broadcast begins). It is characterized by the following. This feature makes it possible to prevent inconvenience caused by the viewpoints of different viewer users overlapping at the same position.

[0012] The virtual space content delivery system of claim 7 is the virtual space content delivery system according to claim 6, The specific position is determined by a specific position determination operation by a viewer user before the distribution of the virtual space content starts (for example, as shown in FIG. 22, in a modified example, a portion where an area in which a viewer avatar will virtually participate can be selected before the start of a live broadcast). It is characterized by the following. According to this feature, it is possible to prevent inconvenience caused by the specific position being determined after distribution of the virtual space content has started.

[0013] The virtual space content delivery system of claim 8 is the virtual space content delivery system according to claim 6, The tracking mode of the viewer viewpoint position when the performer avatar is performing the special action (e.g., virtual flight) differs depending on the specific position (e.g., in the modified example shown in FIG. 22, the flying image with the viewpoint corresponding to the position of the selected area is changed to a right front viewpoint for viewer users who have selected A1 to C6, and a left front viewpoint for viewer users who have selected I1 to K6). It is characterized by the following. This feature can improve the sense of reality of the virtual space.

[0014] The virtual space content delivery system of claim 9 is the virtual space content delivery system according to claim 6, The viewer user can view, on the viewer user terminal, a viewpoint image including the viewer avatar of another person different from the viewer avatar corresponding to the viewer user (for example, as shown in FIG. 13(e) and FIG. 13(f), a portion in which a viewpoint image including another viewer avatar virtually participating in the third area is displayed). It is characterized by the following. This feature can improve the sense of reality of the virtual space.

[0015] The virtual space content delivery system of claim 10 is the virtual space content delivery system according to any one of claims 1 to 3, When the performer avatar is performing a specific action other than the special action (for example, as shown in FIG. 17, a movement of suddenly moving to the right on the virtual stage G), the viewer viewpoint position changing means changes the viewer viewpoint position in a tracking manner different from that in the case of the special action (for example, as shown in FIG. 17(b), a portion in which the viewpoint is changed so as to follow the performer avatar 1 laterally without the viewer user's operation so that the face (head) of the performer avatar does not move outside the display area). It is characterized by the following. This feature allows for tracking of the performer's gaze in accordance with their movements.

[0016] The virtual space content delivery program according to claim 11, A virtual space content distribution program (e.g., each program stored in storage 103 of distribution server computer 100 and storage 303 of viewer terminal 300) for distributing virtual space content (e.g., a virtual space live) by performer avatars over a network (e.g., the Internet N) using at least a server computer (e.g., distribution server computer 100), The avatars operating in the virtual space (e.g., a virtual live concert venue) that is the subject of the virtual space content include the performer avatar and viewer avatars (e.g., viewer avatars 11 to 14 and viewer avatars 21 to 32) that operate in the virtual space by operating a viewer user terminal (e.g., viewer terminal 300) that is a terminal available to a viewer user and connected to the server computer via a network, a virtual space content output step of outputting, as the virtual space content, viewpoint images from a plurality of viewer viewpoint positions including at least an avatar viewpoint corresponding to the viewer avatar in the virtual space on the viewer user terminal (for example, a portion in which a viewer avatar viewpoint image and a flying viewpoint image are displayed by executing step S11 and step S22 in the viewer viewpoint image control process shown in FIG. 20); a viewer viewpoint position changing step (e.g., step S22 in the viewer viewpoint video control process in FIG. 20 performed by the viewer terminal) of changing the viewer viewpoint position so as to follow the special action of the performer avatar (e.g., the virtual flying action shown in FIG. 19 or the rapid lateral movement shown in FIG. 17); Includes It is characterized by the following. According to this feature, the viewpoint from which the viewer watches can be made to correspond to the movements of the performer.

[0017] The virtual space content delivery method according to claim 12 comprises: A virtual space content distribution method for distributing virtual space content (e.g., a virtual space live) by a performer avatar via a network (e.g., the Internet N) using at least a server computer (e.g., a distribution server computer 100), comprising: The avatars operating in the virtual space (e.g., a virtual live concert venue) that is the subject of the virtual space content include the performer avatar and viewer avatars (e.g., viewer avatars 11 to 14 and viewer avatars 21 to 32) that operate in the virtual space by operating a viewer user terminal (e.g., viewer terminal 300) that is a terminal available to a viewer user and connected to the server computer via a network, A procedure for outputting, as the virtual space content, viewpoint images from a plurality of viewer viewpoint positions including at least an avatar viewpoint corresponding to the viewer avatar in the virtual space in the viewer user terminal (for example, the procedures of steps S11 and S22 for outputting the viewpoint image of the viewer avatar and the flying viewpoint image in the viewer viewpoint image control process shown in FIG. 20 ); a viewer viewpoint position change procedure for changing the viewer viewpoint position when the performer avatar is performing a special action so as to follow the special action (for example, the procedure of step S22 in the viewer viewpoint video control process of FIG. 20 performed by the viewer terminal); Includes It is characterized by the following. According to this feature, the viewpoint from which the viewer watches can be made to correspond to the movements of the performer. Furthermore, the present invention may have only the invention-specific matters described in the claims of the present invention, or may have the invention-specific matters described in the claims of the present invention as well as configurations other than the invention-specific matters. [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram showing an example of a system configuration of a virtual space content delivery system according to an embodiment of the present invention. [Diagram 2]1 is a diagram illustrating an example of a configuration of a delivery server computer in a virtual space content delivery system according to an embodiment of the present invention. [Diagram 3] 2 is a diagram showing various data stored in a storage device of a delivery server computer according to an embodiment of the present invention; FIG. [Figure 4] 2 is a diagram illustrating an example of the configuration of a performer terminal in the virtual space content distribution system according to the embodiment of the present invention. FIG. [Diagram 5] 2 is a diagram illustrating an example of the configuration of a viewer terminal in the virtual space content delivery system according to the embodiment of the present invention. [Figure 6] 2 is a diagram illustrating an example of the configuration of an administrator terminal in the virtual space content delivery system according to the embodiment of the present invention. FIG. [Figure 7] 3 is a diagram showing an example of a configuration of event data in the virtual space content delivery system according to the embodiment of the present invention; FIG. [Figure 8] 2 is a diagram showing an example of the configuration of performer user data in the virtual space content distribution system according to the embodiment of the present invention. FIG. [Figure 9] 2 is a diagram showing an example of the configuration of viewer user data in the virtual space content delivery system according to the embodiment of the present invention. FIG. [Figure 10] 2 is an explanatory diagram showing data exchanged between each terminal and a server computer constituting the virtual space content distribution system according to the embodiment of the present invention; FIG. [Figure 11] 1 is a schematic diagram showing studio facilities used by performer users in a virtual space content delivery system according to an embodiment of the present invention. FIG. [Figure 12] 1 is a diagram showing a virtual live venue, which is a virtual space used in a virtual space content distribution system according to an embodiment of the present invention. FIG. [Figure 13] 1A to 1C are diagrams illustrating examples of displays on a viewer terminal constituting a virtual space content delivery system according to an embodiment of the present invention. [Figure 14] 1A to 1C are diagrams illustrating examples of displays on a viewer terminal constituting a virtual space content delivery system according to an embodiment of the present invention. [Figure 15]1A to 1C are diagrams illustrating examples of displays on a performer terminal constituting a virtual space content distribution system according to an embodiment of the present invention. [Figure 16] 1A to 1C are diagrams illustrating an example of a display by a zoom operation on a viewer terminal constituting the virtual space content delivery system according to an embodiment of the present invention. [Figure 17] 1A to 1C are diagrams illustrating an example of a display by a zoom operation on a viewer terminal constituting the virtual space content delivery system according to an embodiment of the present invention. [Figure 18] 1A to 1C are diagrams illustrating examples of displays on a viewer terminal constituting a virtual space content delivery system according to an embodiment of the present invention. [Figure 19] A figure showing an example of the display of a flying performance period on a viewer terminal constituting a virtual space content distribution system according to an embodiment of the present invention. [Figure 20] 11 is a flow diagram showing an example of a viewer viewpoint video control process executed in a viewer terminal constituting the virtual space content delivery system according to an embodiment of the present invention. FIG. [Figure 21] FIG. 13 is a diagram showing a virtual live concert venue in a modified example. [Figure 22] FIG. 13 shows a participation position selection screen for a virtual live concert venue in a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] The following describes an embodiment of the virtual space content distribution system of the present invention with reference to the drawings. The same or similar components in the drawings are given the same reference numerals, and redundant explanations are omitted. The virtual space content distribution program of the present invention may be a program used in the entire system consisting of a server computer and each terminal described later that constitute the virtual space content distribution system, or may be a program that operates only on a part of the devices that constitute the virtual space content distribution system, such as a server computer or each terminal. EXAMPLES

[0020] FIG. 1 is a block diagram showing an example of the system configuration of a virtual space content distribution system in an embodiment which is one embodiment of the present invention. In the system of this embodiment shown in FIG. 1, a virtual space live performance in which a performer avatar 1 corresponding to a performer user performs a live event in a virtual live venue shown in FIG. 12 is distributed as virtual space content.

[0021] As shown in FIG. 1, the virtual space content distribution system of this embodiment is mainly composed of a distribution server computer 100 capable of executing various processes related to the virtual live venue shown in FIG. 11, which is a virtual space, and various processes related to the distribution of a virtual space live event, an administrator terminal 150 used by an event manager K of an event management organization that operates and manages the virtual space live event to manage the distribution of the virtual space live event, a performer terminal 200 used by a performer user, and a viewer terminal 300 that can be used by a viewer user who virtually participates in and watches the virtual space live event, and the distribution server computer 100, the administrator terminal 150, the performer terminal 200 and the viewer terminal 300 are connected to each other so that they can communicate data with each other via the Internet N, which is an open computer network.

[0022] In this embodiment, the administrator terminal 150 and the performer terminal 200 are also shown as being connected to the distribution server computer 100 via the Internet network N, but the present invention is not limited to this. For example, if the studio shown in FIG. 11 is a studio operated by an event management organization which provides the studio to a performer user, and the distribution server computer 100 is installed in this studio, the administrator terminal 150 and the performer terminal 200 may be locally connected to the distribution server computer 100 via data communication, or may be connected via a local area network LAN within the studio facility.

[0023] 1, there are multiple viewer users, such as viewer user A, viewer user B, viewer user C, etc., and viewer terminals 300 include viewer terminals 300a, 300b, 300c, etc. that can be used by each of viewer users A, B, C, etc. Note that there are usually multiple viewer users, but there may be one or four or more, and the number may be appropriately determined depending on the virtual space, such as a virtual live venue. Hereinafter, viewer user A, viewer user B, viewer user C, etc. may be collectively referred to as viewer users, and viewer terminals 300a, 300b, 300c, etc. may be collectively referred to as viewer terminals 300.

[0024] Also, in this embodiment, an example is given of a form in which there is one performer user, but the present invention is not limited to this, and the virtual space live event may be a collaborative event in which multiple performer users working individually virtually participate, or a group event in which a group of multiple people working together virtually participate.In this way, when multiple performers are appearing, performer terminal 200a, performer terminal 200b, performer terminal 200c... may be provided for each performer user, or if a group of multiple people are performing, one performer terminal 200 may be used by multiple people.

[0025] In addition, although FIG. 1 illustrates the distribution server computer 100 as a single unit, the present invention is not limited to this. These distribution server computers 100 may be configured with multiple server computers, such as a first server computer that mainly performs virtual space processing and a second server computer that mainly performs distribution processing, or may be configured as a cloud server, etc., so that the number of server computers can be changed appropriately depending on the scale of the event, the number of participating listener users, etc.

[0026] Similarly, although the administrator terminal 150 is illustrated as a single unit in FIG. 1, the present invention is not limited to this, and it goes without saying that there may be multiple administrator terminals 150 depending on the number of performer users and viewer users virtually participating in the live performance.

[0027] <Distribution server computer> Fig. 2 is a diagram showing the configuration of a distribution server computer 100 used in this embodiment. As described above, the distribution server computer 100 used in this embodiment is a normal server computer with relatively excellent processing power installed in a studio operated by an event management organization, and as shown in Fig. 2, it has a processor (CPU) 101 connected to a data bus 108, a memory (RAM) 102, a storage 103 such as a hard disk, a communication interface (I / F) 104 capable of two-way communication by TCP / IP protocol via the Internet, which is an open computer network, and an input / output interface (I / F) 105 to which an input device such as a keyboard can be connected.

[0028] In this embodiment, an input / output interface (I / F) 105 is provided to enable local operation, but if remote operation is possible via a communication interface (I / F) 104, the input / output interface (I / F) 105 may not be provided.

[0029] In addition to an operation system (server OS) for providing server functions (not shown), storage 103 of distribution server computer 100 stores various data and programs as shown in Fig. 3. Specifically, mainly stored are an event management program 110 for providing an event management function for managing live events, a virtual space generation program 120 for providing a virtual space of a virtual live venue, which is a virtual space, a performer user management program 130 for providing a performer user management function for managing performer users, and a viewer user management program 132 for providing a viewer user management function for managing viewer users.

[0030] The event management program 110 is a program for managing a virtual space live event, which is an event held at a virtual live venue (see FIG. 12), in cooperation with the administrator terminal 150, and is capable of managing the progress of the virtual space live event by using event data 111 (see FIG. 7), time schedule data (TS data) 112, and music data 113 stored in the storage 103 together with the event management program 110.

[0031] An example of the event data 111 used in this embodiment is shown in Fig. 7. The event data 111 stores information such as an event name, a file name of time schedule (TS) data, a performer ID uniquely assigned to a performer user who is a performer, a file name of a music list, a file name of a participating viewer list in which information of special viewer users who are permitted to participate virtually is registered in a third area of ​​the virtual audience area described later, and a file name of a participating viewer list in which information of viewer users who have reserved virtual participation is registered in a fourth area of ​​the virtual audience area described later, in association with an event ID uniquely assigned to the event.

[0032] The time schedule data (TS data) 112 is data in which the order of progress of the virtual space live event to be held and the programs such as songs are described in chronological order, and the virtual space live event is managed so that it progresses based on the time schedule described in the time schedule data (TS data) 112. Moreover, the time schedule data (TS data) 112, event data 111, etc. can be changed, updated, or added by the event manager K operating the manager terminal 150.

[0033] Furthermore, the song data 113 is data for the performance parts (so-called karaoke) of each song sung by the performer user, and is played by the audio control program 124 (described later) based on the time schedule data (TS data) 112 by the event management program 110, and transmitted to the performer terminal 200 and the viewer terminal 300 (see FIG. 11).

[0034] The virtual space generation program 120 is a program for providing a virtual live venue, which is the virtual space shown in Figure 12, and has the function of providing a virtual live venue in which performer avatars and viewer avatars are virtually participating, using virtual space data 121 describing the structure of the virtual live venue, which is stored in storage 103 together with the virtual space generation program 120, object data 122 for placing various objects (including star decorations 43 on the stage and star objects 50-52 in the sky) to be placed in the virtual live venue within the virtual space, and performer avatar data contained in performer user data 131 described below and viewer avatar data contained in viewer user data 133.

[0035] 12, viewpoint image control program 123 is a program for providing a function to generate viewpoint images from virtual cameras C1-C4 virtually set in the virtual live venue and an airborne virtual camera (not shown), and also to provide a function to display images from the viewpoints of viewer avatars 11-14 virtually participating in the third area of ​​the virtual live venue and viewer avatars 21-32 virtually participating in the fourth area on performer terminal 200 in cooperation with viewer terminal 300. Virtual camera C2 is a virtual camera set to provide an image from the viewpoint of performer avatar 1 on the stage of the virtual live venue, and the viewpoint image from virtual camera C2 is projected onto a screen S installed in the studio by projector 211 (described later) connected to viewer terminal 300.

[0036] The audio control program 124 is a program for controlling the audio (including music) distributed during a virtual space live event, and specifically provides a function for playing music data 113 and a function for synthesizing audio produced by a performer user with the played music and distributing it to the administrator terminal 150 and the viewer terminal 300.

[0037] The comment control program 125, in cooperation with a viewer comment control program stored in the viewer terminal 300 described below, provides a function for displaying various comments entered by each viewer user in the viewer terminal 300 during a virtual space live event in chronological order on the performer terminal 200 and the viewer terminal 300.

[0038] The gift control program 126 is a program for controlling gifts given from viewer users to performer users in cooperation with a viewer program stored in the viewer terminal 300 described below, and specifically provides a function for the viewer user to purchase gifts based on information on the unit purchase price set for various gifts stored in the gift data 134, a processing function for the viewer user to give a gift purchased using the gift object data included in the gift data 134 to the performer user (including processing for the presentation when giving the gift), and a processing function for changing the owner of a gift given from the viewer user to the performer user.

[0039] The performer user management program 130 provides a management function for information relating to performer users, an authentication function, and the like, based on performer user data 131 shown in FIG. 8, which is stored in the storage 103 together with the performer user management program 130 .

[0040] The performer user data 131 used in this embodiment is illustrated in Fig. 8. The performer user data 131 stores various information, such as the performer user's account (email address), name, authentication information, the file name of the avatar data of the avatar used in the virtual space, and the file name of the item list in which the items owned are registered, in association with a performer user ID uniquely assigned to the performer user. Although not shown in Fig. 8, information on virtual value such as points owned by each performer user may also be stored as the performer user data 131.

[0041] Performer users can be authenticated by matching their performer user ID, account, and authentication information, and each performer user can identify from an item list the items (gifts) that they own as a result of being given to them by viewers.

[0042] The viewer / user management program 132 provides functions such as management of information relating to viewer / users and authentication based on viewer / user data 133 shown in FIG. 9 stored in the storage 103 together with the program.

[0043] The viewer user data 133 used in this embodiment is illustrated in Fig. 9. The viewer user data 133 stores various information, such as the viewer user's account (email address), authentication information, the file name of the avatar data of the avatar to be used in the virtual space, owned points which are the number of points that can be used in the virtual space, the file name of an item list in which items (gifts) purchased using points are registered, and personal information such as name, date of birth, and telephone number, in association with a viewer user ID uniquely assigned to the viewer user. Although not shown in Fig. 9, a nickname (comment name) to be displayed together with a comment is also registered in the viewer user data 133, and the nickname (comment name) is displayed together with the comment. In addition, points can be increased by purchasing points, for example, from a predetermined operating company.

[0044] <Performer terminal> Fig. 4 is a diagram showing an example of the configuration of a performer terminal 200 in the virtual space content delivery system of this embodiment. In this embodiment, as shown in Fig. 11, the performer terminal 200 is installed in a control room adjacent to a studio where a performer user performs his / her acting motion, and uses a normal computer with relatively excellent processing power, and as shown in Fig. 4, has a processor (CPU) 201 connected to a data bus 208, a memory (RAM) 202, a storage 203 such as a hard disk, a communication interface (I / F) 204 capable of two-way communication by the TCP / IP protocol via the Internet, which is an open computer network, an image processing unit 206 including a graphics processing unit (GPU) to which display devices A210 to C212 are connected, and an input / output interface (I / F) 205 to which various input / output devices are connected.

[0045] The input / output interface (I / F) 205 is connected with input devices such as a motion sensor 220 including multiple wearable sensors 220C1-220C5 (see FIG. 11) worn by the performer user on the left and right limbs and waist, a facial expression input device 221 consisting of an imaging camera for inputting the performer's facial expressions, a voice input device 222 consisting of a sound collecting microphone worn on the performer user's head, and an operation input device 224 such as a keyboard or touch panel operable by an assistant operator O who assists the performer user. Note that by providing a controller that can be held by the performer user to perform various operations as these input devices, the performer himself may be able to perform various operations without the assistance of the assistant operator O.

[0046] In this embodiment, since the performer user performs performance actions (music actions) in accordance with the music, an assistant operator O assists with operations during the performance. However, it is also possible to have multiple assistant operators O, or to set the operation content in advance in the performer terminal 200, for example, in a sequence program, thereby eliminating the need for an assistant operator O.

[0047] An audio output device 223 including high-performance earphones (in-ear monitors) worn by the performer user and speakers placed in the control room is connected to the input / output interface (I / F) 205 as an output device, so that the sound of the played music transmitted from the distribution server computer 100 is output to the performer user via the high-performance earphones (in-ear monitors), and the music audio is output from the speakers along with the performer's voice, allowing an assistant operator O, etc. to check the status of the music, including the voice produced by the performer user, in the control room.

[0048] In this embodiment, high-performance earphones (in-ear monitors) are used to avoid the inconvenience caused by the played music audio being input from the sound collection microphone worn by the performer user, but the present invention is not limited to this, and the music audio may be output from speakers in the studio, etc., as long as the inconvenience caused by the input of the music audio can be avoided.

[0049] In this embodiment, the sound collecting microphones are worn by the performers, but these sound collecting microphones may be fixed to the floor, wall, or ceiling of the studio.

[0050] Here, the motion sensor 220 used in the performer terminal 300 of this embodiment will be described. Any motion sensor can be used as long as it can properly detect (measure) the body movements (movements) of the performer user. In this embodiment, in order to enable more accurate and shorter periodic detection of the performer's movements, a number of wearable sensors 220C1 to 220C5 are used that are worn by the performer user on the body. Note that the motion sensor 220 may be of a type that does not require the performer to wear any equipment, such as LiDAR that uses laser light or the like.

[0051] In this embodiment, in order to reduce the burden on the performer user's head, particularly the burden on the performer user's head, caused by wearing sensors when performing movements, as described below, the performer user's head movements are detected by image recognition using images captured by the imaging camera constituting the facial expression input device 221, and movements other than the head are detected using five wearable sensors 220C1 to 220C5. However, for example, wearable sensors may also be provided on the performer user's head, or more wearable sensors (seven or more) may be worn to detect more detailed movements.

[0052] As shown in FIG. 11, the mounting sensors 220C1 to 220C5 detect their own positions and orientations in cooperation with base stations 220a and 220b installed in a studio room separated from the adjacent control room by a glass window.

[0053] Base station 220a and base station 220b may use, for example, a multi-axis laser emitter. Base station 220a may emit a flashing light for synchronization, and then scan the laser light around, for example, a vertical axis. Base station 220b may scan the laser light around, for example, a horizontal axis.

[0054] Each of the mounting sensors 220C1 to 220C5 may include a plurality of optical sensors that detect the incidence of blinking light and laser light from the base station 220a and the base station 220b.

[0055] Each of the wearing sensors 220C1 to 220C5 is capable of detecting its own position and orientation based on the time difference between the timing of the incidence of the flashing light and the timing of the incidence of the laser light, the light reception time at each optical sensor, the incidence angle of the laser light detected by each optical sensor, and other information as necessary. For example, the ViveTracker or base station provided by HTC CORPORATION can be suitably used.

[0056] Because base station 220a and base station 220b emit blinking light and perform laser light scanning at regular intervals, the detection information of each of worn sensors 220C1-220C5 is updated at each interval. Detection information indicating the position and orientation of each motion sensor calculated by each of these worn sensors 220C1-220C5 is input to performer terminal 200 by short-range wireless communication and transmitted to distribution server computer 100 as performer avatar information together with facial motion information (see FIG. 10).

[0057] In this embodiment, an example is shown in which two base stations, base station 220a and base base station 220b, but the present invention is not limited to this, and the number of base stations may be three or more.

[0058] In addition, the motion sensor 220 may be configured such that each of multiple motion sensors worn by the performer user is equipped with a number of infrared LEDs or visible light LEDs, and the position and orientation of each of the motion sensors is detected by detecting the light from these LEDs with an infrared camera installed on the floor or wall of the studio.

[0059] Also, in order to reduce the burden on the performer user, the motion sensor 220 may be replaced with a motion sensor using a lighter reflective marker instead of the wearable sensors 220C1-220C5. That is, the motion sensor may detect the motion of the performer user by capturing images of reflective markers attached to various parts of the performer user's body with adhesive tape or the like to generate image data, and then processing the captured image data to detect the positions and orientations of the reflective markers. In this case, a cameraman who captures the performer user may be placed in the studio to capture the performer user's motion from a direction suitable for detection, so that the motion of the performer user can be detected well.

[0060] In addition, the motion sensor 220 may be a suit with a built-in inertial sensor, such as a suit with a built-in inertial sensor for the MVN motion capture system marketed by Xsens.com, and the movements of the performer user may be detected by analyzing the sensor signal output from the inertial sensor.

[0061] Any facial expression input device 221 can be used as long as it can appropriately detect the movement of the head including the face of the performer user. In this embodiment, specifically, it is configured with an imaging camera (digital camera) arranged so as to be able to continuously capture head images including the face of the performer user, and the head movement and facial movement (facial expression) of the performer user captured by the imaging camera are detected by image recognition, and the facial motion information, which is the detected head movement and facial movement (facial expression), is transmitted to the delivery server computer 100 as performer avatar information together with body motion information, as shown in Fig. 10.

[0062] The facial expression input device 221 is not limited to the imaging camera (digital camera) used in this embodiment, and may be, for example, a 3D camera capable of detecting the depth of a person's face, or may be a mobile terminal such as a smartphone equipped with a LiDER device. In this case, the performer user may wear such a mobile terminal.

[0063] As described above, the performer terminal 200 of this embodiment has three display devices A210 to C212. Display device A210 is an LCD monitor or the like installed in a control room as shown in FIG. 11, display device B211 is a projector that projects images onto a screen S installed in the studio, and display device C212 is a large vertical display installed adjacent to the screen S in the studio.

[0064] Display device A210, which is placed in the adjustment room of the studio, displays images from each viewpoint of virtual cameras C1 to C4, viewpoint images from a virtual camera moving in the air during the flying performance period described below, the contents of comments, and a message input window into which assistant operator O inputs any message he or she wishes to convey to the performer user.

[0065] Meanwhile, on the screen S projected from a projector, which is a display device B211 installed in the studio, an image from the viewpoint of the virtual camera C2 (see FIG. 12), which is the viewpoint of the performer, is displayed, for example, as shown in FIG. 15, including an image including avatars of viewers who are virtually participating in the live performance in the third and fourth areas, thereby enabling the performer user to check the status of the viewer users who are virtually participating in the virtual space live performance through the image.

[0066] In addition, comments from viewer users, as well as comments and messages entered by the assistant operator O, are displayed on a display device C212 installed in the studio, allowing the performer user to check the comments from viewer users and messages from the assistant operator O even during the performance.

[0067] In addition to an operating system (OS) for operating the computer that is the performer terminal 200, the storage 203 also stores performer programs that, when executed by the processor (CPU) 201 etc., work in cooperation with the distribution server computer 100 to provide various functions, including the function of allowing the performer user to control the movement of the performer avatar 1 in the virtual space live, which is the virtual space content.

[0068] As shown in FIG. 4, the performer programs include a performer avatar control program, a performer viewpoint image control program, a performer viewpoint image control program, a performer voice control program, a performer comment control program, performer authentication data, etc.

[0069] The performer avatar control program is a program that mainly provides the function of scanning the body motion of the performer user using the motion sensor 220 and scanning the facial motion (facial expression) of the performer user using the facial expression input device 221, generating performer avatar information for operating the performer avatar, and transmitting it to the distribution server computer 100.

[0070] The performer's viewpoint image control program is a program that provides a function for generating and outputting a viewpoint image of virtual camera C2, which is the performer's viewpoint, based on the virtual space update data distributed from distribution server computer 100.

[0071] The performer's audio control program is a program that provides the function of outputting the playback sound of a song based on the playback music data distributed from the distribution server computer 100 via the audio output device 223, converting the audio input from the audio input device 222 into data and transmitting it to the distribution server computer 100 as performer audio data, and generating and outputting monitor audio consisting of the playback sound of the song and the performer's audio.

[0072] The performer's comment control program is a program that provides a function for displaying each comment by an audience user that is distributed from the distribution server computer 100 .

[0073] The performer authentication data is data for authenticating the identity of the performer user in a communication connection with the distribution server computer 100 .

[0074] In addition, the performer avatar control program includes a motion parameter generation program capable of generating parameter information for the positions and rotation angles of the bones that make up the rig data (sometimes called "skeleton data") that indicates the skeleton of the performer avatar contained in the performer avatar data, as well as an image recognition processing program for image-recognizing the position of the performer user's head and facial expression from images captured by an imaging camera.

[0075] In addition, in this embodiment, the performer terminal 200 is illustrated as a form in which a computer serving as performer terminal 200 installed in a studio operated by an event management organization is used jointly by multiple performer users to hold a live event, and each performer user can use the computer as their own performer terminal 200 by inputting performer authentication data; however, the present invention is not limited to this, and the performer terminal 200 may be used only by the performer user who owns it, such as when a performer user appears in a virtual space live event from their home or the like, and the performer terminal 200 is a computer installed in the performer user's home and dedicated to the performer user's own use, and in such cases, it is not necessarily necessary to store performer authentication data.

[0076] In this way, by having a performer user or the like use the performer terminal 200 that constitutes the virtual space content distribution system of this embodiment, the performer user can move the performer avatar 1 placed on the virtual stage G in the virtual live venue in conjunction with his or her own movements, can reflect his or her own facial expressions in the facial expressions of the performer avatar 1, and can distribute his or her own voice as the voice of the performer avatar 1 to viewers via the distribution server computer 100.

[0077] <Viewer device> Fig. 5 is a diagram showing an example of the configuration of a viewer terminal 300 in the virtual space content delivery system of this embodiment. In this embodiment, the viewer terminal 300 is a smartphone P carried by a viewer user or a regular desktop computer (PC) installed at a viewer user's home or the like, and as shown in Fig. 5, has a processor (CPU) 301 connected to a data bus 308, a memory (RAM) 302, a storage 303 such as a hard disk or non-volatile memory, a communication interface (I / F) 304 capable of two-way communication by TCP / IP protocol via the Internet, which is an open computer network, an image processing unit 306 including a graphics processing unit (GPU) to which a display device 310 is connected, and an input / output interface (I / F) 305 to which various input / output devices are connected.

[0078] The display device 310 may display each viewpoint video generated based on the virtual space data of the virtual live venue, which is a virtual space updated by the virtual space update data (see FIG. 10) distributed from the distribution server computer 100, by the viewer viewpoint video control program included in the viewer program stored in the storage 303, as described later, and the flying viewpoint video distributed from the distribution server computer 100 during the flying production period described later. If the viewer terminal 300 is a smartphone P, the display device 310 corresponds to the display device of the smartphone P, and if the viewer terminal 300 is a normal computer (PC), the display device 310 corresponds to the display device of the stationary type constituting the computer (PC). Note that these displays are not limited to those that display planar images (2D images), and may be, for example, those that can display stereoscopic images (3D images) by displaying right-eye images and left-eye images, such as a head-mounted display (HMD) that can be worn by the viewer.

[0079] The input / output interface (I / F) 305 is connected to an operation input device 321 consisting of one or more devices that allow the viewer to perform various operations as input devices, and an audio output device 322 such as earphones or speakers that can output live audio, etc., distributed from the distribution server computer 100.

[0080] It goes without saying that the audio output device 322 need not be an independent device, but may be integrated with the display device 310 like the above-mentioned head mounted display (HMD).

[0081] The operation input device 321 may be any device that allows the viewer to perform various operations such as motion operations including moving the viewer avatar, zoom operations, operations related to comments, operations related to gifts, etc., and may be composed of at least one of, for example, a transparent touch panel formed on the display surface of a smartphone, a keyboard constituting a computer, a game controller, etc.

[0082] In this embodiment, as described below, the virtual audience area in the virtual live venue is divided into a third area near the virtual stage G in which only viewer users who have been specially permitted by the performer user can virtually participate, and a fourth area around the third area in which general viewer users can virtually participate, as shown in FIG. 12. Viewer users who virtually participate in the fourth area may use an operation input device 321 such as a touch panel, keyboard, game controller, etc., as described above, but viewer users who virtually participate in the third area possess an operation input device 321 such as an MVN motion capture suit equipped with the above-mentioned inertial sensor, and are viewer users who, like performer users, can precisely move their viewer avatars with their own movements.

[0083] However, even viewer users who are permitted to virtually participate in the third area cannot change the facial expression of the viewer avatar, but the present invention is not limited to this, and the facial expressions of these viewer avatars may be changed, for example, as described above, by the viewer user using the camera function of a smartphone or a camera connected to a computer to recognize the facial expression of the viewer user, even if it is to a different level than the level of change made by the performer user.

[0084] The various input / output devices connected to the input / output interface (I / F) 305 may be built-in or external, and in the case of external devices, the connection form may be either wired or wireless.

[0085] In addition to an operating system (OS) for operating the viewer terminal 300, which is a smartphone or computer, the storage 303 also stores a viewer program as a viewing application that, when executed by a processor (CPU) 301 or the like, works in cooperation with the distribution server computer 100 to provide various functions, including a live viewing function for viewer users in the virtual space live content, which is virtual space content.

[0086] As shown in FIG. 5, the viewer program includes a viewer viewpoint image control program which provides the function of generating each viewpoint image of the virtual space live to be displayed on the display device 310, a viewer audio control program which provides the function of outputting live audio from the audio output device 322 based on the live audio (data) of the virtual space live distributed from the distribution server computer 100, a viewer comment control program which displays comments based on comment data distributed from the distribution server computer 100 and provides functions related to comments such as comments entered by the viewer and sending them to the distribution server computer 100, and viewer authentication data for authenticating the identity of the viewer user in a communication connection with the distribution server computer 100.

[0087] In addition, although not shown in Figure 5, in addition to the above-mentioned programs, there is also included a gift control program that controls gifts, and an area designation program for reserving an area (position) in which a viewer avatar will virtually participate in the virtual live venue in advance before the start of a live event, as shown in Figures 21 and 22, but other programs other than these may also be included.

[0088] In this embodiment, the comments input by the viewer are short messages in text format, but are not limited to these short messages in text format and may be, for example, still image messages, video messages, or any other electronic message format. The comments are displayed on the display device 310, superimposed on each viewpoint video.

[0089] In this way, by using the viewer terminal 300 that constitutes the virtual space content distribution system of this embodiment, a viewer user can move and operate the viewer avatar that is virtually participating in the virtual live venue, and by changing the viewpoint of the viewer avatar, the viewpoint image displayed on the display device 310 can be changed, and the zoom state (viewpoint state) can also be changed, allowing the viewer user to operate the viewer avatar to cheer on the performer avatar 1.

[0090] It should be noted that a viewer avatar who has virtually participated in the third area is only allowed to move within the third area, and cannot move to the fourth area. On the other hand, a viewer avatar who has virtually participated in the fourth area is only allowed to move within the fourth area, and cannot move to the third area. However, the present invention is not limited to this, and a viewer avatar who has virtually participated in three areas may be allowed to move to the fourth area. It should be noted that a viewer avatar cannot naturally move to the virtual stage G, etc., so the visibility of the performer avatar 1 is not obstructed by the viewer avatar moving onto the virtual stage G.

[0091] In addition, by displaying viewpoint images on these display devices 310 and outputting live audio, the user can enjoy a virtual space live performance with the feeling that he or she is actually present at the virtual live venue.

[0092] Although detailed explanations are omitted, the comment function provided by the viewer comment control program allows viewers to enjoy the virtual space live show while checking comments entered by themselves and other viewer users, and the gift function provided by the gift control program allows viewers to liven up the virtual space live show by giving items they own to the performer avatar 1.

[0093] <Administrator terminal> Fig. 6 is a diagram showing an example of the configuration of an administrator terminal 150 in the virtual space content delivery system of this embodiment. In this embodiment, the administrator terminal 150 uses a normal computer installed in a control room adjacent to the studio together with the performer terminal 200, and as shown in Fig. 6, has a processor (CPU) 151 connected to a data bus 158, a memory (RAM) 152, a storage 153 such as a hard disk, a communication interface (I / F) 154 capable of two-way communication by the TCP / IP protocol via the Internet, which is an open computer network, an image processing unit 156 including a graphics processing unit (GPU) to which a display device 160 is connected, and an input / output interface (I / F) 155 to which various input / output devices are connected.

[0094] As described below, the display device 150 is capable of individually displaying each viewpoint image of each of the virtual cameras C1 to C4 described below and each viewpoint image of the virtual camera moving in the air during the flying performance period, which are generated based on virtual space data of the virtual live venue, which is a virtual space updated by virtual space update data (see Figure 10) distributed from the distribution server computer 100, by an administrator viewpoint image control program included in the administrator program stored in storage 153, and is also capable of displaying comments based on comment data distributed from the distribution server computer 100, and is composed of one or more stationary displays.

[0095] The input / output interface (I / F) 155 is connected to an operation input device 161 consisting of one or more devices that enable the administrator K to perform various operations as input devices, and an audio output device 162 such as earphones, headphones, or speakers that can output live audio, etc., distributed from the distribution server computer 100.

[0096] The operation input device 161 may be any device that allows the administrator K to execute various operations such as input and settings related to the virtual space live event, and may be composed of at least one of, for example, a keyboard constituting a computer, a transparent touch panel formed on the surface of a display, a viewpoint switching controller, etc.

[0097] The various input / output devices connected to the input / output interface (I / F) 155 may be connected in either a wired or wireless manner.

[0098] In addition to an operating system (OS) for operating the computer that is the administrator terminal 150, the storage 153 also stores an administrator program that, when executed by a processor (CPU) 151 or the like, works in cooperation with the distribution server computer 100 to provide various functions, including a distribution management function for the distribution of virtual space live content, which is virtual space content.

[0099] As shown in FIG. 6, the administrator programs include an administrator viewpoint video control program, an administrator audio control program, an administrator comment control program, an audience management program, a performer management program, and the like.

[0100] The viewpoint video control program for administrator is a program that provides a function for generating viewpoint videos from each virtual camera to be displayed on the display device 160, as well as a function for changing and switching each viewpoint.

[0101] The administrator audio control program is a program that provides a function for outputting live audio from the audio output device 162 based on the virtual space live audio data distributed from the distribution server computer 100 .

[0102] The administrator's comment control program is a program that displays comments based on comment data distributed from the distribution server computer 100, and provides comment-related functions such as prohibiting comments from being distributed and selecting viewers to prevent them from being distributed.

[0103] The viewer management program is a program for managing viewer users who virtually participate in the virtual space live, and the performer management program is a program for managing performer users who virtually participate in the virtual space live.

[0104] Although not shown in FIG. 6, the system includes an authentication program for authenticating whether the operator, that is, the administrator, is the actual person, and a schedule program for editing the time schedule and song order of the virtual space live performance, but may also include other programs.

[0105] In this way, by having administrator K operate administrator terminal 150 which constitutes the virtual space content distribution system of this embodiment, various settings for executing the event can be implemented, such as settings related to the program, order of songs, and performance, as well as the time schedule and the trajectory of the virtual camera moving in the air during the flying performance period, and the virtual space live performance can be managed based on the settings thus made.

[0106] <Virtual live venue> The virtual live venue, which is the virtual space used in this embodiment, is shown in Fig. 12. As shown in Fig. 12, the virtual live venue, like a live venue in real space, is a virtual space that has a virtual stage G where performer users appear as performer avatars 1 and a virtual audience area where viewer users who will be in the audience virtually participate as viewer avatars.

[0107] 12, virtual stage G has a relatively large trapezoidal floor surface, and performer avatar 1 can move on virtual stage G in the same way as in a live venue in real space. On the side of virtual stage G opposite the virtual audience seats, as shown in Fig. 12, a virtual stage wall is formed that is divided into three parts, a center part, a right part, and a left part, and virtual displays 40, 41, and 42 are provided in front of these virtual stage walls, and images and videos for performance are virtually displayed on these virtual displays 40, 41, and 42.

[0108] A star virtual arrangement object 43 related to the performer user is placed between the virtual displays 40, 41, 42, and large star virtual aerial objects 50-52 are placed in a second area, which is an area on the virtual stage G and is above a first area surrounded by the virtual stage walls, as shown in Fig. 12. The virtual aerial objects 50-52 are movable within the second area.

[0109] A virtual audience area where viewer users can virtually participate in the virtual live venue as viewer avatars is provided in front of the virtual stage G. As shown in Fig. 12, this virtual audience area is made up of a third area, which is a special area close to the center of the virtual stage G where the performer avatar 1 appears, and a fourth area, which is formed to surround the third area and is farther away from the center of the virtual stage G than the third area.

[0110] These third areas are available for virtual participation by special viewer users who are permitted to participate virtually, up to a predetermined upper limit. In this embodiment, as shown in FIG. 12, for example, viewer avatars 11 to 14 of four viewer users who have met a predetermined virtual participation record and have been permitted to participate virtually by the performer user are arranged.

[0111] In addition, the fourth area includes viewer avatars 21-32 of general viewer users who have reserved virtual participation before the start of the virtual space live. Note that, in Fig. 12, the viewer avatars 21-32 are shown in a simplified manner for the sake of convenience, but are the same avatars as the viewer avatars 11-14.

[0112] Furthermore, each viewer avatar 11-14 placed in the third area can be moved within the third area by the viewer user operating the viewer terminal, and each viewer avatar 21-32 placed in the fourth area can be moved within the fourth area by the viewer user operating the viewer terminal.

[0113] Four virtual cameras C1 to C4 are virtually arranged (set) in the virtual live performance venue, as shown in Fig. 12. Virtual camera C1 is virtually arranged in front of performer avatar 1 so as to face the performer avatar 1, and the viewpoint image captured by virtual camera C1 is a viewpoint image viewed from a position close to the front of the performer avatar 1, as shown in Fig. 13(a).

[0114] Virtual camera C2 is a camera virtually placed (set) above the head of performer avatar 1, and the viewpoint image captured by virtual camera C2 is a performer viewpoint image captured by viewing the virtual audience area from performer avatar 1, as shown in FIG.

[0115] Virtual camera C3 is a virtual camera virtually placed on the virtual display 41 diagonally behind the performer avatar 1, and the viewpoint image from virtual camera C3 is a viewpoint image looking at the virtual audience area from diagonally behind the performer avatar 1, as shown in Figure 13(c).

[0116] Virtual camera C4 is a virtual camera virtually placed on the virtual display 42 diagonally behind the performer avatar 1, and the viewpoint image from virtual camera C4 is a viewpoint image looking at the virtual audience area from diagonally behind the performer avatar 1, as shown in Figure 13(b).

[0117] In this embodiment, in addition to the fixedly placed (set) virtual cameras C1 to C4, an air-moving virtual camera (not shown) is virtually placed so as to be movable in the air within the virtual live venue space, which is a virtual space. The air trajectories along which these air-moving virtual cameras move and the viewpoint directions (angles) at each position on the air trajectory are preset by the administrator terminal 150, and during the flying performance period described later, the performer avatar 1 virtually flies so as to follow the air trajectory of the air-moving virtual camera, thereby generating a viewpoint image of the virtually flying performer avatar 1 captured by the air-moving virtual camera.

[0118] In this embodiment, as shown in FIG. 12, an example is given in which four virtual cameras C1 to C4 are virtually arranged as fixedly arranged (set) virtual cameras, but the present invention is not limited to this, and the number of these fixedly arranged (set) virtual cameras may be five or more, or conversely, without providing fixedly arranged (set) virtual cameras, only the viewpoint images seen from the viewpoints corresponding to each avatar virtually participating in the virtual live venue may be displayed on the performer terminal 200 and the viewer terminal 300.

[0119] In addition, while this embodiment illustrates an example in which there is one virtual camera moving through the air, the present invention is not limited to this, and it may also be possible to have a configuration in which there are multiple virtual cameras moving through the air, or conversely, a configuration in which no virtual camera moving through the air is virtually positioned.

[0120] <Studio> 11 is a diagram showing the studio used in this embodiment, and as described above, the studio is adjacent to the control room separated by a glass window, and the performer user performs each movement corresponding to singing in the studio, while the assistant operator O assists with the operation of the performer terminal 300 installed in the control room. Around the performer user wearing the wearable sensors 220C1 to 220C5, base stations 220a and 220b are mounted on a stand, and an imaging camera constituting the facial expression input device 221 is mounted on a stand in a position almost in front of the performer user.

[0121] A screen S is provided on the wall facing the performer user in the studio, and a projector (display device B211) installed on the ceiling of the studio projects the viewpoint image of virtual camera C2 onto screen S, so that the viewpoint image seen from performer avatar 1 as looking at the virtual audience area is displayed on screen S, as shown in Figure 15. This allows the performer user to proceed with the live performance while constantly keeping track of the behavior of the viewer avatars virtually participating in the virtual audience area (the viewer's reaction to the live performance).

[0122] A vertical large display is arranged to the side of the screen S as the display device C212, and comments from the audience and messages that the assistant operator O wants to convey to the performer users are displayed on the large display. The display device C212 may be configured to display information (performer support information) such as the lyrics of the song sung by the performer during the live performance and lines.

[0123] <Operation of virtual space content delivery system> FIG. 10 is an explanatory diagram illustrating the operation of the virtual space content delivery system of this embodiment, specifically, the flow of delivery of video and audio of a virtual live performance.

[0124] First, the video system will be described. As described above, the distribution server computer 100 is capable of providing a virtual space of a virtual live venue based on the virtual space generation program 120 and the avatar data of the performer users and the viewer users, and the actions of the avatars of the performer users and the viewer users are reflected in the actions of the performer avatar 1 and the viewer avatar 1 actions (operations) of the viewer users are reflected in the actions of the viewer avatar by updating the virtual space data in which the state of the virtual space is described based on the performer avatar information transmitted from the performer terminal 200 and the viewer avatar information transmitted from the viewer terminal 300.

[0125] Although not shown in Figure 10, when the performer terminal 200 and the viewer terminal 300 communicate with the distribution server computer 100 to virtually participate in the virtual space live performance, virtual space data of the virtual live venue is delivered to and stored in advance from the distribution server computer 100.As will be described later, the virtual space data stored in advance in this manner is successively updated by virtual space update data delivered successively from the distribution server computer 100, allowing the performer terminal 200 and the viewer terminal 300 to identify the latest state of the virtual live venue.

[0126] Specifically, the movements and facial expressions of the performer user in the studio are scanned at predetermined time intervals by the body motion scan and face motion scan functions of the performer avatar control program described above, and performer avatar information is generated and transmitted to the distribution server computer 100.

[0127] On the other hand, viewer users who are permitted to virtually participate in the third area use motion sensors, just like the performers, and are scanned at specified intervals using the body motion scan and face motion scan functions, and viewer avatar information is generated and transmitted to the distribution server computer 100.

[0128] Furthermore, a viewer user permitted to virtually participate in the fourth area can use the touch panel, keyboard, or controller of the viewer terminal 300 to perform various operations such as movement, changing the viewpoint direction, changing the viewpoint state (zoom), raising hands, clapping hands, jumping, etc., to make the viewer avatar perform the corresponding action. Then, of these operations, viewer avatar information is generated based on the operation related to the viewer avatar's action and transmitted to the delivery server computer 100.

[0129] In this way, based on the performer avatar information transmitted from the performer terminal 200 and the viewer avatar information transmitted from the viewer terminal 300, the distribution server computer 100 executes a virtual space update process that updates the virtual space data, thereby reflecting the actions (operations) of the performer user and the viewer user in each avatar virtually participating in the virtual live venue.

[0130] Then, after executing a virtual space update data generation process for generating virtual space update data based on the virtual space data before the update and the virtual space data after the update, the generated virtual space update data is distributed to the performer terminal 200 and the viewer terminal 300.

[0131] At the performer terminal 200 and viewer terminal 300 to which the virtual space update data has been delivered, as shown in FIG. 10, the pre-stored virtual space data is updated based on the delivered virtual space update data, and a performer perspective video generation process and a viewer perspective video generation process are executed using the updated virtual space data, thereby generating performer perspective video and viewer perspective video based on the updated virtual space data, and the generated performer perspective video and viewer perspective video are displayed on the performer terminal 200 and the viewer terminal 300 (viewer perspective video output process).

[0132] In this manner, in this embodiment, the viewpoint video of each viewer user who is virtually participating in the virtual live venue through a viewer avatar is generated and displayed on the viewer terminal 300 of each viewer user, and by generating the viewer avatar viewpoint video of each viewer user on the distribution server computer 100, it is possible to avoid a situation in which the processing load on the distribution server computer 100 becomes significantly large, which would result in many viewer users being unable to participate in the virtual space live, and it is also possible to avoid a situation in which distribution becomes difficult due to an increased processing load.

[0133] In this embodiment, a viewpoint image is generated and displayed on the display device 310 of the viewer terminal 300 of the viewer user of the viewer avatar 13 who is virtually participating in the third area close to the performer avatar 1 on the virtual stage G, in which a viewpoint image of almost the front of the performer avatar 1 is displayed large, which is the viewpoint image seen from the viewer avatar 13 close to the performer avatar 1, as shown in Figure 13 (d), while a viewpoint image is generated and displayed on the display device 310 of the viewer terminal 300 of the viewer user of the viewer avatar 28 who is virtually participating in the fourth area far from the performer avatar 1 on the virtual stage G, for example, at a position behind the viewer avatar 13, in which a viewpoint image of almost the front of the performer avatar 1 appears relatively small through the viewer avatar 13 and viewer avatar 14 who are virtually participating in the third area, which is the viewpoint image seen from the viewer avatar 28 far from the performer avatar 1, as shown in Figure 13 (e).

[0134] In addition, the display device 310 of the viewer terminal 300 of the viewer user of the viewer avatar 32 who is virtually participating at the edge position of the virtual stage G in the fourth area, which is set up as a U-shape when viewed from above, displays a viewpoint image seen from the viewer avatar 32 positioned diagonally forward of the performer avatar 1, as shown in Figure 13 (f), in which the performer avatar 1 appears relatively small through the viewer avatar 14 who is virtually participating in the third area.

[0135] Note that all of the display examples shown in Figure 13 are examples in which the viewer terminal 300 is a stationary computer (PC) and the display device 310 is a stationary display. However, for example, if the viewer terminal 300 is a smartphone P, tablet, etc., the viewpoint image of the virtual camera C1 is displayed as shown in Figure 14(a), and if the viewer terminal 300 is the viewer terminal 300 of the viewer avatar 13, it is displayed as shown in Figure 14(b).

[0136] Next, the audio system will be described. In the distribution server computer 100, music data 113 is played back by an audio control program 124 based on time schedule data (TS data) 112, and transmitted to the performer terminal 200 as played music (data) (music playback process).

[0137] In the performer terminal 200, the music audio from the played music (data) transmitted from the distribution server computer 100 is output to the performer user through high-performance earphones (in-ear monitors) worn by the performer user (played music output process), and when the performer user sings along with the output played music, the performer voice input from the sound collection microphone (audio input device 222) is digitized and transmitted to the distribution server computer 100 as the performer voice (data).

[0138] The distribution server computer 100 generates distribution audio (data) from the performer audio (data) received from the performer terminal 200 and the played music played in the music playback process described above (distribution audio generation process), and distributes the generated distribution audio (data) to the viewer terminal 300 (generated audio distribution process).

[0139] In the viewer terminal 300, the distribution audio (data) distributed from the distribution server computer 100 is output from an audio output device 322 such as earphones or speakers, allowing the viewer to hear the live audio as if the performer avatar 1 were singing in the virtual live venue.

[0140] Furthermore, in the performer terminal 200, as described above, the performer's voice (data) is transmitted to the distribution server computer 100, and as shown in Figure 10, a monitor audio generation process and a monitor audio output process are executed, so that the music audio of the played music and the performer's voice are output from speakers placed in the control room, allowing an assistant operator O, etc. to check the singing status of the performer user in the control room.

[0141] <Viewer's viewpoint video control processing> Next, a viewer's viewpoint video control process executed in the viewer terminal 300 based on a viewer's viewpoint video control program included in the viewer's program will be described with reference to FIG.

[0142] In the viewer viewpoint video control process, first, it is determined whether or not the performer avatar 1 is in a flying performance period in which the performer avatar 1 is virtually flying, as shown in Fig. 19 (step S1). Whether or not the performer avatar 1 is in a flying performance period can be identified by performance status data indicating that the performer avatar 1 is in a flying performance period, which is sent from the delivery server computer 100. The delivery server computer 100 transmits performance status data indicating that the performer avatar 1 is in a flying performance period to the viewer terminal 300 when the performer avatar 1 is in a flying performance period based on the time schedule data.

[0143] If it is during the flying performance period (Y in step S1), proceed to step S21, whereas if it is not during the flying performance period (N in step S1), it is determined whether viewpoint-related operations including viewpoint movement operations (including changing the viewpoint by avatar movement operations and motion operations) and zoom operations are disabled (step S2).

[0144] If the viewpoint-related operation is invalid (Y in step S2), the process proceeds to step S7. If the viewpoint-related operation is not invalid (N in step S2), the process determines whether or not there is an operation that changes the viewpoint, such as an operation of moving an avatar or an operation of moving an avatar (step S3), and if there is an operation (Y in step S3), the viewer viewpoint is changed according to the operation, and if there is no operation (N in step S3), the process proceeds to step S5, and determines whether or not there is a zoom operation (step S5).

[0145] If a zoom operation is performed (Y in step S5), the image area displayed on the display device 310 is changed to an image area corresponding to the operation, and if a zoom operation is not performed (N in step S5), it is determined whether or not a motion tracking setting is present (step S7).

[0146] This motion tracking setting can be arbitrarily set by the viewer user on the viewer terminal 300, for example, before the start of a live performance or during a live performance, depending on the type of device used by the viewer user and his / her level of proficiency in the operation. Specifically, the motion tracking setting is set in cases where, for example, the viewer user is not familiar with operating the operation input device 321 such as a touch panel, keyboard, or game controller, or is not familiar with changing the viewpoint by operating the avatar to move or move, or, as will be described later, the virtual participation position of the viewer user's avatar is far from the performer avatar 1 and the viewer user is zoomed in and displayed in an enlarged manner, so that even a slight movement of the performer avatar 1 prevents the performer avatar 1 from being displayed appropriately, and so on, and the viewer user is unable to change the viewpoint in response to the movement or movement of the performer avatar 1.

[0147] If there is no motion following setting (N in step S7), the process proceeds to step S11. If there is a motion following setting (Y in step S7), it is further determined whether or not the motion following condition is satisfied (step S8).

[0148] In this embodiment, the motion tracking condition is a condition under which it is almost certain that the performer avatar 1 will no longer be displayed clearly on the display device 310. Specifically, when the head (or face) of the performer avatar 1 is within a specified range around the display area of ​​the display device 310, it is highly likely that the movement or motion of the performer avatar 1 will cause the head (face) to move outside the display area, causing it to no longer be displayed clearly, and therefore it is determined that the motion tracking condition is met.

[0149] In this way, in this embodiment, by determining that the motion tracking condition is met before the head (face) of the performer avatar 1 disappears from the display area, it is possible to significantly reduce the occurrence of the head (face) of the performer avatar 1 disappearing from the display area, but the present invention is not limited to this, and these motion tracking conditions may be the condition that the head (face) moves out of the display area, or other conditions may be used as the motion tracking conditions. In other words, any condition may be used as the motion tracking condition as long as it can prevent the performer avatar 1 from disappearing from the display area in a good manner.

[0150] If the action tracking condition is met (Y in step S8), after disabling the viewpoint-related operations (step S12), the viewpoint image displayed in the display area of ​​display device 310 is automatically changed to a motion tracking viewpoint in which the head (face) of performer avatar 1 is within a non-metable range surrounded by a predetermined range around the display area, and the viewpoint image seen from the motion tracking viewpoint is displayed in the display area of ​​display device 310 (step S13).

[0151] On the other hand, if the action tracking condition is not met (N in step S8), it is further determined whether or not the viewpoint-related operation is disabled (step S9). If the viewpoint-related operation is disabled (Y in step S9), the disabled viewpoint-related operation is enabled (step S10) and then the process proceeds to step S11. If the viewpoint-related operation is not disabled (N in step S9), the process proceeds to step S11 without going through step S10.

[0152] In step 11, since the motion tracking setting is not set or the motion tracking condition is not satisfied, the viewpoint video of the viewer avatar is continuously displayed in the display area of ​​display device 310 (step S11).

[0153] Also, as mentioned above, if the flying performance period in which the performer avatar 1 virtually flies during a live performance has begun, the answer in step S1 is determined to be Y, and the process proceeds to step S21, where it is determined whether or not it is the start timing of the flying performance period (step S21).

[0154] If it is the start timing of the flying performance period (Y in step S21), the flying viewpoint image distributed from the distribution server computer 100 is displayed (step S22) in place of the viewer avatar's viewpoint image or action-tracking viewpoint image that was displayed in the display area of ​​the display device 310 before the start of the flying performance period, and then viewpoint-related operations are disabled (step S23) and the process returns to step S1.

[0155] In this embodiment, during the flying performance period, the same flying viewpoint video is displayed on the viewer terminals 300 of all viewer users. Instead of generating the flying viewpoint video on each viewer terminal 300 side, as is the case with viewer avatar viewpoint videos which are different for each viewer user, the flying viewpoint video is generated on the distribution server computer 100 side and distributed to the viewer terminals 300. This allows the flying viewpoint video, which involves movement of the viewpoint in the air and therefore imposes a large rendering load on the viewer terminals 300, to be displayed well even on viewer terminals 300 with low processing capabilities. However, the present invention is not limited to this, and these flying viewpoint videos may be generated on the individual viewer terminals 300 side, in the same way as viewer avatar viewpoint videos.

[0156] On the other hand, if it is not the start timing of the flying effect period (N in step S21), it is further determined whether it is the end timing of the flying effect period (step S24).

[0157] Whether or not it is the end timing of the flying performance period may be determined based on data indicating the end contained in the flying viewpoint video data transmitted from the distribution server computer 100, or may be determined based on the aforementioned performance status data that is different from the flying viewpoint video.

[0158] If it is not the end of the flying performance period (N in step S24), that is, if it is during the flying performance period, proceed to step S26, update the flying viewpoint video, and then return to step 1; on the other hand, if it is the end of the flying performance period (Y in step S24), enable viewpoint-related operations (step S25), proceed to step S11, display the viewer avatar's viewpoint video in the display area of ​​display device 310 in place of the flying viewpoint video that was displayed in the display area of ​​display device 310 during the flying performance period, and then return to step S1.

[0159] Thus, each viewpoint video displayed on the viewer terminal 300 by executing the viewer viewpoint video control process shown in FIG. 20 in the viewer terminal 300 of this embodiment will be described with reference to FIGS. 16 to 19.

[0160] Fig. 16 is a diagram showing the display contents of the display device 310 when a zoom operation is performed on the viewer terminal 300 of this embodiment, and in the default state where no zoom operation is performed, as shown in Fig. 16(a), a viewpoint video of the video area centered on the upper body of the performer avatar 1 is displayed. In this default state, when the viewer user performs a zoom-up operation, for example, by touching the "+" display (not shown) if the viewer terminal 300 is a smartphone P, or by operating the "+" key on the keyboard if the viewer terminal 300 is a computer (PC), for example, a viewpoint video of the video area centered on the head (face) of the performer avatar 1 is displayed as shown in Fig. 16(b).

[0161] On the other hand, if the viewer user performs a zoom-down operation, for example, by touching the "- " display (not shown) if the viewer terminal 300 is a smartphone P, or by operating the "- " key on the keyboard if the viewer terminal 300 is a computer (PC), a viewpoint image of the image area including the entire body of the performer avatar 1 will be displayed, as shown in Figure 16 (c), for example.

[0162] In Figure 16, in order to make the zoom function easier to understand, the explanation was given using the viewpoint image of virtual camera C1 rather than the viewpoint of the viewer avatar, but as shown in Figure 17, zooming up and down can also be performed in the same way on the viewpoint of the viewer avatar.

[0163] Also, although not described in detail in the viewer viewpoint image control process of Figure 20, for example, when performer avatar 1 appears on virtual stage G for the first time when a live performance begins, the viewpoint of the viewer avatar will not be directed toward performer avatar 1, and there is a possibility that the appearing performer avatar 1 will not be displayed on the viewer terminal 300 for a long period of time. Therefore, the period when performer avatar 1 first appears may be designated as an appearance performance period, and during this performance period, the viewpoint image displayed on the viewer terminal 300 may be automatically switched to the viewpoint image of virtual camera C1, etc., instead of the viewpoint image of the viewer avatar, and individual viewer users may be allowed to zoom in and perform other operations on the viewpoint images of virtual camera C1, etc.

[0164] 17 shows, as an example of a viewpoint image of a viewer avatar, the viewpoint image of viewer avatar 28 who is virtually participating in the fourth area. As shown in Fig. 17, the viewpoint image of viewer avatar 28 includes the heads of viewer avatar 13 and viewer avatar 14 who are virtually participating in the third area, and the image shows performer avatar 1 seen through the heads of viewer avatar 13 and viewer avatar 14.

[0165] Therefore, the viewer user of viewer avatar 28 can enjoy the live performance by, for example, performing a zoom-in operation to change the image area so that the upper body of performer avatar 1 is displayed primarily, as shown in Figure 17(b), thereby preventing viewer avatar 13 and viewer avatar 14 from being displayed.

[0166] However, in the zoomed-in state as shown in Figure 17(b), for example, as the live performance progresses and the performer avatar 1 begins to move and make large movements, if the viewer user is unable to properly change the viewpoint of the viewer avatar 28 to match the movements of the performer avatar 1, the performer avatar 1 may not be displayed properly, as shown in Figures 18(d)-(f).If the viewer user has to pay attention to adjusting the viewpoint, it may become difficult to concentrate on the live performance and they may not be able to enjoy it well.

[0167] However, in this embodiment, when the viewer user sets the motion tracking setting as described above, as shown in Figures 18(a) to (c), the head of the performer avatar 1 enters within a specified area surrounding the display area, the motion tracking condition is met, the viewpoint is automatically changed to the motion tracking viewpoint, and the viewpoint image of the motion tracking viewpoint is displayed. Therefore, the viewer user does not need to pay attention to operating the viewpoint, and can concentrate on the live performance and enjoy it to the fullest.

[0168] As the live performance progresses and the flying performance period begins, as shown in Figure 19, The viewpoint image of each viewer avatar is automatically changed to a flying viewpoint image seen from a virtual camera moving in the air, without any operation by the viewer user.

[0169] Specifically, before the performer avatar 1 takes off in virtual flight, an image is displayed in which a wing object grows on the back of the performer avatar 1. Then, the performer avatar 1 looks up at a virtual aerial star object 50 virtually placed in the second area of ​​the virtual live venue, and the viewpoint of the viewer avatar moves in the same way. At this time, light effects 61 appear around the performer avatar 1.

[0170] Thereafter, when the performer avatar 1 enters virtual flight, the viewpoint image displayed on the display device 310 automatically switches from the viewpoint image of the viewer avatar in FIG. 19(b) to a flying viewpoint image seen from a virtual camera moving in the air, as shown in FIG. 19(c).

[0171] During the flying performance period, the aerial moving virtual cameras move within the second area based on a pre-set aerial trajectory as described above, and the flying viewpoint images seen from these aerial moving virtual cameras moving in the air are displayed on the display device 310 as shown in Figure 19(d).

[0172] The present invention has been described above based on the embodiments, but the specific configuration is not limited to the above-mentioned embodiments, and the present invention also includes modifications and additions that do not deviate from the gist of the present invention.

[0173] For example, in the above embodiment, a viewer avatar is shown to be able to move freely within the third or fourth area in which it is virtually participating. However, doing so may result in problems such as errors in the virtual space data due to different viewer avatars participating in the fourth area colliding virtually with each other or overlapping in the same virtual position.

[0174] In order to solve such problems, for example, the modified examples shown in Figures 21 and 22 may be used. Specifically, the third and fourth areas are divided into smaller areas as shown in Figure 21, and the viewer user is asked in advance before the start of the live performance which of these areas he or she will participate in, as shown in Figure 22. When the viewer user virtually participates with a viewer avatar, he or she is only able to participate in the area selected in advance, and movement is restricted to within the selected area, thereby preventing virtual collisions between viewer avatars or overlapping in the same virtual position, which can cause errors in the virtual space data.

[0175] In addition, in the case where each viewer user is allowed to select the virtual participation position of the viewer avatar in advance as described above, for example, for a viewer user who selects areas A1 to C6 on the right side facing the virtual stage G in the fourth area, a viewpoint image from the right front position of the virtually flying performer avatar 1 is generated and displayed as the viewpoint image during the flying performance period, for a viewer user who selects areas D4 to H6 on the center facing the virtual stage G in the fourth area, a viewpoint image from the central front position of the virtually flying performer avatar 1 is generated and displayed as the viewpoint image during the flying performance period, and for a viewer user who selects areas I1 to K6 on the left side facing the virtual stage G in the fourth area, a viewpoint image from the left front position of the virtually flying performer avatar 1 is generated and displayed as the viewpoint image during the flying performance period. In this way, depending on the position selected by the viewer user, the viewpoint image during the virtual flight, which is a special action, may be a viewpoint image corresponding to the position (area) selected by each viewer user.

[0176] In the above embodiment, the viewpoint video from the viewer avatar is generated and displayed on the viewer terminal 200 side, but the present invention is not limited to this, and for example, in the case of a live event with few virtual participants, the viewpoint video from the viewer avatar may also be generated and distributed on the delivery server computer 100 side. Also, the viewer user may be allowed to select whether to generate the video on the viewer terminal 200 side or the delivery server computer 100 side depending on the processing capacity of the terminal he or she owns and the data communication environment, or the delivery server computer 100 may specify the processing capacity of the viewer terminal 200 with which it is connected for communication, and determine whether to generate the video on the viewer terminal 200 side or the delivery server computer 100 side based on the specified processing capacity.

[0177] In addition, in the above embodiment, the distribution server computer 100 is installed in a studio operated by an event management organization that hosts the virtual space live event, but the present invention is not limited to this. As the distribution server computer 100, a server computer owned in a data center or the like by a company that rents out the server computer may be used, or as described above, a server computer provided by a cloud service may be used. The installation form of these server computers may be any form that provides the functions of the distribution server computer 100.

[0178] In the above embodiment, a form in which a single performer user sings and performs is exemplified, but the present invention is not limited to this, and the performer users may be a group of multiple people, and when a virtual space live performance is performed by a group of multiple people, the multiple people may share one performer terminal 200, or a performer terminal 200 may be provided for each member of the group. When the performer users are a group of multiple members, for example, a viewpoint tracking target designation process may be executed by at least one of the distribution server computer 100 and the viewer terminal 300, so that the viewer user can select a member to be the target of the viewpoint tracking.

[0179] In addition, in the above embodiment, the singer who sings and the actor who performs the performance may be separate persons, in which case the singer's performer terminal and the actor's performer terminal may be separate, and the singer may sing in a location other than the studio where the actor performs the performance.

[0180] In addition, in the above embodiment, the Internet network is used as an example of the computer communication network connecting the distribution server computer 100, the administrator terminal 150, the performer terminal 200, and the viewer terminal 300, but the present invention is not limited to this, and these communication networks may be composed of local area networks that are capable of data communication only within a specific area, or may include a local area network and local data communication as part of them.

[0181] In addition, in the above embodiment, the performer user sings or performs singing actions (performance actions) in real time during the distribution of the virtual space live performance, allowing the performer user to proceed with the live performance while checking the reactions of the viewer users who are virtually participating in the virtual space live performance, but the present invention is not limited to this, and the performer user's singing or singing actions (performance actions) do not have to be done in real time, but may be performed in advance, and the singing audio and singing actions (performance actions) may be recorded in the distribution server computer 100 as performer avatar information, and the recorded singing audio and singing actions (performance actions) may be played back as the live performance progresses, thereby holding a virtual space live performance.

[0182] In addition, instead of the singing and singing actions (performance) of a performer user, the singing and singing actions (performance actions) of a virtual performer created using computer graphics, etc., can be recorded as performer avatar information on a distribution server computer, and the recorded singing audio and singing actions (performance actions) can be played back as the live performance progresses, thereby holding a virtual space live performance.

[0183] In this way, when singing or singing actions (performance actions) are performed in advance to record and play back performer avatar information on the distribution server computer 100 and distribute a virtual space live performance, the configuration of the virtual space content distribution system at the time of distribution will be made up of the distribution server computer 100, administrator terminal 150, and viewer terminal 200, and will not include the performer terminal 200, which is a performer user terminal that can be used by a performer user to perform operations to move the performer avatar 1 in the virtual space.

[0184] In addition, in the above embodiments, the viewer terminal 300 is exemplified as a smartphone P or a computer (PC), however, the present invention is not limited to this, and these computers may be portable notebook computers, tablet terminals, e-book readers, wearable computers, game consoles, and various other information processing devices capable of outputting at least 2D video and audio.

[0185] In addition, in the above embodiment, an example was given of a form in which a performer user performs in a studio, including singing, but the present invention is not limited to this. For example, a performer user may use a performer terminal 200 installed in a home or the like to appear in a virtual space live performance from a home or the like.

[0186] In the above-described embodiment, a virtual space live performance was given as an example of virtual space content, but the present invention is not limited to this, and such virtual space content may be any content that is beneficial to viewer users, such as a fan meeting where performer users gather with their fans, or a talk session, and that is related to a virtual space in which performer avatar 1 and viewer avatars virtually participate.

[0187] In the above embodiment, the general action (first action) is an action of the performer avatar 1 standing on the virtual stage G without moving, the special action (second action) is an action of the performer avatar 1 flying virtually, and the specific action (third action) is an action of the performer avatar 1 moving quickly sideways. However, the present invention is not limited to this, and the actions may be actions according to the content to be distributed. For example, the general action (first action) may be an action (normal action) that the performer avatar performs for the longest period in the content to be distributed, the special action (second action) may be an action that is performed in a special period that is the shortest period in the content to be distributed or an action that is performed least frequently, and the specific action (third action) may be an action that is performed for a shorter period than the general action (first action) but longer than the special action, or an operation that is performed less frequently than the general action (first action) but more frequently than the special action. The special action (second action) may include the specific action (third action).

[0188] In the above embodiment, an example was given of a form having both a function of switching the viewpoint corresponding to a special action (second action) and a function of following a specific action (third action), but the present invention is not limited to this, and the virtual space content distribution system may have only one of these functions.

[0189] In the above-described embodiment, an example is given of a form in which viewpoint switching of the four virtual cameras C1 to C4 as fixedly positioned (set) virtual cameras and the airborne moving virtual camera can be performed in advance by settings and operations on the administrator terminal 150, but the present invention is not limited to this. For example, these viewpoint switching may be performed in real time by the administrator K on the administrator terminal 150 using a viewpoint switching controller or the like constituting the operation input device 161.

[0190] In the above-described embodiment, an example is given of a form in which virtual cameras C1 to C4 are fixedly placed (set) in the virtual space as virtual cameras, but the present invention is not limited to this. For example, in addition to or instead of the virtual cameras C1 to C4, a movable virtual moving camera may be placed in the virtual live venue, and the position and imaging direction of the virtual moving camera in the virtual space may be controlled by an administrator K or a cameraman, etc., via operation of the administrator terminal 150, to produce the viewpoint image of the virtual moving camera.

[0191] In the above embodiment, a viewpoint image of the virtual camera C2 is displayed on the screen S so that the facial expression of the performer user can be detected and reflected in the facial expression of the performer avatar, but the present invention is not limited to this, and for example, if the performer user has a high-performance head-mounted display with a sensor function to detect the facial expression of the performer user, the performer user may wear such a high-performance head-mounted display to perform the performance, which may improve the performer user's immersive feeling in the virtual live performance. Also, if the facial expression of the performer user does not need to be reflected in detail in the facial expression of the performer avatar, the performer user may wear a normal head-mounted display to perform the performance.

[0192] In the above-described embodiment, an example was given of the viewer user's viewpoint being changed and following the movements of the performer avatar, but the present invention is not limited to this. For example, these viewpoint changes and following may be performed in response to specific effects such as smoke or fireworks (pillars of fire) devices, special lighting, or deformation of stage G, which are props on stage G. [Explanation of symbols]

[0193] 1. Performer Avatar 100 Distribution server computer 150 Administrator terminal 200 Performer terminals 300 Viewer terminals

Claims

1. A virtual space content distribution system that distributes virtual space content created by performer avatars so that multiple viewer users can view it, A viewer user terminal capable of outputting video of the virtual space that is the target of the aforementioned virtual space content so that it can be viewed by the viewer user, A server computer connected to the aforementioned viewer user terminal via a network and capable of performing at least the processing related to the virtual space and the processing related to the distribution of the virtual space content, Includes, The aforementioned viewer user terminal is capable of outputting viewpoint video from at least multiple viewer viewpoint positions that are located in different locations within the virtual space as virtual space content. When the performer avatar is performing a special action, the system includes a viewer viewpoint position changing means that changes the viewer viewpoint position to follow the special action. The way the performer avatar tracks the viewer's viewpoint position when performing the special action differs depending on the viewer's viewpoint position. A virtual space content distribution system characterized by the following features.

2. A virtual space content distribution program for distributing virtual space content using performer avatars so that it can be viewed by multiple viewer users via a network using at least a server computer, The server computer is connected via the network to a viewer user terminal that is capable of outputting video of the virtual space that is the target of the virtual space content so that it can be viewed by the viewer user. A virtual space content output step that causes the viewer user terminal to output viewpoint videos from multiple viewer viewpoint positions that are located at least in the virtual space as virtual space content, When the performer avatar is performing a special action, the viewer viewpoint position change step involves changing the viewer viewpoint position to follow the special action, Includes, The tracking behavior of the viewer viewpoint position in the viewer viewpoint position change step differs depending on the viewer viewpoint position. A virtual space content distribution program characterized by the following features.

3. A virtual space content distribution method for distributing virtual space content using performer avatars so that it can be viewed by multiple viewer users via a network using at least a server computer, The server computer is connected via the network to a viewer user terminal that is capable of outputting video of the virtual space that is the target of the virtual space content so that it can be viewed by the viewer user. A virtual space content output procedure that causes the viewer user terminal to output viewpoint videos from multiple viewer viewpoint positions that are located at least in the virtual space as virtual space content, When the performer avatar is performing a special action, the viewer viewpoint position change procedure is used to change the viewer viewpoint position to follow the special action. Includes, The tracking pattern of the viewer viewpoint position in the viewer viewpoint position changing procedure differs depending on the viewer viewpoint position. A method for distributing virtual space content characterized by the following features.

Images