Information processing system, information processing method, and program

Abstract

[Problem] To allow a performer performing in content distributed in real time to perform in response to the reaction of a viewer in a remote location. [Solution] This information processing system includes a control unit. The control unit acquires, from a viewer terminal reproducing content in which a performance by a performer has been captured in real time over a network, a sight line parameter indicating the sight line of the viewer in a coordinate system for a space in which the viewer is present, and identification information identifying the viewer. The control unit converts the acquired sight line parameter to a sight line parameter indicating the virtual-sight-line of the viewer in the coordinate system of the space in which the performer is present. On the basis of the converted sight line parameter, the control unit outputs, to an output device in the space in which the performer is present, the sight line information indicating the virtual sight line of the viewer.

Description

[Technical field]

[0001] The present technology relates to an information processing system, an information processing method, and a program capable of presenting information about viewers to performers in a system that distributes content filmed from a performance by a performer to viewers in real time via a network. [Background technology]

[0002] With the development of network technology, content distribution services to remote locations are continuing to grow. For example, video distribution, including movie content, has been realized through a one-way system in which content data is sent from the distributor to the viewer, and the viewer enjoys the distributed content.

[0003] In recent years, for example, in real-time video distribution services, interactive systems have been realized that allow distributors and viewers to communicate with each other. This system provides new value not only in the content itself, but also in the experience of communication through the content.

[0004] In such a system, the main means of communication between viewers and distributors is text and audio information. With regard to text information in particular, by superimposing text information entered by the viewer on the video being distributed, communication is realized not only between the distributor and the viewer, but also among viewers themselves.

[0005] As a means for realizing communication via a network, for example, Patent Document 1 listed below discloses a means for a plurality of users to communicate with each other on an equal footing in the same virtual space using a text base.

[0006] Furthermore, the following Patent Document 2 discloses a means for users who are using the same content to know each other's status. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2009-211528 A [Patent Document 2] Patent No. 6519468 Summary of the Invention [Problem to be solved by the invention]

[0008] In conventional technology, text and audio information have been the main means of communication between broadcasters and viewers. However, there is a big gap between the experience of communication via these networks and the experience of communication that has traditionally been conducted at close physical distances (where the other person can be seen).

[0009] For example, consider a service in which a distributor captures performers' images and audio data in real time, converts the data into content data for distribution, and distributes it to movie theaters and homes via a network so that customers in remote locations can experience a live music concert held at a limited location. In this case, the content is distributed to movie theaters as content to be projected on a screen, and to homes as content that can be viewed on a television or a head mounted display (HMD).

[0010] In a venue where a live music concert is actually being held, the performer can decide the next action appropriately while checking how many people are in the audience, where they are located, how they are physically distributed, how they are reacting to his / her acting, direction, and performance, etc. Also, from the audience's point of view, since the performer decides the next action while checking their position and reaction, the audience can enjoy the so-called "call and response" performance, such as having the performer look in their direction, or responding to the performer's call, and can feel the reaction of the audience other than themselves, allowing them to experience communication other than listening to music.

[0011] On the other hand, it is difficult to reproduce these communication experiences using conventional communication methods via networks such as text and voice. Therefore, even for the same live music concert audience, there is a large gap between the experience they get at the actual venue and the experience they get at a remote location via a network.

[0012] Furthermore, with the development of network technology, content distribution using live streaming services on an individual basis has become popular. This type of content distribution is often carried out in a form where the audience is not present at the actual shooting location, but only in a remote location via a network. In this environment, performers cannot see the audience directly or feel their gaze, so they do not know where to look or in which direction to face while performing, and are unable to act appropriately. Furthermore, because they cannot see the audience's reaction to their performance, they are unable to decide on their next action appropriately, which is another problem.

[0013] The techniques described in Patent Documents 1 and 2 above also make no disclosure regarding communication between performers appearing in content and viewers, or conveying viewers' reactions to the performers.

[0014] In consideration of the above circumstances, the object of the present technology is to provide an information processing system, an information processing method, and a program that enable performers appearing in content distributed in real time to perform in response to the reactions of viewers in remote locations. [Means for solving the problem]

[0015] In order to achieve the above object, an information processing system according to an embodiment of the present technology includes a control unit. The control unit acquires, from a terminal of a viewer playing back content in which a performer's performance is captured in real time via a network, a line-of-sight parameter indicating the line of sight of the viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information for identifying the viewer. The control unit also converts the acquired line-of-sight parameter into a line-of-sight parameter indicating a virtual line of sight of the viewer in a coordinate system of the space in which the performer exists. The control unit then outputs line-of-sight information indicating the virtual line of sight of the viewer to an output device in the space in which the performer exists, based on the converted line-of-sight parameter.

[0016] This information processing system enables the performer to grasp the virtual gaze of the audience, and allows the performer to perform an appropriate performance according to the audience's reaction, even for audiences in remote locations. The gaze information may be image information or audio information, and may also include the virtual position information of the audience.

[0017] The output device may be a display. In this case, the control unit may calculate, based on the converted line-of-sight parameters, coordinates of an intersection between the display and the virtual line of sight, and output, as the line-of-sight information, an image corresponding to the viewer at a position on the display corresponding to the coordinates of the intersection.

[0018] This allows the performer to know when a remote viewer is looking at him / her simply by looking at the display, and to respond appropriately by turning his / her gaze in response, performing, etc. The image may be, for example, an avatar image of the viewer.

[0019] The control unit may, when the intersection coordinates corresponding to a predetermined number or more of viewers are present in a predetermined area of ​​the display, output one predetermined image showing a group of viewers instead of the images corresponding to each of the viewers.

[0020] This enables the information processing system to prevent images corresponding to a plurality of viewers from being displayed overlapping each other, thereby preventing a decrease in visibility of the performer.

[0021] The control unit may acquire attribute information indicating an attribute of the viewer together with the line-of-sight parameter, and change an output mode of the image in accordance with the attribute information.

[0022] This allows the information processing system to change the image according to the attributes of each viewer, allowing the performer to respond in a detailed manner accordingly. Here, attributes include, for example, age, sex, nationality, place of residence, viewing time, number of times a content in which the same performer appears has been viewed or purchased, distance from the performer in the coordinate system of the content, etc. Changing the attribute mode includes, for example, adding a frame of a different color to the avatar image, changing the size of the avatar image, changing the transparency, etc.

[0023] The control unit may determine whether or not the viewer is directing his / her gaze at the performer based on the converted gaze parameters, and change the output mode of the image depending on the determination result.

[0024] This allows the information processing system to determine whether each viewer is facing the performer or not, and allows the performer to perform in accordance with the viewer's line of sight, for example, by performing toward the viewer whose line of sight is being directed.

[0025] When first intersection coordinates corresponding to a first viewer having first viewer identification information calculated at a first time are different from second intersection coordinates corresponding to the first viewer calculated at a second time after the first time, the control unit may display an image corresponding to the viewer while moving it on a trajectory connecting the first intersection coordinates to the second intersection coordinates.

[0026] This enables the information processing system to allow the performer to grasp the movement of the audience during the performance and perform accordingly.

[0027] The information processing system may further include a storage unit that stores information indicating a plurality of types of effects that can be played together with the image in association with effect identification information that identifies the effect. In this case, when the control unit receives an effect playback request including the viewer identification information and the effect identification information from a terminal of the viewer, the control unit may output an effect corresponding to the effect identification information from a vicinity of the intersection coordinates that correspond to the viewer identification information.

[0028] This allows the information processing system to allow the performer to grasp the audience's reaction and excitement, and to perform accordingly. The effect that is the target of the effect playback request may be linked to any input (gesture, button, etc.) on the viewer's terminal.

[0029] When there are effect playback requests corresponding to the predetermined number or more viewers for a predetermined area of ​​the display, the control unit may output one predetermined effect instead of the effects corresponding to each of the viewers.

[0030] This enables the information processing system to prevent effects corresponding to multiple viewers from being displayed overlapping each other, resulting in reduced visibility of the performer.

[0031] When effect reproduction requests having the same effect identification information are received from the predetermined number or more of viewers, the control unit may output one predetermined effect instead of the effect corresponding to each of the viewers.

[0032] This allows the information processing system to allow the performer to grasp the excitement of the entire audience.

[0033] A plurality of speakers may be provided at different positions on the display. In this case, when the effect corresponding to the effect identification information included in the effect reproduction request is a sound effect, the control unit may output the sound effect from a speaker located in the vicinity of the intersection coordinates corresponding to the viewer identification information.

[0034] This enables the information processing system to reproduce an effect that makes it seem as if the audience is calling out to the performer, and allows the performer to understand this.

[0035] The control unit may acquire gaze parameters indicating the performer's gaze, and if it determines that the absolute value of the dot product of the gaze vector obtained from the performer's gaze parameters and the gaze vector obtained from the gaze parameters indicating the viewer's virtual gaze is less than a predetermined threshold, it may output a predetermined effect from the vicinity of the intersection coordinates corresponding to the viewer identification information.

[0036] This enables the information processing system to allow the performer to recognize when he or she has made eye contact with an audience member and to perform accordingly.

[0037] The control unit may acquire gaze parameters indicating the performer's gaze, count the number of times, for each of a plurality of viewers, that the absolute value of the dot product of the gaze vector obtained from the performer's gaze parameters and the gaze vector obtained from the gaze parameters indicating the virtual gaze of each of the viewers is less than a predetermined threshold, and cause the display to display a histogram in which values ​​corresponding to each number of times for each of the viewers are associated with the vicinity of the intersection coordinates corresponding to each viewer.

[0038] This allows performers to use this histogram to perform in a direction where they make eye contact with the audience less frequently, thereby increasing the satisfaction of the entire audience.

[0039] An information processing method according to another aspect of the present technology includes: acquiring, from a terminal of a viewer in which content in which a performer's performance is captured and played back in real time via a network, a line-of-sight parameter indicating the line of sight of the viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information for identifying the viewer; converting the acquired gaze parameters into gaze parameters that indicate a virtual gaze of the viewer in a coordinate system of a space in which the performer is present; This includes outputting line-of-sight information indicating the virtual line of sight of the viewer based on the converted line-of-sight parameters to an output device within the space in which the performer is present.

[0040] A program according to another aspect of the present technology includes: acquiring, from a terminal of a viewer during real-time playback of content in which a performer's performance is captured via a network, a line-of-sight parameter indicating the line of sight of the viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information for identifying the viewer; converting the acquired gaze parameters into gaze parameters that indicate a virtual gaze of the viewer in a coordinate system of a space in which the performer is present; and outputting line-of-sight information indicating the virtual line of sight of the viewer to an output device within the space in which the performer is present, based on the converted line-of-sight parameters. Effect of the Invention

[0041] As described above, according to the present technology, it is possible to allow performers appearing in content distributed in real time to perform in response to the reactions of viewers in remote locations. However, this effect is not a limitation of the present technology. [Brief description of the drawings]

[0042] [Figure 1] 1 is a diagram illustrating a configuration of a content distribution system according to an embodiment of the present technology. [Diagram 2] FIG. 2 is a diagram showing an example of equipment installation in a studio of the content distribution system. [Diagram 3] 2 is a diagram showing a hardware configuration of a viewer information management server of the content distribution system. FIG. [Figure 4] 10 is a flowchart showing a flow of a content distribution process performed by the content distribution system. [Diagram 5] 13A and 13B are diagrams showing examples of display in the content distribution system when there are viewers with different positions and postures relative to the content. [Figure 6] 10 is a flowchart showing a flow of a viewer information display process performed by the content distribution system. [Figure 7] 13 is a diagram showing an example of presentation of viewer's line of sight information to a performer in the above content distribution system. FIG. [Figure 8] 11A to 11C are diagrams showing examples of display of gaze information in accordance with viewer movement in the content distribution system. [Figure 9] FIG. 13 is a diagram showing a modified example of equipment installation in a studio of the content distribution system. [Figure 10]FIG. 13 is a diagram showing an example in which information of different viewers having different lines of sight are displayed at the same position in the above content distribution system. [Figure 11] 13A and 13B are diagrams showing examples of display of gaze information according to whether the viewer's gaze is directed toward the performer in the content distribution system. [Figure 12] 13 is a table showing the relationship between effects for performers that can be requested by viewers in the above content distribution system and the actions required for performing the effects. [Figure 13] 13A and 13B are diagrams showing examples of effects displayed to a performer from a viewer in the content distribution system. [Figure 14] FIG. 13 is a diagram showing an example of a histogram showing the number of times that a performer and an audience member made eye contact in the content distribution system. [Figure 15] 11 is a flowchart showing a process in which the content distribution system adds an effect to content in response to a request from a viewer. [Figure 16] 1 is a conceptual diagram showing a flow of adding an effect to content in response to a request from a viewer in the content distribution system. [Figure 17] 13A and 13B are diagrams illustrating a modified example of the process of applying an effect to content in the content distribution system. [Figure 18] FIG. 13 is a diagram showing a case where a performer, multiple viewers, and a display have a certain positional relationship in the above content distribution system. [Figure 19] FIG. 19 is a diagram showing a state in which information expressing the viewing states of other viewers is added to the content viewed by a certain viewer in the case of FIG. 18. [Figure 20] In the case of FIG. 18, this figure shows a state in which information expressing the viewing state of a viewer by virtually moving and enlarging the display is added to the content viewed by a certain viewer. [Figure 21] FIG. 13 is a diagram showing an example of playback of a light effect in response to a request from a viewer in the content distribution system. [Figure 22] 22A to 22C are diagrams showing examples of adjustment of the playback position of the effect in FIG. 21. [Diagram 23] 13 is a diagram showing an area in which an effect generation center set around a performer can be set in the above content distribution system. FIG. [Figure 24] FIG. 24 is a diagram showing how the effect generation center is set for each viewer using the areas set in the example of FIG. 23. [Diagram 25] FIG. 13 is a diagram showing an example in which background content having different plane parameters is arranged for each viewer with a different line of sight in the content distribution system. [Figure 26] FIG. 26 is a diagram showing an example of effect reproduction in the case of FIG. 25. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0043] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0044] [System Overview] FIG. 1 is a diagram showing a configuration of a content delivery system according to an embodiment of the present technology.

[0045] FIG. 1 is a diagram showing the overall configuration of the system, and FIG. 2 is a diagram showing an example of equipment installation in a content shooting studio that the system has.

[0046] As shown in both figures, the system comprises a viewer information management server in a content shooting studio, a performer output system 300, a content creation server 400, a content distribution server 500, and multiple viewer output systems 200 connected to these via a network 50 such as the Internet.

[0047] The content creation server 400 uses the above-mentioned studio dedicated to content creation to create content by filming the performers' performance in real time. The created content is streamed to viewers via the network 50.

[0048] The content distributed to viewers is VR (Virtual Reality) content based on 3D models and surround sound. As shown in Fig. 2, the studio is equipped with filming equipment including one or more cameras 51 and microphones 52 for content creation, and a content creation server 400 creates the content to be distributed based on the captured data.

[0049] The viewer information management server 100 appropriately acquires information on the viewing state of the viewer, such as the viewer's virtual line of sight and virtual position, from the viewer output system 200 and manages the information.

[0050] The performer output system 300 has one or more displays 53 for outputting information about the viewing state of the audience to a performer appearing in the content.

[0051] The viewer information management server 100 transmits information such as the viewer's viewing status received from the viewer output system 200 to the content creation server 400, and the content creation server 400 can also make changes to the distributed content in accordance with the information.

[0052] The content created or modified by the content creation server 400 is distributed from the content distribution server 500 via the network 50 to each content viewer (viewer output system 200).

[0053] [Hardware configuration of viewer information management server] FIG. 3 is a diagram showing a hardware configuration of the viewer information management server 100. As shown in FIG.

[0054] As shown in the figure, the viewer information management server 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, and a RAM (Random Access Memory) 13. The viewer information management server 100 may also include a host bus 14, a bridge 15, an external bus 16, an interface 17, an input device 18, an output device 19, a storage device 20, a drive 21, a connection port 22, and a communication device 23. The viewer information management server 100 may also include an imaging device 26 and a sensor 27 as necessary. The viewer information management server 100 may have a processing circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array) instead of or in addition to the CPU 11.

[0055] The CPU 11 functions as an arithmetic processing device and a control device, and controls all or part of the operations in the viewer information management server 100 according to various programs recorded in the ROM 12, the RAM 13, the storage device 20, or the removable recording medium 24. The ROM 12 stores programs and arithmetic parameters used by the CPU 11. The RAM 13 temporarily stores programs used in the execution of the CPU 11 and parameters that change appropriately during the execution. The CPU 11, the ROM 12, and the RAM 13 are interconnected by a host bus 14 that is constituted by an internal bus such as a CPU bus. Furthermore, the host bus 14 is connected to an external bus 16 such as a viewer information management server I (Peripheral Component Interconnect / Interface) bus via a bridge 15.

[0056] The input device 18 is a device operated by a user, such as a touch panel, a physical button, a switch, or a lever. The input device 18 may be, for example, a remote control device using infrared or other radio waves, or an external connection device 25 such as a smartphone or a smart watch that supports the operation of the viewer information management server 100. The input device 18 includes an input control circuit that generates an input signal based on information input by the user and outputs the signal to the CPU 11. The user operates the input device 18 to input various data to the viewer information management server 100 and to instruct processing operations.

[0057] The output device 19 is configured with a device capable of notifying the user of acquired information using senses such as vision, hearing, and touch. The output device 19 can be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker. The output device 19 outputs the results obtained by the processing of the viewer information management server 100 as video such as text or images, sound such as voice or audio, or vibration.

[0058] The storage device 20 is a data storage device configured as an example of a storage unit of the viewer information management server 100. The storage device 20 is configured, for example, with a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 20 stores, for example, programs and various data executed by the CPU 11, various data acquired from the outside, and data acquired from the viewer output system 200 (such as gaze parameters and avatar images of each viewer, which will be described later).

[0059] The drive 21 is a reader / writer for a removable recording medium 24 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built into or attached externally to the viewer information management server 100. The drive 21 reads out information recorded on the attached removable recording medium 24 and outputs it to the RAM 13. The drive 21 also writes information onto the attached removable recording medium 24.

[0060] The connection port 22 is a port for connecting a device to the viewer information management server 100. The connection port 22 may be, for example, a Universal Serial Bus (USB) port, an IEEE1394 port, a Small Computer System Interface (SCSI) port, or the like. The connection port 22 may also be an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, or the like. By connecting an external connection device 25 to the connection port 22, various types of data may be exchanged between the viewer information management server 100 and the external connection device 25.

[0061] The communication device 23 is, for example, a communication interface configured with a communication device for connecting to the communication network 50. The communication device 23 may be, for example, a communication card for a LAN (Local Area Network), Bluetooth (registered trademark), Wi-Fi, or WUSB (Wireless USB). The communication device 23 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 23 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The communication network 50 connected to the communication device 23 is a network connected by wire or wirelessly, and may include, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0062] The imaging device 26 is a camera that captures real space and generates a captured image using an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) and various members such as a lens for controlling the formation of a subject image on the imaging element. The imaging device 26 may capture a still image or a moving image.

[0063] The sensor 27 is, for example, one of various sensors such as an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, an illuminance sensor, a temperature sensor, an air pressure sensor, a depth sensor, or a sound sensor (microphone).

[0064] Each of the above components may be configured using general-purpose materials, or may be configured using hardware specialized for the function of each component. Such configurations may be appropriately changed depending on the technical level at the time of implementation.

[0065] Although not shown, the viewer output system 200, the performer output system 300, the content creation server 400 and the content distribution server 500 also have hardware for functioning as a computer, similar to the viewer information management server 100 described above.

[0066] [Content distribution system operation] Next, the operation of the content distribution system configured as above will be described. The operation is performed by the cooperation of hardware such as the CPU 11 and communication unit of the viewer information management server 100 and software stored in the ROM 12, RAM 13, storage device 20, or removable recording medium 24.

[0067] Fig. 4 is a diagram showing a flowchart of the content distribution process, and Fig. 5 is a diagram showing an example of content display when there are viewers with different positions and orientations with respect to the content.

[0068] The content viewers receive the content and view the content through their own viewer output systems 200. Here, the viewer output systems 200 are, for example, head-mounted displays with a head tracking function capable of estimating the position and orientation of the viewer's head.

[0069] As shown in FIG. 4, the viewer output system 200 initializes the position and orientation of the viewer's head in the content coordinate system (the coordinate system in the space in which the performer exists) (step 41), and estimates the position and orientation of the viewer's head in the content coordinate system using a head tracking function (step 42).

[0070] The viewer output system 200 then projects the delivered 3D content onto the virtual image plane according to this position and orientation (step 43), and outputs the projected content to a display (step 44).

[0071] One example of a technology that can realize a head tracking function is the Simultaneous Localization And Mapping (SLAM) technology that uses a camera and an Inertial Measurement Unit (IMU) sensor.

[0072] Binocular stereoscopic vision, which is generally used to view VR content, requires the position and orientation of the viewer's left and right eyes, which can be calculated using offsets from the estimated head position to the two eyes.

[0073] As shown in Figure 5, viewer 1 viewing the content from directly beside him (Figure 5A) and viewer 2 viewing the content from directly in front (Figure 5B) will see the content differently depending on the position and posture of their heads in the content coordinate system.

[0074] Regardless of whether or not the head tracking function is present, instead of the viewer output system 200 estimating and using the viewer's actual head position and orientation, the viewer can virtually move the head position and orientation using an input device such as a controller.

[0075] The content distribution system of this embodiment is capable of presenting the performer with the viewer's virtual line of sight information (including the viewer's virtual position information) and effects indicating the viewer's reaction during the content distribution process. On the other hand, the content distribution system is capable of adding effects indicating the viewer's reaction to the content during the content distribution process. Details of these processes will be described below.

[0076] [Displaying viewers' gaze information to the performer] FIG. 6 is a flowchart showing the flow of the process of presenting information on the viewer's line of sight and effects to the performer.

[0077] As shown in the flow chart on the left side of the figure, first, viewer output system 200 calculates viewer line-of-sight parameters in the content coordinate system (step 51).

[0078] The viewer output system 200 may obtain this by converting gaze parameters previously defined in a head mounted display coordinate system (the coordinate system of the space in which the viewer exists) into the content coordinate system, or, if the viewer output system 200 has a device for estimating the viewer's gaze direction in real time, by converting those parameters into the content coordinate system.

[0079] Although the gaze parameters may be output separately for the right eye and the left eye, it is considered here that the gaze parameters are limited to one parameter in some way, such as by adopting one of them or by taking the average of the left and right. Also, instead of using the position of the viewer's eyes, viewer output system 200 may use, for example, a straight line connecting the performer's head position and the viewer's head position as the gaze parameter, on the premise that the viewer is always facing the performer. Also, viewer output system 200 may determine the gaze parameter by setting a specific direction in the body coordinate system of the head mounted display as the gaze direction.

[0080] The gaze parameters in the content coordinate system may be calculated by the viewer information management server 100 in the studio, instead of the viewer output system 200. In this case, the viewer output system 200 transmits the viewer's gaze parameters in the head mounted display coordinate system to the viewer information management server 100, and the viewer information management server 100 converts the gaze parameters into gaze parameters in the content coordinate system.

[0081] Next, the viewer output system 200 transmits the viewer's line of sight parameters expressed in the content coordinate system to the viewer information management server 100 (step 52).

[0082] The viewer information management server 100 performs processing required by the performer output system 300 on the line-of-sight parameters sent from each viewer.

[0083] For example, when the viewer information management server 100 outputs an avatar image of a viewer as gaze information in the performer output system 300, the viewer information management server 100 may perform a process of linking gaze parameters with the avatar image of the viewer who sent the information.

[0084] The viewer information management server 100 (CPU 11) has position and orientation information in the content coordinate system of the display 53 installed in the studio, and calculates the coordinates of the intersection between the display 53 and the viewer's line of sight, based on the viewer's line of sight parameters also expressed in the content coordinate system (step 53).

[0085] For example, if the performer output system 300 is composed of multiple displays 53, the viewer information management server 100 can calculate the coordinates of the intersection of the display 53 and the viewer's line of sight by expressing each display 53 with a plane equation and expressing the viewer's line of sight parameters with a straight line equation.

[0086] This can also be applied to the case where the display 53 has a curved surface. Alternatively, the viewer information management server 100 may convert the line of sight parameters into each display coordinate system, and then obtain the intersection coordinates in each display coordinate system.

[0087] Next, the viewer information management server 100 (CPU 11) causes the performer output system 300 to output the viewer's line of sight information to the display 53 in a form recognizable by the performer, based on the calculated intersection coordinates (step 54).

[0088] This can be achieved by converting the intersection coordinate I expressed in the content coordinate system into the display coordinate system and displaying avatar images 71 of each viewer at the corresponding position, as shown in Fig. 7. In the example of the figure, avatar images 71a, 71b, and 71c corresponding to the intersection coordinate I of the virtual line of sight VL1 of viewer 1, the virtual line of sight VL2 of viewer 2, and the virtual line of sight VL3 of viewer 3 and display 53 are displayed.

[0089] 8, by looking at avatar image 71 displayed on display 53, performer P can recognize the line of sight of viewer V in a remote location and the direction in which viewer V is located in real time, and can take appropriate action, such as directing his / her gaze there or performing toward there. In the example of the figure, as viewer V moves his / her line of sight L from right to left in time series from t1, t2, to t3 as shown in FIG. 8B, the virtual line of sight VL also moves as shown in FIG. 8A, and the avatar image 71 also moves accordingly.

[0090] This also allows viewer V to have a communication experience (e.g., making eye contact) as if he or she were physically close to performer P.

[0091] [Modifications regarding the presentation of gaze information] A modified example of presenting the viewer's line of sight information to the performer will be described below.

[0092] When the number of viewers increases, the viewers' gazes may be concentrated on the same coordinates on the display 53 of the performer output system 300. In this case, if an avatar image 71 or the like set for each viewer is displayed, the multiple avatar images 71 may overlap, reducing the visibility of the performers.

[0093] In such a case, viewer information management server 100 may cause performer output system 300 to replace multiple avatar images 71 for each viewer with other images or the like that represent the concentration of the gazes of the multiple viewers.

[0094] For example, when X or more viewpoints are gathered within an area on the display 53 that is equal to or smaller than a predetermined area, the performer output system 300 may replace the group of avatar images 71 of the viewers with image A, and when Y or more viewpoints are gathered within the above-mentioned area, the performer output system 300 may replace the group of avatar images 71 with image B, which is different from image A. Also, instead of the avatar images 71, the performer output system 300 may display on the display 53 a heat map that indicates the concentration of gazes.

[0095] The viewer information management server 100 may use viewer attribute information managed by the viewer information management server 100 or viewer attribute information that has been attached to gaze parameter information obtained from the viewer output system 200 to change or process the viewer's avatar image 71 displayed on the display of the performer output system 300.

[0096] For example, viewer information management server 100 may add a frame of a different color to avatar image 71, change the size of avatar image 71, or change the transparency depending on the viewer's age, sex, nationality, place of residence, viewing time, number of times content featuring the same performer has been viewed or purchased, distance from the performer in the content coordinate system, and other parameters linked to the viewer.

[0097] As the video output device of the performer output system 300, a device other than the display 53 may be used. For example, when a projector is used, the viewer information management server 100 can calculate the position where the viewer's avatar image 71, etc. should be drawn, in the same way as when the display 53 is used, by expressing the plane onto which the projector is projected in the content coordinate system.

[0098] In addition, in order to improve the visibility of the display 51 for the performers, instead of the configuration shown in Figure 2, a display device in which multiple cameras 51 and microphones 52 are embedded (for example, in a matrix form) on the same plane as the display 53 may be used, as shown in Figure 9.

[0099] As shown in FIG. 10, even if information of different viewers, such as avatar images 71, is displayed at the same position on the display 53 in the performer output system 300, it is possible that the viewers are not directing their gaze toward the performers (the gaze VL1 is directed toward performer P, but the gaze VL2 is not directed toward performer P).

[0100] Therefore, in order to convey more accurate gaze information of viewer V to performer P, the viewer information management server 100 may change the size or frame color of avatar image 71, or may not display avatar image 71 itself, depending on whether viewer V's gaze is directed toward performer P in the content coordinate system or not, as shown in Figure 11, for example.

[0101] In the example shown in the figure, virtual lines of sight VL1 and VL2 are directed toward performer P, and therefore the corresponding avatar images 71A and 71B are displayed at normal size; however, virtual line of sight VL3 is not directed toward performer P, and therefore the corresponding avatar image 71C is displayed smaller than avatar images 71A and 71B.

[0102] Whether or not the viewer's line of sight is directed toward performer P can be determined, for example, by whether or not the performer is included within a viewing cone of any size centered on the viewer's line of sight.

[0103] The position of the viewer information (avatar image 71) displayed on the performer output system 300 may be updated at any interval. When the intersection position c(t) of a viewer's line of sight and the display is different from the intersection position c(t-1) for the same viewer that was calculated immediately before, the viewer information management server 100 may move the viewer information so that it moves on a trajectory connecting the two intersection points.

[0104] [Presenting the effect of the audience on the performers] Next, a method for conveying information such as the audience's reaction and excitement to the performers by displaying additional information other than the audience's line of sight in the performer output system 300 will be described.

[0105] By obtaining this information, performers can give appropriate performances for the audience. The content distributor creates effects that the audience can request to the performer output system to play, such as the effect table in Fig. 12A. In this effect table, effect IDs that identify effects are associated with the contents of the effects that they indicate. This effect table is stored, for example, in the storage device 20 of the audience information management server 100.

[0106] Each viewer registers an action for issuing a playback request for each effect according to the input device that the viewer owns, as in the viewer action tables of B1 to B3 of Fig. 12. The action here means the input of a specific command or movement to a device owned by viewer output system 200.

[0107] As shown in the flow on the right side of FIG. 6, first, viewer output system 200 acquires the effect ID of the effect that is the subject of a playback request from the viewer's action (step 61).

[0108] For example, viewer 1 in Fig. 12B makes repeated up and down movements of his / her head to issue a request to play an effect in the performer output system 300 for the effect with effect ID: 1000. A viewer with a viewing environment that has a head tracking function may use head movement as in the case of viewer 1 in Fig. 12B, and a viewer using a motion controller may use a specific motion as in the case of viewer 2.

[0109] Next, the viewer output system 200 transmits an effect reproduction request corresponding to the effect ID to the viewer information management server 100 (step 62).

[0110] The effect reproduction request of each viewer is sent to the viewer information management server 100 as data in which a viewer ID for identifying the viewer is associated with the effect ID.

[0111] Then, based on the intersection coordinates, viewer information management server 100 (CPU 11) reproduces the effect corresponding to the effect ID at a position in performer output system 300 corresponding to the intersection coordinates (eg, near avatar image 71) (step 63).

[0112] For example, as shown in FIG. 13, in response to a playback request from viewer 1, visual effect 72 with effect ID: 1004 (rainbow) in FIG. 12A is played, in response to a playback request from viewer 2, visual effect 72 with effect ID: 1003 (star) in FIG. 12A is played, and in response to a playback request from viewer 3, visual effect 72 with effect ID: 1000 (speech bubble comment "cute") in FIG. 12A is played.

[0113] [Variations of the process of presenting effects to performers] Below, a modified example of the process of presenting effects to the performer will be described.

[0114] As with the above-mentioned gaze information, when the number of viewers increases, effect playback requests may be concentrated around the same coordinates on the display 51 of the performer output system 300. In this case, if the effects requested by each viewer are played back, multiple effects may overlap, reducing the visibility of the performer.

[0115] In such a case, the viewer information management server 100 may cause the performer output system 300 to replace the effects for the multiple viewers with other effects that represent the concentration of the multiple effect playback requests, and play them back.

[0116] For example, if the same effect playback request is received from X or more people within a specified area on the display 53, the performer output system 300 may replace the effects for each viewer with a special effect that represents a concentration of effects and play it back.

[0117] The viewer information management server 100 may use viewer attribute information managed by the viewer information management server 100 or viewer attribute information that has been attached to gaze parameter information obtained from the viewer output system 200 to control the size of the effects played by the performer output system 300 and the types of effects that can be requested.

[0118] For example, the viewer information management server 100 may control the types of effects that can be requested according to the viewing time of the viewer, the number of times a content in which the same performer appears has been viewed or purchased, or other parameters associated with the viewer.

[0119] The viewer information management server 100 may play back effects that are not dependent on the gaze position in order to express the excitement of the entire audience.

[0120] For example, if the viewer information management server 100 receives the same effect playback request from a specific number of viewers or more at a specific timing, it may cause the performer output system 300 to play a special effect indicating this (such as a visual effect displayed across the entire display 53).

[0121] The performer output system 300 may also include an audio reproduction device, such as a speaker, allowing the audience to request sound effects as well as visual effects.

[0122] For example, by using a display device in which multiple speakers are embedded in display 53 in a format similar to that of Figure 9, it becomes possible to play the sound effect requested by the viewer who issued a request to play the sound effect from a speaker near avatar image 71 (intersection coordinate I) corresponding to the viewer.

[0123] [Adding effects to content] Next, a method for adding specific effects to the distributed content in response to the actions of the viewers, thereby allowing viewers watching the same distributed content to know in real time how each other is reacting to the content, will be described.

[0124] Similar to the effects presented to the performers, the content distributor creates effects that viewers can request to be applied to the distributed content. Similarly, each viewer registers an action for issuing a request to apply each effect according to the input device that he or she owns. A table relating to these effects (for example, one having a format similar to that shown in FIG. 12) is also stored, for example, in the storage device 20 of the viewer management server 100.

[0125] Fig. 15 is a flowchart showing the flow from a viewer's request for applying an effect to the delivery of VR content with the applied effect to the viewer. Fig. 16 is a conceptual diagram showing the flow.

[0126] 15, first, the CPU 11 of the viewer information management server 100 receives an effect application request for each viewer from the viewer output system 200 of each viewer (step 151). The effect application request is received as data in which a viewer ID and an effect ID are associated with each other.

[0127] Next, the CPU 11 identifies the effect ID from the effect application request (step 152).

[0128] Next, the CPU 11 transmits an effect application request including the effect ID to the content creating server 400 (step 153).

[0129] The content to which the effect corresponding to the effect ID has been applied by the content creation server 400 is then distributed from the content distribution server 500 to the viewer output system 200 (step 154).

[0130] The effect application request may be sent directly to the content creating server 400 without passing through the viewer information management server 100 .

[0131] As shown in Fig. 16, when a visual request for, for example, illuminating the area around performer P is sent from viewer 2 to viewer information management server 100, the effect is applied to the content by content creation server 400 and distributed to each viewer's viewer output system 200. This allows each viewer to view the applied effect from different line of sight L1, L2, L3. In particular, viewers 1 and 3 can know in real time how viewer 2 is reacting to the content.

[0132] [Variations regarding the application of effects to content] A modified example of the process of applying an effect to content will be described below.

[0133] The same modifications as those described above in the process of presenting effects to the performer are possible.

[0134] That is, as with the gaze information and effects presented to the performer, as the number of viewers increases, effect application requests may be concentrated around the same position in the content (for example, around the performer). In this case, if the effects requested by each viewer are applied, multiple effects may overlap, reducing viewer visibility.

[0135] In such a case, the viewer information management server 100 may cause the content creation server 400 to replace the effects of the multiple viewers with other effects that express the concentration of the multiple effect application requests.

[0136] For example, when the content creation server 400 receives the same effect application request from X or more viewers, the content creation server 400 may apply a special effect that replaces the effect of each viewer with a special effect that expresses a concentration of the effect.

[0137] The viewer information management server 100 may use viewer attribute information managed by the viewer information management server 100 or viewer attribute information that has been added to gaze parameter information obtained from the viewer output system 200 to control the size of the effect added to the content and the type of effect that can be requested.

[0138] For example, the viewer information management server 100 may control the types of effects that can be requested to be applied depending on the viewing time of the viewer, the number of times a content in which the same performer appears has been viewed or purchased, or other parameters associated with the viewer.

[0139] Furthermore, when the content creation server 400 receives the same effect application request from a specific number of viewers or more at a specific timing, the content creation server 400 may apply a special effect indicating this (for example, a visual effect that is displayed across the entire content).

[0140] The viewer information management server 100 may use viewer attribute information managed by the viewer information management server 100 or viewer attribute information that has been added to gaze parameter information to change the VR content without the viewer intentionally issuing a request.

[0141] For example, as shown in Fig. 17, the viewer information management server 100 stores a viewer residential area attribute table indicating the number of viewers for each residential area around the world. The content creation server 400 may then change the display size of a 3D model of a landmark representative of each area (e.g., Tokyo Tower in Japan, the Statue of Liberty in the United States, the Leaning Tower of Pisa in Italy, the Merlion in Singapore, etc.) and synthesize it against the background of the performer P to create content, depending on the number of viewers in that residential area.

[0142] In this example, the number of viewers is largest in Japan, followed by the United States, Italy, and Singapore, so the sizes of the 3D models in the background of the content are set in the following order: Tokyo Tower, the Statue of Liberty, the Leaning Tower of Pisa, and the Merlion.

[0143] [Example of extension of effects to content] By using the gaze information and position information of the viewer and the gaze information and position information of the performer, it is possible to more effectively share the viewing state between viewers and add effects to the content. Some examples are described below.

[0144] - Share viewing status with other viewers One method of sharing viewing status is to add the viewing status (viewer's position, etc.) of other viewers to the distributed content and then view the content. In this case, if the viewing status of other viewers is added to the distributed content unconditionally, problems may occur, such as the additional content (effects) appearing in a position that interferes with the viewing of the distributed content, or the distributed content being buried in the additional content and becoming unviewable.

[0145] Fig. 18 shows a case where a performer P, multiple virtual viewers V, and display 53 have a certain positional relationship. In this case, if avatar content expressing the viewing states of other viewers based on the viewing positions of the other viewers or the positions of the avatar images 71 of the other viewers is added to the content viewed by viewer V1, the added content (avatar content 72) will appear near intersection coordinate I within the viewing cone of viewer 1, as shown in Fig. 19, which may interfere with viewer V1's viewing of the distributed content.

[0146] To address this problem, the content creation server 400 virtually moves and enlarges the display 53 based on the position of the viewer V1, as shown in Figure 20, and uses the intersection of the line of sight of the other viewers and the virtual display 53 as the display position for the additional content, thereby making it possible to add content that represents the viewing state of the other viewers to the distributed content without interfering with the viewing of viewer V1.

[0147] The position and size of the virtual display may be changed as desired. For example, the content creation server 400 may set the virtual display so that it is always located behind the viewing position of the viewer V1 with respect to the content.

[0148] Furthermore, the content creating server 400 may use any plane or sphere, or a combination thereof, instead of a virtual display to find an intersection with the lines of sight of other viewers, and use this as the display position of the additional content.

[0149] When there are a large number of viewers, adding avatar images 71 of all viewers to the content may increase the processing load on content creation server 400 or may result in an overcrowding of added content, degrading the overall quality of the content.

[0150] In contrast, the viewer information management server 100 may share the viewing state only with members of a group or community to which each viewer belongs (obtained, for example, from an SNS, etc.). The content creation server 400 may replace the viewer's avatar content 72 with an image that is more simply drawn (lower resolution).

[0151] -Adjusting the effect playback position When the effect applied by the effect application request is shared among viewers, the effect application position may be appropriately adjusted. Three specific cases are described here, but the present invention is not limited to these.

[0152] (Case 1: Effect played within each viewer's viewing cone) If an effect is played in response to a request from another viewer, the viewer will not notice it unless the effect is played within their viewing cone.

[0153] For example, consider an effect called "Random Rays" that plays a random ray effect in the content coordinate system, as shown in Figure 21. When this effect is played, the number of rays each viewer can see will vary depending on the position of each viewer's viewing cone.

[0154] In response to this, the content creation server 400 can maintain a consistent quality of the playback effect seen by each viewer by adjusting the playback position of the effect so that the effect is played back within the viewing cone of each viewer.

[0155] Figure 22 shows how the effect playback position of "Random Rays" requested by other viewers is adjusted to match the viewing cone VC of viewer 1. The upper diagram in the figure shows the state before adjustment, and the lower diagram shows the state after adjustment. In the upper diagram, the playback positions of ray2 and ray4, which were positioned outside the viewing cone VC, have been adjusted so that they are visible within the viewing cone VC in the lower diagram.

[0156] The creation of the viewing cone VC that determines the effect reproduction position in this way may be centered on the viewer's line of sight or on the viewer's head.

[0157] (Case 2: Adjust the center of the effect according to the gaze direction of each viewer and the position of the performer) If an effect requested by one viewer is also applied to content distributed to other viewers, this could disrupt viewing for viewers who are viewing the distributed content from a different perspective than the viewer who made the request.

[0158] For example, an effect requested by another viewer may be reproduced in the space between the viewer and the performer. One possible way to avoid this problem is to stop the reproduction of an effect with or without a specific attribute in the viewer output system 200 of the viewer when the effect is about to be reproduced in the space between the viewer and the performer. However, if this method is used, there is a possibility that a viewer may not be able to see an effect requested by another viewer with a different viewpoint.

[0159] To address this issue, each viewer output system 200 may adjust the center of the playback effect depending on the viewing direction of each viewer and the position of the performer.

[0160] Here, we consider setting the effect generation center in the performer coordinate system defined based on the performer's position. The effect generation center is the reference coordinate that determines the playback position of an effect with or without a specific attribute.

[0161] Fig. 23 shows area A where an effect generation center can be set with performer P at the center. As an example, a circle with a radius of r [m] horizontal to the ground and a height of h [m] is set with the performer P at its center, but this does not limit the method of setting area A.

[0162] Fig. 24 shows how the effect generation center C is set for each viewer using the above-mentioned set area A. The effect generation center C is set by mapping the line of sight L of each viewer onto the plane in which the effect generation center settable area A exists, and by determining the intersection of the mapped line of sight and the effect generation center settable area A, the intersection is set as the farthest from the viewer.

[0163] In the figure, it can be seen that different effect generation centers C1 to C3 are set for each viewer. In the state shown in the figure, for example, when viewer 2 (corresponding to line of sight L2) issues an effect playback request, viewer output system 200A for viewer 1 (corresponding to line of sight L1) plays an effect at the position of effect generation center C1 for viewer 1, and output system 200C for viewer 3 plays an effect at the position of effect generation center C3 for viewer 3.

[0164] This allows any viewer to view the effects requested by other viewers without interfering with the viewing of the distributed content.

[0165] The setting process of the above-mentioned area A and effect generation center C may be performed not by each viewer output system 200, but by the content creation server 400, which receives the gaze parameters of each viewer from each viewer output system 200 via the viewer information management server 100 or directly.

[0166] (Case 3: Adjusting the effect playback position according to each viewer's line of sight and effect attributes) For effects with specific attributes, the playback position is adjusted using the viewer's line of sight and the attributes of the background content, making it possible to play effects appropriate to each viewer.

[0167] For example, let us define that the effect to be played has a text attribute, and that the effect with the text attribute is played on the plane of the background content with a certain extent or more. In Fig. 25, the background contents Plane1 and Plane2, which have different plane parameters, are placed ahead of the line of sight (L1 and L2) of the viewers 1 and 2, respectively.

[0168] In this case, if a request to play an effect having a text attribute occurs, viewer output system 200A of viewer 1 plays the effect on Plane 1, and viewer output system 200B of viewer 2 plays the effect on Plane 2, as shown in FIG. 26.

[0169] This allows effects with text attributes to be played in a form appropriate for each viewer. If there is no suitable plane in the line of sight of the viewer, it is possible to create a virtual plane behind the performer as seen by the viewer and play the effect on that plane.

[0170] The display process of each of the above background contents may be performed by the content creation server 400, rather than by each viewer output system 200, by receiving the gaze parameters of each viewer from each viewer output system 200 via the viewer information management server 100 or directly.

[0171] -Adding effects to distributed content and processing position When a large number of effect playback requests occur, problems such as delays in distribution due to the increased amount of processing required to add effects to the distributed content and an increase in communication data can occur. In order to avoid these problems, it is conceivable to filter playback requests using viewer line-of-sight information.

[0172] For example, the content creation server 400 may reflect, in the content to be distributed to a certain viewer, only effect playback requests from other viewers who have similar line-of-sight parameters to the viewer.

[0173] It is also possible to adjust the effect of an effect depending on the number of simultaneous viewers. For example, in the case of the "Random Rays" effect described in Figures 21 and 22, if the number of simultaneous viewers is X or more, the content creation server 400 may set the number of rays to be played back in response to one playback request to n, and if the number of simultaneous viewers is Y or more (greater than X), the number may be y (greater than x).

[0174] Furthermore, for effects whose playback position is determined according to the viewer's gaze information, the content creation server 400 does not apply them to the content distributed from the content distribution server 500, but instead transmits information about the effect to each viewer's output system 200 to have the effect applied, thereby reducing the load on the content creation server 400 and the content distribution server 500.

[0175] -Use different playback methods depending on the effect attributes In addition to the above, the following attributes can be used to change the playback method depending on the attributes of the effect.

[0176] For example, possible effects include effects that change or do not change the display posture depending on the viewer's line of sight, effects that change or do not change the display posture depending on the direction of the performer, effects that are not displayed within the viewing cone between the viewer and performer, effects that are played using the distance between the viewer and performer as a parameter, and effects that are played using the degree of alignment of the gazes of the performer and viewer as a parameter.

[0177] As described above, according to this embodiment, the content distribution system enables the performer to grasp the virtual gaze of the audience in the same space as the performer, and enables the performer to perform appropriately in response to the audience's reaction, even for viewers in remote locations.

[0178] As a result of the above, performers and audience members can communicate with each other even when they are in distant locations, as if they were physically close to each other.

[0179] In addition, each viewer can set effect playback requests according to their own input device, so the same request can be sent from any device (differences due to device ownership are absorbed).

[0180] Furthermore, since the viewer's actions are mapped to an effect ID and then sent to the viewer information management server 100, the amount of communication data required to express the viewer's actions is significantly reduced.

[0181] Additionally, because content providers create effects that can be requested, malicious actions that may occur, for example, in arbitrary text-enabled communications, are eliminated.

[0182] Furthermore, the content distribution system allows viewers who are viewing the same content to share their experiences by reflecting the viewer's actions in the distributed content.

[0183] In addition, the content delivery system can differentiate the services provided to each viewer by controlling the effects that can be requested to be played and applied for each viewer.

[0184] [Variations] The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention.

[0185] In the above-mentioned embodiment, when it is determined that a specific communication has been established between a performer and a viewer, the viewer information providing server 100 or the content creating server 300 may enhance the communication experience by adding a special effect to the distribution content for the target viewer or to the distribution content for all viewers. Examples of the establishment of a specific communication include when the performer and a viewer make eye contact, or when a viewer requests the playback of a specific effect in response to a specific performance by the performer.

[0186] The viewer information management server 100 or the content creation server 300 can determine whether the gaze of the performer and viewer has met by, for example, checking whether the performer's gaze is directed in the direction of a viewer's avatar image 71 on the display and whether the absolute value of the dot product of the gaze vectors of the performer and viewer is less than a predetermined threshold value such that both gaze vectors are approximately parallel.

[0187] In addition, when the performer and the viewer make eye contact, the viewer information management server 100 may output a special visual effect or sound effect from the vicinity of the avatar image (intersection coordinates) on the display 53 that corresponds to the viewer.

[0188] Furthermore, the viewer information management server 100 can count the number of times that the gaze of each viewer was met, and display the values ​​indicating the frequency of gaze meeting in each direction as a histogram in association with each intersection coordinate I of the display 53. Based on this information, the performer can increase the satisfaction of the entire audience by performing in a direction in which the gaze of the viewer was met less frequently.

[0189] 14 shows an example of a frequency histogram 73 indicating the above frequency, displayed on the display 53. The value of the frequency histogram 73 can be calculated by dividing the sum of the number of times that eye contact occurred with viewers in each direction by the number of viewers in that direction.

[0190] In addition, by enabling performers and viewers in remote locations to communicate in a manner that is similar to a real-life experience, it will be possible to add value to specific viewing locations on live streaming services.

[0191] For example, a content distribution system can create high-added-value viewing locations by assuming that performers will frequently communicate toward a specific viewing location and charging viewers who use that location a higher viewing fee than usual while limiting the number of viewers who can use that viewing location.

[0192] In the above embodiment, the content is shot by the camera 51 fixed in the shooting studio. However, instead of the camera 51, the content may be shot by, for example, a drone moving around.

[0193] [others] The present technology can also be configured as follows. (1) acquiring, from a terminal of a viewer in which content in which a performer's performance is captured and played back in real time via a network, a line-of-sight parameter indicating the line of sight of the viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information for identifying the viewer; converting the acquired line of sight parameters into line of sight parameters indicating a virtual line of sight of the viewer in a coordinate system of a space in which the performer is present; Based on the converted line of sight parameters, line of sight information indicating the virtual line of sight of the viewer is output to an output device in the space in which the performer is present. Control Unit An information processing system comprising: (2) The information processing system according to (1) above, the output device is a display; The control unit calculates an intersection coordinate between the display and the virtual line of sight based on the converted line of sight parameters, and outputs an image corresponding to the viewer at a position on the display corresponding to the intersection coordinate as the line of sight information. Information processing system. (3) The information processing system according to (2) above, When the intersection coordinates corresponding to a predetermined number or more of viewers are present in a predetermined area of ​​the display, the control unit outputs one predetermined image showing a group of viewers instead of the images corresponding to each of the viewers. Information processing system. (4) The information processing system according to (2) or (3), The control unit acquires attribute information indicating an attribute of the viewer together with the line-of-sight parameter, and changes an output mode of the image in accordance with the attribute information. Information processing system. (5) The information processing system according to any one of (2) to (4), The control unit determines whether the viewer is directing his / her gaze at the performer based on the converted gaze parameter, and changes the output mode of the image in accordance with the determination result. Information processing system. (6) The information processing system according to any one of (2) to (5), When first intersection coordinates corresponding to a first viewer having first viewer identification information calculated at a first time point are different from second intersection coordinates corresponding to the first viewer calculated at a second time point after the first time point, the control unit displays an image corresponding to the viewer while moving the image on a locus connecting the first intersection coordinates to the second intersection coordinates. Information processing system. (7) The information processing system according to any one of (2) to (6), a storage unit that stores information indicating a plurality of types of effects that can be reproduced together with the image in association with effect identification information that identifies the effect; When the control unit receives an effect reproduction request including the viewer identification information and the effect identification information from the terminal of the viewer, the control unit outputs an effect corresponding to the effect identification information from a vicinity of the intersection coordinates corresponding to the viewer identification information. Information processing system. (8) The information processing system according to (7) above, When an effect reproduction request corresponding to the predetermined number or more viewers exists for a predetermined area of ​​the display, the control unit outputs one predetermined effect instead of the effect corresponding to each of the viewers. Information processing system. (9) The information processing system according to (7) or (8), When the control unit receives effect reproduction requests having the same effect identification information from the predetermined number or more of viewers, the control unit outputs one predetermined effect instead of the effect corresponding to each of the viewers. Information processing system. (10) The information processing system according to any one of (7) to (9), A plurality of speakers are provided at different positions on the display; When the effect corresponding to the effect identification information included in the effect reproduction request is a sound effect, the control unit outputs the sound effect from a speaker located in the vicinity of the intersection coordinates corresponding to the viewer identification information. Information processing system. (11) The information processing system according to any one of (2) to (10), The control unit acquires a line-of-sight parameter indicating the line of sight of the performer, and when it determines that an absolute value of an inner product of a line-of-sight vector obtained from the line-of-sight parameter of the performer and a line-of-sight vector obtained from a line-of-sight parameter indicating a virtual line of sight of the viewer is less than a predetermined threshold, outputs a predetermined effect from the vicinity of the intersection coordinates corresponding to the viewer identification information. Information processing system. (12) The information processing system according to any one of (2) to (10), The control unit acquires gaze parameters indicating the gaze of the performer, counts the number of times that an absolute value of an inner product of a gaze vector obtained from the gaze parameters of the performer and a gaze vector obtained from a gaze parameter indicating a virtual gaze of each of the viewers is less than a predetermined threshold, and displays on the display a histogram in which a value corresponding to each count of the number of times for each of the viewers is associated with a neighborhood of the intersection coordinates corresponding to each of the viewers. Information processing system. (13) acquiring, from a terminal of a viewer in which content in which a performer's performance is captured and played back in real time via a network, a line-of-sight parameter indicating the line of sight of the viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information for identifying the viewer; converting the acquired line of sight parameters into line of sight parameters indicating a virtual line of sight of the viewer in a coordinate system of a space in which the performer is present; Based on the converted line of sight parameters, line of sight information indicating the virtual line of sight of the viewer is output to an output device in the space in which the performer is present. Information processing methods. (14) In the information processing device, acquiring, from a terminal of a viewer on which content in which a performer's performance is captured and played back in real time via a network, a line-of-sight parameter indicating the line of sight of the viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information for identifying the viewer; converting the acquired gaze parameters into gaze parameters indicating a virtual gaze of the viewer in a coordinate system of a space in which the performer is present; outputting line-of-sight information indicating the virtual line of sight of the viewer to an output device in a space in which the performer is present, based on the converted line-of-sight parameters; A program that executes the following. [Explanation of symbols]

[0194] 11...CPU 18...Input device 19...Output device 20...Storage device 26...Imaging device 23...Communication equipment 51...Camera 52…Mike 53…Display 71...Avatar image 72...Effects 73...Histogram 100...Viewer information management server 200…Performer output system 300... Viewer output system 400: Content creation server 500: Content distribution server P...Performer V: Viewer L… Gaze VL…Virtual line of sight

Claims

1. acquiring, from a terminal of a viewer in which content in which a performer's performance is captured and played back in real time via a network, a line-of-sight parameter indicating the line of sight of the viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information for identifying the viewer; converting the acquired line of sight parameters into line of sight parameters indicating a virtual line of sight of the viewer in a coordinate system of a space in which the performer is present; Based on the converted line of sight parameters, line of sight information indicating the virtual line of sight of the viewer is output to a display in the space in which the performer is present. Control Unit Equipped with The control unit calculates an intersection coordinate between the display and the virtual line of sight based on the converted line of sight parameters, and outputs an image corresponding to the viewer at a position on the display corresponding to the intersection coordinate as the line of sight information. Information processing system.

2. 2. The information processing system according to claim 1, When the intersection coordinates corresponding to a predetermined number or more of viewers are present in a predetermined area of ​​the display, the control unit outputs one predetermined image showing a group of viewers instead of the images corresponding to each of the viewers. Information processing system.

3. 2. The information processing system according to claim 1, The control unit acquires attribute information indicating an attribute of the viewer together with the line-of-sight parameter, and changes an output mode of the image in accordance with the attribute information. Information processing system.

4. 2. The information processing system according to claim 1, The control unit determines whether the viewer is directing his / her gaze at the performer based on the converted gaze parameter, and changes the output mode of the image in accordance with the determination result. Information processing system.

5. 2. The information processing system according to claim 1, When first intersection coordinates corresponding to a first viewer having first viewer identification information calculated at a first time and second intersection coordinates corresponding to the first viewer calculated at a second time after the first time are different, the control unit displays an image corresponding to the viewer while moving it on a trajectory connecting the first intersection coordinates to the second intersection coordinates. Information processing system.

6. 2. The information processing system according to claim 1, a storage unit that stores information indicating a plurality of types of effects that can be reproduced together with the image in association with effect identification information that identifies the effect; When the control unit receives an effect reproduction request including the viewer identification information and the effect identification information from the terminal of the viewer, the control unit outputs an effect corresponding to the effect identification information from a vicinity of the intersection coordinates corresponding to the viewer identification information. Information processing system.

7. 7. The information processing system according to claim 6, When effect reproduction requests corresponding to a predetermined number or more viewers exist for a predetermined area of ​​the display, the control unit outputs one predetermined effect instead of the effect corresponding to each of the viewers. Information processing system.

8. 7. The information processing system according to claim 6, When the control unit receives effect reproduction requests having the same effect identification information from a predetermined number or more of viewers, the control unit outputs one predetermined effect instead of the effect corresponding to each of the viewers. Information processing system.

9. 7. The information processing system according to claim 6, A plurality of speakers are provided at different positions on the display; When the effect corresponding to the effect identification information included in the effect reproduction request is a sound effect, the control unit outputs the sound effect from a speaker located in the vicinity of the intersection coordinates corresponding to the viewer identification information. Information processing system.

10. 2. The information processing system according to claim 1, The control unit acquires a line-of-sight parameter indicating the line of sight of the performer, and when it determines that an absolute value of an inner product of a line-of-sight vector obtained from the line-of-sight parameter of the performer and a line-of-sight vector obtained from a line-of-sight parameter indicating a virtual line of sight of the viewer is less than a predetermined threshold, outputs a predetermined effect from the vicinity of the intersection coordinates corresponding to the viewer identification information. Information processing system.

11. 2. The information processing system according to claim 1, The control unit acquires gaze parameters indicating the gaze of the performer, counts the number of times that an absolute value of an inner product of a gaze vector obtained from the gaze parameters of the performer and a gaze vector obtained from a gaze parameter indicating a virtual gaze of each of the viewers is less than a predetermined threshold, and displays on the display a histogram in which a value corresponding to each count of the number of times for each of the viewers is associated with a neighborhood of the intersection coordinates corresponding to each of the viewers. Information processing system.

12. acquiring, from a terminal of a viewer in which content in which a performer's performance is captured and played back in real time via a network, a line-of-sight parameter indicating the line of sight of the viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information for identifying the viewer; converting the acquired line of sight parameters into line of sight parameters indicating a virtual line of sight of the viewer in a coordinate system of a space in which the performer is present; outputting line-of-sight information indicating the virtual line of sight of the viewer to a display in the space in which the performer is present, based on the converted line-of-sight parameters; Based on the converted line-of-sight parameters, coordinates of an intersection between the display and the virtual line of sight are calculated, and an image corresponding to the viewer is output at a position on the display corresponding to the coordinates of the intersection as the line-of-sight information. Information processing methods.

13. In the information processing device, acquiring, from a terminal of a viewer on which content in which a performer's performance is captured and played back in real time via a network, a line-of-sight parameter indicating the line of sight of the viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information for identifying the viewer; converting the acquired gaze parameters into gaze parameters indicating a virtual gaze of the viewer in a coordinate system of a space in which the performer is present; outputting line-of-sight information indicating the virtual line of sight of the viewer to a display in a space in which the performer is present, based on the converted line-of-sight parameters; calculating an intersection coordinate between the display and the virtual line of sight based on the converted line of sight parameters, and outputting, as the line of sight information, an image corresponding to the viewer at a position on the display corresponding to the intersection coordinate; A program that executes the following.

Images