Information processing system, information processing method and program

Abstract

This information processing system includes a control unit. The control unit acquires, from a plurality of viewer terminals currently engaged in real-time reproduction of content obtained by imaging performer's performance over a network, sets of viewing state information indicating the sight lines or the positions of viewers on a coordinate system of a space in which the viewers are present together with identification information identifying the viewers. The control unit adds effects to the content for the respective viewers on the basis of the respective acquired sets of viewing state information.

Description

[Technical Field]

[0001] The present technology relates to an information processing system, an information processing method, and a program that can present information about viewers to performers in a system that distributes content filmed from a performer's performance to viewers in real time via a network. [Background technology]

[0002] With the development of network technology, content distribution services to remote locations are continuing to grow. For example, video distribution, including movie content, has been realized through a one-way system in which content data is sent from the distributor to the viewer, who then enjoys the content.

[0003] In recent years, for example, in real-time video streaming services, interactive systems have been realized that allow streamers and viewers to communicate with each other. These systems provide new value not only in the content itself, but also in the experience of communication through the content.

[0004] In such systems, the main means of communication between viewers and broadcasters are text and audio. In particular, text input by viewers is superimposed on the broadcast video, enabling communication not only between broadcasters and viewers, but also between viewers themselves.

[0005] As a means for realizing communication via a network, for example, Patent Document 1 below discloses a means for a plurality of users to communicate with each other on an equal footing in the same virtual space using text.

[0006] Furthermore, Patent Document 2 below discloses a means for users who are using the same content to understand each other's status. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-211528 [Patent Document 2] Patent No. 6519468 Summary of the Invention [Problem to be solved by the invention]

[0008] In conventional technology, text and audio information have been the primary means of communication between broadcasters and viewers. However, there is a significant difference in the experience between communication via these networks and communication that has traditionally taken place at close physical distances (where the other person can be seen).

[0009] For example, consider a service in which a distributor captures performers' images and audio data in real time, converts this data into content data for distribution, and distributes it over a network to movie theaters and homes so that customers in remote locations can experience a live music concert held in a limited location. In this case, the content is distributed to movie theaters as content to be projected on a screen, and to homes as content that can be viewed on television or with an HMD (Head Mounted Display).

[0010] At an actual live music concert venue, performers can determine their next steps by checking how many people are in the audience, where they are located, how they are physically distributed, and how they are reacting to their performance. Furthermore, as audience members decide their next steps based on the performers' position and reactions, they can enjoy the "call and response" effect of having people look in their direction, respond to the performers' calls, and feel the reactions of the audience members other than themselves, providing a communication experience beyond just listening to music.

[0011] However, it is difficult to recreate these communication experiences using conventional communication methods such as text or voice over a network. Therefore, even for the same live music concert audience, there is a large gap between the experience they get at the actual venue and the experience they get at a remote location via a network.

[0012] There are also systems that allow viewers to apply effects to content distributed over a network, but because the appearance of such effects is the same for all viewers, viewers are unable to get a strong sense of realism from the content.

[0013] In particular, in content such as VR (Virtual Reality) content, where each viewer can view from a virtual line of sight (position), if the effects appear uniformly, the experience will lack realism.

[0014] In view of the above circumstances, the object of the present technology is to provide an information processing system, an information processing method, and a program that can add effects to content distributed in real time according to the line of sight and position of viewers in remote locations. [Means for solving the problem]

[0015] To achieve the above object, an information processing system according to one embodiment of the present technology includes a control unit. The control unit acquires, from terminals of a plurality of viewers currently playing content in which a performer's performance has been captured via a network in real time, viewing state information indicating the line of sight or position of each viewer in a coordinate system of a space in which the viewer exists, along with viewer identification information that identifies the viewer. The control unit also applies an effect to the content for each viewer based on the acquired viewing state information.

[0016] This allows the information processing system to apply effects to content delivered in real time according to the line of sight and position of remote viewers. Here, effects include everything from color and light effects to avatar images.

[0017] The control unit may acquire attribute information indicating attributes of the viewer together with the viewing state information, and change the effect in accordance with the attribute information.

[0018] When applying the effect to a first viewer among the plurality of viewers, the control unit may calculate the intersection coordinates between a virtual plane set in the coordinate system of the space and the line of sight of a second viewer different from the first viewer, and apply the effect to a position of the content corresponding to the intersection coordinates.

[0019] The control unit may set the virtual plane behind a viewing position of the first viewer in the coordinate system of the space.

[0020] This allows the information processing system to add effects to the content that represent the viewing states of other viewers without interfering with the viewing of the content by the viewer.

[0021] The control unit may set a lower resolution for each effect according to the second viewers as the number of the second viewers increases.

[0022] This allows the information processing system to prevent a deterioration in the quality of the content itself due to an increase in the processing load for adding effects caused by an increase in the number of other viewers viewing the content.

[0023] If the coordinates of the intersection with the line of sight of the second viewer are not included within a viewing cone in the coordinate system of the first viewer, the control unit may change the position at which the effect is applied to within the viewing cone.

[0024] This allows the information processing system to allow a viewer to always view the effects applied by other viewers, regardless of the gaze positions of the other viewers.

[0025] The control unit may apply the effect for each viewer at a position corresponding to the coordinates of the intersection between an area set centered on the position of the performer in a coordinate system of the space in which the performer exists and the line of sight of the multiple viewers, which is farther from each viewer.

[0026] This allows the information processing system to add effects to the content for each viewer and play it back so as not to interfere with the viewing of each viewer who is focusing on the performers in the content.

[0027] The control unit applies the effect in response to an effect application request received from each viewer's terminal, the effect request including effect identification information indicating the attributes of the effect, and if the attribute indicated by the effect identification information is a planar effect, the control unit may set a predetermined plane for each viewer behind the performer in the coordinate system of the space in which the performer exists and in the line of sight of the viewer, and apply the effect on the predetermined plane.

[0028] This allows the information processing system to reproduce effects having specific attributes in a form suitable for each viewer.

[0029] The control unit applies the effect in response to an effect application request received from the terminal of each of the viewers, and when applying the effect to a first viewer among the multiple viewers, the control unit may apply the effect only in response to an effect application request from a second viewer, among the multiple viewers, whose line of sight or position is within a predetermined distance from the line of sight or position of the first viewer.

[0030] This allows the information processing system to filter out only those requests that are close to the viewer, rather than applying effects in response to all effect application requests, thereby preventing delays in content distribution and increases in communication data due to increased processing volume.

[0031] The control unit may acquire information indicating the number of terminals of viewers currently playing the content, and increase the effectiveness of the applied effect in accordance with the number.

[0032] In this way, the information processing system can allow the viewer to grasp the number of other viewers watching at the same time through the effect of the effect. Here, the effect of the effect means, for example, the number of rays in the case of a light effect, or the brightness in the case of a light effect, but is not limited to these.

[0033] An information processing method according to another aspect of the present technology includes: Obtaining viewing state information indicating the line of sight or position of each viewer in a coordinate system of the space in which the viewer exists, together with viewer identification information that identifies each viewer, from each of the terminals of a plurality of viewers who are playing back content in which a performer's performance has been captured via a network in real time; This includes adding an effect to the content for each viewer based on the acquired viewing state information.

[0034] A program according to another aspect of the present technology includes: a step of acquiring viewing state information indicating the line of sight or position of each viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information that identifies the viewer, from each of the terminals of a plurality of viewers who are playing back content in which a performer's performance has been filmed in real time via a network; and a step of applying an effect to the content for each viewer based on the acquired viewing state information. [Effects of the Invention]

[0035] As described above, according to the present technology, it is possible to add effects to content distributed in real time according to the line of sight and position of a remote viewer. However, the present technology is not limited to these effects. [Brief explanation of the drawings]

[0036] [Figure 1] 1 is a diagram illustrating a configuration of a content distribution system according to an embodiment of the present technology. [Figure 2] FIG. 2 is a diagram showing an example of equipment installation in a studio of the content distribution system. [Figure 3] 2 is a diagram showing the hardware configuration of a viewer information management server included in the content distribution system. FIG. [Figure 4] 10 is a flowchart showing a flow of a content distribution process performed by the content distribution system. [Figure 5] 10A and 10B are diagrams showing display examples in the case where there are viewers with different positions and orientations relative to the content in the content distribution system. [Figure 6] 10 is a flowchart showing the flow of a viewer information display process performed by the content distribution system. [Figure 7] 10A and 10B are diagrams showing examples of presentation of viewer line-of-sight information toward a performer in the content distribution system. [Figure 8] 10A and 10B are diagrams showing examples of displaying line-of-sight information in accordance with viewer movement in the content distribution system. [Figure 9] FIG. 10 is a diagram showing a modified example of equipment installation in a studio of the content distribution system. [Figure 10] FIG. 10 is a diagram showing an example in which information about different viewers with different lines of sight is displayed at the same position in the content distribution system. [Figure 11] 10A and 10B are diagrams showing examples of displaying gaze information according to whether the viewer's gaze is directed toward the performer in the content distribution system. [Figure 12] 10 is a table showing the relationship between effects that can be requested by viewers for performers and the actions required for those effects in the content distribution system. [Figure 13] 10A and 10B are diagrams showing examples of effects displayed to performers from viewers in the content distribution system. [Figure 14] FIG. 10 is a diagram showing an example of a histogram display showing the number of times a performer's gaze meets a viewer's gaze in the content distribution system. [Figure 15] 10 is a flowchart showing a flow in which the content distribution system applies an effect to content in response to a request from a viewer. [Figure 16] FIG. 10 is a conceptual diagram showing a flow of adding an effect to content in response to a request from a viewer in the content distribution system. [Figure 17] 10A and 10B are diagrams illustrating a modified example of the process of applying an effect to content in the content distribution system. [Figure 18] FIG. 10 is a diagram showing a case where a performer, multiple viewers, and a display have a certain positional relationship in the content distribution system. [Figure 19] FIG. 19 is a diagram showing how information expressing the viewing states of other viewers is added to the content viewed by a certain viewer in the case of FIG. 18. [Figure 20] In the case of FIG. 18, this figure shows a state in which information expressing the viewing state of a viewer by virtually moving and enlarging the display is added to the content being viewed by a certain viewer. [Figure 21] 10A and 10B are diagrams showing an example of playback of a light effect in response to a request from a viewer in the content distribution system. [Figure 22] 22A and 22B are diagrams showing an example of adjusting the playback position of the effect in FIG. 21. [Figure 23] FIG. 10 is a diagram showing an area in which an effect generation center set around a performer can be set in the content distribution system. [Figure 24] FIG. 24 is a diagram showing how the effect generation center is set for each viewer using the areas set in the example of FIG. 23. [Figure 25] FIG. 10 is a diagram showing an example in which background content having different plane parameters is arranged for each viewer with a different line of sight in the content distribution system. [Figure 26] FIG. 26 is a diagram showing an example of effect reproduction in the case of FIG. 25. DETAILED DESCRIPTION OF THE INVENTION

[0037] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0038] [System Overview] FIG. 1 is a diagram showing a configuration of a content distribution system according to an embodiment of the present technology.

[0039] FIG. 1 is a diagram showing the overall configuration of the system, and FIG. 2 is a diagram showing an example of equipment installation in a content shooting studio that the system has.

[0040] As shown in both figures, this system comprises a viewer information management server in a content shooting studio, a performer output system 300, a content creation server 400, a content distribution server 500, and multiple viewer output systems 200 connected to these via a network 50 such as the Internet.

[0041] The content creation server 400 uses the studio dedicated to content creation to create content by filming the performers' performances in real time. The created content is streamed to viewers via the network 50.

[0042] The content delivered to viewers is VR (Virtual Reality) content based on 3D models and surround sound. As shown in Fig. 2, the studio is equipped with filming equipment including one or more cameras 51 and microphones 52 for content creation, and a content creation server 400 creates the content to be delivered based on the captured data.

[0043] The viewer information management server 100 appropriately acquires information on the viewer's viewing state, such as the viewer's virtual line of sight and virtual position, from the viewer output system 200 and manages the information.

[0044] The performer output system 300 has one or more displays 53 for outputting information about the viewing state of the audience to the performers appearing in the content.

[0045] The viewer information management server 100 transmits information such as the viewer's viewing status received from the viewer output system 200 to the content creation server 400, and the content creation server 400 can also make changes to the distributed content in accordance with the information.

[0046] The content created and modified by the content creation server 400 is distributed from the content distribution server 500 to each content viewer (viewer output system 200) via the network 50.

[0047] [Hardware configuration of viewer information management server] FIG. 3 is a diagram showing the hardware configuration of the viewer information management server 100. As shown in FIG.

[0048] As shown in the figure, the viewer information management server 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, and a RAM (Random Access Memory) 13. The viewer information management server 100 may also include a host bus 14, a bridge 15, an external bus 16, an interface 17, an input device 18, an output device 19, a storage device 20, a drive 21, a connection port 22, and a communication device 23. The viewer information management server 100 may also include an imaging device 26 and a sensor 27 as necessary. The viewer information management server 100 may have a processing circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array) instead of or in addition to the CPU 11.

[0049] The CPU 11 functions as an arithmetic processing device and control device, and controls all or part of the operations within the viewer information management server 100 in accordance with various programs recorded in the ROM 12, the RAM 13, the storage device 20, or the removable recording medium 24. The ROM 12 stores programs and calculation parameters used by the CPU 11. The RAM 13 temporarily stores programs used in the execution of the CPU 11 and parameters that change as appropriate during the execution. The CPU 11, the ROM 12, and the RAM 13 are interconnected by a host bus 14, which is constituted by an internal bus such as a CPU bus. Furthermore, the host bus 14 is connected via a bridge 15 to an external bus 16, such as a viewer information management server I (Peripheral Component Interconnect / Interface) bus.

[0050] The input device 18 is a device operated by a user, such as a touch panel, physical buttons, switches, or levers. The input device 18 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device 25 such as a smartphone or smartwatch that is compatible with operation of the viewer information management server 100. The input device 18 includes an input control circuit that generates an input signal based on information input by the user and outputs the signal to the CPU 11. By operating the input device 18, the user inputs various data to the viewer information management server 100 and instructs the viewer information management server 100 to perform processing operations.

[0051] The output device 19 is configured as a device capable of notifying the user of acquired information using senses such as sight, hearing, and touch. The output device 19 can be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker. The output device 19 outputs the results obtained by the processing of the viewer information management server 100 as video such as text or images, sound such as voice or audio, or vibration.

[0052] Storage device 20 is a data storage device configured as an example of a storage unit of viewer information management server 100. Storage device 20 is configured, for example, by a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. Storage device 20 stores, for example, programs and various data executed by CPU 11, various data acquired from the outside, and data acquired from viewer output system 200 (such as gaze parameters and avatar images of each viewer, which will be described later).

[0053] The drive 21 is a reader / writer for a removable recording medium 24 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the viewer information management server 100. The drive 21 reads information recorded on the attached removable recording medium 24 and outputs the information to the RAM 13. The drive 21 also writes information onto the attached removable recording medium 24.

[0054] The connection port 22 is a port for connecting a device to the viewer information management server 100. The connection port 22 may be, for example, a USB (Universal Serial Bus) port, an IEEE1394 port, or a SCSI (Small Computer System Interface) port. The connection port 22 may also be an RS-232C port, an optical audio terminal, or an HDMI (registered trademark) (High-Definition Multimedia Interface) port. By connecting an externally connected device 25 to the connection port 22, various types of data can be exchanged between the viewer information management server 100 and the externally connected device 25.

[0055] The communication device 23 is, for example, a communication interface configured with a communication device for connecting to the communication network 50. The communication device 23 may be, for example, a communication card for a local area network (LAN), Bluetooth (registered trademark), Wi-Fi, or wireless USB (WUSB). The communication device 23 may also be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), or a modem for various types of communication. The communication device 23 transmits and receives signals between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The communication network 50 connected to the communication device 23 is a wired or wireless network and may include, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0056] The imaging device 26 is a camera that captures real space and generates a captured image using various components such as an imaging element, such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device), and a lens for controlling the formation of a subject image on the imaging element. The imaging device 26 may capture a still image or a moving image.

[0057] The sensor 27 is, for example, one of various sensors such as an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, an illuminance sensor, a temperature sensor, an air pressure sensor, a depth sensor, or a sound sensor (microphone).

[0058] Each of the above components may be configured using general-purpose components or may be configured using hardware specialized for the function of each component. Such configurations may be modified as appropriate depending on the technical level at the time of implementation.

[0059] Although not shown, the viewer output system 200, the performer output system 300, the content creation server 400, and the content distribution server 500 also have hardware for functioning as computers, similar to the viewer information management server 100.

[0060] [Content distribution system operation] Next, the operation of the content distribution system configured as above will be described. The operation is performed by the cooperation of hardware such as the CPU 11 and communication unit of the viewer information management server 100 and software stored in the ROM 12, RAM 13, storage device 20, or removable recording medium 24.

[0061] Fig. 4 is a diagram showing a flowchart of a display process for content distributed to viewer output system 200. Fig. 5 is a diagram showing an example of content display when there are viewers with different positions and orientations relative to the content.

[0062] The content viewers receive the content and view it through their own viewer output systems 200. Here, the viewer output systems 200 are, for example, head-mounted displays with a head tracking function that can estimate the position and orientation of the viewer's head.

[0063] As shown in Figure 4, the viewer output system 200 initializes the position and orientation of the viewer's head in the content coordinate system (the coordinate system in the space where the performer exists) (step 41), and estimates the position and orientation of the viewer's head in the content coordinate system using the head tracking function (step 42).

[0064] Next, viewer output system 200 projects the distributed 3D content onto a virtual image plane according to this position and orientation (step 43), and outputs the projected content to a display (step 44).

[0065] One example of a technology that realizes head tracking functionality is SLAM (Simultaneous Localization And Mapping) technology, which uses a camera and an IMU (Inertial Measurement Unit) sensor.

[0066] Binocular stereoscopic vision, which is generally used to view VR content, requires the position and orientation of the viewer's left and right eyes, which can be calculated using the offset from the estimated head position to the two eyes.

[0067] As shown in Figure 5, viewer 1 (A in the same figure) viewing the content from the side in the content coordinate system and viewer 2 (B in the same figure) viewing the content from the front will see the content differently depending on the position and posture of their heads.

[0068] Regardless of whether or not head tracking functionality is provided, instead of the viewer output system 200 estimating and using the viewer's actual head position and orientation, the viewer can virtually move the head position and orientation using an input device such as a controller.

[0069] The content distribution system of this embodiment is capable of presenting the performer with virtual viewer line-of-sight information (including virtual viewer position information) and effects indicating the viewer's reaction during the content distribution process. Furthermore, the content distribution system is also capable of adding effects indicating the viewer's reaction to the content during the content distribution process. Details of these processes are described below.

[0070] [Displaying audience gaze information to the performer] FIG. 6 is a flowchart showing the flow of the process of presenting information on the viewer's line of sight and effects to the performer.

[0071] As shown in the flow on the left side of the figure, first, viewer output system 200 calculates the viewer's line of sight parameters in the content coordinate system (step 51).

[0072] The viewer output system 200 may determine this by converting gaze parameters that are predefined in a head-mounted display coordinate system (the coordinate system of the space in which the viewer exists) into the content coordinate system, or, if the viewer output system 200 has a device that estimates the viewer's gaze direction in real time, by converting those parameters into the content coordinate system.

[0073] Although the gaze parameter may be output separately for the right eye and the left eye, here we consider limiting it to one parameter in some way, such as by using one of them or by taking the average of the left and right. Furthermore, instead of using the viewer's eye position, viewer output system 200 may use, for example, a straight line connecting the performer's head position and the viewer's head position as the gaze parameter, assuming that the viewer is always facing the performer. Furthermore, viewer output system 200 may determine the gaze parameter by using a specific direction in the body coordinate system of the head-mounted display as the gaze direction.

[0074] The gaze parameters in the content coordinate system may be calculated by the viewer information management server 100 in the studio, rather than by the viewer output system 200. In this case, the viewer output system 200 transmits the viewer's gaze parameters in the head-mounted display coordinate system to the viewer information management server 100, and the viewer information management server 100 converts the gaze parameters into gaze parameters in the content coordinate system.

[0075] Next, viewer output system 200 transmits the viewer's line of sight parameters expressed in the content coordinate system to viewer information management server 100 (step 52).

[0076] The viewer information management server 100 performs processing required by the performer output system 300 on the line-of-sight parameters sent from each viewer.

[0077] For example, when the viewer information management server 100 outputs an avatar image of a viewer as gaze information in the performer output system 300, the viewer information management server 100 may perform a process of linking the gaze parameters with the avatar image of the viewer who sent the information.

[0078] The viewer information management server 100 (CPU 11) has position and orientation information in the content coordinate system of the display 53 installed in the studio, and calculates the coordinates of the intersection between the display 53 and the viewer's line of sight based on the viewer's line of sight parameters also expressed in the content coordinate system (step 53).

[0079] For example, if the performer output system 300 is composed of multiple displays 53, the viewer information management server 100 can calculate the coordinates of the intersection of the display 53 and the viewer's line of sight by expressing each display 53 with a plane equation and expressing the viewer's line of sight parameters with a linear equation.

[0080] This can also be applied to the case where the display 53 has a curved surface. Alternatively, the viewer information management server 100 may convert the line-of-sight parameters into each display coordinate system, and then find the intersection coordinates in each display coordinate system.

[0081] Next, the viewer information management server 100 (CPU 11) causes the performer output system 300 to output the viewer's line of sight information to the display 53 in a form that can be recognized by the performer, based on the calculated intersection coordinates (step 54).

[0082] This can be achieved by converting the intersection coordinate I expressed in the content coordinate system into the display coordinate system and displaying avatar images 71 of each viewer at the corresponding position, as shown in Fig. 7. In the example of Fig. 7, avatar images 71a, 71b, and 71c corresponding to the intersection coordinate I of the virtual line of sight VL1 of viewer 1, the virtual line of sight VL2 of viewer 2, and the virtual line of sight VL3 of viewer 3 and display 53 are displayed.

[0083] As shown in Fig. 8, by looking at avatar image 71 displayed on display 53, performer P can recognize the line of sight of viewer V in a remote location and the direction in which viewer V is located in real time, and can take appropriate action, such as directing his / her gaze there or performing in that direction. In the example shown in Fig. 8, as viewer V moves his / her line of sight L from right to left in time series from t1 to t2 to t3 as shown in Fig. 8B, the virtual line of sight VL also moves as shown in Fig. 8A, and the avatar image 71 also moves accordingly.

[0084] This also allows the viewer V to have a communication experience (for example, making eye contact) as if they were physically close to the performer P.

[0085] [Modifications regarding presentation of gaze information] A modified example of presenting the viewer's line of sight information to the performer will be described below.

[0086] As the number of viewers increases, the viewers' gazes may converge on the same coordinates on display 53 of performer output system 300. In this case, if an avatar image 71 or the like set for each viewer is displayed, multiple avatar images 71 may overlap, reducing the visibility of the performer.

[0087] In such a case, viewer information management server 100 may cause performer output system 300 to replace multiple avatar images 71 for each viewer with other images that represent the concentration of the gazes of multiple viewers.

[0088] For example, when X or more viewpoints are gathered in an area on the display 53 that is equal to or smaller than a predetermined area, the performer output system 300 may replace the group of viewer avatar images 71 with image A, and when Y or more viewpoints are gathered in the same area, the performer output system 300 may replace the group of viewer avatar images 71 with image B, which is different from image A. Furthermore, instead of avatar images 71, the performer output system 300 may display a heat map on the display 53 that indicates the degree of gaze concentration.

[0089] The viewer information management server 100 may use the viewer attribute information managed by the viewer information management server 100 or the viewer attribute information attached to the gaze parameter information obtained from the viewer output system 200 to change or process the viewer avatar image 71 displayed on the display of the performer output system 300.

[0090] For example, the viewer information management server 100 may add a frame of a different color to the avatar image 71, change the size of the avatar image 71, or change the transparency depending on the viewer's age, gender, nationality, place of residence, viewing time, number of times content featuring the same performer has been viewed or purchased, distance from the performer in the content coordinate system, and other parameters linked to the viewer.

[0091] It is also possible to use a device other than display 53 as the video output device of performer output system 300. For example, when a projector is used, viewer information management server 100 can calculate the position where viewer avatar image 71, etc. should be drawn, in the same way as when display 53 is used, by expressing the plane onto which the projector is projecting in the content coordinate system.

[0092] In addition, in order to improve the visibility of the display 51 to the performers, instead of the configuration shown in Figure 2, a display device may be used in which multiple cameras 51 and microphones 52 are embedded (for example, in a matrix) on the same plane as the display 53, as shown in Figure 9.

[0093] As shown in Figure 10, in the performer output system 300, even if different viewer information, such as avatar image 71, is displayed at the same position on the display 53, it is possible that the viewer may not be looking toward the performer (gaze VL1 is directed toward performer P, but gaze VL2 is not directed toward performer P).

[0094] Therefore, in order to convey more accurate gaze information of viewer V to performer P, the viewer information management server 100 may change the size or frame color of avatar image 71, or may not display avatar image 71 itself, depending on whether viewer V's gaze is directed toward performer P in the content coordinate system or not, as shown in Figure 11, for example.

[0095] In the example shown in the figure, virtual gazes VL1 and VL2 are directed toward performer P, and therefore the corresponding avatar images 71A and 71B are displayed at normal size, but virtual gaze VL3 is not directed toward performer P, and therefore the corresponding avatar image 71C is displayed smaller than avatar images 71A and 71B.

[0096] Whether or not the viewer's line of sight is directed toward performer P can be determined, for example, by whether or not the performer is included within a viewing cone of any size centered on the viewer's line of sight.

[0097] The position of the viewer information (avatar image 71) displayed on the performer output system 300 may be updated at any interval. If the intersection position c(t) of a viewer's line of sight and the display differs from the intersection position c(t-1) for the same viewer calculated immediately before, the viewer information management server 100 may move the viewer information so that it moves along a trajectory connecting the two intersection points.

[0098] [Presenting the effect of the audience on the performer] Next, a method for conveying information such as audience reaction and excitement to the performer by displaying additional information other than the viewer's line of sight in the performer output system 300 will be described.

[0099] By obtaining this information, performers can deliver appropriate performances for the audience. The content distributor creates effects that viewers can request to the performer output system to play, such as the effect table in FIG. 12A. The effect table associates effect IDs that identify effects with the contents of the effects they represent. The effect table is stored, for example, in the storage device 20 of the viewer information management server 100.

[0100] Each viewer registers an action for issuing a playback request for each effect according to the input device he or she owns, as shown in the viewer action tables of Figures 12B1 to 12B3. Here, an action means inputting a specific command or movement to a device owned by viewer output system 200.

[0101] As shown in the flow on the right side of FIG. 6, first, viewer output system 200 acquires the effect ID of the effect to be requested to be played back from the viewer's action (step 61).

[0102] For example, viewer 1 in Fig. 12B makes repeated up and down movements of his / her head to issue a request to play an effect in the performer output system 300 for the effect with effect ID: 1000. A viewer in a viewing environment with a head tracking function may use head movements as in viewer 1 in Fig. 12B, while a viewer using a motion controller may use a specific motion as in viewer 2.

[0103] Next, the viewer output system 200 transmits an effect reproduction request corresponding to the effect ID to the viewer information management server 100 (step 62).

[0104] The effect reproduction request from each viewer is sent to the viewer information management server 100 as data in which the viewer ID for identifying the viewer is associated with the effect ID.

[0105] Then, based on the intersection coordinates, the viewer information management server 100 (CPU 11) plays the effect corresponding to the effect ID at a position on the performer output system 300 corresponding to the intersection coordinates (for example, near the avatar image 71) (step 63).

[0106] For example, as shown in FIG. 13, in response to a playback request from viewer 1, visual effect 72 with effect ID: 1004 (rainbow) in FIG. 12A is played, in response to a playback request from viewer 2, visual effect 72 with effect ID: 1003 (star) in FIG. 12A is played, and in response to a playback request from viewer 3, visual effect 72 with effect ID: 1000 (speech bubble comment "cute") in FIG. 12A is played.

[0107] [Variations of the process of presenting effects to performers] A modified example of the effect presentation process for the performer will be described below.

[0108] As with the above-mentioned gaze information, as the number of viewers increases, effect playback requests may be concentrated around the same coordinates on the display 51 of the performer output system 300. In this case, if the effects requested by each viewer are played back, multiple effects may overlap, reducing the visibility of the performer.

[0109] In such a case, the viewer information management server 100 may cause the performer output system 300 to replace the effects of the multiple viewers with other effects that represent the concentration of the multiple effect playback requests, and then play them.

[0110] For example, if the same effect playback request is received from X or more people within a specified area on the display 53, the performer output system 300 may replace each viewer's effect with a special effect that expresses a concentration of effects and play it back.

[0111] The viewer information management server 100 may use the viewer attribute information managed by the viewer information management server 100 or the viewer attribute information attached to the gaze parameter information obtained from the viewer output system 200 to control the size of the effects played by the performer output system 300 and the types of effects that can be requested.

[0112] For example, the viewer information management server 100 may control the types of effects that can be requested depending on the viewing time of the viewer, the number of times content featuring the same performer has been viewed or purchased, or other parameters linked to the viewer.

[0113] The viewer information management server 100 may play back effects that are not dependent on the gaze position in order to express the excitement of the entire audience.

[0114] For example, if the viewer information management server 100 receives the same effect playback request from a specific number of viewers at a specific timing, it may cause the performer output system 300 to play a special effect indicating this (for example, a visual effect displayed across the entire display 53).

[0115] The performer output system 300 may also include an audio reproduction device, such as a speaker, to allow the audience to request sound effects as well as visual effects.

[0116] For example, by using a display device in which multiple speakers are embedded in display 53 in a format similar to that shown in Figure 9, it becomes possible to play the sound effect requested by the viewer who issued the sound effect playback request from a speaker near avatar image 71 (intersection coordinate I) corresponding to that viewer.

[0117] [Adding effects to content] Next, we will explain how viewers who are watching the same content can know how each other is reacting in real time by adding specific effects to the content in response to the viewer's actions.

[0118] Similar to the effects presented to the performers, content distributors create effects that viewers can request to be applied to distributed content. Each viewer also registers an action for issuing a request to apply each effect, based on their own input device. A table related to these effects (e.g., one with a format similar to that shown in FIG. 12) is also stored, for example, in the storage device 20 of the viewer management server 100.

[0119] Fig. 15 is a flowchart showing the flow from a viewer's request for applying an effect to the delivery of VR content with the applied effect to the viewer, and Fig. 16 is a conceptual diagram showing this flow.

[0120] 15, first, the CPU 11 of the viewer information management server 100 receives an effect application request for each viewer from the viewer output system 200 of each viewer (step 151). The effect application request is received as data in which the viewer ID and the effect ID are associated with each other.

[0121] Next, the CPU 11 identifies the effect ID from the effect application request (step 152).

[0122] Next, the CPU 11 transmits an effect application request including the effect ID to the content creation server 400 (step 153).

[0123] Then, the content to which the effect corresponding to the effect ID has been added by the content creation server 400 is distributed from the content distribution server 500 to the viewer output system 200 (step 154).

[0124] The effect application request may be sent directly to the content creation server 400 without going through the viewer information management server 100 .

[0125] 16, when viewer 2 sends a visual request to the viewer information management server 100 to brighten the area around performer P, the content creation server 400 applies the effect to the content and distributes it to each viewer's viewer output system 200. This allows each viewer to view the applied effect from different gaze points L1, L2, and L3. In particular, viewers 1 and 3 can know in real time how viewer 2 is reacting to the content.

[0126] [Variations regarding the addition of effects to content] A modified example of the process of applying an effect to content will be described below.

[0127] The same modifications as those described above in the process of presenting effects to the performer are possible.

[0128] In other words, as with the gaze information and effects presented to the performer, as the number of viewers increases, effect application requests may be concentrated around the same position in the content (for example, around the performer). In this case, if the effects requested by each viewer are applied, multiple effects may overlap, reducing viewer visibility.

[0129] In such a case, the viewer information management server 100 may cause the content creation server 400 to replace the effects of the multiple viewers with other effects that represent the concentration of the multiple effect application requests.

[0130] For example, when the content creation server 400 receives the same effect application request from X or more viewers, the content creation server 400 may replace the effect of each viewer with a special effect that expresses a concentration of effects.

[0131] The viewer information management server 100 may use viewer attribute information managed by the viewer information management server 100 or viewer attribute information attached to gaze parameter information obtained from the viewer output system 200 to control the size of the effect applied to the content and the type of effect that can be requested.

[0132] For example, the viewer information management server 100 may control the types of effects that can be requested to be applied depending on the viewing time of the viewer, the number of times content featuring the same performer has been viewed or purchased, and other parameters linked to the viewer.

[0133] Furthermore, if the content creation server 400 receives the same effect application request from a specific number of viewers or more at a specific timing, it may apply a special effect indicating this (for example, a visual effect that is displayed across the entire content).

[0134] The viewer information management server 100 may use viewer attribute information managed by the viewer information management server 100 or viewer attribute information that has been attached to gaze parameter information to change the VR content without the viewer intentionally issuing a request.

[0135] For example, as shown in Fig. 17, the viewer information management server 100 stores a viewer residential area attribute table that indicates the number of viewers for each residential area around the world. The content creation server 400 may then change the display size of a 3D model of a landmark representative of each area (for example, Tokyo Tower in Japan, the Statue of Liberty in the United States, the Leaning Tower of Pisa in Italy, or the Merlion in Singapore) depending on the number of viewers in that residential area, and create content by compositing it against the background of performer P.

[0136] In the example shown in the figure, the largest number of viewers are from Japan, the United States, Italy, and Singapore, in that order, so the sizes of the 3D models used as the background for the content are set in the following order: Tokyo Tower, the Statue of Liberty, the Leaning Tower of Pisa, and the Merlion.

[0137] [Example of extension for adding effects to content] By using the viewer's gaze information and position information and the performer's gaze information and position information, viewers can share their viewing state with each other and add effects to content more effectively. Some examples are described below.

[0138] - Share viewing status with other viewers One method of sharing viewing states is to add the viewing states of other viewers (such as the viewer's location) to the distributed content and then view the content. However, if the viewing states of other viewers are added to the distributed content unconditionally, problems may arise, such as the additional content (effects) appearing in a position that interferes with the viewing of the distributed content, or the distributed content becoming buried in the additional content and becoming unviewable.

[0139] 18 shows a case where a performer P, multiple virtual viewers V, and display 53 have a certain positional relationship. In this case, if avatar content representing the viewing states of other viewers based on the viewing positions of the other viewers or the positions of the avatar images 71 of the other viewers is added to the content being viewed by viewer V1, the added content (avatar content 72) will appear near intersection coordinate I within the viewing cone of viewer 1, as shown in FIG. 19, which may interfere with viewer V1's viewing of the distributed content.

[0140] To address this problem, the content creation server 400 virtually moves and enlarges the display 53 based on the position of viewer V1, as shown in Figure 20, and uses the intersection of the line of sight of the other viewers and the virtual display 53 as the display position for the additional content, thereby making it possible to add content that represents the viewing state of other viewers to the distributed content without interfering with viewer V1's viewing.

[0141] The position and size of the virtual display may be changed as desired. For example, the content creation server 400 may set the virtual display so that it is always located behind the viewing position of the viewer V1 relative to the content.

[0142] Furthermore, instead of a virtual display, the content creation server 400 may use any plane or sphere, or a combination thereof, to find the intersection with the line of sight of other viewers, and use this as the display position of the additional content.

[0143] If there are a large number of viewers, adding avatar images 71 of all viewers to the content may increase the processing load on content creation server 400 or may result in an overcrowding of added content, degrading the overall quality of the content.

[0144] In contrast, viewer information management server 100 may share viewing statuses only with members of a group or community to which each viewer belongs (obtained, for example, from a social networking site, etc.). Content creation server 400 may also replace viewer avatar content 72 with an image that is simpler to render (lower resolution).

[0145] -Adjusting the effect playback position When the effect applied by the effect application request is shared among viewers, the effect application position may be appropriately adjusted. Here, three specific cases are described, but the present invention is not limited to these.

[0146] (Case 1: Play the effect within each viewer's viewing cone) If an effect is played in response to a request from another viewer, the viewer will not notice it unless the effect is played within the viewing cone.

[0147] For example, consider an effect called "Random Rays" that plays a random ray effect in the content coordinate system, as shown in Figure 21. When this effect is played, the number of rays that each viewer can see will vary depending on the position of each viewer's viewing cone.

[0148] In response to this, the content creation server 400 can maintain a consistent quality of the playback effect seen by each viewer by adjusting the playback position of the effect so that the effect is played back within the viewing cone of each viewer.

[0149] Figure 22 shows how the effect playback position of "Random Rays," requested by another viewer, is adjusted to fit within the viewing cone VC of viewer 1. The top diagram in the figure shows the state before adjustment, and the bottom diagram shows the state after adjustment. In the top diagram, the playback positions of ray2 and ray4, which were positioned outside the viewing cone VC, have been adjusted so that they are visible within the viewing cone VC in the bottom diagram.

[0150] The creation of the viewing cone VC that determines the effect playback position in this way may be centered on the viewer's line of sight or head direction.

[0151] (Case 2: Adjust the center of the effect depending on the viewer's line of sight and the performer's position) If an effect requested by one viewer is also applied to content distributed to other viewers, it may interfere with viewing for viewers who are viewing the distributed content from a different perspective than the viewer who made the request.

[0152] For example, an effect requested by another viewer may be played in the space between the viewer and the performer. One way to avoid this problem is to have the viewer output system 200 of the viewer stop playing the effect if an effect with or without a specific attribute is about to be played in the space between the viewer and the performer. However, this approach may prevent a viewer from seeing an effect requested by another viewer with a different perspective.

[0153] To address this issue, each viewer output system 200 may adjust the center of the playback effect depending on the viewing direction of each viewer and the position of the performer.

[0154] Here, we consider setting the effect generation center in the performer coordinate system, which is defined based on the performer's position. The effect generation center is the coordinate that serves as the reference for determining the playback position of effects with or without specific attributes.

[0155] Figure 23 shows area A, which can be set as the effect generation center, with performer P at its center. As an example, a circle with radius r [m] horizontal to the ground and height h [m] is set with its center as performer P, but this does not limit the way area A is set.

[0156] 24 shows how the effect generation center C is set for each viewer using the set area A. The effect generation center C is set by mapping each viewer's line of sight L onto the plane where the effect generation center settable area A exists, and then setting the effect generation center C as the intersection point between the mapped line of sight and the effect generation center settable area A, which is farthest from the viewer.

[0157] In the figure, it can be seen that different effect generation centers C1 to C3 are set for each viewer. In the state shown in the figure, for example, if viewer 2 (corresponding to line of sight L2) issues an effect playback request, viewer output system 200A for viewer 1 (corresponding to line of sight L1) plays the effect at the position of effect generation center C1 for viewer 1, and output system 200C for viewer 3 plays the effect at the position of effect generation center C3 for viewer 3.

[0158] This allows any viewer to view the effects requested by other viewers without interfering with the viewing of the distributed content.

[0159] The setting process of the above-mentioned area A and effect generation center C may be performed by the content creation server 400, rather than by each viewer output system 200, by receiving the gaze parameters of each viewer from each viewer output system 200 via the viewer information management server 100 or directly.

[0160] (Case 3: Adjust the effect playback position based on each viewer's line of sight and effect attributes) For effects with specific attributes, the playback position is adjusted using the viewer's line of sight and the attributes of the background content, making it possible to play effects that are appropriate for each viewer.

[0161] For example, let's say the effect to be played has a text attribute, and the effect with the text attribute is defined to be played on the plane of background content that has a certain extent or more. In Figure 25, background content Plane 1 and Plane 2, each with different plane parameters, are placed ahead of the line of sight (L1 and L2) of Viewer 1 and Viewer 2.

[0162] At this time, if a request to play an effect with a text attribute occurs, viewer output system 200A of viewer 1 plays the effect on Plane 1, and viewer output system 200B of viewer 2 plays the effect on Plane 2, as shown in FIG. 26.

[0163] This allows effects with text attributes to be played in a form appropriate for each viewer. If there is no suitable plane in the viewer's line of sight, it is possible to create a virtual plane behind the performer as seen from the viewer's perspective and play the effect on that plane.

[0164] The display process of each of the above background contents may be performed by the content creation server 400, rather than by each viewer output system 200, by receiving the gaze parameters of each viewer from each viewer output system 200 via the viewer information management server 100 or directly.

[0165] - Adding effects to content and processing location When a large number of effect playback requests occur, problems such as delays in distribution due to the increased processing volume required to add effects to the distributed content and an increase in communication data can occur. To avoid these problems, it is possible to filter playback requests using viewer gaze information.

[0166] For example, the content creation server 400 may reflect, in the content delivered to a certain viewer, only effect playback requests from other viewers who have similar line-of-sight parameters to the viewer.

[0167] It is also possible to adjust the effectiveness of an effect depending on the number of simultaneous viewers. For example, in the case of the "Random Rays" effect described in Figures 21 and 22, if the number of simultaneous viewers is X or more, the content creation server 400 may set the number of rays to be played back in response to one playback request to n, and if the number of simultaneous viewers is Y or more (which is greater than X), the number may be set to y (which is greater than x).

[0168] Furthermore, the content creation server 400 does not apply effects whose playback position is determined according to the viewer's gaze information to the content distributed from the content distribution server 500, but instead transmits information about the effect to each viewer's output system 200 to have the effect applied, thereby reducing the load on the content creation server 400 and the content distribution server 500.

[0169] Use different playback methods depending on the effect attributes In addition to the above, the following attributes can be considered as a method for changing the playback method depending on the attributes of the effect.

[0170] For example, possible effects include an effect that changes or does not change the display posture depending on the viewer's line of sight, an effect that changes or does not change the display posture depending on the performer's direction, an effect that is not displayed within the viewing cone between the viewer and performer, an effect that is played using the distance between the viewer and performer as a parameter, and an effect that is played using the degree of line of sight between the performer and viewer as a parameter.

[0171] As described above, according to this embodiment, the content distribution system enables performers to grasp the virtual gaze of viewers in the same space as themselves, and enables performers to perform appropriately in response to viewers' reactions, even for viewers in remote locations.

[0172] Furthermore, as a result of the above, performers and viewers can communicate as if they were physically close to each other, even when they are in distant locations.

[0173] In addition, each viewer can set effect playback requests to suit their own input device, so the same request can be sent from any device (differences due to device ownership are absorbed).

[0174] Furthermore, since the viewer's actions are mapped to effect IDs and sent to the viewer information management server 100, the amount of communication data required to represent the viewer's actions is significantly reduced.

[0175] Additionally, content providers can create effects that can be requested, eliminating malicious actions that may occur, for example, in communications where arbitrary text is available.

[0176] Furthermore, the content distribution system allows viewers who are viewing the same content to share their experiences by reflecting the actions of the viewers in the distributed content.

[0177] In addition, the content distribution system can differentiate the services provided to each viewer by controlling the effects that can be requested to be played and applied for each viewer.

[0178] [Variations] The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention.

[0179] In the above-described embodiment, when it is determined that specific communication has been established between a performer and a viewer, the viewer information providing server 100 or the content creation server 300 may enhance the communication experience by adding special effects to the content distributed to the target viewer or to the content distributed to all viewers. Examples of the establishment of specific communication include when the performer and a viewer make eye contact, or when a viewer requests the playback of a specific effect in response to a specific performance by the performer.

[0180] The viewer information management server 100 or the content creation server 300 can determine whether the performer and viewer have made eye contact by, for example, determining whether the performer's eye contact is directed toward a viewer's avatar image 71 on the display and whether the absolute value of the dot product of the performer's and viewer's eye gaze vectors is less than a predetermined threshold value such that the two eye gaze vectors are approximately parallel.

[0181] In addition, when the performer and the viewer make eye contact, the viewer information management server 100 may output a special visual effect or sound effect from near the avatar image (intersection coordinates) corresponding to the viewer on the display 53.

[0182] Furthermore, the viewer information management server 100 can count the number of times that each viewer's gaze met, and display the values indicating the frequency of gaze contact in each direction as a histogram in association with each intersection coordinate I on the display 53. Based on this information, the performer can increase the satisfaction of the entire audience by performing in a direction where the viewer's gaze met less frequently.

[0183] 14 shows an example of a frequency histogram 73 indicating the above frequency displayed on the display 53. The value of the frequency histogram 73 can be calculated by dividing the total number of times that the line of sight of viewers in each direction was met by the number of viewers in that direction.

[0184] Furthermore, by enabling performers and viewers to communicate in a manner similar to a real-life experience even when they are in remote locations, it becomes possible to add value to specific viewing locations in live streaming services.

[0185] For example, a content distribution system can create high-value-added viewing locations by assuming that performers will frequently communicate with viewers at a specific viewing location, charging viewers who use that location a higher viewing fee than usual and limiting the number of viewers who can use that viewing location.

[0186] In the above embodiment, the content is shot by the camera 51 fixed in the studio, but instead of the camera 51, the content may be shot while moving by, for example, a drone.

[0187] [others] The present technology can also be configured as follows. (1) Obtaining viewing state information indicating the line of sight or position of each viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information that identifies the viewer, from each of the terminals of a plurality of viewers who are playing back content in which a performer's performance has been captured via a network in real time; Based on the acquired viewing state information, an effect is applied to the content for each of the viewers. Control unit An information processing system comprising: (2) The information processing system according to (1) above, The control unit acquires attribute information indicating attributes of the viewer together with the viewing state information, and changes the effect in accordance with the attribute information. Information processing system. (3) The information processing system according to (1) or (2) above, When the control unit applies the effect to a first viewer among the plurality of viewers, the control unit calculates coordinates of an intersection between a virtual plane set in a coordinate system of the space and the line of sight of a second viewer different from the first viewer, and applies the effect to a position of the content corresponding to the coordinates of the intersection. Information processing system. (4) The information processing system according to (3) above, The control unit sets the virtual plane behind a viewing position of the first viewer in the coordinate system of the space. Information processing system. (5) The information processing system according to (3) or (4), The control unit sets a lower resolution of each effect according to the second viewers as the number of the second viewers increases. Information processing system. (6) The information processing system according to any one of (3) to (5) above, When the coordinates of the intersection with the line of sight of the second viewer are not included within a viewing cone in the coordinate system of the first viewer, the control unit changes the position at which the effect is applied to within the viewing cone. Information processing system. (7) The information processing system according to (1) or (2) above, The control unit applies the effect for each viewer to a position corresponding to the coordinates of the intersection between an area set around the position of the performer in a coordinate system of the space in which the performer exists and the line of sight of the plurality of viewers, the intersection being farther from each viewer. Information processing system. (8) The information processing system according to (1) or (2) above, The control unit applies the effect in response to an effect application request received from each of the viewers' terminals and including effect identification information indicating the attribute of the effect, and if the attribute indicated by the effect identification information is a planar effect, sets a predetermined plane for each of the viewers behind the performer in a coordinate system of the space in which the performer exists and on the line of sight of the performer, and applies the effect on the predetermined plane. Information processing system. (9) The information processing system according to any one of (1) to (8) above, The control unit applies the effect in response to an effect application request received from a terminal of each of the viewers, and when applying the effect to a first viewer among the plurality of viewers, applies the effect only in response to an effect application request from a second viewer who has a line of sight or a position within a predetermined distance from the line of sight or a position of the first viewer among the effect application requests from the plurality of viewers. Information processing system. (10) The information processing system according to any one of (1) to (9) above, The control unit acquires information indicating the number of terminals of viewers currently reproducing the content, and increases the effectiveness of the applied effect in accordance with the number. Information processing system. (11) Obtaining viewing state information indicating the line of sight or position of each viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information that identifies the viewer, from each of the terminals of a plurality of viewers who are playing back content in which a performer's performance has been captured via a network in real time; Based on the acquired viewing state information, an effect is applied to the content for each of the viewers. Information processing methods. (12) In the information processing device, acquiring viewing state information indicating the line of sight or position of each viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information that identifies the viewer, from each of the terminals of a plurality of viewers who are playing back content in which a performer's performance has been filmed in real time via a network; applying an effect to the content for each of the viewers based on the acquired viewing state information; A program that executes the following. [Explanation of symbols]

[0188] 11...CPU 18...Input device 19...Output device 20...Storage device 26...imaging device 23...Communication equipment 51...Camera 52...Mike 53...Display 71...Avatar image 72...Effects 73...Histogram 100...Viewer information management server 200...Performer output system 300...Audience output system 400...Content creation server 500...Content distribution server P...Performer V... Viewers L… Gaze VL...Virtual line of sight

Claims

1. Obtaining viewing state information indicating the line of sight or position of each viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information that identifies the viewer, from each of the terminals of a plurality of viewers who are playing back content in which a performer's performance has been captured via a network in real time; Based on the acquired viewing state information, an effect is applied to the content for each of the viewers. Control unit Equipped with When the control unit applies the effect to a first viewer among the plurality of viewers, the control unit calculates coordinates of an intersection between a virtual plane set in a coordinate system of the space and the line of sight of a second viewer different from the first viewer, and applies the effect to a position of the content corresponding to the coordinates of the intersection. Information processing system.

2. 2. The information processing system according to claim 1, The control unit acquires attribute information indicating attributes of the viewer together with the viewing state information, and changes the effect in accordance with the attribute information. Information processing system.

3. 2. The information processing system according to claim 1, The control unit sets the virtual plane behind a viewing position of the first viewer in the coordinate system of the space. Information processing system.

4. 2. The information processing system according to claim 1, The control unit sets a lower resolution for each effect according to the second viewers as the number of the second viewers increases. Information processing system.

5. 2. The information processing system according to claim 1, When the coordinates of the intersection with the line of sight of the second viewer are not included within a viewing cone in the coordinate system of the first viewer, the control unit changes the position at which the effect is applied to within the viewing cone. Information processing system.

6. 2. The information processing system according to claim 1, The control unit acquires information indicating the number of terminals of viewers currently reproducing the content, and increases the effectiveness of the applied effect in accordance with the number. Information processing system.

7. Obtaining viewing state information indicating the line of sight or position of each viewer in a coordinate system of the space in which the viewer exists, together with viewer identification information that identifies the viewer, from the terminals of a plurality of viewers who are playing back content in which a performer's performance has been filmed in real time via a network; Based on the acquired viewing state information, an effect is applied to the content for each of the viewers. Control unit Equipped with The control unit applies the effect for each viewer to a position corresponding to the coordinates of the intersection between an area set around the position of the performer in a coordinate system of the space in which the performer exists and the line of sight of the plurality of viewers, the intersection being farther from each viewer. Information processing system.

8. Obtaining viewing state information indicating the line of sight or position of each viewer in a coordinate system of the space in which the viewer exists, together with viewer identification information that identifies the viewer, from each of the terminals of a plurality of viewers who are playing back content in which a performer's performance has been filmed in real time via a network; Based on the acquired viewing state information, an effect is applied to the content for each of the viewers. Control unit Equipped with The control unit applies the effect in response to an effect application request received from each of the viewers' terminals and including effect identification information indicating the attribute of the effect, and if the attribute indicated by the effect identification information is a planar effect, sets a predetermined plane for each of the viewers behind the performer in a coordinate system of the space in which the performer exists and on the line of sight of the performer, and applies the effect on the predetermined plane. Information processing system.

9. Obtaining viewing state information indicating the line of sight or position of each viewer in a coordinate system of the space in which the viewer exists, together with viewer identification information that identifies the viewer, from the terminals of a plurality of viewers who are playing back content in which a performer's performance has been filmed in real time via a network; Based on the acquired viewing state information, an effect is applied to the content for each of the viewers. Control unit Equipped with The control unit applies the effect in response to an effect application request received from a terminal of each of the viewers, and when applying the effect to a first viewer among the plurality of viewers, applies the effect only in response to an effect application request from a second viewer who has a line of sight or a position within a predetermined distance from the line of sight or a position of the first viewer among the effect application requests from the plurality of viewers. Information processing system.

10. Obtaining viewing state information indicating the line of sight or position of each viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information that identifies the viewer, from each of the terminals of a plurality of viewers who are playing back content in which a performer's performance has been captured via a network in real time; applying an effect to the content for each of the viewers based on the acquired viewing state information; When the effect is to be applied to a first viewer among the plurality of viewers, coordinates of an intersection between a virtual plane set in the coordinate system of the space and the line of sight of a second viewer different from the first viewer are calculated, and the effect is applied to a position of the content corresponding to the coordinates of the intersection. Information processing methods.

11. In the information processing device, acquiring viewing state information indicating the line of sight or position of each viewer in a coordinate system of a space in which the viewer exists, together with viewer identification information that identifies the viewer, from each of the terminals of a plurality of viewers who are playing back content in which a performer's performance has been filmed in real time via a network; applying an effect to the content for each of the viewers based on the acquired viewing state information; When the effect is to be applied to a first viewer among the plurality of viewers, calculating coordinates of an intersection between a virtual plane set in a coordinate system of the space and the line of sight of a second viewer different from the first viewer, and applying the effect to a position of the content corresponding to the coordinates of the intersection; A program that executes the following.

Images