Information processing device, information processing method, and program

Abstract

[Problem] To provide an information processing device, an information processing method, and a program which can achieve a reduction in processing load and can enhance the level of entertainment. [Solution] This information processing device comprises a control unit which controls to distribute, to one or more viewer terminals, information about a virtual space in which a virtual object of a performer and virtual objects of viewers are disposed, wherein the control unit uses, as motion information about virtual objects of viewers at distribution destinations of another information processing device among the virtual objects of all the viewers disposed in the virtual space, information lesser than motion information that is received from the other information processing device and is used in virtual objects of viewers at distribution destinations by the control unit.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, virtual reality (VR) technology has become widespread, allowing users to view virtual spaces in which 3D models are placed from any viewpoint. A head-mounted display (HMD), which completely covers the user's field of view to enhance the sense of immersion in the virtual space, may be used as a device for viewing such virtual space (VR space). Character objects (called avatars) that act as the user's avatar may also be placed in the virtual space. Furthermore, by displaying not only the user's avatar but also the avatars of other users in the virtual space, the user can experience the feeling of sharing the venue with others.

[0003] Furthermore, Patent Document 1 below discloses a technology in which the reactions of each user watching video content distributed in real time are detected by an HMD worn by each user, and the detected information (user movement information based on output from acceleration sensors and angular velocity sensors) is transmitted to the location where the content is being filmed and presented at the real venue. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-126101 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is a problem of processing load when reflecting the movements of many users (viewers) in a virtual space.

[0006] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that reduce the processing load and improve entertainment value. [Means for solving the problem]

[0007] According to the present disclosure, an information processing device is proposed which includes a control unit that controls the distribution of information about a virtual space in which performer virtual objects and viewer virtual objects are placed to one or more viewer terminals, and the control unit uses, of all viewer virtual objects placed in the virtual space, less information as movement information for the viewer's virtual objects that are the distribution destination of another information processing device than the movement information received from the other information processing device and used for the viewer's virtual objects that are the distribution destination of the control unit.

[0008] According to the present disclosure, an information processing method is proposed, which includes a processor controlling the distribution of information about a virtual space in which performer virtual objects and viewer virtual objects are placed to one or more viewer terminals, and using, among all viewer virtual objects placed in the virtual space, less information as movement information for the viewer's virtual objects that are the destination of distribution by another information processing device than the movement information received from the other information processing device and used for the viewer's virtual objects that are the destination of distribution by the distribution control.

[0009] According to the present disclosure, a program is proposed in which a computer is made to function as a control unit that controls the distribution of information about a virtual space in which performer virtual objects and viewer virtual objects are placed to one or more viewer terminals, and the control unit uses, of all viewer virtual objects placed in the virtual space, less information as movement information for the viewer's virtual objects to which another information processing device is distributed than the movement information received from the other information processing device and used for the viewer's virtual objects to which the control unit is distributed. [Brief explanation of the drawings]

[0010] [Figure 1]1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating the arrangement of cameras and displays around a performer according to this embodiment. [Figure 3] 1 is a block diagram showing an example of the configuration of an imaging system according to an embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a distribution server according to the present embodiment. [Figure 5] FIG. 2 is a diagram showing an example of the arrangement of virtual objects in a virtual space according to the present embodiment. [Figure 6] 10A to 10C are diagrams illustrating adjustment of the direction of a 3D model of a performer according to this embodiment. [Figure 7] FIG. 2 is a block diagram showing an example of the configuration of a viewer terminal according to the present embodiment. [Figure 8] 10 is a flowchart illustrating an example of the flow of operation processing of a distribution server according to the present embodiment. [Figure 9] An example of the hardware configuration of an information processing device according to an embodiment of the present disclosure will be described. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] The explanation will be given in the following order: 1. Overview of information processing system according to one embodiment of the present disclosure 2.Configuration example 2-1. Imaging System 31 2-2. Distribution Server 20 2-3. Viewer terminal 10 3. Operation processing 4. Hardware Configuration 5. Supplementary Information

[0013] <<1. Overview of Information Processing System According to One Embodiment of the Present Disclosure>> Fig. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present disclosure. As shown in Fig. 1, the information processing system according to this embodiment includes a plurality of distribution servers 20 (20A to 20C...), an imaging system 31 in a studio 3 where video images to be distributed are acquired, viewer terminals 10 to which videos are distributed from each distribution server 20, and a display system 33 in the studio 3 that displays what the viewers are seeing.

[0014] The distribution server 20 has a function of providing a virtual space. Concerts, lectures, plays, festivals, and various other events can be held in the virtual space. In this embodiment, as an example, it is assumed that a concert by performers is held in the virtual space and a large number of viewers watch the concert in the virtual space.

[0015] Viewer avatars that reflect the movements of viewers are placed in the virtual space provided by the distribution server 20. A viewer avatar is an example of a virtual object of a viewer. More specifically, a viewer avatar is a 3D character model that reflects the movements of the viewer. Viewer movement information is transmitted in real time from the viewer terminal 10 to the distribution server 20. Furthermore, each viewer can experience the feeling of sharing the venue with others, as avatars other than their own are also displayed in the virtual space.

[0016] Video of the performers is provided to the distribution server 20 in real time from the imaging system 31. The video of the performers acquired by the imaging system 31 and provided to the distribution server 20 may be a 3D live-action avatar (referred to herein as a "live-action 3D model") generated using technology for converting live-action video of the performers into 3DCG (e.g., volumetric capture technology). This allows each viewer to view a more realistic performer. Note that, although a live-action 3D model has been given here as an example of a virtual object of the performer, this embodiment is not limited to this, and a 3DCG character that reflects the movements of the performer may also be used as the virtual object of the performer.

[0017] Display system 33 displays in real time an image of the entire concert venue in the virtual space and an image of the audience as seen from the performer's viewpoint in the virtual space (an image of the audience seats where the viewer avatars are located), and provides the performers with the audience's reactions. Note that while Fig. 1 shows a system configuration including display system 33, this embodiment is not limited to this, and a configuration that does not include display system 33 is also possible.

[0018] (Identifying issues) As mentioned above, reflecting the movements of a large number of viewers in a virtual space can cause problems with processing load. When depicting the movements of a large number of viewers in a virtual space, processing must be performed in proportion to the number of viewers, making it difficult to share a virtual space with a number of viewers exceeding the server's processing capacity. For example, currently, depending on the server's processing capacity, the limit may be around 100 people. Therefore, it has been difficult to provide an experience in a virtual space where tens of thousands of viewers can share a venue, as in an actual concert venue.

[0019] Therefore, in an embodiment of the present disclosure, by linking multiple distribution servers 20 (20A to 20C...), it is possible to arrange and share a single virtual space with more people than can be handled by a single server. In other words, it is possible to reduce the processing load of reflecting the movements of many viewers in the virtual space in avatars, and to improve the entertainment value by allowing a larger number of people to share the virtual space.

[0020] In the example shown in FIG. 1, first, each distribution server 20 is connected to a number of viewer terminals 10 that does not exceed its processing capacity. In the virtual space, each distribution server 20 places viewer avatars corresponding to the viewer terminals 10 with which it is connected in communication close to each other, and places viewer avatars corresponding to viewer terminals 10 connected in communication with other distribution servers 20 in distant locations. By placing them in distant locations, these viewer avatars appear from a distance in the virtual space image from the viewer's perspective (viewer avatar's perspective) displayed on the viewer terminal 10 with which the distribution server 20 is connected in communication. Therefore, it does not look unnatural even if individual detailed movements are not reflected, and processing load is reduced by performing simple display control as a crowd. Simple display control, for example, means that only general body movements of each viewer avatar are reflected, without controlling the facial expressions, hair movement, or clothing movement of each viewer avatar. Re The general body movements are, for example, changes in posture, and more specifically, bone information of a 3D character. Also, if a crowd consists of 100 viewer avatars, the same body movements (for example, only changes in posture) can be reflected for every few dozen people.

[0021] Based on the movement information of each viewer received from the many viewer terminals 10 to which the distribution server 20 is connected for communication, the distribution server 20 generates crowd data for simple display control as a crowd, and transmits the crowd data to other distribution servers 20. The crowd data is the movement information of each viewer received from the many viewer terminals 10, with the data volume reduced. For example, the movement information of each viewer received from the many viewer terminals 10 includes the facial expressions of the viewer avatars, the movement information of the viewer avatars' hair, the movement information of the viewer avatars' clothes, and the body movements of the viewer avatars (head direction, changes in posture (e.g., bone information), hand and arm movements, finger movements, etc.). However, by limiting this to "posture changes" only (e.g., bone information only), the data volume can be significantly reduced. In this embodiment, "crowd data," which is the movement information of the many viewer avatars with the data volume reduced, is distributed to other distribution servers 20.

[0022] More specifically, for example, distribution server 20A shown in FIG. 1 generates crowd data based on the movement information of each viewer received from viewer terminals 10A-1 to 10A-3, and transmits the crowd data to other distribution servers 20B, 20C, and so on. Based on the movement information of each viewer received from viewer terminals 10B-1 to 10B-3, and so on connected to itself, distribution server 20B reflects minute movements (e.g., the above-mentioned facial expressions, hair movements, clothing movements, body movements, etc.) in the corresponding viewer avatars. Furthermore, distribution server 20B places these viewer avatars in locations close to each other (e.g., one of the audience seating areas in a virtual concert venue). Furthermore, distribution server 20B places viewer terminals 10A-1 to 10A-3, and so on connected to other distribution servers 20A, in locations far away from the viewer avatars placed in the close locations (e.g., other areas). Based on the crowd data received from distribution server 20A, viewer avatars corresponding to viewer terminals 10A-1 to 10A-3... connected to distribution server 20A are simply displayed as a crowd (for example, by reflecting only posture changes based on bone information), thereby enabling light-load processing. Because the viewer avatars corresponding to viewer terminals 10A-1 to 10A-3... are located far away, the image of the virtual space from the viewer's perspective (viewer avatar's perspective) displayed on each of viewer terminals 10B-1 to 10B-3... does not reflect the individual small movements of the viewer avatars seen from a distance, and therefore the viewer avatars are simply displayed and controlled, so it is unlikely to appear unnatural. It is expected that viewers will not notice or be bothered by the simplified control.

[0023] The above describes an overview of an information processing system according to an embodiment of the present disclosure. This system is not limited to concerts, but can also reduce processing load when a large number of users share a virtual space, such as in games. A large number of users may be, for example, tens of thousands of users, but is not limited to this. Although this system does not mention audio, it is processed separately in implementation, and the performer's audio may be transmitted to the viewer, and the viewer's audio may be transmitted to the performer or other viewers.

[0024] Next, the specific configuration of each device included in the information processing system according to this embodiment will be described with reference to the drawings.

[0025] <<2. Configuration Example>> <2-1. Imaging System 31> First, the imaging system 31 that acquires information about the performers will be described with reference to FIGS.

[0026] 2 is a diagram illustrating the arrangement of cameras 311a and displays 331 around a performer according to this embodiment. As shown in FIG. 2, the imaging system 31 according to this embodiment acquires information for generating a 3D model of the performer using n cameras 311a (an example of a data acquisition unit 311) arranged around performer A (for example, in a circular shape) in studio 3. The n cameras 311a (311a-1 to 311a-1n) simultaneously capture images under the control of the imaging system 31, which acquires multi-viewpoint data.

[0027] Additionally, multiple displays 331 (for example, LED displays) may be placed around performer A. Each display 331 displays a viewer avatar as seen from the viewpoint of the performer's live-action 3D model in the virtual space.

[0028] 2, n cameras 311a and multiple displays 331 are arranged in a circle, but they may be arranged in a square or other shape, and the cameras 311a and displays 331 may be arranged in different shapes. Furthermore, the cameras 311a and displays 331 are not limited to being arranged in a single row around performer A, but may be arranged in multiple rows in the vertical direction.

[0029] FIG. 3 is a block diagram showing an example of the configuration of an imaging system 31 according to this embodiment. As shown in FIG. 3, the imaging system 31 includes a data acquisition unit 311, a 3D model generation unit 312, an encoding unit 313, and a transmission unit 314. The imaging system 31 may be composed of multiple information processing devices or a single information processing device. The 3D model generation unit 312 and the encoding unit 313 are examples of functions of a control unit of the information processing device that realizes the imaging system 31. The transmission unit 314 is examples of functions of a communication unit of the information processing device that realizes the imaging system 31. The data acquisition unit 311 is examples of functions of an input unit of the information processing device that realizes the imaging system 31. The information processing device that realizes the imaging system 31 may be, for example, a PC (personal computer), a tablet terminal, or a smartphone located in the same space as performer A, or may be a server on a network.

[0030] (Data acquisition unit 311) The data acquisition unit 311 acquires information for generating a 3D model of the performer. As described with reference to FIG. 2, one example of the data acquisition unit 311 is a large number (e.g., several tens) of cameras 311a that acquire video signals (captured images). The cameras 311a simultaneously capture images of the performer from various angles under the control of the imaging system 31, acquiring a large number of captured images. The data acquisition unit 311 integrates the captured images and outputs them as multi-viewpoint data to the 3D model generation unit 312, which generates a 3D model of the performer. Note that the cameras 311a may include various devices that sense depth information. In this case, the multi-viewpoint data may include not only RGB signals but also depth signals and their source sensing signals (e.g., infrared signals).

[0031] (3D model generation unit 312) The 3D model generation unit 312 generates (performs modeling of) a 3D model of the performer as a virtual object of the performer based on the data acquired by the data acquisition unit 311. The 3D modeling method is not particularly limited, but may be, for example, a Shape from Silhouette method (SFS method) such as Visual Hull (visual volume intersection method) or a Multi-View Stereo method (MVS method). The data format of the 3D model may be any representation format such as Point Cloud, voxel, or mesh.

[0032] In this embodiment, as an example, a 3D model is represented by a combination of mesh shape data and surface texture information. In this embodiment, for example, Volumetric Capture technology is used to generate a live-action 3D model of a performer from a large number of captured images. More specifically, the 3D model generation unit 312 generates a live-action 3D model based on a group of several dozen captured images (multiple viewpoint data) obtained by simultaneously capturing images of the performer from various directions using several dozen cameras 311a. Then, the distribution server 20 or the viewer terminal 10 generates a high-quality live-action 3D video of the performer by viewing the live-action 3D model of the performer from any direction. Volumetric Capture technology can also generate video from viewpoints (virtual viewpoints) where no cameras exist, allowing for more flexible viewpoint manipulation.

[0033] (encoding unit 313) The encoding unit 313 has a function of compressing the 3D model generated by the 3D model generation unit 312 for transmission. The 3D model is represented by, for example, shape data and texture data. The encoding unit 313 compresses the shape data and texture data by applying a compression format appropriate for each format. For example, MPEG-4 AFX (Animation Framework eXtension) or other open source compression methods may be used to compress the shape data. Furthermore, for example, AVC (H.264) or HEVC (H.265) may be used to compress the texture data.

[0034] (Transmitter 314) The transmitter 314 transmits the data compressed by the encoder 313 to a plurality of distribution servers 20 (for example, distribution servers 20A to 20C shown in FIG. 1). The transmitter 314 also adds a timestamp or the like to the compressed data before transmitting it.

[0035] The transmitting unit 314 may also transmit performer mode information acquired from the data acquiring unit 311 to multiple distribution servers 20. The performer mode information indicates whether the mode is a normal mode, in which the performer's direction is adjusted, or an interaction mode, in which the performer's direction is not adjusted. The performer's direction refers to the performer's direction relative to the audience. For example, in a virtual space, if viewer avatars are positioned around a stage on which a live-action 3D model of the performer is located, the distribution server 20 can appropriately adjust the direction (front direction) of the live-action 3D model of the performer so that it faces the viewer avatar of the viewer at the distribution destination, thereby creating a positional relationship in which the viewer is watching from a seat directly in front of the performer. Direction adjustment may be performed during a performance such as normal singing or dancing. On the other hand, if the performer is speaking in a specific direction or toward a specific viewer avatar, it is preferable not to adjust the performer's direction (it would be unnatural for the performer to look at the performer while speaking to other viewers). When the performer is speaking to the viewer avatar, the mode is set to "interaction mode," and the distribution server 20 does not adjust the performer's direction. The "interaction mode" may be set by operating a button on the microphone held by the performer, or may be automatically determined by the control unit of the imaging system 31 based on the video captured by the camera 311a and the audio picked up by the microphone. Alternatively, the mode may be set by a staff member of the distributor, including the performer, by operating a button as appropriate.

[0036] <2-2. Distribution Server 20> Next, the configuration of the distribution server 20 that arranges virtual objects in a virtual space and provides information about the virtual space will be described with reference to Fig. 4. The multiple distribution servers 20A to 20C according to this embodiment all have the same configuration.

[0037] Fig. 4 is a block diagram showing an example of the configuration of the distribution server 20 according to this embodiment. As shown in Fig. 4, the distribution server 20 has a 3D model data receiving unit 210, a mode information receiving unit 211, a control unit 220, an inter-server communication unit 230, and a device communication unit 240. The distribution server 20 may be composed of multiple information processing devices, or may be a single information processing device. The 3D model data receiving unit 210, the mode information receiving unit 211, the inter-server communication unit 230, and the device communication unit 240 are listed as functions of the communication unit of the distribution server 20.

[0038] (3D model data receiving unit 210) The 3D model data receiving unit 210 receives 3D model data of the performer from the imaging system 31. In this embodiment, compressed shape data and compressed texture data (data representing live-action 3D data) are received as the 3D model data of the performer.

[0039] (Mode information receiving unit 211) The mode information receiving unit 211 receives mode information from the imaging system 31, which indicates whether the state of the performer is in the normal mode or the interaction mode.

[0040] (control unit 220) The control unit 220 functions as a virtual object placement unit 221 and a crowd data processing unit 222. The virtual object placement unit 221 controls the placement of all virtual objects in the virtual space. All virtual objects include a virtual object of the performer, which is the main content (in this embodiment, a live-action 3D model of the performer), and virtual objects of all viewers (viewer avatars). Furthermore, all viewers include each viewer avatar associated with multiple viewer terminals 10 with which the distribution server 20 is communicatively connected, and each viewer avatar associated with multiple viewer terminals 10 with which other distribution servers 20 are communicatively connected. Here, an example of the placement of each virtual object in the virtual space will be described with reference to FIG. 5.

[0041] 5 is a diagram showing an example of the arrangement of virtual objects in a virtual space according to this embodiment. For example, in the virtual space 40 shown in FIG. 5, a stage 430 is arranged in the center, and 3 A plurality of areas 410 (first area 410A to fifth area 410E) are arranged (for example, in a circular shape) around a stage 430. A virtual object (for example, a live-action 3D model 440) of a performer, which is the main content, is positioned on a stage 430, and a large number of viewer avatars (viewer avatar group 420) are arranged in each area 410 as spectators. In this way, a large number of viewer avatars watch a concert by performers together in virtual space 40. Note that the arrangement of the plurality of areas 410 (410A to 410E) in which the viewer avatar group 420 is respectively arranged (positional relationship with the 3D model of the performer) may correspond to the arrangement of the plurality of displays 331 (331-1 to 331-5) arranged around real performer A shown in FIG. 2 (positional relationship with the performer). That is, for example, viewer avatar group 420A arranged in first area 410A may be displayed on display 331-1, and viewer avatar group 420B arranged in second area 410B may be displayed on display 331-2.

[0042] For each frame, the virtual object placement unit 221 transmits data of the live-action 3D model of the performer (compressed shape data and compressed texture data), the position of the live-action 3D model of the performer (and direction information if direction adjustment has been performed), and the position and posture of each viewer avatar (and various parameters for precisely moving the viewer avatar) as update information for the virtual space to the viewer terminal 10 with which the distribution server 20 is connected for communication. Note that the control unit 220 of the distribution server 20 transmits data necessary for rendering the virtual space, such as data for each viewer avatar (3D model data, etc.), background, stage, and lighting data, to the viewer terminal 10 in advance. The update information for the virtual space that the virtual object placement unit 221 transmits to the viewer terminal 10 for each frame also includes information regarding production, such as changes in lighting.

[0043] The virtual object placement unit 221 also controls the transmission of crowd data received from each of the other distribution servers via the inter-server communication unit 230 and processed as necessary by the crowd data processing unit 222 to the viewer terminal 10 to which the distribution server 20 is connected for communication. The crowd data is data used when the viewer terminal 10 performs simple display control (drawing control) on a large number of viewer avatars that are mainly processed by each of the other distribution servers. The crowd data may consist of, for example, only bone information indicating the general body movements of the large number of viewer avatars. Alternatively, the crowd data may consist of only bone information for a portion of the large number of viewer avatars that have made large movements (are excited). In this case, the crowd data processing unit 222 expands the crowd data so that the same bone information is reflected in all of the large number of viewer avatars (or, for example, in units of several tens of people), and the expanded crowd data is output to the virtual object placement unit 221.

[0044] The virtual object placement unit 221 also outputs movement information of each viewer avatar received from one or more viewer terminals 10 via the terminal communication unit 240 to the crowd data processing unit 222. The crowd data processing unit 222 extracts a portion of the movement information of each viewer avatar to generate crowd data. The generated crowd data is transmitted to each of the other distribution servers 20 via the inter-server communication unit 230.

[0045] Next, the arrangement of audience avatars will be described in more detail with reference to FIG. 5. In this embodiment, in each area 410 (410A to 410E) shown in FIG. 5, a viewer avatar group 420 is arranged, which is made up of viewer avatars corresponding to a plurality of viewer terminals 10 (viewer terminals 10 receiving distribution from a single distribution server 20) that are communicatively connected to the corresponding distribution server 20. For example, the first area 410A is associated with the distribution server 20A, and a viewer avatar group 420A is arranged, which is made up of viewer avatars corresponding to a plurality of viewer terminals 10A-1 to A-3... that are communicatively connected to the distribution server 20A. In addition, the second area 410B is associated with the distribution server 20B, and a viewer avatar group 420B is arranged, which is made up of viewer avatars corresponding to a plurality of viewer terminals 10B-1 to B-3... that are communicatively connected to the distribution server 20B. Similarly, in each of the following areas, First A distribution server 20 is associated with each of the viewer terminals 10, and each viewer avatar is arranged corresponding to each of the viewer terminals 10 that are communicatively connected to the distribution server 20. In other words, viewer avatars that are mainly processed by one distribution server 20 are arranged in the same area.

[0046] Note that, although five areas are used in the example shown in FIG. 5, this embodiment is not limited to this. The virtual concert venue may have the same number of areas as the number of distribution servers 20 included in this system. Furthermore, each area 410 may be made up of multiple audience seats, or may be surrounded by a fence or the like. Furthermore, each area 410 may be represented by an island floating on water, a cloud floating in the sky, or the like. By using a fence or the like, or by making the area appear to float on water or in the sky, viewers can intuitively understand that the other areas are separate groups.

[0047] Furthermore, each distribution server 20 may place areas other than the area to which it is associated (other areas) at a distance (far away). The distance is not particularly limited, but may be, for example, such that the viewer avatars placed in other areas are not sufficiently visible. Specifically, the distance may be such that, from the viewpoint of a viewer avatar placed in an area, the facial expressions of viewer avatars placed in other areas cannot be clearly seen. As an example, it is assumed that, from the viewpoint of a viewer avatar, the numerous viewer avatars placed in other areas appear as a single mass, like a crowd.

[0048] Each viewer avatar placed within the area may be able to move freely within the area, or may have a predetermined viewing position (audience seating). This may be set appropriately by the distributor distributing content such as a live performance, or may be set in advance by the administrator of the virtual space providing the VR venue. When the viewer terminal 10 renders an image of the virtual space from the viewpoint (viewing position) of the viewer avatar, it also renders other viewer avatars next to or nearby the viewer avatar, thereby providing the viewer with the experience of sharing the space with others.

[0049] In this system, when the viewer terminal 10 renders an image of the virtual space from the viewpoint of a viewer avatar, detailed rendering control can be performed on the viewer avatars in the same area. Detailed rendering control refers to moving many parts and locations of the viewer avatar, such as the viewer avatar's facial expression, facial direction, hair, clothing, limbs, fingers, and body (posture). The movement of each viewer avatar can be controlled based on information (movement information) acquired from the associated real viewer. Information regarding the movements of viewer avatars in the same area can be transmitted to the viewer terminal 10 from the distribution server 20 as updated information (various parameters for detailed movement of the viewer avatar, transmitted together with the above-mentioned "position and posture of each viewer avatar"). This allows each viewer's reaction to be reflected in detail in real time on each viewer avatar in the same area, enabling communication with others and sharing the virtual space in a more realistic manner.

[0050] On the other hand, when the viewer terminal 10 renders an image of the virtual space from the viewpoint of a viewer avatar, simplified rendering control is performed on a large number of viewer avatars located in other areas (for example, a group of viewer avatars that appear as a single mass from a distance, like a crowd). The simplified rendering control includes rough body movements based only on bone information. Such simplified rendering control is performed using crowd data (movement information with reduced data volume) transmitted from the distribution server 20 connected via communication.

[0051] That is, the numerous viewer avatars placed in other areas are viewer avatars corresponding to a viewer terminal 10 (e.g., viewer terminal 10B) that is communicatively connected to a distribution server 20 (e.g., distribution server 20B) other than the distribution server 20 (e.g., distribution server 20A) that is communicatively connected to the viewer terminal 10 (e.g., viewer terminal 10A). Also, the crowd data is data transmitted from the other distribution server (e.g., distribution server 20B) to the distribution server (e.g., distribution server 20A) that is communicatively connected to the viewer terminal 10 (e.g., viewer terminal 10A).

[0052] As described above, by locating other areas in distant locations, even if a large number of viewer avatars placed in other areas are simply drawn (for example, by reflecting only general body movements and keeping facial expressions fixed), viewers are unlikely to notice that they are being simply controlled, or that the control is rough or inaccurate. Note that an example of a simple drawing would be to reflect the same movements on dozens of viewer avatars (without individually controlling each viewer avatar).

[0053] Simple drawing control (display control) reduces the processing load compared to detailed drawing control (display control). In this embodiment, viewer avatars that are primarily processed by other distribution servers are displayed in other areas of the virtual space, and simple drawing control is performed based on crowd data transmitted from the other distribution servers, making it possible to share the virtual space with more people than can be handled by a single distribution server.

[0054] In this embodiment, the rendering control of the virtual space (including the generation of video from the viewpoint of a viewer avatar in the virtual space) is performed by each viewer terminal 10 as an example, but this embodiment is not limited to this and may be performed by the distribution server 20. The distribution server 20 may generate video from the viewpoint of a viewer avatar and transmit it to the viewer terminal 10.

[0055] Furthermore, the virtual object placement unit 221 may change the position of the corresponding viewer avatar placed in the virtual space based on movement information included in the movement information of the viewer avatar transmitted from each viewer terminal 10, and transmit the new position information as update information to each viewer terminal 10. Furthermore, the virtual object placement unit 221 may change the position of the 3D model of the performer based on movement information added to the 3D model data of the performer transmitted from the imaging system 31, and transmit the new position information as update information to each viewer terminal 10. The imaging system 31 may extract movement information of the performer (amount of change in position and three-dimensional position information) based on the captured image and depth information of the performer, instruction information input by the staff of the distributor, etc., and transmit the information added to the 3D model data.

[0056] The virtual object placement unit 221 may also adjust the orientation of the 3D model of the performer in the virtual space based on mode information transmitted from the imaging system 31. Specifically, in the "normal mode" in which the performer is performing a normal performance such as singing or dancing, the virtual object placement unit 221 controls the orientation of the performer's 3D model to face the position (viewing position) of the viewer avatar of each viewer terminal 10 at the distribution destination. This makes it possible to provide all viewers with the experience of viewing the performer from the front. The virtual object placement unit 221 transmits information about the adjusted orientation to the viewer terminal 10 together with position information of the performer's 3D model.

[0057] Furthermore, in the "interaction mode" where it is natural for the performer to face a specific direction, such as when the performer is talking while facing a specific direction, the virtual object placement unit 221 does not adjust the orientation of the performer's 3D model. This allows all viewers to share the state in which the performer is facing a certain direction and interacting with a specific viewer avatar.

[0058] Fig. 6 is a diagram illustrating the adjustment of the orientation of a 3D model of a performer according to this embodiment. As shown in Fig. 6, this embodiment assumes that the arrangement of multiple displays 331 (331-1 to 331-5) arranged around performer A in a studio where real-life filming of performer A is taking place corresponds to the arrangement of multiple areas 410 (410A to 410E) arranged around a stage on which a 3D model 440 of the performer is positioned in the virtual space.

[0059] Each display 331 can be controlled by the display system 33 to display the viewer avatar of the corresponding area 410. There are no particular limitations on the method of displaying each viewer avatar, but for example, the display system 33 may display an image of each viewer avatar arranged in a panel-like manner, or may display an image of the entire area visible from the stage in the virtual space (an image in which many viewer avatars are arranged within the area). This allows performer A to perform while watching the reactions (excitement, etc.) of the audience (viewer avatars).

[0060] As shown on the left side of Figure 6, if performer A is facing the direction of display 331-1, for example, and the placement of the performer's live-action 3D model 440 in virtual space 40 is controlled without adjusting the direction, the performer's live-action 3D model 440 will be facing the direction of first area 410A corresponding to display 331-1, as shown on the top right side of Figure 6. In "interaction mode," where it is natural for the performer to face a specific direction, such as when the performer is talking to the viewer avatar in first area 410A, no direction adjustment is made. Therefore, viewers in all areas will be able to see that the performer is facing the direction of first area 410A. 410A It is possible to share the state of facing the direction.

[0061] On the other hand, the lower right of Figure 6 shows an example of a case where direction adjustment is performed. For example, when distribution server 20C associated with third area 410C adjusts direction for the distribution destination, as shown in the lower right of Figure 6, it controls the placement of live-action 3D model 440 of the performer so that it faces the viewer avatar placed in third area 410C (i.e., the viewer to whom distribution server 20C is distributing). This makes it possible to view the performer from the front from the viewpoint of the viewer avatar placed in third area 410C. Similar control is performed by each distribution server 20. This makes it possible to provide all viewers with the experience of viewing the performer from the front.

[0062] In this embodiment, as an example, displays 331 displaying viewer avatars are placed around performer A in the studio. However, the present disclosure is not limited to this. Viewer avatars may not be displayed, and displays 331 may not be placed. In this case, information indicating at least which areas are located in the virtual space and in which directions to performer A may be clearly displayed. For example, the names of the areas may be displayed in text on displays placed in each direction, or papers with the names of the areas printed on them may be placed in each direction. This allows performer A to perform actions such as speaking to a specific area.

[0063] (Inter-server communication unit 230) The inter-server communication unit 230 has a function of connecting to other distribution servers 20 for communication and transmitting and receiving crowd data.

[0064] (Terminal communication unit 240) The terminal communication unit 240 has a function of establishing a communication connection with the viewer terminal 10 and transmitting and receiving data. For example, the terminal communication unit 240 continuously receives movement information of a viewer avatar from the connected viewer terminal 10 and inputs the information to the virtual object placement unit 221. The terminal communication unit 240 may also receive data for drawing a viewer avatar (such as data of a 3DCG character) from the viewer terminal 10, for example, at the time of initial connection. The terminal communication unit 240 also transmits virtual space information (data necessary for drawing the virtual space) to the viewer terminal 10. For example, at the time of initial connection, the terminal communication unit 240 transmits data of each viewer avatar (such as data of a 3DCG character), background, stage, lighting data, etc., under the control of the control unit 220. The terminal communication unit 240 also continuously transmits update information of the virtual space output from the virtual object placement unit 221 to the viewer terminal 10 (for example, once per frame).

[0065] <2-3. Viewer terminal 10> Next, the configuration of the viewer terminal 10 used by the viewer when viewing the video of the virtual space will be described with reference to FIG.

[0066] Fig. 7 is a block diagram showing an example of the configuration of a viewer terminal according to this embodiment. As shown in Fig. 7, the viewer terminal 10 has a communication unit 110, a decoding unit 120, a rendering unit 130, a display unit 140, and a sensor unit 150. The viewer terminal 10 may be made up of multiple information processing devices, or may be a single information processing device. The decoding unit 120 and the rendering unit 130 are included as functions of the control unit of the viewer terminal 10.

[0067] The viewer terminal 10 may be realized, for example, by a non-transparent HMD (Head Mounted Display) that covers the viewer's entire field of vision. The viewer terminal 10 may also be realized by various devices such as a smartphone, tablet terminal, PC, projector, game terminal, television device, or wearable device.

[0068] (Communication unit 110) The communication unit 110 is connected to the distribution server 20 for communication and transmits and receives data. For example, the communication unit 110 receives data for rendering a virtual space from the distribution server 20. The communication unit 110 also transmits various sensing data detected by the sensor unit 150 to the distribution server 20 as movement information of a viewer avatar. The sensing data may be, for example, position and orientation information of the viewer. The communication unit 110 may also transmit to the distribution server 20 movement information of the viewer avatar that reflects the sensing data detected by the sensor unit 150 by the rendering unit 130 (described later).

[0069] (Decoding unit 120) The decoding unit 120 decodes data received from the distribution server 20 via the communication unit 110 as necessary. The decoding unit 120 performs decoding processing using the corresponding decoders. For example, the decoding unit 120 decodes the compressed shape data and compressed texture data, which are the live-action 3D model data of the performer received from the distribution server 20, and inputs them to the rendering unit 130.

[0070] (sensor unit 150) The sensor unit 150 is realized by various sensors that detect the state of the viewer. For example, the sensor unit 150 may be an IMU (Inertial Measurement Unit), a camera, an IR sensor, a microphone, a biosensor, or the like. The sensor unit 150 may detect the viewer's movements (position, posture, limb movements, head movements, etc.) using an IMU, or may detect the viewer's facial expressions (movement of eyes, eyebrows, mouth, etc.) and gaze using a camera. The sensor unit 150 may also detect the viewer's speech using a microphone, for example. The sensor unit 150 may also have multiple types of sensors.

[0071] In this embodiment, the sensing data detected by the sensor unit 150 is used as information (operation information) for controlling the viewer avatar, but the present disclosure is not limited to this. The operation information of the viewer avatar may be detected, for example, from a controller operated by the viewer (button / switch operation, joystick operation, tilt operation, etc.). Furthermore, the operation information of the viewer avatar may be input from various operation input units such as a keyboard, mouse, touchpad, etc.

[0072] (Rendering unit 130) The rendering unit 130 has a function of controlling the rendering of the virtual space based on information received from the distribution server 20. Specifically, the rendering unit 130 constructs a virtual space and generates an image from the viewpoint of a viewer avatar placed in the virtual space. The image from the viewpoint of the viewer avatar may or may not include the viewer avatar of the viewer.

[0073] As described above, the virtual space contains live-action 3D models of the performers, who are the main content, and viewer avatars, who are the audience. The live-action 3D models of the performers can be represented by decoded format data and texture data. The layout of each area, the layout of each viewer avatar, and the simple and detailed drawing control of each viewer avatar are as described above with reference to Figure 7.

[0074] Furthermore, the image of the virtual space from the viewer avatar's viewpoint generated by rendering unit 130 may be a three-dimensional image or a two-dimensional image.

[0075] Furthermore, the rendering unit 130 controls the viewer avatar's viewpoint position and viewing direction based on the sensing data detected by the sensor unit 150. For example, the rendering unit 130 controls the viewer avatar's viewpoint and viewing direction in accordance with the head and line of sight movements (up and down, left and right, tilt, forward and backward movements, etc.) detected by the sensor unit 150.

[0076] The rendering unit 130 may also operate the viewer avatar based on sensing data detected by the sensor unit 150. Specifically, the rendering unit 130 controls the viewer avatar's facial expression, hair movement, clothing movement, position, posture, and body movement (raising an arm, crouching, jumping, etc.). For example, the rendering unit 130 may control the viewer avatar's mouth movement and facial expression based on speech sounds picked up by a microphone. The rendering unit 130 may also control the viewer avatar's facial expression (including its line of sight) based on captured images of the viewer's face and eyes captured by a camera. The rendering unit 130 may also control the viewer avatar's facial expression, hair movement, clothing movement, position, posture, and body movement (raising an arm, crouching, jumping, etc.) based on motion data such as the acceleration and angular velocity of the head or other body parts.

[0077] The viewer terminal 10 continuously transmits, from the communication unit 110 to the distribution server 20, movement information of the viewer avatar (various parameters for moving the viewer avatar) in which various sensing data of the viewer is reflected by the rendering unit 130. The movement information of the viewer avatar transmitted to the distribution server 20 may be information including, for example, at least the position and posture.

[0078] Furthermore, the rendering unit 130 generates a real-life 3D image of the performer in the virtual space based on the data of the real-life 3D model of the performer.

[0079] The rendering unit 130 then controls the display of the generated video on the display unit 140.

[0080] (Display section 140) The display unit 140 has a function of presenting an image of a virtual space to a viewer. For example, the display unit 140 can be realized by a 2D display, a 3D display, a projector, or a device for presenting a 3D hologram.

[0081] The above describes the configuration of the viewer terminal 10. Note that the configuration of the viewer terminal 10 is not limited to the example shown in Fig. 7. For example, the viewer terminal 10 may be configured as a display device having at least a display unit 140 and a sensor unit 150, and a control device having at least a communication unit 110 and a rendering unit 130.

[0082] <<3. Operation Processing>> FIG. 8 is a flowchart showing an example of the flow of operational processing of the distribution server 20 according to this embodiment.

[0083] As shown in FIG. 8, first, the distribution server 20 acquires 3D model data of the performer from the imaging system 31 (step S103).

[0084] Next, the distribution server 20 acquires crowd data from other distribution servers (step S106).

[0085] Next, the distribution server 20 acquires movement information of the viewer avatar from the viewer terminal 10 (step S109).

[0086] Next, the control unit 220 of the distribution server 20 determines the mode of the performer based on the mode information acquired from the imaging system 31 (step S112). In this embodiment, mode information indicating whether the mode is the normal mode or the interaction mode may be transmitted from the imaging system 31.

[0087] Next, in the normal mode (step S112 / normal), the virtual object placement unit 221 of the distribution server 20 adjusts the orientation of the 3D model of the performer placed in the virtual space so that it faces the viewer avatar of the distribution destination viewer (step S115). On the other hand, in the interaction mode (step S112 / interaction), the virtual object placement unit 221 does not adjust the orientation.

[0088] Next, the virtual object placement unit 221 performs control to place all virtual objects in the virtual space (step S118). Specifically, the virtual object placement unit 221 updates the placement of each virtual object in the virtual space. The placement update can be performed based on the direction adjustment result, movement information of the viewer avatar transmitted from the viewer terminal 10, etc.

[0089] Next, the virtual object placement unit 221 transmits the information of the virtual space to the viewer terminal 10 as updated information of the virtual space (step S121). The information of the virtual space includes information on the placement (position) of the updated virtual objects, movement information of the viewer avatar for detailed drawing, and crowd data (for simple drawing) acquired from another distribution server.

[0090] The distribution server 20 performs the processes shown in steps S109 to S121 for each destination viewer (step S124).

[0091] Next, when the processing for all destination viewers to whom distribution is to be performed by the distribution server 20 is completed (step S124 / No), the crowd data processing unit 222 generates crowd data by extracting a portion of the movement information of the viewer avatars received from the destination viewer terminals 10 (step S127). Specifically, for example, the crowd data processing unit 222 may extract bone information from the movement information of the viewer avatars of all destination viewers (all viewer avatars primarily processed by the distribution server 20) to generate crowd data. Alternatively, the crowd data processing unit 222 may extract bone information of viewer avatars that exhibit a certain level of movement from among the viewer avatars of all destination viewers (all viewer avatars primarily processed by the distribution server 20) to generate crowd data. For example, only bone information of viewer avatars that are moving their bodies vigorously may be extracted, or only bone information of viewer avatars that are waving their arms vigorously may be extracted. In this way, by extracting only bone information of viewer avatars that are exhibiting characteristic movements, crowd data with a reduced amount of data may be generated.

[0092] Then, the distribution server 20 transmits the crowd data from the inter-server communication unit 230 to the other distribution servers (step S130).

[0093] The operational process of the distribution server 20 according to this embodiment has been described above. 8 The flow of the operation process shown in FIG. 1 is an example, and the present disclosure is not limited to this. 8 The processes shown in steps S103 to S112 in Fig. 8 may be performed in parallel or in a different order. Also, the processes shown in Fig. 8 may be performed once per frame.

[0094] <<4. Hardware configuration example>> Next, an example hardware configuration of an information processing device according to an embodiment of the present disclosure will be described with reference to Fig. 9. The processes performed by the viewer terminal 10, the distribution server 20, the imaging system 31, and the display system 33 described above can be realized by one or more information processing devices. Fig. 9 is a block diagram showing an example hardware configuration of an information processing device 900 that realizes the viewer terminal 10, the distribution server 20, the imaging system 31, or the display system 33 according to an embodiment of the present disclosure.

[0095] It should be noted that the information processing device 900 does not necessarily have to have all of the hardware configuration shown in Fig. 9. Furthermore, some of the hardware configuration shown in Fig. 9 may not be present in the viewer terminal 10, the distribution server 20, the imaging system 31, or the display system 33.

[0096] 9, the information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 903, and a RAM (Random Access Memory) 905. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Instead of or in addition to the CPU 901, the information processing device 900 may include a processing circuit such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an ASIC (Application Specific Integrated Circuit).

[0097] The CPU 901 functions as an arithmetic processing unit and control unit, and controls all or part of the operation of the information processing device 900 in accordance with various programs recorded in the ROM 903, RAM 905, storage device 919, or removable recording medium 927. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. The CPU 901, ROM 903, and RAM 905 are interconnected by a host bus 907, which is an internal bus such as a CPU bus. The host bus 907 is further connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909. For example, the 3D model generation unit 312, encoding unit 313, control unit 220, or decoding unit 120 and rendering unit 130 according to this embodiment may be realized by the CPU 901.

[0098] The input device 915 is a device operated by a user, such as a button. The input device 915 may include a mouse, a keyboard, a touch panel, a switch, a lever, or the like. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device 929 such as a mobile phone that supports operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the signal to the CPU 901. The user operates the input device 915 to input various data to the information processing device 900 and to instruct processing operations.

[0099] The input device 915 may also include an imaging device and a sensor. The imaging device is a device that captures real space and generates a captured image using an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) and various components such as a lens for controlling the formation of a subject image on the imaging element. The imaging device may capture a still image or a moving image.

[0100] The sensors include various types of sensors such as a distance measurement sensor, an acceleration sensor, a gyro sensor, a geomagnetic sensor, a vibration sensor, an optical sensor, and a sound sensor. The sensors acquire information about the state of the information processing device 900 itself, such as the attitude of the housing of the information processing device 900, and information about the surrounding environment of the information processing device 900, such as the brightness and noise around the information processing device 900. The sensors may also include a GPS (Global Positioning System) sensor that receives a GPS signal and measures the latitude, longitude, and altitude of the device.

[0101] The output device 917 is configured with a device capable of visually or audibly notifying the user of acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, or the like. The output device 917 outputs the results obtained by processing by the information processing device 900 as video such as text or images, or as sound such as voice or audio. The output device 917 may also include a lighting device that brightens the surroundings.

[0102] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 is configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs and various data executed by the CPU 901, as well as various data acquired from the outside.

[0103] The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905. The drive 921 also writes information to the attached removable recording medium 927.

[0104] The connection port 923 is a port for directly connecting a device to the information processing device 900. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. The connection port 923 may also be an RS-232C port, an optical audio terminal, or an HDMI (registered trademark) (High-Definition Multimedia Interface) port. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 900 and the external device 929.

[0105] The communication device 925 is, for example, a communication interface configured with a communication device for connecting to the network 931. The communication device 925 may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (registered trademark), Wi-Fi (registered trademark), or WUSB (Wireless USB). The communication device 925 may also be a router for optical communication, a router for an asymmetric digital subscriber line (ADSL), or a modem for various types of communication. The communication device 925 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The network 931 connected to the communication device 925 is a network connected by wire or wirelessly, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0106] <<5. Supplementary Information>> Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0107] For example, the 3D model of the performer is not limited to a live-action 3D model, but may be a 3DCG character.

[0108] Furthermore, the distribution server 20 may be configured without the mode information receiving unit 211. In other words, the distribution server 20 may not have a function for adjusting the direction of the performer based on the mode information.

[0109] Furthermore, the information processing system according to this embodiment may not have the display system 33 shown in FIG.

[0110] It is also possible to create one or more computer programs for causing hardware such as a CPU, ROM, and RAM built into the information processing device 900 to perform the processing functions of the viewer terminal 10, distribution server 20, imaging system 31, or display system 33. A computer-readable storage medium storing the one or more computer programs is also provided.

[0111] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0112] The present technology can also be configured as follows. (1) a control unit that controls the distribution of information about the virtual space in which the performer's virtual object and the viewer's virtual object are placed to one or more viewer terminals; The control unit An information processing device that uses, as movement information for virtual objects of viewers who are distribution destinations of other information processing devices, less information than the movement information used for virtual objects of viewers who are distribution destinations by the control unit, received from the other information processing devices. (2) A plurality of areas are set in the virtual space, in which virtual objects of one or more viewers are respectively placed; The control unit The information processing device described in (1), wherein the control unit places virtual objects of all viewers of distribution destinations in one area, and places virtual objects of all viewers of distribution destinations of the other information processing device in another area different from the one area. (3) The information processing device according to (2), wherein the one area and the other area are arranged at a predetermined distance in the virtual space. (4) The information processing device according to (2) or (3), wherein the movement control of the virtual object of the viewer placed in the other area is performed by simple display control using less information than the movement information. (5) The information processing device according to (4), wherein the information less than the movement information is bone information of a viewer avatar, which is a virtual object of the viewer. (6) The information processing device according to any one of (1) to (5), wherein the number of virtual objects placed in the virtual space exceeds the number of virtual objects for viewers that can be processed by a single information processing device. (7) The control unit The information processing device according to any one of (1) to (6), wherein the control unit controls the transmission of information generated by extracting a portion of the movement information of virtual objects of all viewers to which the control unit is distributed to other information processing devices. (8) The information processing device according to (7), wherein the information generated by extracting a portion of the motion information is less information than the motion information. (9) The information processing device according to (7) or (8), wherein the information generated by extracting a part of the movement information is bone information of a viewer avatar, which is a virtual object of the viewer. (10) The information processing device described in any one of (1) to (9), wherein the control unit adjusts the direction of the performer's virtual object placed in the virtual space so that it faces the direction in which the virtual object of the viewer to whom the distribution is made is located. (11) The information processing device described in any one of (1) to (10), wherein the virtual object of the performer is a real-life 3D model generated based on multiple captured images obtained by simultaneously capturing the performer from multiple directions. (12) The processor: Controlling the distribution of information about the virtual space in which the performer's virtual objects and the viewer's virtual objects are placed to one or more viewer terminals; using, as movement information of the virtual objects of the viewers who are distribution destinations of the other information processing devices among all the virtual objects of the viewers who are arranged in the virtual space, less information than movement information used for the virtual objects of the viewers who are distribution destinations under the distribution control received from the other information processing devices; An information processing method, including: (13) Computer, functioning as a control unit that controls the distribution of information about the virtual space in which the performer's virtual object and the viewer's virtual object are placed to one or more viewer terminals; The control unit A program that uses, as movement information for virtual objects of viewers who are distribution destinations of other information processing devices, less information than the movement information used for virtual objects of viewers who are distribution destinations by the control unit, received from the other information processing devices. [Explanation of symbols]

[0113] 10 (10A, 10B, 10C) Viewer terminal 110 Communications Department 120 Decryption Department 130 Rendering Department 140 Display section 150 Sensor unit 20(20A, 20B, 20C) Distribution server 210 3D model data receiving unit 211 Mode information receiving unit 220 Control Unit 221 Virtual Object Placement 222 Crowd Data Processing Unit 230 Inter-server communication section 240 Terminal communication unit 31 Imaging System 311 Data Acquisition Department 311a Camera 312 3D model generation unit 313 Encoding section 314 Transmitter 33 Display System 331 Display 900 Information Processing Equipment

Claims

1. a control unit that controls distribution of information about a virtual space in which the performer's virtual object and the viewer's virtual object are placed to one or more viewer terminals; a plurality of areas respectively associated with a plurality of information processing devices that perform distribution are set in the virtual space; The control unit placing each virtual object of the one or more viewers as a distribution destination by the control unit in one area corresponding to the information processing device among the plurality of areas; placing each virtual object of the one or more viewers as distribution destinations by the other information processing device in another area, among the plurality of areas, that is associated with the other information processing device and that is spaced a predetermined distance from the one area; Each virtual object can only move within the area in which it is placed. With respect to a virtual object arranged in the other area that is visible from a viewpoint of the virtual object arranged in the one area, simple display control is performed using, as its movement information, less information than the movement information used for the virtual object arranged in the one area, which is received from the other information processing device. Information processing device.

2. The information processing device according to claim 1 , wherein the control unit reflects the same movement in a predetermined number of units based on movement information of some of the virtual objects arranged in the other area as simple display control for the virtual objects arranged in the other area.

3. The information processing device according to claim 2 , wherein the information less than the movement information is bone information of a viewer avatar that is a virtual object of the viewer.

4. The information processing device according to claim 1 , wherein the virtual space contains virtual objects in a number exceeding the number of virtual objects of viewers that can be processed by a single information processing device.

5. The control unit The information processing device according to claim 1 , wherein the control unit controls to transmit information generated by extracting a portion of movement information of each virtual object of a plurality of viewers as distribution destinations to another information processing device.

6. The information processing apparatus according to claim 5 , wherein the information generated by extracting a portion of the motion information is less information than the motion information.

7. The information processing device according to claim 6 , wherein the information generated by extracting a part of the movement information is bone information of a viewer avatar that is a virtual object of the viewer.

8. The information processing device according to claim 1 , wherein the control unit adjusts the direction of the virtual object of the performer placed in the virtual space so that the virtual object faces in a direction toward which a virtual object of a viewer of distribution destination is located.

9. The information processing device according to claim 1 , wherein the virtual object of the performer is a real-life 3D model generated based on a plurality of captured images obtained by simultaneously capturing images of the performer from a plurality of directions.

10. The processor: Controlling the distribution of information about the virtual space in which the performer's virtual objects and the viewer's virtual objects are placed to one or more viewer terminals; setting a plurality of areas in the virtual space, each of the areas corresponding to a plurality of information processing devices that perform distribution; placing each virtual object of the one or more viewers as a distribution destination in one area among the plurality of areas that is associated with the information processing device; placing each virtual object of the one or more viewers as distribution destinations by the other information processing device in another area, among the plurality of areas, that is associated with the other information processing device and that is spaced a predetermined distance from the one area; For a virtual object arranged in the other area that is visible from a viewpoint of the virtual object arranged in the one area, simple display control is performed using, as its movement information, less information than the movement information used for the virtual object arranged in the one area, which is received from the other information processing device; Including, Each virtual object can only move within the area in which it is placed. Information processing methods.

11. Computer, functioning as a control unit that controls the distribution of information about the virtual space in which the performer's virtual object and the viewer's virtual object are placed to one or more viewer terminals; a plurality of areas respectively associated with a plurality of information processing devices that perform distribution are set in the virtual space; The control unit each virtual object of the one or more viewers to whom the control unit delivers the content is placed in one area corresponding to the information processing device among the plurality of areas; each virtual object of one or more viewers to whom the other information processing device delivers the content is placed in another area among the plurality of areas that is associated with the other information processing device and that is spaced a predetermined distance from the one area; Each virtual object can only move within the area in which it is placed. With respect to a virtual object arranged in the other area that is visible from a viewpoint of the virtual object arranged in the one area, simple display control is performed using, as its movement information, less information than the movement information used for the virtual object arranged in the one area, which is received from the other information processing device. program.

Images