Information processing system, information processing method, and program

Abstract

An information processing system according to the present disclosure comprises a first information processing device and a second information processing device connected to the first information processing device, wherein the first information processing device or the second information processing device includes: a determining unit that determines, on the basis of the position, in a virtual space, of a person on a third information processing device that is connected to the first information processing device and that shares the virtual space with the first information processing device, whether to display either a captured image of the person to be displayed on the first information processing device or a generated avatar image corresponding to the captured image of the person; and a rendering unit that renders the captured image or the avatar image determined by the determining unit.

Description

Information processing system, information processing method and program

[0001] The present disclosure relates to an information processing system, an information processing method, and a program.

[0002] There are known communication systems in which a user's avatar at a location and other users' avatars at other locations gather and communicate in a virtual space. Examples of such communication systems include conference systems and game systems.

[0003] In such communication systems, there is a technology that determines the transmission rate between locations based on the distance in virtual space between the user's avatar and another user's avatar. With this technology, the closer the distance in virtual space between the user's avatar and another user's avatar, the higher the transmission rate allocated between the locations where the user is located.

[0004] Japanese Patent Application Laid-Open No. 2018-28789

[0005] In virtual spaces, not only avatars but also people captured by cameras are sometimes used. However, when using avatars or people captured in virtual spaces, the amount of data required for the people captured can be large, which can impair the real-time display of the avatars or people captured in virtual spaces.

[0006] The present disclosure has been made in consideration of the above-mentioned circumstances, and aims to prevent the real-time nature of the virtual space from being impaired by appropriately allocating avatars and captured people in the virtual space.

[0007] The information processing system of the present disclosure includes a first information processing device and a second information processing device connected to the first information processing device, and the first information processing device or the second information processing device has a determination unit that determines whether to display a captured image of a person displayed on the first information processing device or a generated avatar image corresponding to the captured image of the person based on a position in the virtual space of the person on a third information processing device that is connected to the first information processing device and shares a virtual space with the first information processing device, and a rendering unit that renders the captured image or the avatar image determined by the determination unit.

[0008] 1 is a diagram illustrating an example in which users from multiple locations participate in a 3D remote conference system according to a first embodiment. FIG. 2 is a diagram illustrating the arrangement of a camera on the capture side. FIG. 3 is a diagram illustrating a skeletal posture displayed by skeletal posture information according to the first embodiment. FIG. 4 is a functional block diagram of an information processing system according to the first embodiment. FIG. 5 is a flowchart illustrating the flow of avatar generation and registration performed when a participant at location A participates. FIG. 6 is a flowchart illustrating an overview of the operation of the information processing device at location A in the information processing system according to the first embodiment. FIG. 7 is a flowchart illustrating the operation of a cloud server according to the first embodiment. FIG. 8 is a diagram illustrating an example of the relationship between the angle of view of a virtual camera at a location and a person at location B. FIG. 9 is a flowchart illustrating the operation of the information processing device at location B in the information processing system according to the first embodiment. FIG. 10 is a diagram illustrating an example of determination by various devices at location B. FIG. 11 is a functional block diagram of an information processing system according to a second embodiment. FIG. 12 is a functional block diagram of an information processing system according to a third embodiment. FIG. 13 is a diagram illustrating the viewpoint (virtual camera) of location B. FIG. 14 is a flowchart illustrating the operation of the information processing device at location B in the information processing system according to the third embodiment. FIG. 15 is a functional block diagram of an information processing system according to a fourth embodiment. FIG. 16 is a diagram illustrating a situation in which a person at location A is viewed from the position of a person at location B in three-dimensional space. 10 is a flowchart for explaining the operation of an information processing device at base B of an information processing system according to a fourth embodiment. FIG. 11 is a hardware configuration diagram showing an example of a computer that realizes the information processing devices and the arithmetic unit of a cloud server according to the first to fourth embodiments.

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. The description will be given in the following order.

[0010] 1. Background of the present disclosure 2. First embodiment 2.1. Overview of information processing system 1 2.1.1. Overview of operations at base A 2.1.2. Overview of operations at base B 2.2. Functional block diagram of information processing system 1 2.3. Explanation of operations 2.3.1. Preparatory operations 2.3.2. Operations of information processing device 2 at base A 2.3.3. Operations of cloud server 3 2.3.3.1. Determination of person information 2.3.3.2. Number of people that can be processed 2.3.4. Operations of information processing device 4 at base B 3. Second embodiment 3.1 Functional block diagram of information processing system 100 4. Third embodiment 5. Fourth embodiment 6. Other embodiments 7. Effects 8. Hardware configuration

[0011] 1. Background of the Disclosure Due to the impact of COVID-19, remote communication methods such as Zoom (registered trademark) have become widely used. However, the current display of 2D solutions (videos) lacks nuance, atmosphere, ambience, and realism, making it difficult to achieve smooth communication.

[0012] For this reason, a method has been devised that allows for 6 DoF (six degrees of freedom) of 3D freedom within the virtual space, and allows each location to display their own video and audio in three dimensions to the other party using two-way transmission and reception communication.

[0013] Currently, in 3D 6DoF two-way communication, people at each location are often represented as avatars. For example, VR (Virtual Reality) Chat provides a virtual space in which each user can move around in 6DoF as an avatar. In the virtual space, users can express their own movements using skeletal sensing with a controller or camera. Avatars can also be made into 3D objects that match the user's appearance by capturing the user from an anime-style object.

[0014] Several methods for efficiently modeling 3D objects have been developed. For example, Japanese Patent Application Laid-Open Publication No. 2014-10805 discloses a technology for achieving high speed and high definition by selecting a camera from multiple cameras based on a virtual viewpoint and optimizing the cameras.

[0015] Furthermore, Japanese Patent Application Laid-Open No. 2019-36790 discloses a technology for changing the resolution of voxels, which are 3D representation formats such as Visual Hull, depending on the distance from a virtual viewpoint.

[0016] The above two technologies can contribute to reducing the processing load, but they do not describe any measures to prevent loss of real-time performance when the processing load becomes high at multiple locations or when processing becomes heavy due to the capabilities of the user's equipment.

[0017] On the other hand, as mentioned above, there is a technology for displaying an avatar as a 3D object. With such a technology, once the 3D object represented by the avatar is transmitted to each terminal, information for changing the user's own movement can be realized with a small amount of transmitted and received data, such as skeletal posture information.

[0018] Using a system that transmits small amounts of data, such as skeletal posture information, can avoid the problem of data size during remote communication between multiple locations, such as online meetings. However, because such systems display users as avatars, they have the problem of being unable to express the user as they really are. Even if the system creates an avatar that is close to the real thing, because it is an avatar, it cannot reproduce the user's detailed finger movements, hair and clothing movements, facial expressions, etc.

[0019] Meanwhile, codec technology has been developed that expresses captured people as 3D data (point cloud, mesh, etc.) and compresses that data. However, this technology has the problem that the amount of data transmitted increases linearly with the number of locations (the number of people captured), making it impossible to support multiple locations.

[0020] Furthermore, such systems have a problem in that the processing load for generating and displaying 3D models increases linearly. This problem is particularly pronounced on devices such as smartphones, which have lower processing power than PCs and require pay-per-use communications. These problems manifest as low frames per second (FPS) and high latency, significantly impairing the user experience.

[0021] The device disclosed herein solves the problems of low FPS and high latency by determining whether to display avatars or real-time capture of people for each frame of 3D person data for each location in a virtual space. This allows the device disclosed herein to realize two-way 3D communication that can withstand expansion of locations by appropriately using avatars rather than real-time capture, without compromising realism, even when multiple locations are expanded.

[0022] Specifically, the device disclosed herein uses the observer's posture information in 3D space, the position information of the capture target (remote capture target person), the observer's gaze information, the processing capabilities of the observer's display device (CPU / GPU / memory usage rate and screen size), the capabilities of the person capture device (capture range, etc.), and transmission information (transmission type and transmission bandwidth, such as LAN / WiFi / 5G) to determine whether to display the user at each location as a captured image or an avatar image.

[0023] Based on the determination result, the device of the present disclosure presents the observer with a real-time captured image of the person as much as possible, while changing the display to an avatar for areas that are difficult for the observer to notice. Furthermore, if the device used by the user cannot maintain its performance, the device of the present disclosure switches the display of the user to an avatar, thereby maintaining low latency and high FPS.

[0024] Furthermore, the device disclosed herein switches the display in the virtual space from the captured image to an avatar when the amount of data transmission exceeds a threshold based on the observer's posture information in the 3D space, the position information of the capture target (a remote person to be captured), the observer's line of sight, etc. For example, the amount of data transmission varies depending on the RGB / Depth image size and the size of the area in which the person is displayed.

[0025] For example, the size of the data transmitted, in the case of the data size required for avatar animation, is: Number of joints (approximately 30) * (Position xyz 12 bytes + Posture xyz 12 bytes) = 720 bytes / frame The data size of the point cloud required for real-time capture data is (calculated using Depth image resolution 640 * 480), Number of points (150,000) + ((Position xyz 12 bytes) + (Color xyz 3 bytes)) = 225 KB / frame When compressed using Draco (compression technology for 3D graphic data), which is required for real-time capture data, the amount is 33 KB / frame. By using avatars, the amount of data transmission required to draw one frame can be reduced to approximately 1 / 30 per location compared to real-time capture data.

[0026] In other words, since the data size of an avatar is smaller than the data size of a captured image, reducing the transmission amount of the captured image can improve the real-time performance of presenting a person. As described above, the device disclosed herein can appropriately reduce the processing amount and transmission amount according to each user's device in 6DoF 3D two-way communication at multiple locations (i.e., many people), enabling person presentation with high FPS and low latency.

[0027] 2. First Embodiment For simplicity, the following explanation will be given assuming that two locations (location A and location B) are transmitting and receiving data. However, this system is characterized by being able to withstand expansion of locations by appropriately using avatars to reduce the amount of information, rather than real capture, even if the number of locations increases, so in some cases an example will be given with multiple locations.

[0028] In this case, the locations after B are designated by alphabetical letters (location C, location D, ...). Each location is equipped with one or more cameras necessary for generating 3D people, and the data captured at each location is sent and received between them.

[0029] Furthermore, each site displays the received data on a head-mounted display (HMD) or smartphone and presents it to the observer at that site. Thus, each site has its own capture system and display system. In the following explanation, site A will be referred to as the capture side, and site B as the display side.

[0030] FIG. 1 is a diagram showing an example in which users from multiple locations participate in a 3D remote conference system according to a first embodiment. FIG. 1 shows an example in which people Ha to Hf participate from their respective remote locations, and a capture image or avatar image of each person is displayed in a virtual space. In FIG. 1, person Hf is shown as a capture image, and people Ha to He are shown as avatar images. Furthermore, in the embodiment, a 3D remote conference system in a VR space such as that shown in FIG. 1 is described as an example, but the present invention can be applied to any use case.

[0031] <2.1. Overview of Information Processing System 1> A general flow of the information processing system 1 according to the embodiment will be described. FIG. 2 is a diagram showing the arrangement of the camera Ca on the capture side. However, it is sufficient for the camera Ca to be able to capture data necessary to construct 3D data. For example, the camera Ca may capture images from a stereo camera, etc. Furthermore, the installation positions of the cameras Ca and the number of cameras Ca are not limited to the format shown in this figure. In the following description, a camera Ca capable of acquiring RGB images and depth images will be used to acquire 3D data.

[0032] Next, the advance preparations for a user to participate in a 3D remote conference system will be described. In the advance preparations, when a user participates in the 3D remote conference system, the user's own real avatar is captured by a camera C at a base on the capture side. The captured real avatar of the user is then distributed to each base on the display side when the user participates. Note that in recent years, it has become possible for smartphones to create one's own real avatar. A real avatar created on a smartphone may also be used in the embodiment. Furthermore, although the display side is assumed to be a display of the VR space using an HMD, smartphone, or the like, the display device on the display side can be applied to any device capable of displaying 3D data.

[0033] <2.1.1. Overview of operations at site A> 1. The information processing device at site A acquires skeletal posture information using RGB images, depth images, and existing technology from a camera at site A. FIG. 3 is a diagram showing a skeletal posture displayed using the skeletal posture information according to the first embodiment. The skeletal posture information includes information representing the positions and rotations of multiple joints R relative to the skeleton. 2. The skeletal posture information is always transmitted to all display-side sites. The RGB images and depth images are transmitted only to necessary sites based on the avatar / person capture determination conditions described below.

[0034] <2.1.2. Overview of Operation at Site B> 1. The information processing device at site B acquires information necessary for determining switching between avatars and person capture, which will be described later, and transmits the acquired information necessary for switching determination to all sites for each frame.

[0035] 2. Next, the information processing device at site B determines whether the RGB image and the depth image have been received. If the information processing device at site B determines that the RGB image and the depth image have been received, it converts the RGB image and the depth image into point cloud or mesh data and displays them. If the information processing device at site B determines that the RGB image and the depth image have not been received, it applies the RGB image and the depth image together with the skeletal posture information to the avatar and displays an animation.

[0036] In the above flow, an example is described in which the information processing device at site B performs modeling, but it is also possible for the information processing device at site A to perform modeling and transmit the modeled data to the information processing device at site B. In addition, in the detailed system explanation below, an explanation is given in which a cloud server is placed between the information processing device at site A and the information processing device at site B, but the information processing device at site A and the information processing device at site B may be connected via peer-to-peer, and data may be transmitted and received between the respective sites.

[0037] <2.2. Functional Block Diagram of Information Processing System 1> Fig. 4 is a functional block diagram of the information processing system 1 according to the first embodiment. In the information processing system 1 according to the first embodiment, 3D modeling is performed at a site B. As shown in Fig. 4, the information processing system 1 includes an information processing device 2 and a cloud server 3 at a site A (capture side), and an information processing device 4 at a site B (display side). The information processing device 2, cloud server 3, and information processing device 4 are connected to each other via a network.

[0038] Although each location is both a capture side and a display side at the same time, the capture side and the display side are shown separately in the embodiment for the sake of simplicity of explanation.

[0039] The information processing device 2 includes a capture unit 11 , a video encoding unit 12 , a skeleton / posture encoding unit 13 , a packetizing unit 14 , a storage unit 15 , and an avatar uploading unit 16 .

[0040] The capture unit 11 acquires an RGB image, a depth image, and skeletal posture information captured by the camera Ca. The skeletal posture information is acquired based on the RGB image and the depth image using existing technology.

[0041] The video encoding unit 12 compresses and encodes the RGB image and the depth image acquired by the capture unit 11 .

[0042] The skeleton posture encoding unit 13 compresses and encodes the skeleton posture information acquired by the capture unit 11 .

[0043] The packetizing unit 14 packetizes the RGB images and depth images compression-encoded by the video encoding unit 12 and transmits them to the cloud server 3 only at necessary locations based on the avatar and person capture switching determination conditions described below. The packetizing unit 14 also packetizes the skeleton posture information compression-encoded by the skeleton posture encoding unit 13 and transmits them to the cloud servers 3 at all locations.

[0044] The storage unit 15 of the information processing device 2 stores rigged 3D live-action avatars. Here, "rig" refers to the mechanism, such as joints, that moves the 3D live-action avatar. The rig is attached to skeletal posture information. The rigged 3D live-action avatars are generated using existing technology. The 3D live-action avatars may have poor image quality, for example, because they are avatars that represent the appearance of the user on that day, but they may also be simple avatars that can be generated on the spot in a few minutes.

[0045] When participating in a 3D remote conference system, the information processing device 2 at the site A captures the real avatar of the user at the site A and stores the captured rigged 3D live-action avatar in the storage unit 15 of the information processing device 2.

[0046] The avatar upload unit 16 uploads the rigged 3D live-action avatar stored in the storage unit 15 of the information processing device 2 to the cloud server 3 .

[0047] The cloud server 3 includes a data transmission control unit 21 , a session management unit 22 , and a storage unit 23 of the cloud server 3 .

[0048] The data transmission control unit 21 receives packetized RGB images, depth images, and skeletal posture information from the information processing device 2 at each site A. The data transmission control unit 21 determines the data (avatar image or captured image) to be transmitted to the information processing device 4 at site B based on the determination information displaying the avatar or captured image of the person transmitted from the avatar / person determination information transmission unit 38. The data transmission control unit 21 transmits the skeletal posture information transmitted by the packetizing unit 14 to all sites. Furthermore, the data transmission control unit 21 transmits the video of the RGB images and depth images transmitted by the packetizing unit 14 to site B only when person capture is required at site B.

[0049] The data transmission control unit 21 sends the necessary amount of person capture data (RGB image and depth image) to the site B based on the determination information notified from the avatar / person determination information transmission unit 38 of the site B. As described above, the skeletal posture information is transmitted to all the sites.

[0050] Specifically, the data transmission control unit 21 determines a capture image to be sent to the information processing device 4 based on the field of view information, line of sight information, and the number of people that can be processed sent by the avatar / person determination information sending unit 38, and sends the determined capture image to the information processing device 4.

[0051] When the participant is notified, the session management unit 22 registers the participant in the cloud server 3. When an avatar is uploaded, the session management unit 22 registers the uploaded avatar in the storage unit 23. When a new participant (base) is added, the session management unit 22 distributes the avatar stored in the storage unit 23 of the cloud server 3 to the information processing device 4 of base B. After downloading the avatar to the information processing device 4 of base B, the session management unit 22 updates the session with the information processing device 2 of base A.

[0052] The storage unit 23 of the cloud server 3 stores the rigged 3D live-action avatar uploaded by the avatar upload unit 16 at the site A.

[0053] The information processing device 4 has an unpacketizing unit 31, a video decoding unit 32, an image processing unit 33, a rendering unit 34, a skeleton posture decoding unit 35, a transmission information monitoring unit 36, a processing capacity monitoring unit 37, an avatar / person determination information transmitting unit 38, a viewpoint / gaze monitoring unit 39, a person presence determination unit 40, an avatar downloading unit 41, a user interface processing unit 42, and a display unit 43.

[0054] The unpacketizing unit 31 receives and unpacketizes the packetized data (RGB image, depth image, and skeleton posture information) sent from the data transmission control unit 21. The data received by the unpacketizing unit 31 includes the field of view information, line of sight information, and captured images of people for the number of people that can be processed, all sent by the avatar / person determination information sending unit.

[0055] The video decoding unit 32 performs video decoding on the person capture data (RGB image and depth image) and extracts an image from the decoded video. The image processing unit 33 performs image processing to display the image extracted by the video decoding unit 32.

[0056] The rendering unit 34 renders the image processed by the image processing unit 33 on the display unit 43. The rendering unit 34 performs modeling and rendering on the person capture data (RGB image and depth image).

[0057] Furthermore, in the case of a person for which there is no person capture data (RGB image and depth image) and only skeletal posture information, the rendering unit 34 applies animation to the avatar corresponding to that person downloaded by the avatar downloading unit 41 and renders it on the display device. In other words, the rendering unit 34 renders the capture image or avatar image of the person received by the unpacketizing unit 31 and displays it on the display unit 43.

[0058] The skeleton posture decoding unit 35 decodes the skeleton posture information unpacketized by the unpacketizing unit 31 to extract the skeleton posture.

[0059] The transmission information monitor 36 monitors the transmission information of the information processing device 4. Specific transmission information will be described later. The processing capacity monitor 37 monitors the processing capacity of the information processing device 4. Specific processing capacity will be described later.

[0060] The avatar / person determination information transmission unit 38 transmits angle of view information and line of sight information of people of the information processing device 4 to the cloud server 3. The avatar / person determination information transmission unit 38 determines the number of people that can be processed in real person capture (first processable number of people) based on the processing capacity of the information processing device 4 monitored by the processing capacity monitor unit 37 and the transmission information of the information processing device 4 monitored by the transmission information monitor unit 36. The avatar / person determination information transmission unit 38 transmits the determined processable number of people to the cloud server 3. The determination information includes angle of view information, line of sight information, and the processable number of people.

[0061] The avatar / person determination information transmitting unit 38 calculates the number of people that can be processed (second number of people that can be processed) based on the processing capacity of the information processing device 4 and the transmission information.

[0062] The viewpoint / line of sight monitor unit 39 monitors the viewpoint and line of sight of the camera in the virtual space at site B based on the skeleton posture information decoded by the skeleton posture decoding unit 35 .

[0063] The person presence determination unit 40 determines whether there is a person not being processed (displayed) within the angle of view of the virtual camera of the information processing device 4, and when it is determined that a person not being displayed exists within the angle of view of the virtual camera, it determines to display the not-displayed person as an avatar image. The person presence determination unit 40 is connected to the information processing device 2 via the cloud server 3 and determines whether to display a captured image of the person displayed on the information processing device 4 or a generated avatar image corresponding to the captured image of the person, based on the position in the virtual space of the person of the information processing device 2 that shares the virtual space with the information processing device 4. When there is a person being displayed within the angle of view of the virtual camera of the information processing device 4, the person presence determination unit 40 determines to display a captured image of the person for the displayed person.

[0064] The avatar download unit 41 downloads the 3D live-action avatar stored in the storage unit 23 of the cloud server 3.

[0065] The user interface processing unit 42 performs processing in response to operations (e.g., head tracking, controller operation, etc.) via the user interface of the information processing device 4. The user interface processing unit 42 acquires field of view information and line of sight information. Of the operations via the user interface, information relating to the viewpoint / line of sight monitor unit 39 is input to the viewpoint / line of sight monitor unit 39, and information relating to the person presence determination unit 40 is input to the person presence determination unit 40.

[0066] The display unit 43 is a display device such as a display of the information processing device 4, and displays a captured image or an avatar image of a person rendered by the rendering unit 34. The display unit 43 may be, for example, a head mounted display (HMD), a smartphone, or the like.

[0067] <2.3. Description of Operation> <2.3.1. Advance Preparation Operation> FIG. 5 is a flowchart for explaining the flow of avatar generation and registration performed when a participant from site A joins.

[0068] The information processing device 2 at the site A generates a real avatar using a smartphone or a capture camera of the system (step S1). Then, the information processing device 2 at the site A notifies the cloud server 3 of participation (step S2). The information processing device 2 registers the participant on the cloud server 3 (step S3).

[0069] Thereafter, the avatar uploader 16 uploads the generated avatar to the cloud server 3 (step S4). When the avatar is uploaded, the session manager 22 of the cloud server 3 registers the uploaded avatar in the storage unit 23 (step S5). The cloud server 3 notifies the base B that a new participant has been added.

[0070] The avatar download unit 41 of the information processing device 4 at site B downloads the avatar of the new participant (step S6). Also, in step S6, the internal parameters and external parameters of each camera at site A are calibrated, and the calibrated internal parameters and external parameters of each camera are transmitted to site B.

[0071] Thereafter, the session management unit 22 downloads the avatar to the information processing device 4 at the site B, and then updates the session with the information processing device 2 at the site A (step S7).

[0072] <2.3.2. Operation of Information Processing Device 2 at Site A> FIG. 6 is a flowchart for explaining an outline of the operation of the information processing device 2 at site A in the information processing system 1 according to the first embodiment.

[0073] The capture unit 11 acquires the RGB image and the depth image acquired by the camera Ca, and the skeleton posture information acquired based on the RGB image and the depth image (step S10). The skeleton posture information is data representing the position and rotation of the skeleton, as shown in FIG.

[0074] The video encoding unit 12 compression-encodes the RGB image and the depth image acquired by the capture unit 11, and the skeleton posture encoding unit 13 compression-encodes the skeleton posture information acquired by the capture unit 11 (step S11).

[0075] The packetizing unit 14 packetizes the RGB image and the depth image compression-encoded by the video encoding unit 12, and transmits them to the cloud server 3. The packetizing unit 14 also packetizes the skeleton posture information compression-encoded by the skeleton posture encoding unit 13, and transmits them to the cloud servers 3 at all bases (step S12). Note that the order of steps S11 and S12 may be reversed. Furthermore, the operations of steps S10 to S12 are performed for each frame. In other words, a captured image is obtained for each frame.

[0076] The cloud server 3 transmits data synchronously using a protocol such as WebRTC (Web Real-Time Communication) or SRT (Secure Reliable Transport), but the present embodiment is not limited to this.

[0077] 2.3.3. Operation of Cloud Server 3 Next, a description will be given of the operation of the cloud server 3. Fig. 7 is a flowchart for explaining the operation of the cloud server 3 according to the first embodiment.

[0078] The data transmission control unit 21 receives the packetized RGB image, depth image, and skeletal posture information from the information processing device 2 at each site A (step S20). The data transmission control unit 21 determines the data (captured image) to be transmitted to the information processing device 4 at site B based on the determination information indicating the captured image of the avatar or person transmitted from the avatar / person determination information transmission unit 38 (step S21). Note that it is obvious that the data from site A is not transmitted to site A.

[0079] In step S21, the data transmission control unit 21 determines a captured image of a person to be sent to the information processing device 4 at base B based on the field of view information of the virtual camera of the information processing device 4 sent from the avatar / person determination information sending unit 38, the gaze information of the user of the information processing device 4, and the number of people that can be processed by the information processing device 4.

[0080] Thereafter, the data transmission control unit 21 transmits the data determined in step S21 to each base station (step S22). The data transmission control unit 21 also transmits the skeleton posture information transmitted by the packetizing unit 14 to all base stations. The data transmission control unit 21 transmits the RGB image and Depth image video transmitted by the packetizing unit 14 to base station B only when person capture is required at base station B.

[0081] <2.3.3.1. Determination of Person Information> A case where person information is determined from the angle of view information sent in step S31 will be described.

[0082] At site B, the viewpoint from the virtual camera is used to determine the range displayed on the HMD or smartphone. Field-of-view information refers to the camera's FOV (Field of View), focal length, and near / far clip. Viewpoint information, in this case, refers to the camera's position and orientation. In step S21, the person presence determination unit 40 calculates, from the field-of-view information and viewpoint information, whether the position information of the skeleton posture information of each site is within the viewing frustum formed by the virtual camera. Here, the "viewing frustum" refers to a 3D volume within the scene positioned relative to the viewport camera. The shape of this volume affects how the model is projected from camera space onto the screen. If at least one skeleton is present within the screen, the person presence determination unit 40 determines that a person image should be sent to that site, and transmits the person image in step S22.

[0083] Fig. 8 is a diagram showing an example of the relationship between the angle of view of virtual camera BCAb at location B and the person at the location. In Fig. 8, the angle of view of virtual camera BCAb at location Bp includes the person Ha to be displayed at location A, the person Hb to be displayed at location B, and the person Hd to be displayed at location D, but does not include the person Hc to be displayed at location C. Therefore, in the example of Fig. 8, the person at location C is not transmitted.

[0084] <2.3.3.2. Processable Number of People> The data transmission control unit 21 of the cloud server 3 receives the processable number of real-time capture people transmitted from the avatar / person determination information transmission unit 38 of base B. The processable number of real-time capture people is obtained from the calculation results of the processing capacity monitor unit 37 and the transmission information monitor unit 36. If the processable number of real-time capture people is less than the number determined in step S31, the data transmission control unit 21 determines the person model to be ultimately transmitted based on the following determination conditions. Note that which determination conditions to combine can be set by the application, and examples of the determination conditions are described here.

[0085] Distance of person from virtual camera The data transmission control unit 21 reduces the amount of data to be transmitted and processed by not transmitting person modeling information for people who are far from the virtual camera. For example, the distance can be calculated by calculating the Euclidean distance from the head position of the skeletal data of each location to the camera position. For example, if the number of people that can be processed is two, in the example shown in FIG. 8, the data transmission control unit 21 does not transmit the RGB and depth images of location E.

[0086] In other words, if the second processable number of people calculated based on the processing capacity of the information processing device 4 and the transmission information is smaller than the first processable number of people transmitted by the avatar / person determination information transmission unit 38, the data transmission control unit 21 determines the capture image to be transmitted to the information processing device 4 based on the distance between the virtual camera and the person.

[0087] Whether important parts are within the field of view The data transmission control unit 21 determines whether the head position and the positions of both hands in the position information of the skeletal posture information are within the field of view. For example, in the example shown in Fig. 8, the data for site D has the head outside the field of view, so the data transmission control unit 21 does not transmit the RGB image and the depth image.

[0088] 8, it is assumed that the person at point E is hiding his / her head from the person at point A. There are various methods for determining whether or not the person is hiding, but an example of a simple method for determining whether or not the person is hiding is a method using ray casting. In the method using ray casting, the data transmission control unit 21 sends a ray from the virtual camera toward the head position of point E, and if the ray collides with another object, it can be determined that the person is hiding.

[0089] That is, if a specific part of a person is not included in the angle of view of the virtual camera of the information processing device 4, the data transmission control unit 21 does not transmit a captured image of the person to the information processing device 4. Therefore, the amount of data transmitted to the information processing device 4 and the amount of data to be processed by the information processing device 4 are further reduced.

[0090] <2.3.4. Operation of information processing device 4 at site B> The following explanation will be based on the situation shown in Fig. 8. Fig. 8 is a diagram showing the positional relationship between the angle of view of the virtual camera at site B and the people at each site. As shown in Fig. 8, the angle of view of the virtual camera at site B does not include the person at site C, but does include the person at site A, the person at site D, and the person at site E.

[0091] FIG. 9 is a flowchart for explaining the operation of the information processing device 4 at the site B of the information processing system 1 according to the first embodiment.

[0092] The user interface processing unit 42 acquires field of view information and line of sight information (step S30). Specifically, the user interface processing unit 42 acquires the virtual camera's FOV, focal length, near / far clip information, and line of sight information (information on the camera's position and orientation). Next, the avatar / person determination information transmission unit 38 transmits the acquired field of view information and line of sight information to the cloud server 3 (step S31).

[0093] Next, the unpacketizing unit 31 receives the packetized data (RGB image, depth image, and skeleton posture information) sent from the data transmission control unit 21 and unpacketizes it (step S32).

[0094] Next, the video decoding unit 32 performs video decoding on the person capture data (RGB image and depth image) and extracts an image from the decoded video. Also, the skeleton posture decoding unit 35 decodes the skeleton posture information unpacketized by the unpacketizing unit 31 and extracts the skeleton posture (step S33).

[0095] Next, the rendering unit 34 models 3D data from the person capture data (RGB image and depth image) and performs rendering (step S34). Furthermore, in the case of a person for which there is no person capture data (RGB image and depth image) and only skeletal posture information, the rendering unit 34 applies animation to the avatar corresponding to that person downloaded by the avatar download unit 41 and performs rendering on the display device.

[0096] In the first embodiment, the modeling may involve generating a point cloud from the depth data, or may involve generating TSDF data from the depth data and generating mesh data using the marching cubes method.

[0097] Next, the avatar / person determination information transmission unit 38 determines the number of people that can be processed based on the processing capacity of the information processing device 4 monitored by the processing capacity monitor unit 37 and the transmission information of the information processing device 4 monitored by the transmission information monitor unit 36 ​​(step S35).

[0098] In step S35, the avatar / person determination information transmission unit 38 saves the CPU / GPU / memory usage rates used in the process of step S34 and the average transmission / reception bit rates used in the process of step S32. If the saved CPU / GPU / memory usage rates or average transmission / reception bit rates exceed thresholds set by the application thereafter, the avatar / person determination information transmission unit 38 uses these values ​​to recalculate the number of people that can be processed. As an example, a method for calculating CPU usage rate is shown below.

[0099] For example, if the threshold CPU usage rate is set to an increase of Z%, and the usage rate increases to Y% when X people are processed in S34, the avatar / person determination information sending unit 38 can calculate the new number of people that can be processed as follows: Number of people that can be processed = X + [(Z - Y) / 100 * X] The same calculation is performed for all indicators to calculate the number of people that can be processed.

[0100] Then, the avatar / person determination information transmission unit 38 transmits the determined number of people that can be handled to the cloud server 3 (step S36).

[0101] Next, the person presence determination unit 40 determines whether there is a person within the field of view of the virtual camera that has not been processed (displayed) (step S37). Specifically, in step S37, the person presence determination unit 40 determines, among the people at the base that should be processed within the field of view, those that were not processed in step S34. This is because the data transmission control unit 21 transmits only the amount of person real-time capture data (RGB image and depth image) that can be processed in step S21, and the remaining people are displayed as avatars.

[0102] Furthermore, since the determination in step S21 is based on the information transmitted from step S31 and step S36, the person presence determination unit 40 needs to calculate the amount of person real-time capture data that can be processed based on the latest virtual camera information. The person presence determination unit 40 uses the latest virtual camera angle of view / viewpoint information to calculate whether the position information of the skeleton posture information of the unprocessed base is within the viewing frustum formed by the virtual camera.

[0103] If the person presence determination unit 40 determines that there is an unprocessed person within the field of view (step S37; Yes), the rendering unit 34 renders an avatar (step S38). If there is no unprocessed person within the field of view (step S37; No), the person presence determination unit 40 determines that the person displayed within the field of view will be displayed in the captured image, and ends the process.

[0104] In step S38, the rendering unit 34 performs rendering to display animated avatars with skeleton posture information applied to people within the field of view and people at bases not displayed in the processing described in step S21.

[0105] Fig. 10 is a diagram showing examples of determinations made by various devices at site B. As shown in Fig. 10, the avatar / person determination information transmission unit 38 displays real person captures for all sites A, D, and E on a high-performance HMD (LAN), and displays site D (whose face is not displayed) as an avatar in order to reduce transmission volume on a high-performance HMD (4G line) (see Fig. 8). Furthermore, on a smartphone (4G line), the avatar / person determination information transmission unit 38 displays site E (whose head is hidden) as an avatar from the perspective of processing capacity.

[0106] 3. Second Embodiment Next, a second embodiment of the present disclosure will be described. In the second embodiment, 3D modeling and rendering are performed by the cloud server 3.

[0107] <3.1 Functional Block Diagram of Information Processing System 100> Figure 11 is a functional block diagram of the information processing system 100 according to the second embodiment. Note that the same functional blocks as those in Figure 4 will be described with the same reference numerals. As shown in Figure 11, much of the processing at base B is moved to the cloud server 3 side, but the basic processing flow is the same.

[0108] In the second embodiment, the cloud server 3 processes the data to be finally displayed on the display unit 43 at the site B and transmits it as a video. This enables the information processing system 100 to perform rapid 3D communication between multiple sites even if the processing capability of the display-side information processing device 4 is low or the transmission bandwidth is narrow.

[0109] Specifically, as shown in FIG. 11 , the cloud server 3 includes a data transmission control unit 21, a session management unit 22, a storage unit 23 of the cloud server 3, an unpacketization unit 31, a video decoding unit 32, an image processing unit 33, a rendering unit 34, a video encoding unit 12, a skeleton posture decoding unit 35, a transmission information monitor unit 36, a processing capacity monitor unit 37, an avatar / person determination information transmission unit 38, a person presence determination unit 40, a viewpoint / gaze monitor unit 39, and a user interface processing unit 42.

[0110] The information processing device 4 at site B has a video decoding unit 32, a rendering unit 34, and a display unit 43. In the second embodiment, most of the image processing according to the embodiment is performed by the cloud server 3, which has better performance than the information processing device 4, and therefore, the real-time display of avatar images or captured images in the virtual space can be improved.

[0111] 4. Third Embodiment Next, a description will be given of an information processing system 200 according to a third embodiment. In the third embodiment, the display of avatars / real-time person captures is selected not only for each person but also for each body part of the same person.

[0112] 12 is a functional block diagram of an information processing system 200 according to the third embodiment. The same parts as those in FIG. 4 are denoted by the same reference numerals, and their description will be omitted. Only the different parts will be described here.

[0113] The avatar / person determination information transmission unit 238 transmits angle of view information and line of sight information of people of the information processing device 4 to the cloud server 3. The avatar / person determination information transmission unit 38 determines the number of people that can be processed in real person capture (first processable number of people) based on the processing capacity of the information processing device 4 monitored by the processing capacity monitor unit 37 and the transmission information of the information processing device 4 monitored by the transmission information monitor unit 36. The avatar / person determination information transmission unit 238 transmits the determined processable number of people to the cloud server 3. The determination information includes angle of view information, line of sight information, and the processable number of people.

[0114] The avatar / person determination information transmission unit 238 also determines whether an important part is within the attention area. For example, in an HMD or the like, the attention area is calculated by tracking gaze information from the viewpoint / gaze monitor unit 39 and calculating which area within the virtual camera the person is gazing at. The human eye can see a range of 135° vertically and 160° horizontally, but it is said that the range within which the details of an image can actually be recognized is approximately within a 5° central circle. Therefore, the avatar / person determination information transmission unit 238 calculates this area. The avatar / person determination information transmission unit 238 projects skeletal posture information of important parts necessary for communication, such as the face and hands, onto the virtual camera and determines whether they are within the attention area.

[0115] The avatar / person determination information transmission unit 238 determines whether an important part is within the attention area. Since the avatar / person determination information transmission unit 238 processes only the part within the attention area as real person data, it marks the Depth image of the part other than the relevant part as an invalid value.

[0116] The rendering unit 234 renders the image processed by the image processing unit 33 on the display unit 43. The rendering unit 234 performs modeling and rendering on the person capture data (RGB image and depth image).

[0117] Furthermore, in the case of a person for which there is no person capture data (RGB image and depth image) and only skeletal posture information, the rendering unit 234 applies animation to the avatar corresponding to that person downloaded by the avatar download unit 41 and renders it on the display device. In other words, the rendering unit 234 renders the capture image or avatar image of the person received by the unpacketizing unit 31 and displays it on the display unit 43.

[0118] If there is a part of the avatar image that is not displayed within the field of view, rendering unit 234 assigns the captured image to a part of the avatar image and performs rendering. The part of the avatar image may be, for example, a part necessary for communication.

[0119] The person presence determination unit 240 determines whether there is a person not being processed (displayed) within the angle of view of the virtual camera of the information processing device 4, and if it is determined that there is a person not being displayed within the angle of view of the virtual camera, it determines to display the not-displayed person as an avatar image. If there is a person being displayed within the angle of view of the virtual camera of the information processing device 4, the person presence determination unit 240 determines to display a captured image of the person for the displayed person.

[0120] Furthermore, the person presence determination unit 240 determines whether there are any people or parts that have not been processed (displayed) within the angle of view of the virtual camera.

[0121] Fig. 13 is a diagram illustrating the viewpoint (virtual camera) of site B. In Fig. 13, a hand within a circled attention area R is displayed as a real-time captured image, and an area outside the attention area is displayed as an avatar image. Fig. 14 is a flowchart illustrating the operation of information processing device 4 at site B in information processing system 200 according to the third embodiment.

[0122] The processing on the capture side (steps S10-S12 in FIG. 6) and the processing on the cloud server 3 (steps S20-S22 in FIG. 7) are the same as those in the third embodiment, and therefore will not be described here. Furthermore, the same steps as those in the flowchart of FIG. 9 of the information processing device 4 at site B described in the first embodiment will be described with the same step numbers.

[0123] 14, the user interface processing unit 42 acquires the field of view information and the line of sight information (step S30). Next, the avatar / person determination information transmission unit 238 transmits the acquired field of view information and line of sight information to the cloud server 3 (step S31).

[0124] The unpacketizing unit 31 receives the packetized data (RGB image, depth image, and skeletal posture information) sent from the data transmission control unit 21 (step S32) and unpacketizes the data.

[0125] Next, the video decoding unit 32 decodes the person capture data (RGB image and depth image) and extracts an image from the decoded video. The skeleton posture decoding unit 35 decodes the skeleton posture information unpacketized by the unpacketizing unit 31 and extracts the skeleton posture (step S33).

[0126] The avatar / person determination information transmission unit 238 determines whether an important part is within the attention area (step S101). Next, the avatar / person determination information transmission unit 238 marks the Depth image of the part other than the important part as an invalid value (step S102) in order to process only the part within the attention area as real person data.

[0127] The rendering unit 234 models 3D data from the person capture data (RGB image and depth image) and performs rendering (step S34).

[0128] The avatar / person determination information transmission unit 238 determines the processable number of people based on the processing capacity of the information processing device 4 monitored by the processing capacity monitor unit 37 and the transmission information of the information processing device 4 monitored by the transmission information monitor unit 36 ​​(step S35). Then, the avatar / person determination information transmission unit 238 transmits the determined processable number of people to the cloud server 3 (step S36).

[0129] The person presence determination unit 240 determines whether there are any people or parts that have not been processed (displayed) within the field of view of the virtual camera (step S103). If there are any people or parts that have not been processed within the field of view (step S103; Yes), the rendering unit 234 renders the avatar (step S104). If there are any parts that have not been displayed within the field of view, the rendering unit 234 assigns a captured image to part of the avatar image and performs rendering. If there are no people or parts that have not been processed within the field of view (step S103; No), the information processing device 4 ends processing.

[0130] In step S104, the rendering unit 234 renders the avatar with respect to parts of the same person that have not been processed (displayed). For this reason, when rendering the avatar from the viewpoint of the virtual camera, the rendering unit 234 performs the rendering by, for example, invalidating parts that exist in the corresponding depth image.

[0131] In the third embodiment, the processing performed by the information processing device 4 at the base B has been described, but the processing in the third embodiment may also be performed by the cloud server 3 as in the second embodiment.

[0132] Therefore, the information processing system 200 according to the third embodiment does not model parts other than the region of interest, so it is possible to reduce the total amount of processing required for modeling. Furthermore, the information processing system 200 according to the third embodiment can apply the improved processing capacity to other image quality improvement processes, etc.

[0133] 5. Fourth Embodiment Next, an information processing system 300 according to a fourth embodiment will be described. In the fourth embodiment, similar to the third embodiment, the display of avatars / real-time person captures is selected not for each person but for each body part of the same person.

[0134] Fig. 15 is a functional block diagram of an information processing system 300 according to the fourth embodiment. Note that the same parts as those in Fig. 4 are given the same reference numerals, and their description will be omitted, and only the different parts will be described here.

[0135] The avatar / person determination information sending unit 338 sends angle of view information and line of sight information of people on the information processing device 4 to the cloud server 3. The avatar / person determination information sending unit 38 determines the number of people that can be processed in real person capture (first processable number of people) based on the processing capacity of the information processing device 4 monitored by the processing capacity monitor unit 37 and the transmission information of the information processing device 4 monitored by the transmission information monitor unit 36. The avatar / person determination information sending unit 238 sends the determined processable number of people to the cloud server 3. The determination information includes angle of view information, line of sight information, and the processable number of people.

[0136] Furthermore, the avatar / person determination information transmission unit 338 determines whether all parts of the person at site A are visible in the camera. If all parts of the person at site A are not visible in the camera, the avatar / person determination information transmission unit 338 generates an avatar that complements the parts of the image from the camera that are not visible.

[0137] The rendering unit 334 renders on a display device the image that has been processed by the image processing unit 33. The rendering unit 34 performs modeling and rendering on the person capture data (RGB image and depth image).

[0138] Furthermore, in the case of a person for which there is no person capture data (RGB image and depth image) and only skeletal posture information, the rendering unit 334 applies animation to the avatar corresponding to that person downloaded by the avatar download unit 41 and renders it on the display unit 43. In other words, the rendering unit 334 renders the capture image or avatar image of the person received by the unpacketizing unit 31 and displays it on the display unit 43.

[0139] The rendering unit 334 models 3D data from the person capture data (RGB image and depth image) and performs rendering. In the information processing system 300 according to the fourth embodiment, the rendering unit 334 renders a part of the capture image of a person not captured by the capture camera using a part of the avatar image.

[0140] 16 is a diagram showing a situation in which person Ha at location A is viewed from the position of person Hb at location B in three-dimensional space. As shown in Fig. 16, person Ha to be displayed at location A is only visible from the position of camera Ca at location A. In such a case, the information processing system 300 of the fourth embodiment may display the entire person Ha to be displayed as an avatar, but only the portion of person Ha to be displayed that is not visible from the camera is displayed as an avatar.

[0141] FIG. 17 is a flowchart for explaining the operation of the information processing device 4 at the site B in the information processing system 300 according to the fourth embodiment.

[0142] Note that the capture-side processing of the information processing system 300 of the fourth embodiment (steps S10 to S12 in FIG. 6) and the processing of the cloud server 3 (steps S20 to S22 in FIG. 7) are the same as the processing of the information processing system 1 of the first embodiment, and therefore a description thereof will be omitted here. Also, in FIG. 17, the same processing as that in the flowchart shown in FIG. 9 that explains the operation of the information processing device 4 at base B will be described with the same step numbers.

[0143] 17, the user interface processing unit 42 acquires the field of view information and the line of sight information (step S30). Next, the avatar / person determination information transmission unit 38 transmits the acquired field of view information and line of sight information to the cloud server 3 (step S31).

[0144] The unpacketizing unit 31 receives the packetized data (RGB image, depth image, and skeletal posture information) sent from the data transmission control unit 21 (step S32) and unpacketizes the data.

[0145] Next, the video decoding unit 32 performs video decoding on the person capture data (RGB image and depth image) and extracts an image from the decoded video. Also, the skeleton posture decoding unit 35 decodes the skeleton posture information unpacketized by the unpacketizing unit 31 and extracts the skeleton posture (step S33).

[0146] The avatar / person determination information transmission unit 338 determines whether all parts of the person at site A are visible through the camera (step S201). This can be achieved by the avatar / person determination information transmission unit 338 determining whether all pixels in the depth image have invalid values. Alternatively, the avatar / person determination information transmission unit 338 can handle this by performing a similar process at the time of capture at site A and not transmitting the RGB image and the depth image.

[0147] In step S201, if not all parts of the person at base A are visible from the camera (step S201; No), the avatar / person determination information transmission unit 338 generates an avatar that complements the part of the image from the camera that is not visible (step S202).

[0148] An avatar that complements a part of an image from a camera that is not visible can be generated by applying skeleton posture information to the avatar and generating an RGB image and a depth image by rendering from a virtual camera. The rendering unit 334 performs modeling using this generated data and the RGB image and depth image from the visible camera.

[0149] The rendering unit 334 models 3D data from the person capture data (RGB image and depth image) and performs rendering (step S203). In the fourth embodiment, the rendering unit 334 renders a part of the capture image of a person not captured by the capture camera using a part of the avatar image.

[0150] The avatar / person determination information transmission unit 338 determines the number of people that can be processed based on the processing capacity of the information processing device 4 monitored by the processing capacity monitor unit 37 and the transmission information of the information processing device 4 monitored by the transmission information monitor unit 36 ​​(step S35).

[0151] Then, the avatar / person determination information transmission unit 338 transmits the determined number of people that can be handled to the cloud server 3 (step S36).

[0152] Next, the person presence determination unit 40 determines whether there is a person within the field of view of the virtual camera that has not been processed (displayed) (step S37). Specifically, in step S37, the person presence determination unit 40 determines, among the people at the base that should be processed within the field of view, those that were not processed in step S34. This is because, since only the amount of person real-time capture data (RGB image and depth image) that can be processed in step S21 is transmitted, the remaining people are displayed as avatars.

[0153] Furthermore, since the determination in step S21 is based on the information transmitted from step S31 or step S36, the person presence determination unit 40 needs to perform calculations based on the latest virtual camera information. The person presence determination unit 40 uses the latest virtual camera angle of view / viewpoint information to calculate whether the position information of the skeleton posture information of the unprocessed base is within the viewing frustum formed by the virtual camera.

[0154] If there is an unprocessed person within the field of view (step S37; Yes), the rendering unit 334 performs rendering of the avatar (step S38). If there is no unprocessed person within the field of view (step S37; No), the information processing device 4 ends the process.

[0155] In step S38, the rendering unit 334 performs rendering to animate the avatars of people who are within the field of view and are located at bases that are not displayed in the processing described in step S21, by applying skeleton posture information to the avatars.

[0156] Therefore, in the case of a viewpoint from a direction not captured by the capture camera (a person at site B looking at the person to be displayed at site A), the information processing system 300 of the fourth embodiment can prevent part of the body from being missing by extracting that part from the avatar. Also, in the information processing system 300 of the fourth embodiment, it is sufficient that any one part of the body is captured by the camera, so the area in which the person to be displayed can move is expanded.

[0157] 6. Other Embodiments The processing according to each of the above-described embodiments may be implemented in various different forms other than the above-described embodiments.

[0158] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. Furthermore, the information, including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0159] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0160] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0161] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0162] 7. Effects An information processing system according to the present disclosure (in the embodiments, information processing systems 1, 100, 200, and 300) includes a first information processing device (in the embodiments, information processing device 4) and a second information processing device (in the embodiments, cloud server 3) connected to the first information processing device. The first information processing device or the second information processing device includes a determination unit (in the embodiments, person presence determination unit 40, etc.) that determines, based on a virtual space position of a person of a third information processing device (in the embodiments, information processing device 2) that is connected to the first information processing device and shares a virtual space with the first information processing device, whether to display a captured image of a person or a generated avatar image corresponding to the captured image of the person displayed on the first information processing device, and a rendering unit (in the embodiments, rendering unit 34) that renders the captured image or the avatar image determined by the determination unit.

[0163] Therefore, the information processing system of the present disclosure can determine whether to display an avatar image or a captured image in a virtual space displayed on the first information processing device, thereby reducing the amount of image data transferred and processed by the information processing system. As a result, the information processing system of the present disclosure can present people at multiple locations (i.e., a large number of people) with high FPS and low latency.

[0164] The determination unit has a data transmission control unit (in the embodiment, the data transmission control unit 21) that determines a captured image of a person to be sent to the first information processing device based on the field of view information of the virtual camera of the first information processing device, the line of sight information of the user of the first information processing device, and the first number of people that the first information processing device can process.

[0165] In the information processing system of the present disclosure, the data transmission control unit determines the captured images of a person to be transmitted to the first information processing device based on the field of view information of the virtual camera of the first information processing device, the line of sight information of the user of the first information processing device, and the first number of people that the first information processing device can process, so that an appropriate number of captured images can be transmitted, thereby reducing the amount of processing related to image processing in the system.

[0166] The determination unit has a person presence determination unit (in the embodiment, a person presence determination unit 40) that determines whether to display an undisplayed person as an avatar image when it is determined that an undisplayed person exists within the field of view of the virtual camera of the first information processing device.

[0167] Therefore, the information processing system of the present disclosure determines whether to display a person who is not displayed within the field of view of the virtual camera of the first information processing device as an avatar image, thereby further reducing the amount of processing related to image processing in the system of the first information processing device.

[0168] The determination unit has an avatar / person determination information transmission unit (avatar / person determination information transmission unit 38 in the embodiment) that transmits the field of view information, line of sight information, and the first processable number of people to the second information processing apparatus.

[0169] Therefore, in the information processing system of the present disclosure, the avatar / person determination information transmission unit sends the field of view information, line of sight information, and the first number of people that can be processed to the second information processing device, so that the process of determining the captured image of the person to be sent to the first information processing device can be performed on the second information processing device (cloud server 3) side, thereby reducing the processing load on the first information processing device and maintaining real-time performance in the virtual space.

[0170] The avatar / person determination information transmitting unit obtains the second processable number of people based on the processing capacity of the first information processing device and the transmission information.

[0171] The information processing system of the present disclosure can determine an appropriate number of people that can be processed according to the capabilities of the first information processing device based on the processing capabilities and transmission information of the first information processing device.

[0172] When the second processable number of people is smaller than the first processable number of people transmitted by the avatar / person determination information transmission unit, the data transmission control unit determines the capture image to be transmitted to the first information processing device based on the distance between the virtual camera and the person.

[0173] As a result, the information processing system of the present disclosure can reduce the amount of data to be transmitted to the first information processing device and the amount of data to be processed by the first information processing device.

[0174] If a specific part of the person is not within the angle of view of the virtual camera of the first information processing device, the data transmission control unit does not transmit a captured image of the person to the first information processing device.

[0175] As a result, the information processing system of the present disclosure further reduces the amount of data to be transmitted to the first information processing device and the amount of data to be processed by the first information processing device.

[0176] Furthermore, the captured image is acquired for each frame, which allows the information processing system of the present disclosure to display the captured image in real time in the virtual space.

[0177] The second information processing device has a determination unit. As a result, in the information processing system of the present disclosure, the determination process, which accounts for most of the image processing, is performed by the second information processing device (cloud server 3) that has better performance than the first information processing device, thereby improving the real-time display of avatar images or captured images in the virtual space.

[0178] A captured image may be allocated to a portion of the avatar image. This allows the information processing system of the present disclosure to reduce the amount of processing required for modeling. Furthermore, the improved processing capacity can be used for other processes such as image quality improvement.

[0179] Furthermore, a part of the avatar image is a part necessary for communication. Thus, the information processing system of the present disclosure displays a captured image of the part necessary for communication, thereby enabling smooth communication in the virtual space.

[0180] The rendering unit renders a portion of a captured image of a person not captured by the capture camera of the first information processing device using a portion of an avatar image. This allows the information processing system of the present disclosure to prevent a missing body part from being displayed. Furthermore, since it is sufficient for the person to be captured by one of the cameras, the area in which the person can move is expanded.

[0181] 8. Hardware Configuration FIG. 18 is a hardware configuration diagram showing an example of a computer 1000 that realizes the arithmetic units of the information processing device 2, the cloud server 3, and the information processing device 4 according to the first to fourth embodiments.

[0182] The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0183] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0184] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, as well as programs that depend on the hardware of the computer 1000 .

[0185] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an application program according to the present disclosure, which is an example of program data 1450.

[0186] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0187] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, and semiconductor memories.

[0188] Although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, as another example, the CPU 1100 may obtain these programs from other devices via an external network 1550.

[0189] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0190] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0191] Note that the present technology may also be configured as follows: (1) An information processing system including: a first information processing device; and a second information processing device connected to the first information processing device, wherein the first information processing device or the second information processing device includes: a determination unit that determines, based on a position in the virtual space of a person of a third information processing device that is connected to the first information processing device and shares a virtual space with the first information processing device, whether to display a captured image of the person displayed on the first information processing device or a generated avatar image corresponding to the captured image of the person; and a rendering unit that renders the captured image or the avatar image determined by the determination unit. (2) The information processing system described in (1), wherein the determination unit includes a data transmission control unit that determines a captured image of the person to be transmitted to the first information processing device based on information about an angle of view of a virtual camera of the first information processing device, line-of-sight information of a user of the first information processing device, and a first number of people that can be processed by the first information processing device. (3) The information processing system according to (1) or (2), wherein the determination unit includes a person presence determination unit that, when it is determined that an undisplayed person exists within the angle of view of a virtual camera of the first information processing device, determines to display the undisplayed person as the avatar image. (4) The information processing system according to (2), wherein the determination unit includes an avatar / person determination information transmission unit that transmits the angle of view information, the line of sight information, and the first processable number of people to the second information processing device. (5) The information processing system according to (4), wherein the avatar / person determination information transmission unit calculates a second processable number of people based on the processing capacity and transmission information of the first information processing device. (6) The information processing system according to (5), wherein the data transmission control unit, when the second processable number of people is smaller than the first processable number of people transmitted by the avatar / person determination information transmission unit, determines the captured image to be transmitted to the first information processing device based on the distance between the virtual camera and the people.(7) The information processing system according to any one of (2), (4) to (6), wherein the data transmission control unit does not transmit the capture image of the person to the first information processing device when a specific part of the person is not included in the angle of view of a virtual camera of the first information processing device. (8) The information processing system according to any one of (1) to (7), wherein the capture image is acquired for each frame. (9) The information processing system according to any one of (1) to (8), wherein the second information processing device includes the determination unit. (10) The information processing system according to any one of (1) to (9), wherein the capture image is assigned to a part of the avatar image. (11) The information processing system according to (10), wherein the part of the avatar image is a part necessary for communication. (12) The information processing system according to any one of (1) to (11), wherein the rendering unit renders a part of the capture image of the person not captured by a capture camera of the first information processing device with a part of the avatar image. (13) An information processing method in an information processing system having a first information processing device and a second information processing device connected to the first information processing device, the information processing method including: a computer in the information processing system determines whether the first information processing device or the second information processing device should display a captured image of a person displayed on the first information processing device or a generated avatar image corresponding to the captured image of the person, based on a virtual space position of a person on a third information processing device connected to the first information processing device and sharing a virtual space with the first information processing device; and rendering the determined captured image or avatar image.(14) A program for an information processing system having a first information processing device and a second information processing device connected to the first information processing device, the program causing a computer in the information processing system to function as: a determination unit that determines whether the first information processing device or the second information processing device should display a captured image of a person displayed on the first information processing device or a generated avatar image corresponding to the captured image of the person, based on a virtual space position of a person on a third information processing device that is connected to the first information processing device and shares a virtual space with the first information processing device; and a rendering unit that renders the captured image or avatar image determined by the determination unit.

[0192] 1, 100, 200, 300 Information processing system 2, 4 Information processing device 3 Cloud server 11 Capture unit 12 Video encoding unit 13 Skeleton posture encoding unit 14 Packetizing unit 15, 23 Storage unit 16 Avatar upload unit 21 Data transmission control unit 22 Session management unit 31 Unpacketizing unit 32 Video decoding unit 33 Image processing unit 34, 234, 334 Rendering unit 35 Skeleton posture decoding unit 36 ​​Transmission information monitor unit 37 Processing capacity monitor unit 38, 238, 338 Avatar / person determination information transmission unit 39 Viewpoint / gaze monitor unit 40, 240 Person presence determination unit 41 Avatar download unit 42 User interface processing unit 43 Display unit

Claims

1. An information processing system comprising: a first information processing device; and a second information processing device connected to the first information processing device, wherein the first information processing device or the second information processing device comprises: a determination unit that determines whether to display a capture image of a person displayed on the first information processing device or a generated avatar image corresponding to the capture image of the person based on a position in the virtual space of the person of a third information processing device that is connected to the first information processing device and shares a virtual space with the first information processing device; and a rendering unit that renders the capture image or the avatar image determined by the determination unit.

2. The information processing system of claim 1, wherein the determination unit has a data transmission control unit that determines a captured image of a person to be transmitted to the first information processing device based on information on the angle of view of the virtual camera of the first information processing device, information on the line of sight of the user of the first information processing device, and the first number of people that can be processed by the first information processing device.

3. The information processing system of claim 1, wherein the determination unit has a person presence determination unit that, when it is determined that a person not displayed is present within the field of view of the virtual camera of the first information processing device, determines that the person not displayed should be displayed in the avatar image.

4. An information processing system as described in claim 2, wherein the determination unit has an avatar / person determination information transmission unit that transmits the field of view information, the line of sight information, and the first processable number of people to the second information processing device.

5. The information processing system according to claim 4, wherein the avatar / person determination information transmission unit determines a second processable number of people based on the processing capacity of the first information processing device and transmission information.

6. The information processing system of claim 5, wherein the data transmission control unit, when the second processable number of people is smaller than the first processable number of people transmitted by the avatar / person determination information transmission unit, determines the capture image to be transmitted to the first information processing device based on the distance between the virtual camera and the person.

7. The information processing system according to claim 2, wherein the data transmission control unit does not transmit the captured image of the person to the first information processing device if a specific part of the person is not included in the angle of view of the virtual camera of the first information processing device.

8. The information processing system according to claim 1, wherein the captured image is obtained frame by frame.

9. The information processing system according to claim 1, wherein the second information processing device has the determination unit.

10. The information processing system according to claim 1, wherein the captured image is assigned to a portion of the avatar image.

11. The information processing system according to claim 10, wherein the portion of the avatar image is a part necessary for communication.

12. The information processing system according to claim 1, wherein the rendering unit renders a portion of the capture image of the person not captured by a capture camera of the first information processing device with a portion of the avatar image.

13. An information processing method in an information processing system having a first information processing device and a second information processing device connected to the first information processing device, wherein a computer in the information processing system determines whether the first information processing device or the second information processing device will display either a captured image of the person displayed on the first information processing device or a generated avatar image corresponding to the captured image of the person, based on a position in the virtual space of a person of a third information processing device connected to the first information processing device and sharing a virtual space with the first information processing device, and renders the determined captured image or avatar image.

14. A program for an information processing system having a first information processing device and a second information processing device connected to the first information processing device, causing a computer in the information processing system to function as: a determination unit that determines whether the first information processing device or the second information processing device will display a capture image of a person displayed on the first information processing device or a generated avatar image corresponding to the capture image of the person, based on a position in the virtual space of the person of a third information processing device that is connected to the first information processing device and shares a virtual space with the first information processing device; and a rendering unit that renders the capture image or the avatar image determined by the determination unit.

Images