Information processing system and information processing method

Abstract

This information processing system includes: a texel coordinate determination unit that determines texel coordinates of a volumetric image on the basis of a depth value of a depth image acquired from a virtual space; a storage unit that stores color information, which is included in a color image, at the volumetric image texel coordinates determined by the texel coordinate determination unit; and an adding unit that adds the color information stored in the volumetric image by the storage unit to a viewpoint video of a space.

Description

Information processing system and information processing method

[0001] The present disclosure relates to an information processing system and an information processing method.

[0002] There is a technology that reduces the processing load by dividing a virtual space into uniform intervals and rendering only the area near the user's location. However, such a technology does not render the appearance of the virtual world at locations far from the user. As a result, content far from the user's location is not rendered, making it difficult to represent a large amount of content in a single CG space.

[0003] There is a technology called "The Volumetric Billboards" [Decaudin and Neyret, 2009] that uses volumetric images for lightweight CG space rendering. This technology stores color information of 3D content contained in a certain region in a volumetric image, and reconstructs its appearance in a CG (computer graphics) space.

[0004] Japanese Patent Application Publication No. 11-328124

[0005] However, this technology uses a method of scanning the space by the resolution of the volumetric image to generate the volumetric image and acquiring color information, and the higher the resolution of the volumetric image, the longer the processing time required to generate the volumetric image. Therefore, this technology is not suitable for cases where the 3D content to be stored in the volumetric image moves or changes in real time.

[0006] The present disclosure has been made in consideration of the above circumstances, and realizes high-speed generation of volumetric images.

[0007] The information processing system of the present disclosure includes a texel coordinate determination unit that determines the texel coordinates of a volumetric image based on the depth values ​​of a depth image acquired from a virtual space, a storage unit that stores color information contained in a color image at the texel coordinates of the volumetric image determined by the texel coordinate determination unit, and an addition unit that adds the color information stored in the volumetric image by the storage unit to a viewpoint image of the space.

[0008] 1 is a functional block diagram of an information processing system according to an embodiment. FIG. 2 is a functional block diagram of a volumetric image generation unit of a server according to an embodiment. FIG. 3 is a functional block diagram of a content display control unit of a client terminal according to an embodiment. FIG. 4 is a functional block diagram of a 3D reconstruction unit according to an embodiment. FIG. 5 is a diagram illustrating an overview of 3D reconstruction processing according to an embodiment. FIG. 6 is a flowchart for explaining a volumetric image generation operation of a server according to an embodiment. FIG. 7 is a diagram for explaining division of a virtual space according to an embodiment. FIG. 8 is a diagram for explaining an overview of processing when dividing a virtual space of a soccer stadium according to an embodiment. FIG. 9 is a diagram for explaining generation of a volumetric image according to an embodiment. FIG. 10 is a flowchart for explaining 3D reconstruction processing of a client terminal according to an embodiment. FIG. 11 is a flowchart for explaining details of 3D reconstruction processing of a client terminal according to an embodiment. FIG. 12 is a flowchart for explaining an overview of operation of an information processing system. FIG. 13 is a diagram for explaining a 3D reconstruction technique using ray tracing according to an embodiment. FIG. 14 is a diagram illustrating a rendering result showing a triangular mesh when a normal LOD is applied. FIG. 15 is a diagram illustrating a rendering result showing a volumetric image when a LOD according to an embodiment is applied. FIG. 16 is a diagram for explaining an example of the range of a small space when moving within the range of a small space of normal rendering around a user according to an embodiment of Modification 3. FIG. 17 is a diagram for explaining setting of a small space in an information processing system according to an embodiment of Modification 4. 10 is a diagram for explaining space division regions when the virtual space of the information processing system according to the embodiment of the modified example 5 is a soccer stadium. FIG. 11 is a hardware configuration diagram showing an example of a computer that realizes the arithmetic units of the server and client terminals that are the information processing device according to the embodiment.

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. The description will be given in the following order.

[0010] 1. Explanation of Terms 2. Configuration of Information Processing System 1 2.1. Configuration of Server 2 2.2. Configuration of Client Terminal 3 3. Explanation of Operation 3.1. Operation of Server 2 3.2. Operation of Client Terminal 3 3.3 Overview of Operation of Information Processing System 1 4. Modifications 4.1. Modification 1 4.2. Modification 2 4.3. Modification 3 4.4. Modification 4 4.5. Modification 5 5. Summary of Embodiments 6. Effects of Information Processing System According to the Present Disclosure 7. Hardware Configuration

[0011] <1. Explanation of terms> The terms used in the embodiments have the following meanings: "Volumetric image": An image format with a three-dimensional data structure of X / Y / Z. Data such as color and density is stored in each grid (texel).

[0012] "Depth image" is a two-dimensional image that stores the distance from the camera or viewpoint to an object. "Content data" is general 3D game content data. Specifically, content data includes 3D model data, sound data, object placement, orientation, size, object animation data, data that defines object behavior, and programs.

[0013] "Area division data" is data on the spatial range when dividing a virtual space into multiple areas. Area division data is generally defined by a bounding box and its position.

[0014] 2. Configuration of Information Processing System 1 In this embodiment, it is assumed that 10,000 users are participating as spectators using their own avatars at a soccer stadium in a virtual reality (VR) world. Each seat is divided into uniformly sized blocks of 100 seats each. Furthermore, it is assumed that the generation of volumetric images is performed by a server, and that 3D reconstruction using the volumetric images and display of the virtual space content on a device are performed by a client terminal.

[0015] The information processing system according to the embodiment is a virtual space system having a plurality of client terminals (such as PCs (Personal Computers), HMDs (Head Mounted Displays), smartphones, etc.) connected to a network, and a server that manages data transmission and reception between the plurality of client terminals.

[0016] Fig. 1 is a functional block diagram of an information processing system 1 according to an embodiment. In Fig. 1, a server 2 of the information processing system 1 and a client terminal 3 are connected via a network (not shown). For the sake of simplicity, Fig. 1 shows only one of the multiple client terminals 3.

[0017] 2.1. Configuration of Server 2 The server 2 includes a volumetric image generation unit 11 , a content logic processing unit 12 , a content drawing unit 13 , a depth and color image acquisition unit 14 , and a communication unit 15 .

[0018] The volumetric image generator 11 generates a volumetric image represented by three-dimensional image data having color and normal information of 3D content included in a region divided into a certain range. Fig. 2 is a functional block diagram of the volumetric image generator 11 of the server 2 according to the embodiment.

[0019] As shown in FIG. 2, the volumetric image generating unit 11 includes a space dividing unit 41 , an updating unit 42 , a texel coordinate determining unit 43 , a saving unit 44 , and a volumetric image transmitting unit 45 .

[0020] The space dividing unit 41 divides the entire virtual space. For example, if the virtual space is a soccer stadium, the space dividing unit 41 divides the space based on the spectator seats.

[0021] The update unit 42 updates the position, posture, and logic of the 3D content based on the update information updated by the content logic processing unit 12. The update unit 42 also performs update processing for the bones, materials, and the like of assets existing in space.

[0022] The texel coordinate determination unit 43 determines the texel coordinates of the volumetric image based on the depth values ​​of the depth image acquired from the virtual space.

[0023] The storage unit 44 stores color information included in the color image at the determined texel coordinates in the volume image. The volume image transmission unit 45 transmits the volume image stored in the storage unit 44 to the client terminal 3.

[0024] Returning to Fig. 1, the content logic processing unit 12 defines the behavior of content that exists in the virtual space using programs, animations, etc., and processes and updates the behavior.

[0025] The content rendering unit 13 performs processing to convert content that exists in the virtual space into a format (image, etc.) that can be displayed on the device.

[0026] The depth and color image acquisition unit 14 photographs a group of contents existing within a predetermined range in the virtual space using virtual cameras positioned in six directions (up, down, left, right, front, back), and obtains corresponding depth images and color images.

[0027] The communication unit 15 is responsible for communication with the outside of the server 2, for example, transmitting data of the generated volumetric image to the client terminal 3 and transmitting the database, and acquiring update information of the VR world transmitted from the client terminal 3.

[0028] The server 2 includes a content data storage unit 31, a volumetric image data storage unit 32, and an area division data storage unit .

[0029] The content data storage unit 31 stores content data. The volumetric image data storage unit 32 stores volumetric image data. The region division data storage unit 34 stores region division data. The stored content data, volumetric image data, and region division data are transmitted to the client terminal 3 via the communication unit 15.

[0030] 2.2. Configuration of the Client Terminal 3 The client terminal 3 includes a content display control unit 21, a content logic processing unit 22, a content drawing unit 23, a three-dimensional reconstruction unit 24, a communication unit 25, a display unit 35, and a device input unit 36.

[0031] The content display control unit 21 is responsible for controlling content display, and performs processes such as determining the area to be displayed by 3D reconstruction using volumetric images and the area to be displayed using a normal 3D content drawing method, depending on the user's position in the virtual space.

[0032] The content logic processing unit 22 and the content drawing unit 23 have the same functions as the content logic processing unit 12 and the content drawing unit 13 of the server 2 .

[0033] The three-dimensional reconstruction unit 24 superimposes the color information contained in the acquired volumetric image onto the viewpoint of the display device, and reconstructs the appearance of a part of the VR world stored in the volumetric image.

[0034] The communication unit 25 controls communication between the client terminal 3 and the outside world, for example, acquiring volumetric image data and transmitting update information on the VR world and input information from users operating devices to the server 2 or other terminals.

[0035] The display unit 35 includes a display for displaying content and a speaker for reproducing sound. The device input unit 36 ​​receives input from the user's device for operating the VR world.

[0036] The client terminal 3 has a content data storage unit 33. The content data storage unit 33 stores content data sent from the server 2 in addition to content data on the client terminal 3 side.

[0037] 3 is a functional block diagram of the content display control unit 21 of the client terminal 3 according to the embodiment. As shown in FIG. 7, the content display control unit 21 includes an area determination unit 51, a drawing unit 52, and a three-dimensional reconstruction unit 53.

[0038] The area determination unit 51 determines, depending on the user's position in the virtual space, a first area in which 3D content in the virtual space is displayed by three-dimensional reconstruction using a volumetric image, and a second area in which 3D content is displayed using a normal drawing method without using a volumetric image.

[0039] Here, the normal drawing method refers to a general method used for drawing 3D space, such as rasterization or ray tracing. If there is a change in the area, the update content is notified to the server 2.

[0040] The drawing unit 52 draws the 3D content in the second region determined by the region determining unit 51 using a normal drawing method without using a volumetric image.

[0041] The 3D reconstruction unit 53 performs 3D reconstruction of the appearance of the small space based on the volumetric image data. Specifically, the 3D reconstruction unit 53 displays 3D content in the virtual space in the first region by 3D reconstruction using the volumetric image.

[0042] 4 is a functional block diagram of the 3D reconstruction unit 53 according to the embodiment. As shown in FIG. 8, the 3D reconstruction unit 53 includes a scanning unit 61, a 3D texture coordinate calculation unit 62, a color information acquisition unit 63, and an addition unit 64.

[0043] The 3D reconstruction according to the embodiment is a method for scanning a space from a distance from the user's viewpoint and performing intersection processing between a cube for displaying a volumetric image and the scanning plane. Fig. 5 is a diagram showing an overview of the 3D reconstruction processing according to the embodiment.

[0044] In Fig. 4, the scanning unit 61 scans the virtual space with planes at predetermined intervals from a distance in the line of sight of the user's viewpoint. Fig. 5 shows a case where scan planes 0, ..., k, ...N are operated from the user's viewpoint.

[0045] The three-dimensional texture coordinate calculation unit 62 forms a new surface at the intersection of the scanning plane and the cube for displaying the volumetric image, and calculates three-dimensional texture coordinates on the formed surface. Here, texture coordinates are coordinate information for attaching a two-dimensional or three-dimensional image to the 3D content.

[0046] FIG. 5 shows an example in which surfaces are formed from the intersections of the scan planes 0, . . . , k, . . . N with a cube, and three-dimensional texture coordinates are calculated from the formed surfaces.

[0047] The color information acquisition unit 63 acquires color information of the texel position of the volumetric image corresponding to the calculated texture coordinates. In Fig. 5, acquiring the texture coordinates and color information of the texel position of the volumetric image is shown as texture mapping.

[0048] The adder 64 adds the color information acquired by the color information acquisition unit 63 to the viewpoint video after the 3D content has been rendered. Fig. 5 shows that the acquired color information is added to the viewpoint video after normal rendering.

[0049] <3. Description of Operation> <3.1. Operation of Server 2> FIG. 6 is a flowchart for explaining the operation of generating a volumetric image by the server 2 according to the embodiment.

[0050] First, the space dividing unit 41 divides the entire virtual space into a plurality of small spaces according to a predetermined rule (step S1). Fig. 7 is a diagram for explaining the division of the virtual space according to the embodiment. Fig. 7 shows an example in which the entire virtual space 71 is divided into a plurality of small spaces 72. The small spaces 73 indicated by hatching indicate the spaces in which the users exist.

[0051] In the embodiment, the space dividing unit 41 assumes an example of a soccer stadium and divides a space containing 10,000 avatars into small spaces of 100 people each based on each spectator seating block. Fig. 8 is a diagram for explaining an outline of the process for dividing the virtual space of a soccer stadium according to the embodiment.

[0052] In Figure 8, the space is divided into seats R1, R2, ..., and RN. A volumetric image is then generated for each seat based on depth images acquired from the space. The generated volumetric image is used to generate an image of the seats as seen from a distance from the user (three-dimensional reconstructed image 81). Meanwhile, an image of the seats at the user's position (normally drawn image 82) is drawn using a normal drawing method from the user's viewpoint.

[0053] The update unit 42 updates the position, posture, and logic of the 3D content for each small space 72 based on the update information updated by the content logic processing unit 12. In addition, update processing is also performed on the bones, materials, and the like of the assets existing in the space (step S2).

[0054] 9 is a diagram for explaining generation of a volumetric image according to an embodiment. As shown in Fig. 6, depth images and color images are obtained by rendering from six directions, i.e., front and rear, top and bottom, left and right, of the divided and interleaved space (step S3).

[0055] Then, in the small space 72, the texel coordinate determination unit 43 determines a texel image of the volumetric image that stores color information contained in the acquired color image based on the depth value of the acquired depth image (step S4).Then, the storage unit 44 stores the color information of the color image at the determined texel coordinates of the volumetric image (step S5).

[0056] Next, it is determined whether processing of depth images and color images for six directions has been completed (step S6), and if it is determined that processing of depth images and color images for six directions has been completed (YES in step S6), the generated volumetric image is sent to the client terminal 3 (step S7), and processing is terminated.

[0057] Although the server 2 and the client terminal 3 may be installed in the same machine, in this embodiment they are separated into different machines, and it is assumed that volumetric images are sent via a network.

[0058] If it is determined that the processing of the depth images and color images for six directions has not been completed (NO in step S6), the process returns to step S1.

[0059] 3.2. Operation of the Client Terminal 3 FIG. 10 is a flowchart for explaining the 3D reconstruction process of the client terminal 3 according to the embodiment.

[0060] The area determination unit 51 determines, depending on the user's position in the virtual space, a first area in which 3D content in the virtual space is displayed by three-dimensional reconstruction using a volumetric image, and a second area in which 3D content is displayed using a normal drawing method without using a volumetric image (step S11).

[0061] The drawing unit 52 draws the 3D content in the second region determined by the region determining unit 51 using a normal drawing method without using a volumetric image (step S12).

[0062] Next, the client terminal 3 receives the volume data from the server 2 (step S13) and performs 3D reconstruction of the appearance of the small space based on the volume image data (step S14). Specifically, the 3D reconstruction unit 53 displays 3D content in the virtual space by 3D reconstruction using the volume image in the first region.

[0063] FIG. 11 is a flowchart for explaining details of the 3D reconstruction process of the client terminal 3 according to the embodiment.

[0064] First, the scanning unit 61 scans the virtual space with a plane at a predetermined interval from a distance in the line of sight of the user's viewpoint (step S21).

[0065] Next, the three-dimensional texture coordinate calculation unit 62 forms a new surface at the intersection of the scanning plane and the cube, and calculates three-dimensional texture coordinates on the formed surface (step S22).

[0066] The color information acquisition unit 63 acquires color information of the texel position of the volumetric image corresponding to the calculated texture coordinates (step S23).

[0067] The adder 64 adds the color information acquired by the color information acquisition unit 63 to the viewpoint video after the 3D content is rendered (step S24).

[0068] Next, it is determined whether processing for all scanning planes has been completed (step S25), and if it is determined that processing for all scanning planes has been completed (YES in step S25), the processing ends. On the other hand, if it is determined that processing for all scanning planes has not been completed (NO in step S25), the processing returns to step S21.

[0069] <3.3 Overview of Operation of Information Processing System 1> FIG. 12 is a flowchart for explaining an overview of operation of the information processing system 1.

[0070] 12, the server 2 performs bone and mesh processing for multiple avatars (step S31), then renders depth and color images (step S32), and generates a volumetric image based on the depth and color images (step S33).

[0071] Next, data required for drawing the volumetric image is transmitted to the client terminal 3 via the network N (step S34). Meanwhile, the server 2 receives the drawing range of the virtual space and update data from the client terminal 3 (step S35).

[0072] Thereafter, it is determined whether the spatial division range has changed (step S36), and if the spatial division range has changed (Yes in step S36), the range of space to be stored in the volumetric image is changed (step S37), and the process proceeds to step S32. If the spatial division range has not changed (No in step S36), the process returns to step S36.

[0073] The server 2 also determines whether there has been a change in the normal drawing range on the client terminal 3 (step S38). If it is determined in step S38 that there has been a change in the normal drawing range (Yes in step S38), the server 2 transmits update data required for normal drawing to the client terminal 3 via the network N (step S39). If it is determined in step S38 that there has been no change in the normal drawing range (No in step S38), the process returns to step S38.

[0074] The operation of the client terminal 3 will now be described. The client terminal 3 updates the normal drawing reconstruction part based on the user position (step S41). Next, each drawing range and the updated content are notified to the server 2 side via the network N (step S42).

[0075] On the other hand, when the client terminal 3 receives the update data from the server 2 (step S46), it updates the 3D content (step S47) and proceeds to the processing of step S43. After the processing of step S42, the client terminal 3 normally renders the 3D content (step S43).

[0076] On the other hand, the client terminal 3 receives the volumetric image from the server 2 via the network N (step S48), performs three-dimensional reconstruction (step S49), and proceeds to the processing of step S44.

[0077] Next, after the process of step S43, the client terminal 3 integrates the three-dimensional reconstruction result and the normal rendering result (step S44), and displays the rendering result on a display unit such as an HMD (step S45).

[0078] 4. Modifications Next, modifications of the information processing system 1 according to the embodiment will be described.

[0079] <4.1. Modification 1> The 3D reconstruction method performed by the client terminal 3 may cause color loss or noise in the appearance of the reconstructed 3D content. Fig. 13 is a diagram for explaining the 3D reconstruction method using ray tracing according to the embodiment.

[0080] In the first modification of the embodiment, a 3D reconstruction method is used in which a ray tracing method with the user's viewpoint as the origin is used to determine whether a ray intersects with the texel values ​​of a volumetric image containing color information, which is expected to reduce color loss and noise in the reconstructed 3D content.

[0081] 4.2. Modification 2 The resolution of a volumetric image used for 3D reconstruction may be changed depending on the importance of the volumetric image from the user's viewpoint. For example, lowering the resolution of a volumetric image located far from the user's viewpoint has the effect of reducing the data load on the server 2 and the client terminal 3.

[0082] A technology that changes the level of detail of content depending on the importance of distance and viewpoint is called LOD (Level of Detail). Conventional measures to reduce the number of polygons in a triangular mesh result in a significant difference in the appearance of the rendered content compared to measures to reduce the resolution of a volumetric image. Fig. 14 shows the rendering result of a triangular mesh when a normal LOD is applied. Fig. 15 shows the rendering result of a volumetric image when an LOD according to an embodiment is applied.

[0083] <4.3. Modification 3> There are various possible methods for setting the range of normal rendering in the client terminal 3. The entire virtual space may be divided into small spaces based on criteria such as spectator seating blocks, and 3D content included in the small space where the user is present may be subject to normal rendering.

[0084] In addition, noise due to the volumetric image is noticeable at the boundary between the range of the small space using normal drawing and the range of the small space using the volumetric image. In the information processing system 1 of the third modification of the embodiment, the small space using normal drawing is set as the small space in which the target user exists.

[0085] 16 is a diagram illustrating an example of the range of a small space in normal drawing when the range of the small space moves around the user in accordance with an embodiment of Modification 3. The range of the small space in normal drawing may be a space of a predetermined range around the small space where the user is present.

[0086] In the information processing system 1 according to the embodiment of the modified example 3, content close to the user is no longer displayed as a volumetric image from the user's viewpoint, so a consistent experience can be provided. On the other hand, information update processing occurs on both the server 2 and the client terminal 3 as the user moves, which may increase the communication bandwidth.

[0087] <4.4. Modification 4> When dividing a virtual space into multiple small spaces, noise that causes 3D content to be divided near the boundaries of the small spaces may occur. In the information processing system 1 according to the embodiment of Modification 4, noise is avoided by setting a larger range for generating a volumetric image so that it spans the boundaries of the small spaces.

[0088] Fig. 17 is a diagram for explaining the setting of a small space in the information processing system 1 according to an embodiment of Modification 4. As shown in Fig. 17 , if an object p (volume image) exists on the boundary of the small space, noise may occur on the boundary when reconstructing the object p, as if the object p were divided.

[0089] In Fig. 17, the solid lines indicate the volume image generation range, and the dotted lines R1 and R2 indicate the expanded volume image generation range. As shown by the dotted lines in Fig. 17, the volume image generation range indicated by the dashed lines of the volume image at the boundary of the small space is set to straddle the boundary of the small space. In other words, the volume image used for reconstruction is switched depending on the size of the volume image at the boundary.

[0090] <4.5. Variation 5> The information processing system 1 according to the embodiment of Variation 5 prevents noise that would cause the avatar to be restored in the volumetric image to be divided by determining how to divide the space based on the aisles of the spectator seats, etc.

[0091] 18 is a diagram illustrating space division regions when the virtual space of the information processing system 1 according to the embodiment of Variation 5 is a soccer stadium. As shown in FIG. 18 , when the virtual space is a soccer stadium, noise can be avoided by using the aisles between the spectator seating blocks as the boundaries of the space division. Furthermore, in the case of a live music concert, noise can be avoided by using areas of the venue where it is expected that little 3D content will be placed, such as the aisles of the venue, as the boundaries of the space division.

[0092] 5. Summary of the embodiment The information processing system 1 according to the embodiment provides a service in which a large number of user avatars and 3D content are shared in a CG space via a network. In order to reduce the processing load imposed on the client terminal for rendering the content, the virtual space is divided into multiple regions, each containing tens to hundreds of pieces of 3D content. Appearance information for each divided region is stored as color information in a volumetric image, and the appearance is reconstructed on the client terminal using the volumetric image.

[0093] The depth image is used to store color information in the volume image in real time. The pixel (texel) positions of the volume image are calculated based on depth values ​​obtained from the depth image of the content from six directions (front / back, left / right, top / bottom) of the space to be stored. Then, a color image corresponding to the depth image is saved at the pixel position where the color information was calculated.

[0094] In the visual reconstruction, the visual appearance of a large amount of 3D content reconstructed from the user's viewpoint is superimposed and displayed using a spatial scanning or ray tracing 3D reconstruction technology that operates in real time.

[0095] However, once 3D content such as avatars is stored in a volumetric image, it is difficult to interact with it independently. Once 3D content such as avatars is stored in a volumetric image, the user information and bone information for each avatar are lost. Therefore, to achieve interaction, it is necessary to add additional information such as user information to each texel of the volumetric image and then implement appropriate processing for the additional information.

[0096] 6. Effects of the Information Processing System According to the Present Disclosure The information processing system according to the embodiment (information processing system 1 in the embodiment) has a texel coordinate determination unit (texel coordinate determination unit 43 in the embodiment) that determines the texel coordinates of a volumetric image based on the depth values ​​of a depth image acquired from a virtual space, a storage unit (storage unit 44 in the embodiment) that stores color information included in a color image at the texel coordinates of the volumetric image determined by the texel coordinate determination unit, and an adder unit (adder unit 64 in the embodiment) that adds the color information stored in the volumetric image by the storage unit to a viewpoint video of the space.

[0097] Therefore, the information processing system according to the embodiment can display a large amount of 3D content, each with its own unique behavior and appearance, in a virtual space with minimal processing and data load through 3D reconstruction using volumetric images. Furthermore, the information processing system according to the embodiment determines the texel coordinates of the volumetric image based on the depth values ​​of the depth image acquired from the virtual space. Therefore, the information processing system according to the embodiment can generate volumetric images faster than conventional methods. As a result, the information processing system according to the embodiment can present the excitement of a large audience at a large-scale music concert, soccer stadium, or other venue in real time in a virtual space.

[0098] The information processing system also includes a server (server 2 in the embodiment) and a client terminal (client terminal 3 in the embodiment) connected to the server via a network, the server including a texel coordinate determination unit and a storage unit, the client terminal 3 including an adder, and the server 2 including a transmission unit (communication unit 15 in the embodiment) that transmits the volumetric image in which color information is stored by the storage unit to the client terminal 3. Therefore, the client terminal can perform processing related to the volumetric image on the server, thereby reducing the processing load.

[0099] The space is a space into which a virtual space is divided, thereby enabling the information processing system to target a divided space of a large-scale virtual space.

[0100] The space may be divided based on the spectator seats. This allows the space dividing unit 41 of the information processing system to easily divide the space based on the spectator seats when the virtual space is a soccer stadium or the like.

[0101] The client terminal has an area determination unit that determines, according to the user's position in the virtual space, a first area in which 3D content in the virtual space is displayed by three-dimensional reconstruction using a volumetric image, and a second area in which the 3D content is displayed without using a volumetric image. Therefore, in a service in which a large number of user avatars and 3D content are shared in a CG space via a network, the client terminal can reduce the processing load for rendering the content by determining the first area and the second area.

[0102] The client terminal has a drawing unit (drawing unit 52 in the embodiment) that draws the 3D content in the second area without using a volumetric image. As a result, the drawing unit draws the 3D content in the second area without using a volumetric image, so that the 3D content can be displayed with high accuracy in the second area.

[0103] The client terminal has a 3D reconstruction unit (in the embodiment, the 3D reconstruction unit 24) that displays 3D content in the virtual space by 3D reconstruction using a volumetric image in the first region. This allows the 3D reconstruction unit to display the first region in the virtual space with a small data load.

[0104] The 3D reconstruction unit includes a scanning unit that scans the virtual space with a plane at a predetermined interval from a distance away from the line of sight of the user's viewpoint, a 3D texture coordinate calculation unit that forms a new surface at the intersection of the scanning plane scanned by the scanning unit and a cube for displaying a volumetric image and calculates 3D texture coordinates on the formed surface, a color information acquisition unit that acquires color information of the texel position of the volumetric image corresponding to the 3D texture coordinates calculated by the 3D texture coordinate calculation unit, and an adder that adds the color information acquired by the color information acquisition unit to the viewpoint image after rendering of the 3D content. This enables intersection processing between the cube for displaying the volumetric image and the scanning plane, thereby reducing the processing load on the client terminal.

[0105] The viewpoint video is a video that depicts 3D spatial content without using volumetric images, which allows the 3D spatial content to be added to the viewpoint video, and the use of volumetric images reduces the processing load for generating the viewpoint video.

[0106] The 3D reconstruction unit uses ray tracing with the user's viewpoint as the origin, which reduces color loss and noise in the reconstructed 3D content.

[0107] The resolution of a volumetric image is changed depending on the importance of the image from the user's viewpoint. This allows, for example, reducing the resolution of a volumetric image located far from the user's viewpoint, thereby reducing the data load on the server and client terminal.

[0108] The area determination unit determines the space where the user is present as the second area, thereby enabling the space where the user is present to be displayed with higher accuracy than the first area, and allowing the user to better observe the space where the user is present.

[0109] The second area is a predetermined range of space centered on the space where the user is present, which allows for better display accuracy for the predetermined range of space centered on the space where the user is present than for the first area, making it possible to clearly observe the predetermined range of space centered on the space where the user is present.

[0110] When a volumetric image is located at a boundary of a space, the generation range of the volumetric image may be set to straddle the boundary, thereby preventing noise that may cause the 3D content to be divided near the boundary of the space.

[0111] The space may be divided based on the aisles of the spectator seats, thereby avoiding noise by using areas where it is expected that 3D content will hardly be placed as boundaries for dividing the space.

[0112] 7. Hardware Configuration FIG. 19 is a hardware configuration diagram showing an example of a computer 1000 that realizes the arithmetic unit of the server 2 and the client terminal 3, which are information processing devices according to the embodiment.

[0113] The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0114] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0115] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, as well as programs that depend on the hardware of the computer 1000 .

[0116] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an application program according to the present disclosure, which is an example of program data 1450.

[0117] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0118] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, and semiconductor memories.

[0119] Although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, as another example, the CPU 1100 may obtain these programs from other devices via an external network 1550.

[0120] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0121] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0122] The present technology can also be configured as follows. (1) An information processing system comprising: a texel coordinate determination unit that determines texel coordinates of a volumetric image based on depth values ​​of a depth image acquired from a virtual space; a storage unit that stores color information contained in a color image at the texel coordinates of the volumetric image determined by the texel coordinate determination unit; and an adder that adds the color information stored in the volumetric image by the storage unit to a viewpoint video of the space. (2) The information processing system described in (1) includes: a server; and a client terminal connected to the server via a network, wherein the server includes the texel coordinate determination unit and the storage unit, the client terminal includes the adder, and the server includes a transmitter that transmits the volumetric image in which the color information is stored by the storage unit to the client terminal. (3) The information processing system described in (1) or (2), wherein the space is a space obtained by dividing the virtual space. (4) The information processing system described in (3), wherein the space is divided based on spectator seats. (5) The information processing system according to (2), wherein the client terminal has: an area determination unit that determines, according to a user's position in the virtual space, a first area in which 3D content in the virtual space is displayed by three-dimensional reconstruction using the volumetric image, and a second area in which the 3D content is displayed without using the volumetric image. (6) The information processing system according to (5), wherein the client terminal has: a drawing unit that draws the 3D content in the second area without using the volumetric image. (7) The information processing system according to (5), wherein the client terminal has: a three-dimensional reconstruction unit that displays the 3D content in the virtual space in the first area by three-dimensional reconstruction using the volumetric image.(8) The information processing system according to (7), wherein the three-dimensional reconstruction unit includes: a scanning unit that scans a virtual space with a plane at a predetermined interval from a distance in the line of sight of a user's viewpoint; a three-dimensional texture coordinate calculation unit that forms a new surface at a portion where the scanning plane scanned by the scanning unit intersects with a cube for displaying the volumetric image and calculates three-dimensional texture coordinates on the formed surface; a color information acquisition unit that acquires color information of a texel position of the volumetric image corresponding to the three-dimensional texture coordinates calculated by the three-dimensional texture coordinate calculation unit; and the adder that adds the color information acquired by the color information acquisition unit to the viewpoint image after rendering of the 3D content. (9) The information processing system according to any one of (1) to (8), wherein the viewpoint image is an image obtained by rendering 3D content of the space without using the volumetric image. (10) The information processing system according to (7) or (8), wherein the three-dimensional reconstruction unit uses a ray tracing method with the user's viewpoint as the origin. (11) The information processing system according to any one of (1) to (10), wherein the resolution of the volumetric image is changed depending on the importance from the user's viewpoint. (12) The information processing system according to any one of (5) to (8), wherein the region determination unit determines a space where the user is present as the second region. (13) The information processing system according to (12), wherein the second region is a space of a predetermined range centered on the space where the user is present. (14) The information processing system according to any one of (1) to (13), wherein, when the volumetric image is at a boundary of the space, the generation range of the volumetric image is set to straddle the boundary. (15) The information processing system according to (4), wherein the space is divided based on the aisles of the spectator seats. (16) An information processing method in which a computer determines texel coordinates of a volumetric image based on depth values ​​of a depth image acquired from a virtual space, stores color information contained in a color image at the determined texel coordinates of the volumetric image, and adds the stored color information to a viewpoint image of the space.

[0123] REFERENCE SIGNS LIST 1 Information processing system 2 Server 3 Client terminal 11 Volumetric image generation unit 12 Content logic processing unit 13 Content drawing unit 14 Depth and color image acquisition unit 15, 25 Communication unit 21 Content display control unit 22 Content logic processing unit 23 Content drawing unit 24 Three-dimensional reconstruction unit 31, 33 Content data storage unit 32 Volumetric image data storage unit 34 Area division data storage unit 35 Display unit 36 ​​Device input unit 41 Space division unit 42 Update unit 43 Texel coordinate determination unit 44 Save unit 45 Volumetric image transmission unit 51 Area determination unit 52 Drawing unit 53 Three-dimensional reconstruction unit 61 Scanning unit 62 Three-dimensional texture coordinate calculation unit 63 Color information acquisition unit 64 Addition unit 71 Entire virtual space 72 Small space 73 Small space where user exists 81 Three-dimensional reconstruction image 82 Normal drawing image N Network p object

Claims

1. An information processing system having a texel coordinate determination unit that determines the texel coordinates of a volumetric image based on the depth values ​​of a depth image acquired from a virtual space, a storage unit that stores color information contained in a color image at the texel coordinates of the volumetric image determined by the texel coordinate determination unit, and an addition unit that adds the color information stored in the volumetric image by the storage unit to a viewpoint image of space.

2. An information processing system as described in claim 1, comprising: a server; and a client terminal connected to the server via a network, wherein the server has the texel coordinate determination unit and the storage unit, the client terminal has the addition unit, and the server has a transmission unit that transmits the volumetric image in which color information is stored by the storage unit to the client terminal.

3. The information processing system according to claim 1, wherein the space is a space into which the virtual space is divided.

4. The information processing system according to claim 3, wherein the space is divided based on spectator seats.

5. The information processing system of claim 2, wherein the client terminal has an area determination unit that determines, depending on the user's position in the virtual space, a first area in which 3D content in the virtual space is displayed by three-dimensional reconstruction using the volumetric image, and a second area in which the 3D content is displayed without using the volumetric image.

6. The information processing system according to claim 5, wherein the client terminal has a drawing unit that draws the 3D content in the second area without using the volumetric image.

7. An information processing system according to claim 5, wherein the client terminal has a three-dimensional reconstruction unit that displays 3D content in the virtual space by three-dimensional reconstruction using the volumetric image in the first area.

8. The information processing system of claim 7, wherein the three-dimensional reconstruction unit comprises: a scanning unit which scans a virtual space with a plane at a determined interval from a distance in the line of sight of a user's viewpoint; a three-dimensional texture coordinate calculation unit which forms a new surface at a portion where the scanning plane scanned by the scanning unit intersects with a cube for displaying the volumetric image and calculates three-dimensional texture coordinates on the formed surface; a color information acquisition unit which acquires color information of a texel position of the volumetric image corresponding to the three-dimensional texture coordinates calculated by the three-dimensional texture coordinate calculation unit; and an adder which adds the color information acquired by the color information acquisition unit to the viewpoint image after the 3D content is rendered.

9. The information processing system according to claim 1, wherein the viewpoint image is an image that depicts 3D content of the space without using the volumetric image.

10. The information processing system according to claim 7, wherein the three-dimensional reconstruction unit uses a ray tracing method with the user's viewpoint as the origin.

11. The information processing system according to claim 1, wherein the resolution of the volumetric image is changed according to the importance of the volumetric image in a user's viewpoint.

12. The information processing system according to claim 5, wherein the area determination unit determines a space in which the user exists as the second area.

13. The information processing system according to claim 12, wherein the second area is a predetermined range of space centered on the space in which the user exists.

14. The information processing system according to claim 1, wherein, when the volumetric image is on a boundary of the space, the generation range of the volumetric image is set to straddle the boundary.

15. The information processing system according to claim 4, wherein the space is divided based on the aisles of the spectator seats.

16. An information processing method in which a computer determines texel coordinates of a volumetric image based on depth values ​​of a depth image obtained from a virtual space, stores color information contained in a color image at the determined texel coordinates of the volumetric image, and adds the stored color information to a viewpoint image of the space.

Images