Information processing device and information processing method

Abstract

[Problem] To provide an information processing device and an information processing method. [Solution] The information processing device is provided with: a receiving unit that receives a request including load information about a load; and a transmitting unit that transmits a data set including three-dimensional shape data having a number of vertices according to the load information and left-eye texture data and right-eye texture data corresponding to the three-dimensional shape data in response to the request.

Description

Technical Field

[0001] The present disclosure relates to an information processing device and an information processing method. Background Art

[0002] 3D movies, 3D TVs, etc. all use a mechanism in which the left eye and the right eye view different images, and use binocular parallax between the images to provide a stereoscopic effect to the user. In addition, some technologies use an HMD (head mounted display) or the like mounted on the user's head to reproduce motion parallax.

[0003] Since the viewpoint needs to be moved according to the user's head position, etc., images viewed from different viewpoints are required to reproduce motion parallax. Examples thereof include a technology for rendering (generating) an image viewed from each viewpoint by detecting / recording information about the three-dimensional shape of an object and using a three-dimensional model reconfigured based on the information, as in the following patent document 1.

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: Specification of U.S. Unexamined Patent Application Publication No. 2015 / 310662 Summary of the invention

[0007] Problems to be solved by the present invention

[0008] However, in the above-mentioned technology, the three-dimensional shape data used for rendering has a great influence on the subjective image quality of the image (display image) viewed from each viewpoint. Therefore, depending on the accuracy of the three-dimensional shape data, etc., the subjective image quality of the display image may be reduced.

[0009] Therefore, the present disclosure proposes a mechanism that makes it possible to suppress a reduction in the subjective image quality of a display image generated based on three-dimensional shape data.

[0010] Means of solving the problem

[0011] According to the present disclosure, an information processing device is provided, comprising: a receiving unit that receives a request including load information about a load; and a sending unit that sends a data set according to the request. The data set includes three-dimensional shape data and left-eye texture data and right-eye texture data. The three-dimensional shape data has a vertex number corresponding to the load information. The left-eye texture data and the right-eye texture data correspond to the three-dimensional shape data.

[0012] In addition, according to the present disclosure, there is provided an information processing device, including: a sending unit that sends a request including load information about a load; a receiving unit that receives a data set including three-dimensional shape data and left-eye texture data and right-eye texture data; and a rendering unit that generates a left-eye display image and a right-eye display image based on the data set. The three-dimensional shape data has a vertex number corresponding to the load information. The left-eye texture data and the right-eye texture data correspond to the three-dimensional shape data.

[0013] In addition, according to the present disclosure, there is provided an information processing method, comprising: receiving a request including load information about a load; and causing a data set to be sent by a processor according to the request. The data set includes three-dimensional shape data and left-eye texture data and right-eye texture data. The three-dimensional shape data has a vertex number corresponding to the load information. The left-eye texture data and the right-eye texture data correspond to the three-dimensional shape data.

[0014] Effects of the Invention

[0015] As described above, according to the present disclosure, it is possible to suppress a decrease in the subjective image quality of a display image generated based on three-dimensional shape data.

[0016] Note that the effects described above are not necessarily limiting. In addition to the above effects, or instead of the above effects, any effect indicated in this specification or other effects that can be understood from this specification can be applied. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is an explanatory diagram schematically showing a configuration of a conveying system according to an embodiment of the present disclosure.

[0018] Figure 2 Schematic diagram schematically showing the flow of a process of generating a data set according to an embodiment.

[0019] Figure 3 FIG. 1 is a diagram schematically showing the flow of a data transmission process according to an embodiment.

[0020] Figure 4 An explanatory diagram for describing the association between vertices of a three-dimensional model and individual texture data.

[0021] Figure 5 is a diagram showing an example of the data structure of three-dimensional shape data.

[0022] Figure 6 This is an explanatory diagram for describing the association between vertices of a three-dimensional model and texture data in the embodiment.

[0023] Figure 7is a schematic diagram for describing rendering using stereo texture data in an embodiment.

[0024] Figure 8 This is an explanatory diagram for explaining errors in a three-dimensional model.

[0025] Fig. 9 FIG. 1 is a schematic diagram schematically showing imaging by a stereo camera device.

[0026] Fig.10 : is a schematic diagram schematically showing the flow of a process of generating texture data.

[0027] Fig.11 is a schematic diagram showing the relationship between a three-dimensional model including errors and a stereo camera.

[0028] Fig.12 is a schematic diagram showing a viewing and listening situation in which the position of the camera and the position of the user's eyes match each other.

[0029] Fig.13 is a schematic diagram illustrating a viewing and listening situation in which the position of a camera and the position of the user's eyes are different from each other.

[0030] Fig.14 is an explanatory diagram for describing a comparative example.

[0031] Fig.15 is an explanatory diagram for describing a comparative example.

[0032] Fig.16 : is a block diagram showing an example of the configuration of the distribution server 1 according to the embodiment.

[0033] Fig.17 is a block diagram showing an example of the configuration of the data set generating unit 11 .

[0034] Fig.18 1 is an explanatory diagram showing the vertex reduction process performed by the vertex reduction unit 111 .

[0035] Fig.19 is a schematic diagram showing the gradual decrease of vertices.

[0036] Fig. 20 is a diagram schematically illustrating parallel generation of data sets in a case where the controller 10 includes a plurality of data set generation units 11 .

[0037] Fig.21 : is a flowchart showing an operation example of the distribution server 1 for generating a data set.

[0038] Fig. 22 : is a flowchart showing an example of the operation of the distribution server 1 for transmitting a data set.

[0039] Fig.23 : is a block diagram showing a configuration example of the display control device 2 according to the present embodiment.

[0040] Fig.24 is a flowchart showing an operation example of the display control device 2 .

[0041] Fig.25 : is an explanatory diagram for describing Modification Example 1.

[0042] Fig.26 is a block diagram showing an example of a hardware configuration. DETAILED DESCRIPTION

[0043] Preferred embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and thus redundant descriptions thereof are omitted.

[0044] Note that the description is given in the following order.

[0045] <<1. System Overview>>

[0046] <1-1. Configuration>

[0047] <1-2. Processing Flow>

[0048] <<2. According to the principle of this technology>>

[0049] <<3. Distribution Server>>

[0050] <3-1. Configuration>

[0051] <3-2. Operation>

[0052] <<4. Display control device>>

[0053] <4-1. Configuration>

[0054] <4-2. Operation>

[0055] <<5. Modification Example>>

[0056] <5-1. Modification Example 1>

[0057] <5-2. Modification Example 2>

[0058] <5-3. Modification Example 3>

[0059] <5-4. Modification Example 4>

[0060] <<6. Hardware Configuration Example>>

[0061] <<7. Conclusion>>

[0062] <<1. System Overview>>

[0063] <1-1. Configuration>

[0064] First, refer to Figure 1 An overview of an information processing system according to an embodiment of the present disclosure is described. Figure 1 is an explanatory diagram schematically showing a configuration of a conveying system according to an embodiment of the present disclosure.

[0065] like Figure 1 As shown, the transmission system 1000 according to the present embodiment is an information processing system including a distribution server 1, a display control device 2, an HMD 3, and a communication network 5. The transmission system 1000 according to the present embodiment provides a stereoscopic effect by binocular parallax and motion parallax to a user U who wears the HMD 3 and views an image displayed on the HMD 3.

[0066] The distribution server 1 and the display control device 2 are coupled via a communication network 5. Information can be sent and received between the distribution server 1 and the display control device 2. In addition, the display control device 2 and the HMD 3 are also coupled in a wired or wireless manner, and information can be sent and received between the display control device 2 and the HMD 3.

[0067] The communication network 5 is a wired or wireless transmission path for information to be sent from a device coupled to the communication network 5. For example, the communication network 5 may include a public network such as the Internet, a telephone network, or a satellite communication network, and various LANs (Local Area Networks) including Ethernet (registered trademark), WAN (Wide Area Network), etc. In addition, the communication network 5 may include a private network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0068] The distribution server 1 stores 3D shape data and texture data corresponding to the 3D shape data, and transmits (transmits) a data set to the display control device 2 in response to a data transmission request (hereinafter also referred to as a request) from the display control device 2. The data set includes the 3D shape data and the texture data.

[0069] The display control device 2 generates (renders) a display image at the viewpoint based on the data set received from the distribution server 1 and the viewpoint information related to the viewpoint of the user U received from the HMD 3, and sends the display image to the HMD 3. In the present embodiment, the display image generated by the display control device 2 includes a left-eye display image and a right-eye display image. The left-eye display image is displayed in front of the left eye of the user U by the HMD 3 described below. The right-eye display image is displayed in front of the right eye of the user U. In addition, the left-eye display image and the right-eye display image are collectively referred to as a stereoscopic display image below, or simply referred to as a display image in some cases.

[0070] The HMD 3 is a display device (display unit) that displays a stereoscopic display image received from the display control device 2. Note that the HMD 3 includes a sensor that acquires viewpoint information about the viewpoint of the user U wearing the HMD 3 and transmits the viewpoint information to the display control device 2. The viewpoint information transmitted by the HMD 3 may include, for example, information indicating the position of the viewpoint of the user U and the posture of the user U.

[0071] Note that Figure 1 The display control device 2 and the HMD 3 are shown as different devices, but the display control device 2 and the HMD 3 may be integrated. That is, one information processing device may also have the function of the display control device 2, and the function of the HMD 3 serving as a display unit that is worn on the user's head and displays a left-eye display image and a right-eye display image.

[0072] The above configuration allows the transmission system 1000 to provide the user U with a stereoscopic effect brought about by binocular parallax and motion parallax.

[0073] Here, for example, the above-mentioned three-dimensional shape data and texture data corresponding to the three-dimensional shape data can be acquired by a well-known three-dimensional capture technology, which uses, for example, a method in which a distance measuring device such as a ToF (Time of Flight) sensor is used or a technology such as stereo matching is used. The three-dimensional shape data and texture data acquired by the three-dimensional capture technology are referred to as original three-dimensional shape data and original texture data, respectively, and in some cases, the two are collectively referred to as original data.

[0074] In some cases, the raw data has an extremely large amount of data. Therefore, the transmission load and the processing load are also heavy, and it is difficult to perform processing within the requested time under some conditions. For example, depending on the frequency band of the transmission path between the distribution server 1 and the display control device 2 (the frequency band of the communication network 5), it may be difficult to transmit the raw data as it is. In addition, depending on the processing performance of the display control device 2 (for example, the processing speed of the processor and the size of the memory), it may be difficult to generate a display image from the raw data.

[0075] Therefore, it is considered to change the amount of data to be transmitted from the distribution server 1 to the display control device 2 according to such a load. Such a mechanism is described below.

[0076] The display control device 2 sends a request including load information about the load to the distribution server 1. The load information may include, for example, at least one of transmission path band information about the frequency band of the transmission path between the distribution server 1 and the display control device 2 or processing performance information about the processing performance of the display control device 2.

[0077] The distribution server 1 transmits a data set including three-dimensional shape data and texture data corresponding to the three-dimensional shape data to the display control device 2. The data set has a data amount corresponding to the load information included in the request received from the display control device 2. Here, each of the three-dimensional shape data and texture data included in the data set to be transmitted may be data having a data amount that is reduced more than the data amount of the original three-dimensional shape data and the original texture data. The process of reducing the data amount may include, for example, a vertex reduction process of reducing the vertices included in the three-dimensional shape data.

[0078] As described above, changing the amount of data to be sent from the distribution server 1 to the display control device 2 according to the load information can make the transmission and generation of the display image smooth. However, the subjective image quality of the display image may decrease as the amount of data decreases. For example, reducing the vertices included in the three-dimensional shape data may reduce the shape accuracy of the three-dimensional model corresponding to the three-dimensional shape data, and the subjective image quality of the display image may decrease. For example, the three-dimensional model corresponding to the three-dimensional shape data loses unevenness, etc., and may cause the user to be unable to obtain a stereoscopic effect.

[0079] Therefore, focusing on the above situation, the present embodiment has been created. The distribution server 1 according to the present embodiment also uses the left camera image and the right camera image to generate the left eye texture data and the right eye texture data. The left camera image and the right camera image are used to generate (capture) three-dimensional shape data. That is, the data set to be sent from the distribution server 1 according to the present embodiment to the display control device 2 does not include a single texture data, but includes multiple texture data: left eye texture data and right eye texture data. The display control device 2 then generates a left eye display image and a right eye display image based on the three-dimensional shape data processed by vertex reduction and the left eye texture data and the right eye texture data. The data set received from the distribution server 1 includes three-dimensional shape data, left eye texture data, and right eye texture data. Even in the case where the three-dimensional shape data processed by vertex reduction is used for display, this configuration suppresses the reduction of the subjective image quality of the displayed image. For example, it is possible to reproduce unevenness that does not exist in the three-dimensional model corresponding to the three-dimensional shape data.

[0080] <1-2. Processing Flow>

[0081] The schematic configuration of the conveying system 1000 according to the present embodiment has been described above. Next, a schematic flow of processing of the conveying system 1000 according to the present embodiment is described.

[0082] As described above, the distribution server 1 according to the present embodiment sends three-dimensional shape data and texture data each having a reduced data amount to the display control device 2. Here, it is desirable to perform processing for reducing the data amount in advance. For example, a plurality of data sets having different data amounts can be pre-generated and stored based on the original data. The distribution server 1 then selects a data set having a data amount corresponding to the load information from the plurality of stored data sets, and sends the selected data set to the display control device 2. This allows the distribution server 1 to quickly respond to a request from the display control device 2.

[0083] Accordingly, the following reference Figure 3 The processing flow used to generate the dataset is described, and then, with reference to Figure 3 The flow of a process for transferring data is described.

[0084] (Processing of generating data set)

[0085] Figure 2 is a schematic diagram schematically showing the process of generating a data set according to the present embodiment. Figure 1 The distribution server 1 shown executes Figure 2 An example of a process of generating a data set is shown. However, the present technology is not limited to this example. Figure 2The processing shown may be performed by another device. A data set generated in advance by another device may be stored in the distribution server 1 .

[0086] The distribution server 1 performs a process of reducing the data included in the data set DS10 and generates a plurality of data sets DS11 to DS13. The generated data sets DS11 to DS13 are stored in the distribution server 1.

[0087] Figure 2 The data set DS10 shown includes original three-dimensional shape data F10, original texture data T10, left camera image G10L, and right camera image G10R. The original three-dimensional shape data F10 includes a plurality of vertex data V101 to V107. In addition, the left camera image G10L and the right camera image G10R may be images used to generate the original three-dimensional shape data F10. For example, the left camera image G10L is an image acquired by performing imaging by the left camera, and the right camera image G10R is an image acquired by performing imaging by the right camera. The left camera images the object from the left side. The right camera images the object from the right side.

[0088] like Figure 2 As shown, the generated data sets DS11 to DS13 include three-dimensional shape data F11 to F13, and left-eye texture data T11L to T13L and right-eye texture data T11R to T13R, respectively. The left-eye texture data T11L to T13L and the right-eye texture data T11R to T13R correspond to the three-dimensional shape data F11 to F13. Figure 2 As shown, the generated data sets DS11 to DS13 have different data amounts.

[0089] exist Figure 2 In the example shown, data set DS11 has the largest amount of data among data sets DS11 to DS13. Figure 2 In the example shown, the three-dimensional shape data F11 included in the data set DS11 has not been subjected to vertex reduction processing. The number of vertex data V111 to V117 included in the three-dimensional shape data F11 is the same as the number of vertex data included in the original three-dimensional shape data F10. However, the present embodiment is not limited to this example. The distribution server 1 can perform vertex reduction processing on the three-dimensional shape data included in all data sets that can be sent.

[0090] In contrast, the three-dimensional shape data F12 and the three-dimensional shape data F13 included in the data set DS12 and the data set DS13 are generated by performing vertex reduction processing on the original three-dimensional shape data F10. Figure 2In the example shown, the vertex data V121 to V124 included in the three-dimensional shape data F12 are smaller than the vertex data V101 to V107 included in the original three-dimensional shape data F10, and are larger than the vertex data V131 to V132 included in the three-dimensional shape data F13. It should be noted that the number of vertices (vertex count) included in each three-dimensional shape data is not limited to Figure 2 The number of vertex data shown.

[0091] Furthermore, the left-eye texture data T11L to T13L and the right-eye texture data T11R to T13R included in the data sets DS11 to DS12 may be generated to have the number of pixels corresponding to the number of vertices of the respective three-dimensional shape data F11 to F13. Figure 2 As shown, since the corresponding three-dimensional shape data has a higher number of vertices, the left eye texture data and the right eye texture data can each have a higher number of pixels. This is because, in the case where the three-dimensional shape data has a smaller number of vertices, even if the number of pixels of each of the left eye texture data and the right eye texture data is increased, it does not contribute much to improving the image quality. This configuration makes it possible to effectively reduce the amount of data.

[0092] Notice, Figure 2 An example is shown in which three data sets DS11 to DS13 are generated, but the number of data sets to be generated is not limited to Figure 2 . As a large number of data sets are generated, the amount of data transmitted and the processing performance can be more finely adjusted according to the bandwidth of the transmission path. However, depending on the number of data sets to be generated, a processing cost for generating the data sets and a retention cost for the data sets are required. Therefore, it is desirable to determine the number of data sets to be generated by considering the processing cost and the retention cost.

[0093] (Processing of transmitted data)

[0094] Figure 3 The diagram schematically shows the flow of data transmission processing according to the present embodiment. Figure 2 A data set to be transmitted to the display control device 2 is selected from the plurality of data sets DS11 to DS13 generated in advance (S11). In step S11, for example, the distribution server 1 can select a data set including three-dimensional shape data having a number of vertices corresponding to the load information based on the load information included in the request received from the display control device 2. Such selection makes it possible to adjust the transmission of the data set from the distribution server 1 to the display control device 2 and the load of the rendering process performed by the display control device 2.

[0095] Subsequently, the data set selected in step S11 is transmitted (sent) from the distribution server 1 to the display control device 2 (S12). Then, the display control device 2 generates (renders) a display image based on the data set received from the distribution server 1 (S13). The display image generated in step S13 includes a left eye display image D10L and a right eye display image D10R. The left eye display image D10L is displayed in front of the user's left eye. The right eye display image D10R is displayed in front of the user's right eye. In addition, in step S13, the left eye display image D10L is generated using the three-dimensional shape data and the left eye texture data, and the right eye display image D10R is generated using the three-dimensional shape data and the right eye texture data.

[0096] The left-eye display image D10L and the right-eye display image D10R generated by the display control device 2 are displayed by the HMD 3 ( S14 ).

[0097] <<2. According to the principle of this technology>>

[0098] An overview of the present embodiment has been described above. As described above, in the present embodiment, instead of transmitting a single texture data, left eye texture data and right eye texture data are transmitted. The transmitted left eye texture data and right eye texture data are then used to render a display image. This makes it possible to suppress degradation in the image quality of the display image while reducing the amount of data. Here, the technical principle of suppressing degradation in the image quality of the display image by performing rendering using the left eye texture data and the right eye texture data in the present embodiment is described. Note that the following describes that in some cases the left eye texture data and the right eye texture data are collectively referred to as stereo texture data.

[0099] First, refer to Figure 4 and Figure 5 , describes the association between three-dimensional shape data and texture data in the case of performing rendering by using single texture data. Figure 4 It is an explanatory diagram for describing the association between the vertices of a three-dimensional model and individual texture data. Figure 5 is a diagram showing an example of the data structure of three-dimensional shape data.

[0100] Figure 4 A three-dimensional model M21 as a cube and a single texture data T21 corresponding to the three-dimensional model M21 are shown. Figure 4 The three-dimensional space coordinates (x 0 ,y 0 , z 0 ) corresponds to the texture coordinate (u 0 , v 0 ).

[0101] like Figure 5 As shown, in the three-dimensional shape data F21 retained in the calculator, (x 0 ,y 0 , z 0 ,u 0 , v 0 ) can be considered as a vertex data. Then, in Figure 5 In the example shown, N arrays of such vertex data are included in the three-dimensional shape data F21. That is, the three-dimensional shape data F21 includes information for associating the coordinates of the vertices in the three-dimensional model with the corresponding coordinates in the texture data.

[0102] Here, when the Figure 5 When the three heads of each of the three-dimensional shape data F21 shown are regarded as triangular patches, Figure 4 A three-dimensional model M21 is formed in the three-dimensional space coordinate system of xyz shown. The three-dimensional model M21 has a solid shape including a group of triangular patches. In addition, the association between the three-dimensional space coordinates (x, y, z) and the texture coordinates (u, v) makes it possible to obtain the triangular area corresponding to each triangular patch in the texture data T21. Mapping (pasting) the triangular area to the triangular patch in the three-dimensional model M21 while transforming the triangular area using an affine transformation makes it possible to render the three-dimensional model using the texture data.

[0103] The above describes the association between three-dimensional shape data and texture data in the case of performing rendering by using single texture data. Subsequently, the association between three-dimensional shape data and texture data by performing rendering by using stereo texture data in the present embodiment is described.

[0104] Figure 6 This is an explanatory diagram for describing the relationship between vertices of a three-dimensional model and texture data in this embodiment. Figure 6 The three-dimensional model M22, and the left-eye texture data T22L and the right-eye texture data T22R are shown. The three-dimensional model M22 is a cube. The left-eye texture data T22L and the right-eye texture data T22R correspond to the three-dimensional model M22.

[0105] instruct Figure 6 The three-dimensional space coordinates (x 0 ,y 0 , z 0 ) corresponds to the texture coordinate (u 0 , v 0 ).also, Figure 6 The three-dimensional space coordinates (x 0 ,y 0、z 0 ) similarly corresponds to the texture coordinate (u 0 、v 0 ).

[0106] like Figure 6 As shown in FIG. 1 , the texture coordinates in the left eye texture data T22L and the texture coordinates in the right eye texture data T22R can be expressed as the same coordinates. The texture coordinates in the left eye texture data T22L and the texture coordinates in the right eye texture data T22R correspond to the same vertex in the three-dimensional model M22. Therefore, the data structure of the three-dimensional shape data according to the present embodiment can be similar to that of the reference Figure 5 Describes the data structure of the example.

[0107] Figure 7 is a schematic diagram for describing rendering using three-dimensional texture data in this embodiment. Figure 7 As shown, the left eye texture data T22L is mapped to the three-dimensional model M22 and rendering is performed at the viewpoint corresponding to the user's left eye so that the left eye display image D22L can be generated (rendered). Figure 7 As shown, mapping the right-eye texture data T22R to the three-dimensional model M22 and performing rendering at a viewpoint corresponding to the user's right eye makes it possible to generate (render) a right-eye display image D22R.

[0108] Figure 7 The process shown for performing the rendering may be performed by Figure 1 The display control device 2 shown in FIG. Figure 5 The information of the association between the vertex positions of the three-dimensional model and the texture data is represented as three-dimensional shape data. Therefore, using the data set received from the distribution server 1 allows the display control device 2 to perform the following operations: Figure 7 The rendering shown. The data set includes three-dimensional shape data and three-dimensional texture data corresponding to the three-dimensional shape data.

[0109] As described above, using stereo texture data for rendering a texture of a common three-dimensional model allows the rendered stereoscopic display image to be fused at a position different from the surface of the three-dimensional model. This is the same principle as that a stereoscopic effect can be provided despite the fact that the display surface is planar, for example, in a stereo display that allows an image displayed on a flat screen to provide binocular parallax. The present technology suppresses a reduction in the subjective image quality of a display image rendered at a user's viewpoint by utilizing the effect of allowing recognition of unevenness different from the unevenness of the shape of the three-dimensional model by performing rendering using such a stereo texture.

[0110] As described above, the original three-dimensional shape data is acquired by a three-dimensional capture technique such as a method using a distance measuring device or a method using a technique such as stereo matching. The original three-dimensional shape data is acquired in various methods, but any method may have errors.

[0111] Furthermore, even if a shape can be acquired with high precision, a complex shape requires a very large amount of three-dimensional shape data or a very large number of vertices to reproduce the shape with high precision. Figures 1 to 3 As described above, in the present embodiment, a data set including three-dimensional shape data subjected to vertex reduction processing is transmitted and used for rendering according to load information about the frequency band of the transmission path, processing performance, etc. Therefore, depending on the load information, rendering may be performed using three-dimensional shape data having a number of vertices insufficient for reproducing the shape with high accuracy or three-dimensional shape data including errors.

[0112] When rendering is performed in this way by using three-dimensional shape data including errors caused by any one of acquisition (measurement), transmission or processing and a single texture, the shape including the error is reproduced and the user who views the stereoscopic display image also recognizes the shape including the error. In contrast, even in the case of using three-dimensional shape data including such errors, rendering is performed using three-dimensional shape data and stereoscopic texture in the present embodiment. This makes the error of the shape appear smaller, so that the reduction in subjective image quality can be suppressed. The following describes the principle that such a shape error appears smaller in the present embodiment.

[0113] Figure 8 This is an explanatory diagram used to describe errors in a three-dimensional model. Figure 8 The actual shape of the three-dimensional object OBJ31 having a protrusion (beak) and the shape of the three-dimensional model M32 are shown. The three-dimensional model M32 is three-dimensionally modeled on the three-dimensional object OBJ31 by a three-dimensional capture technology. The three-dimensional object OBJ31 initially includes Figure 8 The protrusion B shown. In contrast, the three-dimensional model M32 has no protrusion due to errors caused by measurement, etc. Note that Figure 8 The tip position corresponding to the tip of the protrusion B of the three-dimensional object OBJ31 in the three-dimensional space where the three-dimensional model M32 exists is shown as a point P.

[0114] Here, using Figure 8 In the case where rendering is performed using the three-dimensional shape data corresponding to the three-dimensional model M32 shown and a single texture data, it is difficult for a user viewing the generated stereoscopic display image to recognize the protrusion. In contrast, the use of a three-dimensional texture can make the protrusion appear to exist in this embodiment even if rendering is performed using the three-dimensional shape data corresponding to a three-dimensional model including an error, such as the three-dimensional model M32.

[0115] In order to provide a stereoscopic effect brought about by a stereoscopic texture, texture data generated based on a camera image acquired by a stereoscopic camera is used, and the stereoscopic camera is arranged to have a horizontal distance close to the interocular distance of a person. The principle of texture data generation based on the camera image acquired by the stereoscopic camera in this embodiment is described below.

[0116] Fig. 9 FIG. 1 is a schematic diagram schematically showing imaging by a stereo camera device. Fig. 9 In FIG. 1 , the left camera C31L and the right camera C31R are arranged as stereo cameras, the interval of which is substantially the same as the interocular distance of a person to allow a three-dimensional object OBJ31 to be imaged. Note that Fig. 9 The imaging ranges of the left camera C31L and the right camera C31R are shown by single-dot chain lines.

[0117] The positional relationship between the stereo camera and the three-dimensional object corresponds to the positional relationship between the stereo camera and the three-dimensional model in the three-dimensional space. The known positional relationship between the stereo camera and the three-dimensional model in the three-dimensional space makes it possible to generate texture data corresponding to the three-dimensional model as follows based on the camera images acquired by the stereo camera.

[0118] Fig.10 Schematic diagram schematically showing the flow of the process of generating texture data. A perspective projection image can be generated according to the positional relationship between the stereo camera and the three-dimensional model by perspectively projecting the shape of the three-dimensional model viewed from each camera position onto a perspective projection surface. Fig.10 In the example shown, the projection is from Fig. 9 The three-dimensional model M32 is viewed from the position of the left camera C32L shown in FIG. 1 and a left perspective projection image P31L is generated. Similarly, the projection image P31L is viewed from the left camera C32L. Fig. 9 The three-dimensional model M32 is viewed from the position of the right camera C32R as shown, and a right perspective projection image P31R is generated.

[0119] The composition of each perspective projection image is the same as the composition of the camera image acquired by performing imaging by each camera. This makes it possible to establish an association between the perspective projection image and the camera image for each triangular area corresponding to a triangular patch including three vertices in the three-dimensional model. Fig.10 In the example shown, the triangular area A31L corresponding to the triangular patch A30 of the three-dimensional model M32 in the left perspective projection image P31L is associated with the triangular area A32L of the left camera image G32L. Similarly, the triangular area A31R corresponding to the triangular patch A30 in the right perspective projection image P31R is associated with the triangular area A32R of the right camera image G32R.

[0120] Texture data can be generated by mapping (pasting) each triangular region in each camera image based on the association of each triangular region acquired as described above while transforming the triangular region. Fig.10 In the example shown, a triangular area A32L of the left camera image G32L is mapped to a triangular area A33L of the left eye texture data T33L, and a triangular area A32R of the right camera image G32R is mapped to a triangular area A33L of the right eye texture data T33R.

[0121] Note that a perspective projection image generated according to perspective projection from one camera position has a surface without a three-dimensional model, and therefore the texture data may have an area whose texture is not obtained from the camera image. In this case, for the area where the texture is not obtained from the camera image, for example, Figure 2 The raw texture data shown is used to obtain the texture.

[0122] like Fig.10 As shown, consider a case where a three-dimensional texture is generated by using a three-dimensional model M32 including an error. Fig.11 31 is a schematic diagram showing the relationship between the three-dimensional model M32 including an error and the stereo camera (the left camera C31L and the right camera C31R).

[0123] exist Fig.11 In FIG. 1 , the arrows extending from each of the left camera C31L and the right camera C31R to the point P represent the light rays of the point P in the image appearing in each camera. The point P is the tip position of the protrusion existing in the real shape. The three-dimensional model M32 has no protrusion due to the error. Therefore, the arrows extending from the left camera C31L and the right camera C31R to the point P do not intersect at the point P, but intersect at the point P on the surface of the three-dimensional model M32. R and point P L These points P R With point P L The distance d therebetween is mapped as is to the position difference between the left-eye texture data and the right-eye texture data, and is recorded as is as is the left-right parallax.

[0124] Fig.10 The left eye texture data T33L and the right eye texture data T33R shown have a design difference due to this left-right parallax. The three-dimensional model M32 corresponding to the three-dimensional shape data has no protrusions. This allows the pattern of the protrusions to be recorded as having left-right parallax in texture.

[0125] When the stereoscopic texture data generated as described above is mapped from the camera position to the three-dimensional model M32 and viewed and listened to, even if the shape of the three-dimensional model M32 includes an error, a stereoscopic effect similar to that of the real shape can be obtained. Fig.12 is a schematic diagram showing a viewing and listening situation in which the position of the camera and the position of the user's eyes match each other. Fig.12 In the figure, the light ray represented by the arrow starts from point P. L The light ray indicated by the arrow extends from the point P to the user's left eye E32L located at the position of the left camera C31L. R The user's right eye E32R is located at the right camera C31R. Here, the user looks at point P L The texture and point P at R The textures at are merged at the position of point P where the above two light rays intersect, and the surface of the three-dimensional model M32 exists at the position of point P.

[0126] Notice, Fig.12 Only for point P, but in addition to point P, all light rays appearing in the left and right cameras are reproduced. Thus, even if the shape of the three-dimensional model M32 is different from the real shape, the original stereoscopic effect can be obtained. In addition, when viewing and listening with the camera position matching the user's eye position, the light rays when the camera is imaging are reproduced, and even if the three-dimensional model M32 is an arbitrary shape, the real shape of the user is reproduced.

[0127] However, in the case of viewing and listening at a position different from the position of the camera, if the shape of the three-dimensional model M32 contains errors, the actual light cannot be reproduced. Fig.13 is a schematic diagram showing a viewing and listening situation in which the position of the camera and the position of the user's eyes are different from each other. Fig.13 In the case shown, point P appears in the user's left eye E32L L The texture at and the point P that appears in the user's right eye E32R L The textures at the points merge at the point P', and the surface of the three-dimensional model M32 appears to exist at the position of the point P'. Here, Fig.13 The point P' shown is a position different from the point P which is the true tip position of the protrusion. This is because the three-dimensional model M32 includes an error. However, compared with the case where the protrusion does not appear to exist, the point P' appears to exist at the protruding position. This suppresses the reduction in subjective image quality caused by the error even in the case of viewing and listening at a position different from the camera position.

[0128] Here, an example of a case where a technique (hereinafter referred to as a related art) for performing rendering by using single texture data instead of stereo texture data is used for the present embodiment is described as a comparative example. Fig.14 and Fig.15 are explanatory diagrams each for describing a comparative example according to the present embodiment. Fig.14 and Fig.15 A single texture data is mapped to the three-dimensional model M32 and is respectively Fig.12 and Fig.13 The positional relationship is the same as the positional relationship for viewing and listening.

[0129] The prior art does not allow the fusion of the left eye and the right eye points at different positions on the surface of the three-dimensional model M32, such as Fig.12 and Fig.13 Point P in L and P R Therefore, in this comparative example, for example, Fig.14 and Fig.15 As shown, the user uses the left eye and the right eye to view the common point P located on the surface of the three-dimensional model M32. C That is, the prior art does not allow the user to identify points located on a protrusion when the three-dimensional model M32 includes errors but does not have a protrusion. In other words, the prior art cannot render a point that appears to exist on a protrusion. Fig.12 and Fig.13 Points located outside the shape of the three-dimensional model M32 are shown.

[0130] As described above, according to the present embodiment, a user can see a shape closer to the real shape than in the prior art. In other words, rendering is performed using a stereo texture so that even when three-dimensional shape data corresponding to a three-dimensional shape model including errors is used for rendering, in the present embodiment, a reduction in the subjective image quality of the displayed image at each viewpoint can be suppressed.

[0131] The technical principle according to the present embodiment has been described above. Subsequently, in order to realize the above-mentioned functions and processes, configuration examples and operation examples of the distribution server 1 and the display control device 2 are described one by one.

[0132] <<3. Distribution Server>>

[0133] <3-1. Configuration>

[0134] Fig.16 It shows that according to Figure 1 FIG. 1 is a block diagram showing an example of the configuration of the distribution server 1 of the present embodiment. Fig.16 As shown, the distribution server 1 is an information processing device including a controller 10 , a communication unit 17 , and a storage unit 19 .

[0135] The controller 10 controls each component of the distribution server 1. In addition, the controller 10 is also used as Fig.16 The data set generating unit 11 , the communication control unit 13 and the selecting unit 15 are shown.

[0136] As reference Figure 2 As described above, the data set generating unit 11 generates a plurality of data sets having different data amounts based on the original three-dimensional shape data, the original texture data, the left camera image, and the right camera image.

[0137] Fig.17 is a block diagram showing an example of the configuration of the data set generating unit 11. Fig.17 As shown, the data set generating unit 11 includes a vertex reduction unit 111 , a perspective projection unit 112 and a texture generating unit 115 .

[0138] The vertex reduction section 111 performs vertex reduction processing on the original three-dimensional shape data. The three-dimensional shape data output from the vertex reduction section 111 and subjected to the vertex reduction processing is provided to the perspective projection section 112 and is associated with the stereo texture data described below. Then, the three-dimensional shape data is stored in the storage unit 19 as a data set. That is, the three-dimensional shape data included in the data set is generated by performing the vertex reduction processing by the vertex reduction section 111.

[0139] Fig.18 1 is an explanatory diagram showing the vertex reduction process performed by the vertex reduction unit 111. Fig.18 As shown, the six vertices and seven polygons of the three-dimensional model M41 corresponding to the three-dimensional shape data that has not been subjected to the vertex reduction processing are respectively reduced to four vertices and three polygons of the three-dimensional model M42 corresponding to the three-dimensional shape data that has been subjected to the vertex reduction processing. Performing the vertex reduction processing in this way makes the shape rougher, but allows the data to be significantly reduced. This makes it possible to significantly reduce the load for transmitting and processing data. Note that the technology used for the vertex reduction processing of the vertex reduction unit 111 is not particularly limited, but, for example, a well-known vertex reduction technology such as QEM (quadratic error metric) can be used.

[0140] Fig.19 is a schematic diagram showing the gradual decrease of vertices. Fig.19 The illustrated example shows that as the number of vertices becomes smaller in the order of the three-dimensional models M51, M52, and M53, the shapes become different, and each vertex data included in the three-dimensional shape data F51, F52, and F53 corresponding to each shape has a smaller data amount.

[0141] Note that when generating a data set that does not require the reduction of vertices in the 3D shape data, such as Figure 2In the case of the data set DS11 shown, the vertex reduction unit 111 does not need to perform vertex reduction processing, but can output the original three-dimensional shape data as it is.

[0142] Refer again Fig.17 , and continue to describe the data set generation unit 11. As referenced Fig.10 As described above, the perspective projection unit 112 performs perspective projection from left and right camera positions corresponding to left-eye texture data and right-eye texture data by using the three-dimensional shape data, and generates a perspective projection image.

[0143] like Fig.17 As shown, the perspective projection unit 112 includes a left perspective projector 113L and a right perspective projector 113R. The left perspective projector 113L perspectively projects the shape of the three-dimensional model from the position of the left camera device corresponding to the left eye texture data to the perspective projection surface, and generates a left perspective projection image. The three-dimensional model corresponds to the three-dimensional shape data provided from the vertex reduction unit 111. Similarly, the right perspective projector 113R perspectively projects the shape of the three-dimensional model from the position of the right camera device corresponding to the right eye texture data to the perspective projection surface, and generates a right perspective projection image. The three-dimensional model corresponds to the three-dimensional shape data provided from the vertex reduction unit 111.

[0144] As reference Fig.10 As described above, the texture generation unit 115 establishes an association between the perspective projection image generated by the perspective projection unit 12 and the camera image, and maps the camera image to generate left-eye texture data and right-eye texture data. Fig.10 As described above, the texture generator 115 may establish an association between the perspective projection image and the camera image for each region (eg, each triangular region corresponding to a triangular patch) based on the vertices included in the three-dimensional shape data.

[0145] like Fig.17 As shown, the texture generating section 115 includes a left generating processor 116L, a right generating processor 116R, a left resolution changer 117L, and a right resolution changer 117R.

[0146] The left generation processor 116L establishes an association between the left perspective projection image and the left camera image for each triangular area. The left generation processor 116L then maps the triangular area of ​​the left camera image to the corresponding triangular area in the left eye texture data to generate the left eye texture data. Similarly, the right generation processor 116R establishes an association between the right perspective projection image and the right camera image for each triangular area. The right generation processor 116L then maps the triangular area of ​​the left camera image to the corresponding triangular area in the right eye texture data to generate the right eye texture data. Note that the left generation processor 116L and the right generation processor 116R can each obtain a texture from the original texture data for an area whose texture is not obtained from the camera image, and generate the texture data as described above.

[0147] The left resolution changer 117L performs a resolution change process on the left eye texture data generated by the left generation processor 116L, and outputs the left eye texture data. Similarly, the right resolution changer 117R performs a resolution change process on the right eye texture data generated by the right eye generation processor 116R, and outputs the right eye texture data. The left eye texture data output from the left resolution changer 117L and the right eye texture data output from the right resolution changer 117R are associated with the three-dimensional shape data output from the vertex reduction section 111, and are stored as data sets in the storage unit 19.

[0148] The left resolution changer 117L and the right resolution changer 117R may perform resolution change processing so that the left eye texture data and the right eye texture data included in each data set each have a number of pixels corresponding to the number of vertices of the three-dimensional shape data included in the data set. In the case where the three-dimensional shape data has a low number of vertices, reducing the number of pixels of each of the left eye texture data and the right eye texture data does not result in a significant reduction in image quality. Therefore, this configuration makes it possible to effectively reduce the amount of data.

[0149] Reference above Fig.17 A configuration example of the data set generation unit 11 is described. Note that the data set generation unit 11 generates a plurality of data sets having different data amounts as described above. This allows the data set generation unit 11 to repeatedly generate data sets the same number of times as the number of desired data sets while appropriately changing parameters, for example, to gradually change the number of vertices of three-dimensional shape data and the number of pixels of each texture data included in the data set.

[0150] Alternatively, the controller 10 may include a plurality of data set generating units 11 and generate data sets in parallel. Fig. 20 is a diagram schematically illustrating parallel generation of data sets in a case where the controller 10 includes a plurality of data set generation units 11 .

[0151] Fig. 20 The N data set generating units 11-1 to 11-N shown in the figure may respectively have Fig.17 The configuration of the data set generating unit 11 shown in FIG. 1 is similar to that of the data set generating unit 11. The data set generating units 11-1 to 11-N generate data sets DS-1 to DS-N based on the original three-dimensional shape data, the original texture data, the left camera image, and the right camera image. Each data set DS-1 to DS-N includes three-dimensional shape data with different numbers of vertices, and left eye texture data and right eye texture data corresponding to the three-dimensional shape data.

[0152] like Fig. 20 As shown, parallel processing allows multiple data sets with gradually different data amounts to be generated more efficiently.

[0153] Refer again Fig.16 , and the controller 10 of the distribution server 1 is described continuously. The communication control unit 13 controls the communication with another device established by the communication unit 17. For example, the communication control unit 13 controls the communication unit 17 so that the communication unit 17 receives a request including load information from the display control device 2. In addition, the communication control unit 13 controls the communication unit 17 so that the communication unit 17 transmits a data set selected by the selection unit 15 described below to the display control device 2 according to the request received from the display control device 2.

[0154] The selection unit 15 selects a data set to be transmitted by the communication unit 17 from among a plurality of data sets generated by the data set generation unit 11 and stored in the storage unit 19 based on the load information included in the request received by the communication unit 17. As described above, the selection unit 15 can select a data set including three-dimensional shape data having the number of vertices corresponding to the load information, and left-eye texture data and right-eye texture data corresponding to the three-dimensional shape data.

[0155] The load information may include transmission path band information about the frequency band of the transmission path between the distribution server 1 and the display control device 2 as described above. For example, in a case where the bandwidth of the transmission path between the distribution server 1 and the display control device 2 is insufficient to transmit the data amount of the data set including the original three-dimensional shape data, the selection unit 15 selects the data set including the three-dimensional shape data having a lower number of vertices than the number of vertices of the original three-dimensional shape data.

[0156] For example, HMD 3 finds a convenient use when allowing the user to move freely within a certain range. Therefore, it is expected that HMD 3 establishes wireless communication. Then, in the case where the display control device 2 and HMD 3 are integrated to provide the HMD 3 with the functions of the display control device 2, the transmission path between the distribution server 1 and the display control device 2 (integrated with the HMD 3) should have a narrow frequency band. On the contrary, in the case where the display control device 2 and HMD 3 are different devices and the display control device 2 is coupled to the communication network 5 in a wired manner, the transmission path between the distribution server 1 and the display control device 2 should have a wide frequency band. In any such case, the selection unit 15 is able to appropriately select a data set based on the frequency band of the transmission path.

[0157] In addition, the load information may include processing performance information about the processing performance of the display control device 2 as described above. The display control device 2 may have various processing performances. The display control device 2 and the HMD 3 may be different devices, and the display control device 2 may be a high-specification PC with high processing performance. Alternatively, in a case where the display control device 2 and the HMD 3 are integrated and the HMD 3 has the function of the display control device 2, the display control device 2 has a processing performance lower than that of a high-specification PC in some cases. In addition, incorporating a smartphone into the HMD also allows the smartphone to be used as the display control device 2 and the HMD 3. In this case, the display control device 2 should have lower processing performance. In any such case, the selection unit 15 is able to appropriately select a data set based on the processing performance.

[0158] The communication unit 17 performs information communication with another device under the control of the above-mentioned communication control unit 13. For example, the communication unit 17 functions as a receiving unit and receives a request including load information about the load from the display control device 2. In addition, the communication unit 17 functions as a sending unit and sends a data set according to the received request. The data set includes three-dimensional shape data having a vertex number corresponding to the load information, and left eye texture data and right eye texture data corresponding to the three-dimensional shape data.

[0159] The storage unit 19 stores programs and parameters for functioning each component of the distribution server 1. For example, the storage unit 19 stores the above-mentioned original three-dimensional shape data, original texture data, left camera image, and right camera image in advance, and provides them to the data set generation unit 11. In addition, the storage unit 19 stores a plurality of data sets generated by the data set generation unit 11.

[0160] <3-2. Operation>

[0161] The above describes an example of the configuration of the distribution server 1 according to the present embodiment. Next, an example of the operation of the distribution server 1 according to the present embodiment will be described. Note that the distribution server 1 according to the present embodiment generates a plurality of data sets in advance, and transmits a data set selected from the plurality of data sets to the display control device 2 according to a request from the display control device 2 as described above. Therefore, the following refers to Fig.21 An example of the operation of the distribution server 1 for generating a data set is described, and then with reference to Fig. 22 An operation example of the distribution server 1 for transmitting a data set has been described.

[0162] Fig.21 FIG. 1 is a flowchart showing an example of the operation of the distribution server 1 for generating a data set. Fig.21 As shown, the vertex reduction section 111 of the data set generating unit 11 first performs vertex reduction processing on original three-dimensional shape data ( S101 ).

[0163] Next, the perspective projection section 112 of the data set generation unit 11 uses the three-dimensional shape data subjected to the vertex reduction process in step S101 to perform perspective projection from the left and right camera positions, and generates a perspective projection image ( S103 ).

[0164] Subsequently, the texture generation section 115 of the data set generation unit 11 establishes an association between the perspective projection image and the left and right camera images, and generates left-eye texture data and right-eye texture data ( S105 ).

[0165] Furthermore, the texture generation unit 115 performs resolution change processing on the left eye texture data and the right eye texture data generated in step S105 to provide left eye texture data and right eye texture data having a number of pixels corresponding to the number of vertices of the three-dimensional shape data subjected to vertex reduction processing in step S101 (S107).

[0166] The dataset generation unit 11 then associates the three-dimensional shape data that has undergone vertex reduction processing in step S101 with the left eye texture data and right eye texture data that have undergone resolution change processing in step S107, and causes the storage unit 19 to store the three-dimensional shape data and the left eye texture data and right eye texture data as a dataset (S109).

[0167] The above has been referred to Figure 2 An operation example of the distribution server 1 for generating a data set is described. Note that, for example, the distribution server 1 may be gradually changed while appropriately changing the parameters. Fig.21 The series of processes shown is repeated the same number of times as the desired number of data sets to gradually change the number of vertices of the three-dimensional shape data and the number of pixels of each texture data included in the data set. Fig. 20As shown, multiple data set generation units 11 may execute in parallel Fig.21 A series of processing shown.

[0168] Next, refer to Fig. 22 An operation example of the distribution server 1 for transmitting a data set has been described. Fig. 22 FIG. 4 is a flowchart showing an example of the operation of the distribution server 1 for transmitting a data set. Fig. 22 As shown, the communication unit 17 first receives a request including load information related to load from the display control device 2 under the control of the communication control unit 13 (S151).

[0169] Next, the selection unit 15 selects a data set to be transmitted by the communication unit 17 from a plurality of data sets stored in the storage unit 19 based on the load information included in the request received in step S151 (S153). As described above, the data set selected in step S153 includes three-dimensional shape data having the number of vertices corresponding to the load information, and left-eye texture data and right-eye texture data corresponding to the three-dimensional shape data.

[0170] Subsequently, under the control of the communication control unit 13, the communication unit 17 transmits the data set selected in step S153 to the display control device 2 according to the request (S155). In step S151, the request is received from the display control device 2.

[0171] <<4. Display control device>>

[0172] <4-1. Configuration>

[0173] The configuration example and the operation example of the distribution server 1 according to the present embodiment are described above. Next, a configuration example of the display control device 2 is described. Fig.23 2 is a block diagram showing a configuration example of the display control device 2 according to the present embodiment. Fig.23 As shown, the display control device 2 is an information processing device including a controller 20 , a communication unit 27 , and a storage unit 29 .

[0174] The controller 20 controls each component of the display control device 2. Fig.23 As shown, the controller 20 also functions as a rendering unit 21 and a communication control unit 23 .

[0175] As above Figure 3 , Figure 7As shown in FIG. 1 , the rendering unit 21 generates (renders) a left-eye display image and a right-eye display image based on a data set received from the distribution server 1 by the communication unit 27 described below. In addition, based on viewpoint information about the viewpoint of the user, the rendering unit 21 can also generate a left-eye display image and a right-eye display image at the viewpoint. The communication unit 27 receives the viewpoint information from the HMD 3.

[0176] The communication control unit 23 controls communication with another device established by the communication unit 27. For example, the communication control unit 23 controls the communication unit 27 so that the communication unit 27 transmits a request including load information about the load to the distribution server 1. Note that the communication control unit 23 may acquire the load information from the storage unit 29, or may acquire the load information from the outside via the communication unit 27. Furthermore, the communication control unit 23 controls the communication unit 27 so that the communication unit 27 receives a data set from the distribution server 1. Furthermore, the communication control unit 23 controls the communication unit 27 so that the communication unit 27 receives viewpoint information related to the viewpoint of the user from the HMD 3. Furthermore, the communication control unit 23 controls the communication unit 27 so that the communication unit 27 transmits a left eye display image and a right eye display image to the HMD 3, and causes the HMD 3 to display the left eye display image and the right eye display image. The left eye display image and the right eye display image are generated by the rendering unit 21.

[0177] The communication unit 27 performs information communication with another device under the control of the above-mentioned communication control unit 23. For example, the communication unit 27 functions as a sending unit, and sends a request including load information about the load to the distribution server 1. In addition, the communication unit 27 functions as a receiving unit, and receives a data set from the distribution server 1. The data set includes three-dimensional shape data having a vertex number corresponding to the load information, and left-eye texture data and right-eye texture data corresponding to the three-dimensional shape data. In addition, the communication unit 27 receives viewpoint information related to the viewpoint of the user from the HMD 3, and sends the left-eye display image and the right-eye display image generated by the rendering unit 21 to the HMD 3.

[0178] The storage unit 29 stores a program and parameters for functioning each component of the display control device 2. For example, the storage unit 29 may store the above-mentioned load information.

[0179] <4-2. Operation>

[0180] The configuration example of the display control device 2 according to the present embodiment is described above. Next, an operation example of the display control device 2 according to the present embodiment is described. Fig.24 is a flowchart showing an operation example of the display control device 2 .

[0181] like Fig.24As shown, the communication unit 27 first sends a request including load information about the load to the distribution server 1 under the control of the communication control unit 23 (S201). Next, the communication unit 27 receives a data set from the distribution server 1 under the control of the communication control unit 23 (S203). As described above, the data set received in step S203 includes three-dimensional shape data having a number of vertices corresponding to the load information, and left-eye texture data and right-eye texture data corresponding to the three-dimensional shape data.

[0182] Next, the rendering unit 21 generates a left-eye display image and a right-eye display image based on the data set received in step S203. Subsequently, the communication control unit 23 controls the communication unit 27 to cause the communication unit 27 to transmit the left-eye display image and the right-eye display image to the HMD 3, thereby causing the HMD 3 to display the left-eye display image and the right-eye display image. The left-eye display image and the right-eye display image are generated in step S205.

[0183] <<5. Modification Example>>

[0184] The embodiments of the present disclosure have been described above. Some modified examples of the embodiments of the present disclosure are described below. Note that the modified examples described below can be applied to the embodiments of the present disclosure alone, or can be applied to the embodiments of the present disclosure in combination. In addition, each modified example can be applied in place of the configuration described in the embodiments of the present disclosure, or each modified example can be additionally applied to the configuration described in the embodiments of the present disclosure.

[0185] <5-1. Modification Example 1>

[0186] In the above-mentioned embodiment, an example has been described in which the selection unit 15 selects a data set to be transmitted by the communication unit 17 based on the load information, but the present technology is not limited to this example. For example, the selection unit 15 may also select a data set to be transmitted by the communication unit 17 based on the number of objects included in the data set to be transmitted by the communication unit 17. This example is described as Modification Example 1.

[0187] Fig.25 is an explanatory diagram for describing modification example 1. In this modification example, a data set can be generated for each object. Then, in this modification example, the selection unit 15 can select a data set for each object. In addition, since the data set to be sent by the communication unit 17 includes more objects, the selection unit 15 can select a data set with a smaller data amount. For example, the selection unit 15 can select a data set based on the number of objects so that the data set to be sent has a constant total data amount. For example, in the case where the number of objects is doubled, a data set can be selected so that each object has half the data amount.

[0188] Note that the number of objects included in the data set to be sent by the communication unit 17 can be identified, for example, based on the original data stored in the storage unit 19. In addition, in the case where viewpoint information about the user's viewpoint is obtained from the display control device 2, the number of objects included in the data set to be sent by the communication unit 17 can be identified based on the user's field of view identified based on the viewpoint information. Fig.25 The user's fields of view W11 to W13 are shown as examples.

[0189] The user's field of view W11 includes an object OBJ111, and therefore, the data set to be transmitted by the communication unit 17 also includes an object. Fig.25 As shown, the selection unit 15 can select the data set DS21 including the three-dimensional shape data F21 having a large number of vertices and the left eye texture data T21L and the right eye texture data T21R having a high number of pixels, respectively. Fig.25 As shown, the data set DS31 to be transmitted includes three-dimensional shape data F31, and left-eye texture data T31L and right-eye texture data T31R.

[0190] The user's field of view W12 includes two objects OBJ121 and OBJ122, and therefore, the data set to be transmitted by the communication unit 17 also includes the two objects. Fig.25 As shown, the selection unit 15 may select, for each object, a data set DS22 including three-dimensional shape data F22 whose number of vertices is reduced compared to the three-dimensional shape data F21. Fig.25 As shown in FIG. 1 , the left eye texture data T22L and the right eye texture data T22R included in the data set DS22 have a lower number of pixels than the left eye texture data T21L and the right eye texture data T21R. Fig.25 As shown, the data set DS32 to be transmitted includes three-dimensional shape data F32-1 and F32-2, and left-eye texture data T32-1L and T32-2L and right-eye texture data T32-1R and T32-2R.

[0191] The user's field of view W13 includes three objects OBJ131, OBJ132, and OBJ133, and therefore, the data set to be transmitted by the communication unit 17 also includes three objects. Fig.25 As shown, the selection unit 15 may select, for each object, a data set DS23 including three-dimensional shape data F23 whose number of vertices is further reduced compared to the three-dimensional shape data F22. Fig.25 As shown in FIG. 1 , the left eye texture data T23L and the right eye texture data T23R included in the data set DS23 have a lower number of pixels than the left eye texture data T22L and the right eye texture data T22R. Fig.25 As shown, the data set DS33 to be transmitted includes three-dimensional shape data F33-1 to F33-3, and left-eye texture data T33-1L to T33-3L and right-eye texture data T33-1R to T33-3R.

[0192] As described above, according to the present modification example, a data set to be transmitted is selected also based on the number of objects included in the data set so that an appropriate data set can be transmitted.

[0193] <5-2. Modification Example 2>

[0194] In the above embodiment, an example has been described in which the distribution server 1 generates a data set, but the present technology is not limited to this example. For example, the function of the above-mentioned data set generation unit 11 can be installed in another information processing device. Other information processing devices can generate multiple data sets in advance and provide them to the distribution server 1.

[0195] <5-3. Modification Example 3>

[0196] In addition, in the above-mentioned embodiment, an example of generating two texture data based on two camera images of the left camera image and the right camera image has been described, but the present technology is not limited to this example. For example, three or more texture data may be generated based on three or more camera images. In this case, two images of the three or more camera images may be regarded as the left camera image and the right camera image, and two texture data of the three or more generated texture data may be regarded as the left eye texture data and the right eye texture data.

[0197] Note that when three or more texture data are generated, the distribution server 1 may receive viewpoint information from the display control device 2 and select left eye texture and right eye texture data included in a data set to be transmitted from the three or more texture data based on the viewpoint information.

[0198] <5-4. Modification Example 4>

[0199] In the above-mentioned embodiment, an example has been described in which the three-dimensional shape data acquired based on the three-dimensional capture technology is the original three-dimensional shape data, but the present technology is not limited to this example. For example, the three-dimensional shape data generated on a computer instead of based on sensing in the real space is the original three-dimensional shape data, and the present technology is applicable. In this case, for example, a left camera image and a right camera image can be generated by imaging (rendering) a three-dimensional model corresponding to the original three-dimensional shape data using a virtual stereo camera, and the present technology can be applied.

[0200] <<6. Hardware Configuration Example>>

[0201] The embodiments of the present disclosure have been described above. Fig.26 The hardware configuration of the information processing device according to the embodiment of the present disclosure has been described. Fig.26 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present disclosure. Note that, Fig.26 The information processing device 900 shown can implement, for example Figure 1 , Fig.16 and Fig.23 The distribution server 1, display control device 2, and HMD 3 are shown. Information processing by the distribution server 1, display control device 2, and HMD 3 according to the present embodiment is realized by cooperation between software and hardware as described below.

[0202] like Fig.26 As shown, the information processing apparatus 900 includes a CPU (central processing unit) 901, a ROM (read only memory) 902, a RAM (random access memory) 903, and a host bus 904a. In addition, the information processing apparatus 900 includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a coupling port 911, a communication device 913, and a sensor 915. Instead of the CPU 901 or in addition to the CPU 901, the information processing apparatus 900 may include a processing circuit such as a DSP or an ASIC.

[0203] The CPU 901 functions as an arithmetic processing device and a control device, and controls the overall operation of the information processing apparatus 900 according to various programs. In addition, the CPU 901 may be a microprocessor. The ROM 902 stores programs to be used by the CPU 901, operating parameters, and the like. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters appropriately changed in the execution, and the like. The CPU 901 may be included in, for example, the controller 10 and the controller 20.

[0204] The CPU 901, the ROM 902, and the RAM 903 are coupled to each other through a host bus 904a including a CPU bus, etc. The host bus 904a is connected to an external bus 904b such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 904. Note that the host bus 904a, the bridge 904, and the external bus 904b do not necessarily have to be included separately, and the functions of the host bus 904a, the bridge 904, and the external bus 904b may be implemented in a single bus.

[0205] For example, the input device 906 is implemented by a device (e.g., a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a joystick) through which a user inputs information. In addition, the input device 906 may be, for example, a remote control device using infrared or other radio waves or an external coupling device such as a mobile phone or a PDA that supports the operation of the information processing apparatus 900. In addition, the input device 906 may include an input control circuit or the like that generates an input signal based on the information input by the user using the above-mentioned input device, and outputs the generated input signal to the CPU 901, for example. The user of the information processing apparatus 900 can input various data to the information processing apparatus 900 or instruct the information processing apparatus 900 to perform a processing operation by operating the input device 906.

[0206] The output device 907 includes a device capable of visually or audibly notifying the user of the acquired information. Examples of such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices and lamps, audio output devices such as speakers and headphones, printers, etc. The output device 907 outputs the results obtained by various processes performed by, for example, the information processing device 900. Specifically, the display device visually displays the results obtained by various processes performed by the information processing device 900 in various forms (such as text, images, tables, and graphics). At the same time, the audio output device converts an audio signal including reproduced audio data, acoustic data, etc. into an analog signal, and outputs the analog signal audibly.

[0207] The storage device 908 is a device for data storage formed as an example of a storage unit of the information processing apparatus 900. For example, the storage device 908 is implemented by a magnetic storage unit device such as an HDD, a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. The storage device 908 may include a storage medium, a recording device that records data in the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded in the storage medium, etc. The storage device 908 stores a program to be executed by the CPU 901, various types of data, various types of data acquired from the outside, etc. The above-mentioned storage device 908 may be included in, for example, the storage unit 19 and the storage unit 29.

[0208] The drive 909 is a reader / writer of a storage medium, and is incorporated in the information processing apparatus 900 or is externally attached to the information processing apparatus 900. The drive 909 reads out information recorded in a removable storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory mounted thereon, and outputs the information to the RAM 903. Furthermore, the drive 909 can also write information in a removable storage medium.

[0209] The coupling port 911 is an interface coupled to an external device, and is a coupling port to an external device capable of transmitting data by, for example, a USB (Universal Serial Bus) or the like.

[0210] The communication device 913 is a communication interface, which includes, for example, a communication device for coupling to the network 920, etc. The communication device 913 is, for example, a communication card for a wired or wireless LAN (local area network), LTE (long term evolution), Bluetooth (registered trademark), or WUSB (wireless USB), etc. In addition, the communication device 913 may be a router for optical communication, a router for ADSL (asymmetric digital subscriber line), a modem for various types of communication, etc. For example, the communication device 913 may send signals to the Internet and other communication devices according to a predetermined protocol (e.g., TCP / IP), etc., and receive signals from the Internet and other communication devices, etc. The communication device 913 may be included in, for example, the communication unit 17 and the communication unit 27.

[0211] For example, the sensor 915 may be various sensors such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, an optical sensor, a sound sensor, a distance sensor, and a force sensor. The sensor 915 acquires information about the state of the information processing device 900, such as the posture and movement speed of the information processing device 900, and information about the surrounding environment of the information processing device 900, such as the brightness or noise around the information processing device 900. In addition, the sensor 915 may include a GPS sensor that receives a GPS signal and measures the latitude, longitude, and altitude of the device.

[0212] Note that the network 920 is a wired or wireless transmission path for information to be transmitted from a device coupled to the network 920. For example, the network 920 may include a public network such as the Internet, a telephone network, or a satellite communication network, and various LANs (Local Area Networks) including Ethernet (registered trademark), WAN (Wide Area Network), and LAN, etc. In addition, the network 920 may include a private network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0213] An example of a hardware configuration that enables the functions of the information processing device 900 according to an embodiment of the present disclosure to be implemented has been described above. The above-mentioned various components can be implemented by using general-purpose components, or can be implemented by hardware dedicated to the functions of the various components. Therefore, the hardware configuration to be used can be appropriately changed according to the technical level when executing the embodiments of the present disclosure.

[0214] Note that a computer program for realizing each function of the information processing device 900 according to the embodiment of the present disclosure as described above can be created and installed in a PC or the like. In addition, a computer-readable recording medium in which such a computer program is stored can also be provided. For example, the recording medium is a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above-mentioned computer program can be distributed via, for example, a network without using a recording medium.

[0215] <<7. Conclusion>>

[0216] According to the embodiments of the present disclosure as described above, it is possible to suppress a decrease in the subjective image quality of a display image generated based on three-dimensional shape data.

[0217] The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such embodiments. Those skilled in the art may find various alternatives and modifications within the scope of the appended claims, and it should be understood that these alternatives and modifications will naturally fall within the technical scope of the present disclosure.

[0218] In addition, the steps in the above-described embodiments need not always be processed in chronological order according to the order described as a flowchart. For example, the respective steps in the processing according to the above-described embodiments may be processed in an order different from the order described as a flowchart, or may be processed in parallel.

[0219] In addition, the effects described herein are only illustrative and exemplary, and not restrictive. That is, in addition to the above effects, or instead of the above effects, the technology according to the present disclosure can also exert other effects that are obvious to those skilled in the art based on the description herein.

[0220] Note that the following configurations also fall within the technical scope of the present disclosure. (1)

[0222] An information processing device, comprising:

[0223] a receiving unit that receives a request including load information about a load; and

[0224] A sending unit sends a data set according to the request, the data set including three-dimensional shape data and left-eye texture data and right-eye texture data, the three-dimensional shape data having a vertex number corresponding to the load information, and the left-eye texture data and the right-eye texture data corresponding to the three-dimensional shape data. (2)

[0226] The information processing device according to (1) further includes a selection unit, which selects the data set to be sent by the sending unit from multiple data sets based on the load information, and each of the multiple data sets includes the three-dimensional shape data and the left eye texture data and the right eye texture data, and the left eye texture data and the right eye texture data correspond to the three-dimensional shape data. (3)

[0228] The information processing device according to (2), wherein the selection unit performs the selection further based on the number of objects included in the data set to be transmitted by the transmission unit. (4)

[0230] The information processing device according to (2) or (3), further including a data set generation unit that generates the plurality of data sets. (5)

[0232] The information processing device according to (4), wherein the data set generation unit includes a vertex reduction section that generates the three-dimensional shape data included in the data set through a vertex reduction process. (6)

[0234] An information processing device according to (4) or (5), wherein the data set generation unit also includes a perspective projection unit, which performs perspective projection from each camera device position corresponding to the left eye texture data and the right eye texture data by using the three-dimensional shape data, and generates a perspective projection image. (7)

[0236] An information processing device according to (6), wherein the data set generation unit also includes a texture generation unit, which establishes an association between the perspective projection image and a camera device image, and generates the left eye texture data and the right eye texture data, and the camera device image is acquired by performing imaging from the camera device position. (8)

[0238] The information processing device according to (7), wherein the texture generation section establishes an association between the perspective projection image and the camera image for each region based on vertices included in the three-dimensional shape data. (9)

[0240] An information processing device according to any one of (1) to (8), wherein the left eye texture data and the right eye texture data each have a number of pixels corresponding to the number of vertices of the three-dimensional shape data, the left eye texture data and the right eye texture data are included in each data set, and the three-dimensional shape data is included in the data set. (10)

[0242] An information processing device according to any one of (1) to (9), wherein the load information includes transmission path band information about the frequency band of the transmission path between a sending device and the information processing device, or processing performance information about the processing performance of the sending device, and the sending device sends the request. (11)

[0244] An information processing device, comprising:

[0245] a sending unit that sends a request including load information about the load;

[0246] a receiving unit that receives a data set including three-dimensional shape data having a vertex number corresponding to the load information and left-eye texture data and right-eye texture data corresponding to the three-dimensional shape data; and

[0247] A rendering unit generates a left-eye display image and a right-eye display image based on the data set. (12)

[0249] An information processing device according to (11), wherein the load information includes transmission path band information about the frequency band of the transmission path between a receiving device and the information processing device, or processing performance information about the processing performance of the information processing device, and the receiving device receives the request. (13)

[0251] The information processing device according to (11) or (12), wherein the rendering unit generates the left-eye display image and the right-eye display image further based on information about a viewpoint of a user. (14)

[0253] The information processing device according to (13), further including a display unit that is worn on a head of a user and displays the left-eye display image and the right-eye display image. (15)

[0255] An information processing method, comprising:

[0256] receiving a request including load information about a load; and

[0257] The processor causes a data set to be sent according to the request, the data set including three-dimensional shape data and left-eye texture data and right-eye texture data, the three-dimensional shape data having a vertex number corresponding to the load information, the left-eye texture data and the right-eye texture data corresponding to the three-dimensional shape data. (16)

[0259] An information processing method, comprising:

[0260] sending a request including load information about the load;

[0261] receiving a data set including three-dimensional shape data having a number of vertices corresponding to the load information and left-eye texture data and right-eye texture data corresponding to the three-dimensional shape data; and

[0262] A left-eye display image and a right-eye display image are generated by a processor based on the data set.

[0263] Reference numerals list

[0264] 1 Distribution Server

[0265] 2 Display control device

[0266] 3 HMD

[0267] 5 Communication Network

[0268] 10 Controller

[0269] 11 Dataset Generation Unit

[0270] 13 Communication control unit

[0271] 15 Select Unit

[0272] 17 Communication Unit

[0273] 19 Storage Unit

[0274] 20 Controller

[0275] 21 Rendering Units

[0276] 23 Communication control unit

[0277] 27 Communication Unit

[0278] 29 Storage Units

[0279] 111 Vertex reduction unit

[0280] 112 Perspective Projection Department

[0281] 115 Texture Generation Unit

Claims

1. An information processing device, comprising: a receiving unit that receives a request including load information about a load; a sending unit, which sends a data set according to the request, the data set including three-dimensional shape data and left-eye texture data and right-eye texture data, the three-dimensional shape data having a vertex number corresponding to the load information, the left-eye texture data and the right-eye texture data corresponding to the three-dimensional shape data; a selecting unit that selects the data set to be transmitted by the transmitting unit from a plurality of data sets based on the load information, the plurality of data sets each including the three-dimensional shape data and the left-eye texture data and the right-eye texture data, the left-eye texture data and the right-eye texture data corresponding to the three-dimensional shape data; as well as A data set generating unit, wherein the data set generating unit generates the plurality of data sets; Wherein, the data set generating unit comprises: a perspective projection unit that performs perspective projection from each camera position corresponding to the left-eye texture data and the right-eye texture data by using the three-dimensional shape data and generates a perspective projection image; and A texture generating section establishes an association between the perspective projection image and a camera image acquired by performing imaging from the camera position, and generates the left-eye texture data and the right-eye texture data based on the association.

2. The information processing device according to claim 1, wherein: The selection unit also performs the selection based on the number of objects included in the data set to be transmitted by the transmission unit.

3. The information processing device according to claim 1, wherein: The data set generation unit further includes a vertex reduction section that generates the three-dimensional shape data through a vertex reduction process, the three-dimensional shape data being included in the data set.

4. The information processing device according to claim 1, wherein: The texture generation section establishes an association between the perspective projection image and the camera image for each region based on vertices included in the three-dimensional shape data.

5. The information processing device according to claim 1, wherein: The left eye texture data and the right eye texture data each have a number of pixels corresponding to the number of vertices of the three-dimensional shape data, and the left eye texture data and the right eye texture data are included in each data set in which the three-dimensional shape data is included.

6. The information processing device according to claim 1, wherein: The load information includes transmission path band information about a frequency band of a transmission path between a transmission device and the information processing device, or processing performance information about processing performance of the transmission device, the transmission device transmitting the request.

7. An information processing device, comprising: a sending unit that sends a request including load information about the load; a receiving unit receiving a data set including three-dimensional shape data having a vertex number corresponding to the load information and left-eye texture data and right-eye texture data corresponding to the three-dimensional shape data; as well as a rendering unit, which generates a left-eye display image and a right-eye display image based on the data set, wherein the data set is selected from a plurality of data sets based on the load information, wherein each of the plurality of data sets includes the three-dimensional shape data and the left-eye texture data and the right-eye texture data, the left-eye texture data and the right-eye texture data corresponding to the three-dimensional shape data, The left-eye texture data and the right-eye texture data are generated based on the association between the perspective projection image and the camera device image. wherein the perspective projection image is generated by perspective projection from respective camera positions corresponding to the left-eye texture data and the right-eye texture data using the three-dimensional shape data, and Wherein, the camera image is acquired by performing imaging from the camera position.

8. The information processing device according to claim 7, wherein: The load information includes transmission path band information about a frequency band of a transmission path between a reception device and the information processing device, the reception device receiving the request, or processing performance information about processing performance of the information processing device.

9. The information processing device according to claim 7, wherein: The rendering unit also generates the left-eye display image and the right-eye display image based on information about a viewpoint of a user. 10 . The information processing device according to claim 9 , further comprising a display unit which is worn on a head of the user and displays the left-eye display image and the right-eye display image.

11. An information processing method, comprising: receiving a request including load information about a load; selecting the data set to be transmitted from a plurality of data sets based on the load information, the plurality of data sets each including three-dimensional shape data and left-eye texture data and right-eye texture data, the left-eye texture data and the right-eye texture data corresponding to the three-dimensional shape data; as well as causing a data set to be sent by a processor according to the request, the data set including three-dimensional shape data and left-eye texture data and right-eye texture data, the three-dimensional shape data having a vertex number corresponding to the load information, the left-eye texture data and the right-eye texture data corresponding to the three-dimensional shape data, Wherein, the information processing method further includes: performing perspective projection from respective camera positions corresponding to the left-eye texture data and the right-eye texture data by using the three-dimensional shape data, and generating a perspective projection image; and An association is established between the perspective projection image and a camera image acquired by performing imaging from the camera position, and the left-eye texture data and the right-eye texture data are generated based on the association.

Images