Information processing device

Abstract

This information processing device: acquires data expressing the shape of a three-dimensional model and a plurality of textures obtained on the basis of projection of the shape of each of the plurality of images; receives designation of a viewpoint with respect to the three-dimensional model; acquires one or more textures corresponding to the designated viewpoint on the basis of the plurality of textures; and generates a two-dimensional image of the three-dimensional model corresponding to the designated viewpoint on the basis of the one or more textures.

Description

Information processing device

[0001] The present disclosure relates to an information processing device.

[0002] There is a technology for mapping texture onto a three-dimensional model and rendering a two-dimensional image of the three-dimensional model. For example, in a virtual viewpoint image rendering device that renders a virtual viewpoint image based on images from multiple cameras with different viewpoints, a 3D model created based on the camera images is acquired, a virtual viewpoint is selected, and the texture of each camera image is sequentially mapped for each camera based on the virtual viewpoint and the 3D model.

[0003] Japanese Patent Application Laid-Open No. 2022-19073

[0004] According to the present disclosure, it is possible to provide a technology capable of generating highly realistic images.

[0005] An information processing device according to one aspect of the present disclosure has one or more memories and one or more processors, and the one or more processors acquire data representing the shape of a three-dimensional model and a plurality of textures obtained based on projections of the shapes of each of a plurality of images, accept a specification of a viewpoint for the three-dimensional model, acquire one or more textures corresponding to the specified viewpoint based on the plurality of textures, and generate a two-dimensional image of the three-dimensional model corresponding to the specified viewpoint based on the one or more textures.

[0006] FIG. 1 is a block diagram showing an example of the overall configuration of an image processing system according to a first embodiment; FIG. 2 is a diagram for explaining model data; FIG. 3 is a diagram for explaining blending processing; FIG. 4 is a flowchart showing an example of an image processing method; FIG. 5 is a block diagram showing an example of the overall configuration of an image processing system according to a second embodiment; FIG. 6 is a block diagram showing an example of the overall configuration of an image processing system according to a third embodiment; and FIG. 7 is a block diagram showing an example of the hardware configuration of a computer.

[0007] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0008] [First Embodiment] A first embodiment of the present disclosure is an example of an image processing system that generates a two-dimensional image of a three-dimensional model. The two-dimensional image of the three-dimensional model may be, for example, a free-viewpoint image. A free-viewpoint image is a two-dimensional image generated so that a user can view the three-dimensional model from any viewpoint. The image processing system according to this embodiment has a function of accepting a viewpoint specified by a user and displaying a free-viewpoint image corresponding to the specified viewpoint.

[0009] In the present disclosure, a "viewpoint" may include at least one of the position of the viewpoint (e.g., three-dimensional coordinates) or the line of sight of the viewpoint (e.g., three-dimensional orientation). Furthermore, "viewpoint information" may include at least one of information regarding the position of the viewpoint or information regarding the line of sight of the viewpoint. Note that a "free viewpoint image corresponding to a viewpoint" means that the free viewpoint image is generated so that the three-dimensional model can be viewed from that viewpoint.

[0010] <Overall Configuration of Image Processing System> The overall configuration of an image processing system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the overall configuration of an image processing system according to the first embodiment.

[0011] 1, the image processing system 1000 includes a server device 10 and a client device 20. The server device 10 and the client device 20 may be connected to each other so as to be able to communicate data with each other via a communication network such as a local area network (LAN) or the Internet.

[0012] The server device 10 is an example of an information processing device, such as a personal computer, workstation, or server, that manages model data. The model data is electronic data that includes information about a three-dimensional model. The three-dimensional model may be, for example, a three-dimensional CG (Computer Graphics) model of a real object. The real object and the three-dimensional model may correspond to each other.

[0013] The server device 10 stores one or more pieces of model data in advance. The one or more pieces of model data may include model data relating to a three-dimensional model that can be selected as a display target. The server device 10 may store the one or more pieces of model data in a storage device such as a hard disk drive (HDD) or a solid state disk (SSD). The model data may be generated by the server device 10 and stored in the storage device. The model data may be generated by the client device 20 or another information processing device and stored in the storage device.

[0014] In this embodiment, the server device 10 transmits model data to the client device 20 in response to a request from the client device 20. The request from the client device 20 may include information specifying a 3D model to be displayed. The server device 10 may read model data related to the specified 3D model from a storage device and transmit the model data to the client device 20.

[0015] 2 is a diagram illustrating model data. The model data includes (1) mesh data that represents the shape of a three-dimensional model, (2) a plurality of textures for the three-dimensional model, and (3) viewpoint information that indicates viewpoints corresponding to each of the plurality of textures. The server device 10 may transmit the model data to the client device 20 in a data-compressed state according to a predetermined standard. In this case, the client device 20 decompresses the model data received in a data-compressed state before processing it.

[0016] (1) The mesh data may be data indicating the shape of a real object corresponding to a three-dimensional model. For example, the mesh data may be obtained by measuring the shape of the real object using a three-dimensional scanner or the like. For example, the mesh data may be obtained by performing three-dimensional reconstruction based on multiple captured images obtained by capturing images of the real object from multiple different viewpoints. The timing at which the mesh data is obtained is not limited.

[0017] (2) The multiple textures may be image data generated based on multiple captured images of a real object corresponding to the three-dimensional model taken from multiple different viewpoints. Each image data may be image data captured by an imaging device such as an actual physical camera or a virtual camera in CG. Each texture may be generated by projecting the corresponding captured image onto the shape of the three-dimensional model and UV-developing the three-dimensional model onto which the captured image is mapped. The multiple textures are generated in the same number as the viewpoints from which the captured images were taken. The number of viewpoints may be set arbitrarily, but may be 270, for example.

[0018] Each of the multiple textures may be image data expressed in a UV coordinate system. Each of the multiple textures may be image data generated by projecting and mapping the captured image corresponding to the texture onto mesh data from the shooting position of the imaging device when the captured image was captured. For example, each of the multiple textures may be generated by preparing a texture map expressed in a UV coordinate system, in which each pixel is initialized with predetermined color information, for the number of captured images, and, for each captured image, associating each polygon constituting the mesh data onto which the captured image is projected with each portion of the captured image projected onto each polygon using UV coordinates, and copying the image of each portion of the captured image to the corresponding portion of the texture map. As an example, the predetermined color information may be color information in which all RGB values ​​are 0 (i.e., black, where R = G = B = 0).

[0019] Each texture expressed in the UV coordinate system may include a portion that is mapped to the mesh data and a portion that is not mapped. The portion that is mapped to the mesh data may include, for example, a portion derived from the captured image. The portion that is not mapped to the mesh data may include, for example, a portion that is not associated with any polygon that constitutes the mesh data, or a portion that is associated with a polygon that is hidden by another polygon and therefore not visible. In other words, each texture may include a portion (pixel) that is visible from the viewpoint corresponding to that texture (the viewpoint at which the captured image corresponding to that texture was captured) and a portion (pixel) that is not visible, and the hidden portion may be a portion that is not associated with the mesh data and therefore not visible, or a portion that is hidden by another polygon and therefore not visible.

[0020] Each texture may have information (visible portion distinction information) that distinguishes between portions that are mapped to the mesh data (portions that are visible from the viewpoint corresponding to each texture) and portions that are not mapped to the mesh data (portions that are not visible from the viewpoint corresponding to each texture). The portions that are not mapped to the mesh data may have the same color information (e.g., black, represented by R=G=B=0) as an example of the visible portion distinction information. Having the same color information for the unmapped portions increases the compression rate of the data size when the texture is compressed (encoded). As another example of the visible portion distinction information, information about the portions that are not mapped to the mesh data may be included in the model data as separate data, such as a binary mask image or a 256-value mask image, for each texture. The mask image may be the alpha channel of the texture.

[0021] For example, each of the plurality of textures may be compressed into compressed data according to a predetermined still image compression standard. For example, the mask image corresponding to each texture may be compressed into compressed data according to a predetermined still image compression standard. For example, the still image compression standard may be PNG (Portable Network Graphics) or the like. The PNG may be PNG with an alpha channel.

[0022] As another example, each of the multiple textures may be data-compressed into compressed data according to a predetermined video compression standard. A mask image corresponding to each texture may also be data-compressed into compressed data according to a predetermined video compression standard, along with each texture. The video compression standard may be, for example, H.264. When data compression is performed according to a video compression standard, the multiple textures may be data-compressed into a single data file as an image of one frame of consecutive frames according to the video compression standard. For example, when data compression is performed using H.264, the multiple textures are data-compressed into a single data file with an extension such as "mp4."

[0023] (3) The viewpoint information may be information about the position or orientation of an image capturing device that captured an image corresponding to each texture. The viewpoint information may also be information indicating the position or orientation from which a real object corresponding to a three-dimensional model is viewed. The viewpoint information may include, for example, a unit vector indicating the orientation of the image capturing device.

[0024] When a viewpoint for a three-dimensional model (i.e., a position or a direction from which the three-dimensional model is viewed) is specified, the viewpoint information may be referenced in order to select one or more textures corresponding to a position or a direction close to the viewpoint from a plurality of textures. For example, k textures corresponding to the top k (e.g., k=5) pieces of viewpoint information closest to the specified viewpoint may be selected from a plurality of textures.

[0025] The client device 20 is an example of an information processing terminal, such as a personal computer, a smartphone, or a tablet terminal, operated by a user of the image processing system 1000. The client device 20 displays a free-viewpoint image of a 3D model. The client device 20 may also display a free-viewpoint video of the 3D model. The free-viewpoint video of the 3D model is a video in which free-viewpoint images generated from arbitrary viewpoints are played back in time series.

[0026] In this embodiment, the client device 20 may request model data from the server device 10 in response to an operation by a user. The client device 20 may accept selection of a 3D model to be displayed. Upon accepting the selection of the 3D model, the client device 20 may transmit to the server device 10 a request to acquire model data related to the selected 3D model. The client device 20 may receive model data transmitted by the server device 10 in response to the request to acquire model data. The client device 20 may display a free-viewpoint image or a free-viewpoint video of the 3D model based on the model data received from the server device 10.

[0027] A 3D viewer for displaying a free-viewpoint image of a 3D model may be pre-installed in the client device 20. The client device 20 may use the 3D viewer to accept a viewpoint specified by a user. Upon accepting the viewpoint specified by the user, the client device 20 may display a free-viewpoint image corresponding to the specified viewpoint in a display area of ​​the 3D viewer. The 3D viewer may accept the viewpoint specification through a user's operation on a keyboard, pointing device, display, touch screen, or the like provided on the client device 20.

[0028] If multiple textures included in the model data received from the server device 10 are data compressed, the client device 20 may generate the multiple textures by decompressing (decoding) the compressed data. The client device 20 may cache the multiple textures received from the server device 10 or the multiple textures obtained by decompressing the compressed data received from the server device 10 in a memory or the like.

[0029] The client device 20 may select one or more textures corresponding to a default viewpoint from among the plurality of textures. The client device 20 may map the selected textures onto the mesh data. As an example, the client device 20 may map a texture expressed in a UV coordinate system by UV mapping onto the mesh data.

[0030] The client device 20 may generate a free-viewpoint image corresponding to the default viewpoint by rendering the texture-mapped mesh data based on the default viewpoint. The client device 20 may display the generated free-viewpoint image on a display. For example, the client device 20 may display the generated free-viewpoint image in a display area of ​​a 3D viewer displayed on the display.

[0031] The client device 20 may accept a designation of a new viewpoint for the three-dimensional model. For example, the client device 20 may accept a designation of a viewpoint from which a user views the three-dimensional model in a three-dimensional viewer. The designated viewpoint may include, for example, a unit vector indicating the orientation of a virtual imaging device toward the three-dimensional model.

[0032] When the client device 20 receives a designation of a viewpoint for the 3D model, the client device 20 may refer to the viewpoint information corresponding to each texture and select one or more textures corresponding to the top k viewpoint information that matches the designated viewpoint or is closest to the designated viewpoint. As a non-limiting example, a criterion for determining the proximity between the designated viewpoint and the viewpoint information corresponding to a texture may be the cosine similarity or cosine distance between a unit vector indicating the direction from the designated viewpoint toward the 3D model and a unit vector indicating the orientation of the imaging device corresponding to each texture. For example, the client device 20 may select the top k textures in descending order of cosine similarity from textures whose cosine similarity is 0 or greater.

[0033] The client device 20 may generate a free-viewpoint image corresponding to a specified viewpoint based on one or more selected textures. For example, the client device 20 may generate a single blended texture by blending multiple selected textures. The client device 20 may map the blended texture obtained by blending to mesh data. The client device 20 may generate a free-viewpoint image corresponding to the specified viewpoint by rendering the mesh data to which the blended texture is mapped, based on the specified viewpoint.

[0034] As an example, a method for blending textures may be a weighted average of multiple textures. When blending by weighted average, the client device 20 may calculate the weight of each texture based on viewpoint information corresponding to each texture and a specified viewpoint. In other words, the client device 20 may blend one or more textures expressed in a UV coordinate system with a weight according to the viewpoint. The client device 20 may also calculate the weight of each texture for each pixel based on whether or not the pixel in each texture is mapped to mesh data (is visible from the viewpoint corresponding to each texture). In other words, the client device 20 may blend one or more textures expressed in a UV coordinate system with a weight for each pixel according to the viewpoint and visible portion distinction information.

[0035] For example, the client device 20 may use, as a weight for each texture, a value obtained by using the sum of the cosine similarities calculated for the selected k textures as the denominator and the cosine similarity calculated for each texture as the numerator. If each texture is expressed in the UV coordinate system, a single blended texture expressed in the UV coordinate system is generated by blending the selected textures for each UV coordinate.

[0036] For example, the client device 20 may identify pixels to be mapped to the mesh data (pixels visible from the viewpoint corresponding to each texture) for each of the selected k textures based on the visible portion distinction information, and blend these identified mapped pixels (visible pixels) for each of the k textures by weighting them with the corresponding cosine similarity. If each texture is expressed in a UV coordinate system, the client device 20 may obtain color information for pixels at each UV coordinate in the blended texture expressed in the UV coordinate system as follows: The client device 20 may identify pixels to be mapped to the mesh data (visible pixels) from among pixels at the same UV coordinate in each texture based on the visible portion distinction information for each texture, weight-average the color information of the identified pixels with the corresponding cosine similarity, and obtain the color information obtained by the weighted average as the blended color information for the pixels at those UV coordinates. This configuration can suppress the influence of pixels that are invisible from the viewpoint corresponding to each texture when portions not mapped to the mesh data (invisible portions) have the same color information (e.g., black).

[0037] 3 is a diagram for explaining the blending process. As shown in FIG. 3, the client device 20 calculates the cosine similarity θ between the specified viewpoint and the orientation of the image capture device corresponding to each texture. １ , θ ２ , ..., and selects k textures with large cosine similarities. １ , θ ２ Weights w corresponding to , ... １ , w ２ , .... Then, the client device 20 calculates the weights w １ , w ２ , ... to calculate the blended texture. The client device 20 then UV-maps the blended texture onto the mesh data, and renders the mesh data onto which the blended texture has been mapped, thereby generating a free-viewpoint image corresponding to the specified viewpoint.

[0038] Another example of a texture blending method may be a weighted average using weights calculated for each pixel of multiple textures. Because textures are mapped to mesh data, the point (position) on the surface of the 3D model to which a given texture pixel corresponds may differ depending on the texture pixel. In other words, the line of sight from a specified viewpoint to the point on the surface of the 3D model to which that pixel corresponds may differ for each texture pixel. Taking this into consideration, a weighted average may be performed using different weights for each pixel of multiple textures.

[0039] When blending multiple textures using a weighted average with weights calculated on a pixel-by-pixel basis, the client device 20 may, for example, calculate a weight for each pixel of each of the multiple textures based on viewpoint information corresponding to each texture, a specified viewpoint, and a point on the surface of the three-dimensional model corresponding to each pixel of each texture.

[0040] For example, the client device 20 may first calculate, for each pixel in each texture with the same UV coordinates (i.e., each texture mapped to the same point on the surface of the three-dimensional model), the line of sight from the specified viewpoint to the point on the surface of the three-dimensional model corresponding to that pixel. Next, the client device 20 may calculate the cosine similarity of each texture with respect to that line of sight, and calculate a weight for each texture for that pixel. The client device 20 may then generate a blended texture by weighted averaging the textures using the weight calculated for each pixel.

[0041] The client device 20 may repeat the above process each time the user specifies a new viewpoint for the 3D model. The client device 20 may also display free-viewpoint images corresponding to the new viewpoints in chronological order. This allows the image processing system 1000 to provide the user with a free-viewpoint video of the 3D model.

[0042] Note that the overall configuration of the image processing system 1000 shown in FIG. 1 is one example, and various system configuration examples are possible depending on the application and purpose. The image processing system 1000 may be configured with one or more devices. The devices included in the image processing system 1000 may be a system configured with multiple devices. Each function included in the image processing system 1000 may be realized by any device that constitutes the system. Each component included in the image processing system 1000 may be included in any device that constitutes the system.

[0043] The image processing system 1000 may include a plurality of server devices 10 and one or more client devices 20. The server device 10 or the client device 20 may be realized as a system including a plurality of computers, or may be realized as a cloud computing service. The server device 10 and the client device 20 may be realized as standalone computers. The classification of devices such as the server device 10 and the client device 20 shown in FIG. 1 is one example.

[0044] <Flow of Image Processing Method> The image processing method executed by the image processing system 1000 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the image processing method.

[0045] In step S1, the client device 20 launches a 3D viewer in response to a user's operation. The client device 20 accepts a selection of a 3D model to be displayed in response to the user's operation of the 3D viewer. The client device 20 then transmits a request to the server device 10 to obtain model data for the selected 3D model.

[0046] In step S2, the server device 10 receives a request to acquire model data from the client device 20. The server device 10 reads model data related to the three-dimensional model indicated in the request to acquire model data from the storage device. The server device 10 transmits the read model data to the client device 20. When transmitting the model data, the server device 10 may compress the data of multiple textures included in the model data. Note that the data of the multiple textures may be compressed in advance. The client device 20 receives the model data from the server device 10.

[0047] In step S3, the client device 20 determines whether the multiple textures included in the model data are data-compressed. If the multiple textures are data-compressed (YES), the client device 20 proceeds to step S4. On the other hand, if the multiple textures are not data-compressed (NO), the client device 20 skips step S4 and proceeds to step S5.

[0048] In step S4, the client device 20 decompresses the compressed data in which a plurality of textures are compressed, and caches the plurality of textures obtained by decompressing the compressed file in a memory or the like.

[0049] In step S5, the client device 20 selects a texture corresponding to a default viewpoint from the plurality of textures. The default viewpoint may be, for example, predetermined in the 3D viewer, predetermined for each 3D model, or specified by the user when selecting a 3D model.

[0050] The client device 20 maps the selected texture to the mesh data. The client device 20 generates a free-viewpoint image corresponding to the default viewpoint by rendering the mesh data to which the texture has been mapped based on a default viewpoint. The client device 20 displays the generated free-viewpoint image on a display. The client device 20 may display the generated free-viewpoint image in a display area of ​​a 3D viewer.

[0051] In step S6, the client device 20 accepts a new viewpoint designation in response to a user's operation on the 3D viewer. The client device 20 selects the top k pieces of viewpoint information that match or are closest to the designated viewpoint from the viewpoint information corresponding to each texture.

[0052] In step S7, the client device 20 blends k textures corresponding to the viewpoint information selected based on the specified viewpoint. The client device 20 maps the blended texture obtained by blending to mesh data. The client device 20 generates a free-viewpoint image corresponding to the specified viewpoint by rendering the mesh data to which the blended texture is mapped based on the specified viewpoint. The client device 20 displays the generated free-viewpoint image on a display. The client device 20 may update the free-viewpoint image displayed in the display area of ​​the 3D viewer to the generated free-viewpoint image.

[0053] Thereafter, the client device 20 repeatedly accepts the designation of a new viewpoint and displays the free-viewpoint image corresponding to the designated viewpoint until the user stops displaying the free-viewpoint image in the 3D viewer, thereby enabling the client device 20 to display a free-viewpoint video of the 3D model on the display.

[0054] Second Embodiment In the first embodiment, a configuration was described in which the client device 20 receives model data from the server device 10, the client device 20 generates a free-viewpoint image corresponding to a viewpoint specified by a user based on the model data, and displays the generated free-viewpoint image on a 3D viewer. In the second embodiment, a configuration is described in which the client device 20 transmits to the server device 10 a request to generate a free-viewpoint image corresponding to a viewpoint specified by a user, the server device 10 generates a free-viewpoint image corresponding to the specified viewpoint based on the model data, transmits the generated free-viewpoint image to the client device 20, and the client device 20 displays the free-viewpoint image received from the server device 10 on a 3D viewer.

[0055] Here, the configuration according to the second embodiment will be described, focusing on the differences from the first embodiment. Unless otherwise specified, the same processing as in the first embodiment is performed.

[0056] <Overall Configuration of Image Processing System> The overall configuration of the image processing system according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the overall configuration of the image processing system according to the second embodiment.

[0057] The client device 20 transmits a specification of a 3D model to be displayed in the 3D viewer to the server device 10. The server device 10 maps a texture corresponding to a default viewpoint onto mesh data of the specified 3D model.

[0058] The server device 10 generates a free-viewpoint image corresponding to the default viewpoint by rendering the texture-mapped mesh data based on the default viewpoint. The server device 10 transmits the generated free-viewpoint image to the client device 20. The client device 20 displays the free-viewpoint image received from the server device 10 in the display area of ​​the 3D viewer.

[0059] The client device 20 accepts a designation of a new viewpoint in response to a user's operation on the 3D viewer, and transmits to the server device 10 a request to generate a free viewpoint image corresponding to the designated viewpoint.

[0060] When the server device 10 receives a request to generate a free viewpoint image, the server device 10 selects the top k pieces of viewpoint information that match the specified viewpoint or are closest to the specified viewpoint, based on the viewpoint included in the generation request. The server device 10 selects one or more textures that correspond to the viewpoint information selected based on the specified viewpoint from the multiple textures included in the model data.

[0061] Here, if multiple textures are compressed into one compressed data in accordance with a video compression standard and the server device 10 is capable of on-demand video streaming distribution, one or more textures corresponding to viewpoint information selected based on a specified viewpoint may be acquired from the compressed data. When the server device 10 acquires multiple textures, it blends the acquired textures and maps the blended texture obtained by blending to mesh data.

[0062] The server device 10 generates a free-viewpoint image corresponding to the specified viewpoint by rendering the texture-mapped mesh data based on the specified viewpoint. The server device 10 transmits the generated free-viewpoint image to the client device 20. The client device 20 displays the free-viewpoint image received from the server device 10 in the display area of ​​the 3D viewer.

[0063] Thereafter, the client device 20 repeatedly accepts the designation of a new viewpoint and displays the free-viewpoint image corresponding to the designated viewpoint until the user stops displaying the free-viewpoint image in the 3D viewer, thereby enabling the client device 20 to display a free-viewpoint video of the 3D model on the display.

[0064] Third Embodiment In the second embodiment, a configuration has been described in which server device 10 generates, based on model data, a free viewpoint image corresponding to a viewpoint specified by a user on client device 20. In the third embodiment, a configuration will be described in which server device 10 selects one or more textures from a plurality of textures included in the model data based on a viewpoint specified by a user on client device 20, and transmits the selected textures to client device 20.

[0065] Here, the configuration of the third embodiment will be described, focusing on the differences from the first and second embodiments. Unless otherwise specified, the same processing as in the first or second embodiment is performed.

[0066] <Overall Configuration of Image Processing System> The overall configuration of the image processing system according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing an example of the overall configuration of the image processing system according to the third embodiment.

[0067] The client device 20 accepts a designation of a new viewpoint in response to a user's operation on the 3D viewer, and transmits to the server device 10 a request to acquire a texture corresponding to the designated viewpoint.

[0068] When the server device 10 receives a texture acquisition request, it selects the top k pieces of viewpoint information that match or are closest to the specified viewpoint based on the viewpoint included in the acquisition request. The server device 10 selects one or more textures that correspond to the viewpoint information selected based on the specified viewpoint from multiple textures included in the model data. If the server device 10 acquires multiple textures, it blends the acquired textures. The server device 10 transmits the texture selected based on the specified viewpoint or the blended texture obtained by blending to the client device 20.

[0069] The client device 20 maps the texture or blended texture received from the server device 10 onto mesh data. The timing at which the client device 20 acquires the mesh data may be arbitrary. For example, the client device 20 may acquire the mesh data before receiving the texture, when receiving the texture, or after receiving the texture.

[0070] The client device 20 generates a free-viewpoint image corresponding to the specified viewpoint by rendering the texture-mapped mesh data based on the specified viewpoint, and displays the generated free-viewpoint image in the display area of ​​the 3D viewer.

[0071] In the first to third embodiments, each texture is expressed in a UV coordinate system, and texture-mapped mesh data is generated by UV mapping the mesh data. In the fourth embodiment, a captured image is used as a texture as is.

[0072] In this embodiment, the mesh data and viewpoint information included in the model data are the same as those in the first to third embodiments. On the other hand, the multiple textures included in the model data are multiple captured images taken by an imaging device from multiple different positions.

[0073] When textures expressed in a UV coordinate system have the same UV coordinates, the images of each texture are mapped to the same polygon, so when blending multiple textures, it is sufficient to perform blending of the multiple textures for each UV coordinate. On the other hand, in this embodiment, since each texture is not expressed in a UV coordinate system, when blending multiple textures, the correspondence between the texture portion and the polygon is identified for each texture to be blended, and blending is performed between the portions of the texture that are mapped to the same polygon.

[0074] For example, if you want to generate texture C corresponding to a new viewpoint C located between viewpoints A and B by blending texture A corresponding to viewpoint A and texture B corresponding to viewpoint B, you can perform the following process.

[0075] Texture A is projectively mapped from viewpoint A onto the mesh data to generate texture A' expressed in a UV coordinate system. Similarly, texture B is projectively mapped from viewpoint B onto the mesh data to generate texture B' expressed in a UV coordinate system. Next, textures A' and B' are blended (for example, a weighted average is calculated) with weights according to the proximity between viewpoint A and viewpoint C (for example, cosine similarity or cosine distance) and the proximity between viewpoint B and viewpoint C (for example, cosine similarity or cosine distance). At this time, as in the above-described embodiment, blending may be performed based on the portions of textures A' and B' that are mapped onto the mesh data (portions visible from the corresponding viewpoints). This generates texture C expressed in a UV coordinate system that corresponds to a new viewpoint C. Texture C is UV-mapped onto the mesh data, and the mesh data is rendered based on the new viewpoint C to generate a free viewpoint image that corresponds to the new viewpoint C.

[0076] In this embodiment, the plurality of textures may also be data-compressed in accordance with a still image compression standard or a video compression standard.

[0077] Fifth Embodiment In the first to third embodiments, a configuration was described in which a plurality of textures were generated based on a plurality of captured images of a real object corresponding to a three-dimensional model taken from a plurality of different viewpoints. In the fourth embodiment, a configuration was described in which a plurality of textures were generated based on a plurality of captured images of a real object corresponding to a three-dimensional model taken from a plurality of different viewpoints.

[0078] In the fifth embodiment, a configuration will be described in which each pixel included in a plurality of textures is approximated by a base of a plurality of spherical functions. The spherical function may be, for example, a spherical Gaussian function. However, the spherical Gaussian function is merely an example, and any function may be used.

[0079] The number of bases of the spherical function may be smaller than that of the captured images according to the first to fourth embodiments. As an example, the first to fourth embodiments use captured images of a real object captured from 270 viewpoints, but in this embodiment, approximation is performed using 12 or 20 bases. In this embodiment, it is assumed that a sphere is approximated by a regular dodecahedron or a regular icosahedron, and approximation is performed using 12 or 20 bases, but the number of bases is just an example and may be any number.

[0080] When a spherical Gaussian function is used as the basis, the texture included in the model data may be calculated as follows: First, the spherical Gaussian function is defined by the following equation.

[0081]

[0082] where x is a unit vector indicating the direction of the viewpoint, p is a unit vector indicating the axis of the spherical Gaussian function, and λ is a scalar value indicating the spread of the spherical Gaussian function.

[0083] p and λ may be set to fixed values ​​in advance. p and λ may also be optimized. When p and λ are set to fixed values, fixed p and λ may be shared in advance between the server device 10 and the client device 20. When p and λ are not shared between the server device 10 and the client device 20, p and λ may be transmitted and received between the server device 10 and the client device 20 at any timing.

[0084] In this embodiment, M (=270) textures are converted into N (=12 or 20) textures. Each of the N textures is converted into N spherical Gaussian functions G １ (x;p １ , λ １ ), G ２ (x;p ２ , λ ２ ), ..., G Ｎ (x;p Ｎ , λ Ｎ ) Coefficient c for each １＿ｉｊ , c ２＿ｉｊ , ..., c Ｎ＿ｉｊwhere i is an integer between 1 and h, j is an integer between 1 and w, h is the height of the texture, and w is the width of the texture.

[0085] In this embodiment, each pixel of the texture is expressed by an RGB value. Therefore, the nth (1≦n≦N) spherical Gaussian function G ｎ (x;p ｎ , λ ｎ ) the pixel value c of the texture pixel ij corresponding to ｎ＿ｉｊ is the R value c_red ｎ＿ｉｊ , G value c_green ｎ＿ｉｊ , B value c_bleu ｎ＿ｉｊ Includes:

[0086] Texture pixel value c ｎ＿ｉｊ can be calculated by, for example, analytically, approximately, or pseudo-solving the simultaneous equations (2) with respect to c. ｎ＿ｉｊ Here is an example of calculating the G value c_green. ｎ＿ｉｊ and B value c_bleu ｎ＿ｉｊ can be calculated in the same way.

[0087]

[0088] where y_red ｎ＿ｉｊ is the R value of pixel ij in the image captured from the nth viewpoint. ｎ＿ｉｊ or B value c_bleu ｎ＿ｉｊ When calculating the G value y_green of pixel ij in the captured image, ｎ＿ｉｊ or B value y_blue ｎ＿ｉｊ The equation (2) can be solved analytically, approximately, or pseudo-analytically using the above equation.

[0089] Texture pixel value c ｎ＿ｉｊ As another example, it may be calculated using equation (3): Equation (3) is an equation that expresses the difference between the right and left sides of equation (2) using matrix calculation.

[0090]

[0091] The equations (2) and (3) are merely examples, and the pixel values ​​of the texture may be calculated by various calculations. Here, the equation (3) is expressed as y=(y_red １＿ｉｊ ,...,y_red Ｍ＿ｉｊ ), G = (G(x １ ;p １ , λ １ ), ..., G(x Ｍ ;p Ｎ , λ Ｎ )), c=(c_red １＿ｉｊ ,...,c_red Ｎ＿ｉｊ ) and express it as y-Gc. As an example, it is possible to calculate c that minimizes the norm of y-Gc, ∥y-Gc∥. As another example, it is possible to calculate the square norm of y-Gc, ∥y-Gc∥ ２ Based on the least squares method, c that minimizes －１ As another example, the square norm of y-Gc, ||y-Gc|| ２ The c that minimizes may be calculated based on gradient descent.

[0092] However, the pixel value c of the nth texture obtained by the above method ｎ＿ｉｊ is the corresponding n-th basis (for example, the spherical Gaussian function G ｎ ) are the coefficients of the coefficients of the matrix. That is, by using equation (2) or equation (3), N textures (hereinafter referred to as coefficient textures) having the coefficients of the matrix as pixel values ​​can be obtained. Since the coefficients of the matrix can take all real numbers, each pixel value c of each coefficient texture is calculated for each coefficient texture so that it can be treated as a texture having RGB values ​​as pixel values ​​(hereinafter referred to as RGB texture). ｎ＿ｉｊ may be normalized to an integer value between 0 and 255. For example, each pixel value c of the n-th coefficient texture ｎ＿ｉｊ The value (pixel value) obtained by normalizing is c' ｎ＿ｉｊ Then, each pixel value c' of the n-th RGB texture after normalization ｎ＿ｉｊ is c' ｎ＿ｉｊ = int(255×(c ｎ＿ｉｊThe coefficient texture density may be calculated as follows: (-minimum pixel value of the nth coefficient texture) / (maximum pixel value of the nth coefficient texture-minimum pixel value of the nth coefficient texture). int(•) is a function that rounds down the decimal portion of the number • in parentheses to an integer. In this manner, N RGB textures may be generated based on N coefficient textures. When transmitting N RGB textures from the server device 10 to the client device 20, the N coefficient textures must be restored in the client device 20. Therefore, the server device 10 transmits the maximum and minimum pixel values ​​in each coefficient texture along with the N RGB textures to the client device 20.

[0093] In this embodiment, when generating a texture corresponding to a specified viewpoint, all N bases G may be used, or only k bases closest to the specified viewpoint may be used to generate the texture. When only k bases are used, only the bases G selected in equation (2) or equation (3) may be used, and the unselected bases G may be set to 0 for calculation.

[0094] As shown in equation (2) or (3), captured images captured from M viewpoints can be replaced with textures expressed by coefficients c of base G of N spherical functions, which is less than the number (M) of captured images. Therefore, according to this embodiment, the model data only needs to include fewer textures than in the first to fourth embodiments. According to this embodiment, for example, the amount of communication between the server device 10 and the client device 20 can be reduced.

[0095] Other Embodiments The above-described embodiments may be applied to augmented reality (AR). Specifically, the client device 20 may superimpose a free-viewpoint image of a 3D model received from the server device 10 on an image (still image or video) captured in real space.

[0096] In particular, the client device 20 may superimpose a free-viewpoint image of the 3D model on video captured in real time by the client device's own imaging device. In this case, as an example, the client device 20 may use its own sensor to determine its own position and orientation and accept in real time a designation of a viewpoint for the 3D model based on the position and orientation of the client device. As another example, the client device 20 may analyze the captured video to extract multiple feature points and accept in real time a designation of a viewpoint for the 3D model based on the positional relationship of the multiple feature points. By changing the position or orientation of the client device 20 while capturing the real space, the user of the client device 20 can view the video as if the 3D model were present in the real space from any viewpoint.

[0097] <Summary> As is clear from the above explanation, the client device 20 according to one embodiment of the present disclosure acquires data representing the shape of a three-dimensional model and a plurality of textures obtained based on projections onto the shapes of each of a plurality of images, accepts a designation of a viewpoint for the three-dimensional model, acquires one or more textures corresponding to the designated viewpoint based on the plurality of textures, and generates a two-dimensional image of the three-dimensional model corresponding to the designated viewpoint based on the one or more textures.

[0098] The multiple textures may include textures corresponding to multiple viewpoints. The client device 20 may select one or more textures corresponding to a specified viewpoint from the multiple textures. The client device 20 may perform texture mapping on the shape of the three-dimensional model based on the one or more textures and data representing the shape of the three-dimensional model. The client device 20 may generate a two-dimensional image by rendering the texture-mapped three-dimensional model based on the specified viewpoint.

[0099] The multiple textures may be image data expressed in a UV coordinate system. Portions of the multiple textures that are not texture-mapped to the shape of the three-dimensional model may have the same color information. Portions of the multiple textures that are not visible from a viewpoint corresponding to each texture may have the same color information. The multiple textures may have information, as a mask image for each texture, that distinguishes between portions that are visible and portions that are not visible from a viewpoint corresponding to each texture.

[0100] The client device 20 may receive multiple data-compressed textures expressed in a UV coordinate system from another information processing device via a communications network. The client device 20 may decompress the multiple data-compressed textures. The client device 20 may acquire one or more textures based on the multiple decompressed textures. The multiple textures may be data-compressed according to a predetermined video compression standard. Each of the multiple textures may be data-compressed as an image of one frame of consecutive frames in the video compression standard.

[0101] The data representing the shape of the three-dimensional model may be data representing the shape of a real object corresponding to the three-dimensional model. The multiple textures may be data generated based on multiple captured images obtained by photographing the real object from multiple different positions. The client device 20 may acquire viewpoint information corresponding to each texture. The viewpoint information may be information regarding the position or orientation of an imaging device that captured each of the multiple captured images.

[0102] The client device 20 may blend one or more textures expressed in a UV coordinate system with weights according to a specified viewpoint. The client device 20 may texture-map the blended texture obtained by blending onto the shape of a three-dimensional model based on data representing the shape of the three-dimensional model. The client device 20 may generate a two-dimensional image by rendering the texture-mapped three-dimensional model based on a specified viewpoint.

[0103] The client device 20 may identify, for each of one or more textures expressed in a UV coordinate system, a portion that is visible from a corresponding viewpoint. The client device 20 may blend the identified visible portions of each of the one or more textures with a weight according to the specified viewpoint. The client device 20 may texture-map the blended texture obtained by blending onto the shape of the three-dimensional model based on data representing the shape of the three-dimensional model. The client device 20 may generate a two-dimensional image by rendering the texture-mapped three-dimensional model based on the specified viewpoint.

[0104] The multiple textures may include textures that are obtained by approximating textures corresponding to each of the multiple viewpoints using a predetermined number of function bases that is less than the number of viewpoints, and the number of textures may be the same as the number of bases. The client device 20 may generate one or more textures corresponding to the specified viewpoint based on the same number of textures as the number of bases. The client device 20 may select one or more bases that correspond to the specified viewpoint from the predetermined number of function bases. The client device 20 may generate one or more textures corresponding to the specified viewpoint based on the selected one or more bases. The functions may be spherical functions.

[0105] The client device 20 may superimpose a two-dimensional image on an image captured of the real space and display it. The client device 20 may accept a designation of a viewpoint based on at least one of the position and the orientation of the client device 20.

[0106] As a result, according to one embodiment of the present disclosure, it is possible to provide a device capable of generating highly realistic images. In one aspect, according to one embodiment, it is not necessary to calculate colors according to a specified viewpoint, so that highly realistic images can be rendered with a small amount of calculation. In another aspect, according to one embodiment, it is possible to transfer textures corresponding to multiple viewpoints with a small amount of data.

[0107] [Hardware Configuration of Information Processing Device] Some or all of the devices (server device 10 and client device 20) in the above-described embodiments may be configured with hardware, or may be configured with software (program) information processing executed by a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or the like. When configured with software information processing, software that realizes at least some of the functions of each device in the above-described embodiments may be stored on a non-transitory storage medium (non-transitory computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or USB (Universal Serial Bus) memory, and the software information processing may be executed by loading the software into a computer. The software may also be downloaded via a communications network. Furthermore, all or part of the software processing may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), thereby allowing the software information processing to be executed by hardware.

[0108] The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed medium such as a hard disk, memory, etc. The storage medium may be provided inside the computer (such as a main storage device or auxiliary storage device) or outside the computer.

[0109] 7 is a block diagram showing an example of the hardware configuration of each device (the server device 10 and the client device 20) in the above-described embodiment. Each device may be realized as a computer 7 including, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, which are connected via a bus 76.

[0110] Although the computer 7 in FIG. 7 includes one of each component, it may also include multiple of the same component. Also, while FIG. 7 shows one computer 7, the software may be installed on multiple computers, with each of the multiple computers executing the same or different parts of the software. In this case, a distributed computing configuration may be used in which each computer communicates via a network interface 74 or the like to execute the processing. In other words, each device (server device 10 and client device 20) in the above-described embodiment may be configured as a system in which one or more computers execute instructions stored in one or more storage devices to realize its functions. Furthermore, information transmitted from a terminal may be processed by one or more computers provided on a cloud, and the processing results may be transmitted to the terminal.

[0111] The various calculations of each device (server device 10 and client device 20) in the above-described embodiments may be executed in parallel using one or more processors, or using multiple computers via a network. Furthermore, the various calculations may be distributed to multiple processor cores within a processor and executed in parallel. Furthermore, some or all of the processes, means, etc. disclosed herein may be implemented by at least one of a processor and a storage device provided on a cloud that can communicate with computer 7 via a network. Thus, each device in the above-described embodiments may be implemented in the form of parallel computing using one or more computers.

[0112] The processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that at least controls or performs calculations on a computer. The processor 71 may be a general-purpose processor, a dedicated processing circuit designed to perform a specific calculation, or a semiconductor device that includes both a general-purpose processor and a dedicated processing circuit. The processor 71 may also include an optical circuit or a calculation function based on quantum computing.

[0113] The processor 71 may perform arithmetic processing based on data or software input from each device or the like configured internally of the computer 7, and may output arithmetic results or control signals to each device or the like. The processor 71 may control each component constituting the computer 7 by executing the OS (Operating System) of the computer 7, applications, etc.

[0114] Each device (server device 10 and client device 20) in the above-described embodiment may be realized by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other via wire or wirelessly.

[0115] The main memory device 72 may store instructions executed by the processor 71, various data, etc., and information stored in the main memory device 72 may be read by the processor 71. The auxiliary memory device 73 is a memory device other than the main memory device 72. Note that these memory devices refer to any electronic component capable of storing electronic information and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory devices for saving various data, etc. in each device (server device 10 and client device 20) in the above-described embodiments may be realized by the main memory device 72 or the auxiliary memory device 73, or may be realized by internal memory built into the processor 71. For example, each memory unit in the above-described embodiments may be realized by the main memory device 72 or the auxiliary memory device 73.

[0116] When each device (server device 10 and client device 20) in the above-described embodiment is configured with at least one storage device (memory) and at least one processor connected (coupled) to this at least one storage device, at least one processor may be connected to one storage device. Also, at least one storage device may be connected to one processor. Also, a configuration in which at least one processor among multiple processors is connected to at least one storage device among multiple storage devices may be included. Also, this configuration may be realized by storage devices and processors included in multiple computers. Furthermore, a configuration in which a storage device is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

[0117] The network interface 74 is an interface for connecting to the communication network 8 wirelessly or via a wire. The network interface 74 may be an appropriate interface, such as one that conforms to an existing communication standard. The network interface 74 may exchange information with an external device 9A connected via the communication network 8. The communication network 8 may be any one of a wide area network (WAN), a local area network (LAN), a personal area network (PAN), etc., or a combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, an example of a LAN is IEEE 802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.

[0118] The device interface 75 is an interface such as a USB that directly connects to the external device 9B.

[0119] The external device 9A is a device connected to the computer 7 via a network, and the external device 9B is a device connected directly to the computer 7.

[0120] The external device 9A or the external device 9B may be, for example, an input device. The input device is, for example, a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, a touch panel, or the like, and provides acquired information to the computer 7. Alternatively, the external device 9A or the external device 9B may be a device including an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0121] Furthermore, the external device 9A or the external device 9B may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker that outputs sound or the like. Alternatively, the external device 9A or the external device 9B may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0122] The external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage device such as an HDD.

[0123] Furthermore, the external device 9A or the external device 9B may be a device having some of the functions of the components of each device (the server device 10 and the client device 20) in the above-described embodiment. That is, the computer 7 may transmit some or all of the processing results to the external device 9A or the external device 9B, or may receive some or all of the processing results from the external device 9A or the external device 9B.

[0124] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.

[0125] In this specification (including claims), when expressions such as "using / using data as input / based on / according to / in response to data" (including similar expressions) are used, unless otherwise specified, this includes cases where the data itself is used, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.) is used. Furthermore, when a statement is made that a result is obtained "using data as input / based on / according to / in response to data" (including similar expressions), this includes cases where the result is obtained based solely on the data, or where the result is influenced by other data, factors, conditions, and / or states other than the data. Furthermore, when a statement is made that "data is output" (including similar expressions), this includes cases where the data itself is used as output, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of various data, etc.) is used as output, unless otherwise specified.

[0126] When the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that include any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately according to the context in which they are used, but any connection / coupling form that is not intentionally or naturally excluded should be interpreted as being included in these terms without limitation.

[0127] In this specification (including the claims), the expression "A configured to B" may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, if element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0128] Whenever words implying containing or possessing (e.g., "comprising / including," "having," etc.) are used in this specification (including the claims), they are intended to be open-ended terms that include containing or possessing things other than the object designated by the object of the term. When the object of such words implying containing or possessing does not specify a quantity or suggests a singular number (e.g., expressions using the articles "a" or "an"), the expression should be construed as not being limited to a specific number.

[0129] In this specification (including the claims), although expressions such as "one or more" and "at least one" are used in some places and expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0130] In this specification, when a particular advantage / result is described as being obtained with respect to a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained with one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or situations, and that the effect is not necessarily obtained with the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or situations are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same configuration or a similar configuration.

[0131] In this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" (including similar expressions) is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may include an electronic circuit, a device including an electronic circuit, etc.

[0132] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. Also, a configuration in which only some of the multiple storage devices store data may be included.

[0133] In this specification (including the claims), terms such as "first," "second," etc. are used merely as a way of distinguishing between two or more elements, and are not necessarily intended to impose technical meanings such as temporal aspect, spatial aspect, sequence, quantity, etc. on the subject. Thus, for example, a reference to a first element and a second element does not necessarily mean that only two elements may be employed therein, that the first element must precede the second element, that the first element must be present in order for the second element to be present, etc.

[0134] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, partial deletions, etc. are possible within the scope of the conceptual idea and spirit of the present invention, which is derived from the content defined in the claims and their equivalents. For example, when numerical values ​​or formulas are used in the above-described embodiments, they are shown for illustrative purposes and do not limit the scope of the present disclosure. Furthermore, the order of each operation shown in the embodiments is also illustrative and does not limit the scope of the present disclosure.

[0135] The disclosed technology may take the following forms as described below.

[0136] (Supplementary Note 1) An information processing device having one or more memories and one or more processors, wherein the one or more processors acquire data representing the shape of a three-dimensional model and a plurality of textures obtained based on the projection of each of a plurality of images onto the shape, accept a designation of a viewpoint for the three-dimensional model, acquire one or more textures corresponding to the designated viewpoint based on the plurality of textures, and generate a two-dimensional image of the three-dimensional model corresponding to the designated viewpoint based on the one or more textures.

[0137] (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the plurality of textures include textures corresponding to a plurality of viewpoints, and the one or more processors select the one or more textures corresponding to the specified viewpoint from the plurality of textures.

[0138] (Supplementary Note 3) The information processing device according to Supplementary Note 2, wherein the one or more processors perform texture mapping on a shape of the three-dimensional model based on the one or more textures and the data, and generate the two-dimensional image by rendering the three-dimensional model on which the texture mapping has been performed based on the specified viewpoint.

[0139] (Supplementary Note 4) The information processing device according to any one of Supplementary Notes 1 to 3, wherein the plurality of textures are image data expressed in a UV coordinate system.

[0140] (Supplementary Note 5) The information processing device according to Supplementary Note 4, wherein portions of the plurality of textures that are not texture-mapped to the shape of the three-dimensional model have the same color information.

[0141] (Supplementary Note 6) The information processing device according to Supplementary Note 4, wherein portions of the plurality of textures that are not visible from a corresponding viewpoint for each texture have the same color information.

[0142] (Supplementary Note 7) The information processing device according to Supplementary Note 4, wherein the plurality of textures have information for distinguishing between a portion visible from a viewpoint corresponding to each texture and a portion invisible from the viewpoint, as a mask image for each texture.

[0143] (Supplementary Note 8) The information processing device described in any one of Supplementary Notes 4 to 7, wherein the one or more processors receive the plurality of data-compressed textures expressed in a UV coordinate system from another information processing device via a communication network, decompress the plurality of data-compressed textures, and obtain the one or more textures based on the decompressed plurality of textures.

[0144] (Supplementary Note 9) The information processing device according to Supplementary Note 8, wherein the plurality of textures are data-compressed in accordance with a predetermined video compression standard.

[0145] (Supplementary Note 10) The information processing device according to Supplementary Note 9, wherein each of the plurality of textures is data-compressed as an image of one frame of consecutive frames in accordance with the video compression standard.

[0146] (Supplementary Note 11) The information processing device according to any one of Supplementary Notes 1 to 10, wherein the data is data of a shape of a real object corresponding to the three-dimensional model, and the plurality of textures is data generated based on a plurality of captured images obtained by photographing the real object from a plurality of different positions.

[0147] (Supplementary Note 12) The information processing device according to Supplementary Note 11, wherein the one or more processors acquire viewpoint information corresponding to each of the textures, and the viewpoint information is information relating to a position or orientation of an imaging device that captured each of the plurality of captured images.

[0148] (Supplementary Note 13) The information processing device described in any one of Supplementary Notes 1 to 12, wherein the one or more processors blend the one or more textures expressed in a UV coordinate system with weights according to the specified viewpoint, texture-map the blended texture obtained by the blending onto the shape of the three-dimensional model based on the data, and generate the two-dimensional image by rendering the three-dimensional model on which the texture mapping has been performed based on the specified viewpoint.

[0149] (Supplementary Note 14) The information processing device described in any of Supplementary Notes 1 to 13, wherein the one or more processors: identify a portion of each of the one or more textures expressed in a UV coordinate system that is visible from a corresponding viewpoint for each texture; blend the identified visible portions of each of the one or more textures with a weight according to the specified viewpoint; texture-map the blended texture obtained by the blending onto the shape of the three-dimensional model based on the data; and generate the two-dimensional image by rendering the three-dimensional model on which the texture mapping has been performed based on the specified viewpoint.

[0150] (Supplementary Note 15) The information processing device described in any one of Supplementary Notes 1 to 14, wherein the plurality of textures include textures obtained by approximating textures corresponding to each of a plurality of viewpoints using a basis of a predetermined number of functions that is less than the plurality of viewpoints, the number of textures being equal to the number of bases, and the one or more processors generate the one or more textures corresponding to the specified viewpoint based on the same number of textures as the number of bases.

[0151] (Supplementary Note 16) The information processing device according to Supplementary Note 15, wherein the one or more processors select one or more bases corresponding to the specified viewpoint from the bases of the predetermined number of functions, and generate the one or more textures corresponding to the specified viewpoint based on the one or more selected bases.

[0152] (Supplementary Note 17) The information processing device according to Supplementary Note 15 or 16, wherein the function is a spherical function.

[0153] (Supplementary Note 18) The information processing device according to any one of Supplementary Notes 1 to 17, wherein the one or more processors superimpose and display the two-dimensional image on an image captured of a real space.

[0154] (Supplementary Note 19) The information processing device according to Supplementary Note 18, wherein the one or more processors accept the designation of the viewpoint based on at least one of a position or an orientation of the information processing device.

[0155] This application claims priority to Provisional Application No. 63 / 582,025, filed September 12, 2023, in the United States Patent and Trademark Office, the entire contents of which are incorporated herein by reference.

[0156] 10: Server device 20: Client device 1000: Image processing system

Claims

1. An information processing device having one or more memories and one or more processors, wherein the one or more processors acquire data representing the shape of a three-dimensional model and a plurality of textures obtained based on the projection of each of a plurality of images onto the shape, accept a designation of a viewpoint for the three-dimensional model, acquire one or more textures corresponding to the designated viewpoint based on the plurality of textures, and generate a two-dimensional image of the three-dimensional model corresponding to the designated viewpoint based on the one or more textures.

2. The information processing device according to claim 1, wherein the plurality of textures includes textures corresponding to a plurality of viewpoints, and the one or more processors select the one or more textures corresponding to the specified viewpoint from the plurality of textures.

3. The information processing device according to claim 2, wherein the one or more processors perform texture mapping on the shape of the three-dimensional model based on the one or more textures and the data, and generate the two-dimensional image by rendering the three-dimensional model on which the texture mapping has been performed based on the specified viewpoint.

4. The information processing device according to claim 1, wherein the plurality of textures are image data expressed in a UV coordinate system.

5. The information processing device according to claim 4, wherein portions of said plurality of textures that are not texture-mapped onto the shape of said three-dimensional model have the same color information.

6. The information processing device according to claim 4, wherein the portions of the plurality of textures that are not visible from the corresponding viewpoint for each texture have the same color information.

7. The information processing device according to claim 4, wherein the plurality of textures have information for distinguishing between parts that are visible and parts that are not visible from a viewpoint corresponding to each texture, as a mask image for each texture.

8. The information processing device according to claim 4, wherein the one or more processors receive the plurality of data-compressed textures expressed in a UV coordinate system from another information processing device via a communications network, decompress the plurality of data-compressed textures, and obtain the one or more textures based on the plurality of decompressed textures.

9. The information processing device according to claim 8, wherein the plurality of textures are data-compressed in accordance with a predetermined video compression standard.

10. The information processing device according to claim 9, wherein each of the plurality of textures is data-compressed as an image of one frame of successive frames according to the video compression standard.

11. An information processing device according to any one of claims 1 to 10, wherein the data is data on the shape of a real object corresponding to the three-dimensional model, and the multiple textures are data generated based on multiple captured images obtained by photographing the real object from multiple different positions.

12. The information processing device according to claim 11, wherein the one or more processors acquire viewpoint information corresponding to each of the textures, the viewpoint information being information relating to a position or orientation of an imaging device that captured each of the plurality of captured images.

13. An information processing device according to any one of claims 1 to 10, wherein the one or more processors blend the one or more textures expressed in a UV coordinate system with weighting according to the specified viewpoint, texture-map the blended texture obtained by the blending onto the shape of the three-dimensional model based on the data, and generate the two-dimensional image by rendering the three-dimensional model on which the texture mapping has been performed based on the specified viewpoint.

14. An information processing device as described in any one of claims 1 to 10, wherein the one or more processors: identify a portion of each of the one or more textures expressed in a UV coordinate system that is visible from a corresponding viewpoint for each texture; blend the identified visible portions of each of the one or more textures with a weighting corresponding to the specified viewpoint; texture-map the blended texture obtained by the blending onto the shape of the three-dimensional model based on the data; and generate the two-dimensional image by rendering the three-dimensional model on which the texture mapping has been performed based on the specified viewpoint.

15. An information processing device according to any one of claims 1 to 10, wherein the multiple textures include textures that are obtained by approximating textures corresponding to each of a plurality of viewpoints using a basis of a predetermined number of functions that is less than the plurality of viewpoints, the number of textures being equal to the number of bases, and the one or more processors generate the one or more textures corresponding to the specified viewpoint based on the same number of textures as the number of bases.

16. The information processing device according to claim 15, wherein the one or more processors select one or more bases corresponding to the specified viewpoint from the bases of the predetermined number of functions, and generate the one or more textures corresponding to the specified viewpoint based on the one or more selected bases.

17. The information processing device according to claim 15, wherein the function is a spherical function.

18. An information processing device according to any one of claims 1 to 10, wherein the one or more processors superimpose the two-dimensional image on an image captured of real space.

19. The information processing device according to claim 18, wherein the one or more processors accept the designation of the viewpoint based on at least one of a position or an orientation of the information processing device.

Images