Information processing system, information processing method, and information processing program

Abstract

An information processing system according to an embodiment of the present disclosure comprises: an acquisition unit that acquires first input information for adding information pertaining to a 3D object to be generated and second input information indicating the shape of the 3D object to be generated; and a generation unit that generates a plurality of texture maps corresponding to physical base rendering of the 3D object to be generated on the basis of the first input information and the second input information.

Description

Information processing system, information processing method, and information processing program

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program.

[0002] Techniques for generating textures for 3D models are known for creating 3D models, such as a technique for generating high-dimensional textures and textures with special effects by combining multiple texture materials (see, for example, Patent Document 1).

[0003] Also known are, for example, a technique for generating high-dimensional textures by sampling texture images from a group of texture images according to the characteristics of the material (e.g., Patent Document 2), and a technique for generating physically consistent textures by reflecting spectral reflectance from images captured from multiple viewpoints (e.g., Patent Document 3).

[0004] JP 2008-186485 A JP 2006-031561 A JP 2022-035153 A

[0005] Dave Zhenyu Chen et. al., “Text2Tex: Text-driven Texture Synthesis via Diffusion Models”, [online], [Retrieved May 13, 2020], 2023, pp. 18558-18568, https: / / arxiv.org / pdf / 2303.11396Elad Richardson et. al. “TEXTure: Text-Guided Texturing of 3D Shapes”, [online], [Retrieved May 13, 2020], 2023, pp. 1-12, https: / / arxiv.org / pdf / 2302.01721

[0006] However, with conventional techniques, it is difficult to efficiently generate high-precision textures. For example, with conventional techniques, multiple images or materials corresponding to the texture to be generated are required each time a texture is generated, which takes a long time to generate the texture. Furthermore, for example, a high-precision 3D model is a 3D model with physical consistency, and therefore requires a physically consistent texture. However, a physically consistent texture is not necessarily limited to a texture that reflects spectral reflectance, and multiple physically consistent textures are required. Therefore, it may not be possible to efficiently generate high-precision textures.

[0007] Therefore, the present disclosure proposes an information processing system, an information processing method, and an information processing program that can efficiently generate high-precision textures.

[0008] In order to solve the above problem, an information processing system of one embodiment according to the present disclosure includes: an acquisition unit that acquires first input information for adding information about a 3D object to be generated and second input information that indicates a shape of the 3D object to be generated; and a generation unit that generates a plurality of texture maps that correspond to physically based rendering of the 3D object to be generated based on the first input information and the second input information.

[0009] 1 is a diagram illustrating an overview of an information processing system according to a first embodiment. FIG. 2 is a diagram illustrating an example of U-Net processing in the information processing system according to the first embodiment. FIG. 3 is a diagram illustrating an example of attention mechanism processing in the information processing system according to the first embodiment. FIG. 4 is a diagram illustrating an example of a configuration of an information processing device according to the first embodiment. FIG. 5 is a diagram illustrating an example of a texture map according to the first embodiment. FIG. 6 is a diagram illustrating an example of a 3D model according to the first embodiment. FIG. 7 is a diagram illustrating an example of text information conversion processing according to the first embodiment. FIG. 8 is a diagram illustrating an example of inputting text information and image information according to the first embodiment into the same U-Net using an IP-Adapter. FIG. 9 is a diagram illustrating an example of mesh data conversion according to the first embodiment. FIG. 10 is a diagram illustrating an example of inputting three-dimensional coordinates according to the first embodiment into U-Net. FIG. 11 is a diagram illustrating an example of an information processing procedure according to the first embodiment. FIG. 12 is a diagram illustrating an example of texture editing according to a second embodiment. FIG. 13 is a diagram illustrating an example of an information processing procedure according to the second embodiment. FIG. 14 is a flowchart illustrating an example of a user interface of an information processing device according to the second embodiment. FIG. 15 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an information processing device.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] The present disclosure will be described in the following order of items: 1. Problems with existing technology 2. First embodiment 2-1. Overview of information processing system according to first embodiment 2-2. Configuration of information processing device according to first embodiment 2-3. Information processing procedure according to first embodiment 2-4. Effects of information processing system according to first embodiment 3. Second embodiment 3-1. Overview of information processing system according to second embodiment 3-2. Configuration of information processing device according to second embodiment 3-3. Information processing procedure according to second embodiment 3-4. Effects of information processing system according to second embodiment 4. Third embodiment 5. Other embodiments 6. Hardware configuration

[0012] (1. Issues with Existing Technology) In recent years, 3D models have been used in games, video production, and other fields, and have been applied to a variety of fields. For example, a 3D model is created by attaching a texture to a mesh. A mesh is a polygon mesh that represents the shape of a 3D object. A texture is information that represents the surface color and surface irregularities of a 3D object. Furthermore, a 3D texture refers to a texture attached to a UV map. A UV map is data that stores coordinates used to attach an image or the like to a 3D model.

[0013] A high-resolution 3D model is, for example, attached with a texture that corresponds to physically based rendering (also referred to as PBR or physically based rendering). Therefore, the texture attached to the high-resolution 3D model is physically consistent because it can accurately display the surface texture, the reflection of ambient light, and the like during rendering. Physically based rendering refers to rendering using a rendering equation that expresses the relationship between incident light and reflected light on the surface of an object using a bidirectional reflectance distribution function.

[0014] Physically based rendering requires texture maps for surface roughness, reflectivity, etc. Texture maps are textures that contain information for creating physically consistent 3D models, such as diffuse maps and normal maps, including color and unevenness.

[0015] For example, textures that support physically based rendering can naturally express the texture and reflection of an object's surface, while textures that do not support physically based rendering cannot properly express textures and reflections that correspond to ambient light because the reflected light is already incorporated into the texture.

[0016] Traditionally, users create textures manually or using so-called AI-generated textures, and then apply the textures to meshes manually or using AI-generated textures. However, creating textures is labor-intensive, and even using AI-generated textures takes time. Furthermore, textures must be consistent across all surfaces of a 3D object. Furthermore, creating textures is difficult unless the user has specialized knowledge.

[0017] For example, the techniques disclosed in Non-Patent Documents 1 and 2 are techniques for generating textures from text. The techniques disclosed in Non-Patent Documents 1 and 2 capture images along the shape of a 3D object from multiple viewpoints and generate the captured images. Next, the techniques disclosed in Non-Patent Documents 1 and 2 generate a 3D texture by pasting the generated images onto a UV map. Then, the techniques disclosed in Non-Patent Documents 1 and 2 paste the 3D texture generated using generation AI onto a mesh.

[0018] However, the techniques disclosed in Non-Patent Document 1 and Non-Patent Document 2 take a long time to generate textures because they generate images from multiple viewpoints. For example, the technique disclosed in Non-Patent Document 1 takes a long time to generate textures even when a high-performance personal computer (PC) is used.

[0019] Furthermore, the generated 3D model may have unnatural seams or lack three-dimensional consistency. Furthermore, the generated 3D model is an RGB image and is therefore not compatible with physically based rendering.

[0020] Therefore, the present disclosure proposes an information processing system that automatically generates multiple texture maps that support physically based rendering.

[0021] (2. First embodiment) (2-1. Overview of information processing system according to first embodiment) First, an example of information processing according to the first embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing an overview of the information processing system according to the first embodiment. The information processing system 1 is composed of a terminal device 10 and an information processing device 100. The terminal device 10 and the information processing device 100 are connected to each other so as to be able to communicate with each other via a network, whether wired or wireless.

[0022] The terminal device 10 is a computer used by the user U1 to receive first input information 11 and second input information 12 from the user U1 and to display a plurality of texture maps corresponding to physically based rendering (hereinafter, texture maps corresponding to physically based rendering may be simply referred to as texture maps). The terminal device 10 may be a device such as a smartphone, a tablet, a notebook PC, a desktop PC, a mobile phone, or a personal digital assistant (PDA).

[0023] The information processing device 100 is a computer used by the user U1 to acquire multiple texture maps, and may be any device that can perform the processes in the embodiment.

[0024] The number of terminal devices 10 is not limited to the number shown in Fig. 1. Also, Fig. 1 shows a case where the terminal device 10 and the information processing device 100 are separate devices, but the terminal device 10 and the information processing device 100 may be integrated.

[0025] The terminal device 10 receives, for example, text information 11a including sentences describing the 3D object in natural language and image information 11b including images defining the style of the 3D object, as the first input information 11. The terminal device 10 may receive either the text information 11a or the image information 11b, or may receive both the text information 11a and the image information 11b.

[0026] The terminal device 10 receives, as the second input information 12, mesh data 12a that indicates the shape of a 3D object, for example.

[0027] The information processing device 100 receives text information 11a, image information 11b, and mesh data 12a from the terminal device 10 for generating a plurality of texture maps requested by the user U1 (step S11). Subsequently, the information processing device 100 generates a plurality of texture maps 17 corresponding to physically based rendering based on the text information 11a, image information 11b, and mesh data 12a (step S21).

[0028] The process of step S21 will be described in detail below. In this disclosure, an example will be described in which a so-called U-Net based diffusion model is used to generate a texture map.

[0029] 1, the information processing device 100 inputs the acquired text information 11a to the text encoder 13 (step S22). The text encoder 13 outputs an embedded expression based on the input text information 11a.

[0030] The text encoder 13 is a trained model that converts input text information 11a into an embedded representation with extracted features and outputs the embedded representation. The embedded representation is a vector with extracted features so that it can be input to the U-net 16.

[0031] 1, the information processing device 100 inputs the acquired image information 11b to the image encoder 14 (step S23). The image encoder 14 outputs an embedded representation based on the input image information 11b.

[0032] The image encoder 14 is a trained model that converts the input image information 11b into an embedded representation with extracted features and outputs the resulting representation. The embedded representation is a vector with extracted features so that it can be input to the U-net 16. For example, the image encoder 14 converts the image information so that it can be input to the same U-net as the U-net 16 that input the embedded representation of text information.

[0033] The information processing device 100 unfolds the surface of the 3D object including the mesh data 12a and converts it into a UV map. As shown in Fig. 1, the information processing device 100 inputs the UV map to the ControlNet 15 (step S24). Based on the input UV map, the ControlNet 15 converts it into three-dimensional coordinates corresponding to each pixel that constitutes the UV map.

[0034] The ControlNet 15 is a model trained to output three-dimensional coordinates corresponding to each pixel of the input UV map so that they can be input to the U-net 16 .

[0035] Next, the information processing device 100 inputs the text information 11a from which the feature amounts have been extracted to the U-Net 16 (step S25).The information processing device 100 inputs the image information 11b from which the feature amounts have been extracted to the U-Net 16 (step S26).The information processing device 100 inputs the three-dimensional coordinates obtained by converting the mesh data 12a to the U-Net 16 (step S27).

[0036] The U-Net 16 is a trained model that receives the features of the text information 11a, the features of the image information 11b, and three-dimensional coordinates as input, and outputs a texture map 17 that corresponds to physically based rendering.

[0037] Here, the generation process of the texture map 17 corresponding to the physically based rendering performed by the diffusion model will be described in more detail with reference to Fig. 2. Fig. 2 is a diagram showing an example of the process of U-Net in the information processing system according to the first embodiment.

[0038] The information processing device 100 generates multiple texture maps using a different U-Net for each texture map to be generated, as shown in Fig. 2. Specifically, the information processing device 100 has, as each U-Net, a diffuse net 16a that generates a base color (diffuse), a normal net 16b that generates a normal direction (normal), a roughness net 16c that generates surface roughness, and the like.

[0039] 2, the information processing device 100 inputs text information 11a and image information 11b, from which features have been extracted in the embedded representation, to each U-Net. For example, the information processing device 100 uses the attention mechanism 20 to calculate the similarity of the intermediate features output by each U-Net from the information of the content input to each U-Net, thereby calculating the dependency relationship between the vectors of each U-Net.

[0040] Here, the generation process of the texture map 17 corresponding to the physically based rendering in the U-Net will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the process of the attention mechanism of the information processing system according to the first embodiment. Note that the example described here uses the diffuse net 16a among the U-Nets, but similar processes are also performed in other U-Nets.

[0041] As shown in FIG. 3 , the information processing device 100 uses the feature quantities of the diffuse net 16a as a query. The information processing device 100 also calculates the similarity between tokens of the diffuse net 16a and other networks by calculating the inner product of the query and the key, using the feature quantities of other U-Nets as keys. The information processing device 100 then calculates the similarity calculation result as a value. The information processing device 100 uses the query as the feature quantities of other U-Nets 16, such as the normal net 16b, to calculate the similarity between all networks and other networks. The information processing device 100 then generates multiple texture maps corresponding to physically based rendering, taking into account the relationship with each network.

[0042] As shown in FIG. 1 , the U-Net 16 generates a texture map 17a related to the base color. The example described above uses the diffuse net 16a to generate the texture map 17a related to the base color. However, in reality, the information processing device 100 generates multiple texture maps in addition to the texture map 17a related to the base color. For example, the information processing device 100 uses the U-Net 16b that generates the normal direction to generate the texture map 17b related to the normal direction. Furthermore, the information processing device 100 uses the U-Net 16c that generates the surface roughness to generate the texture map 17c related to the surface roughness.

[0043] Furthermore, as shown in FIG. 1, the information processing device 100 provides the user U1 with a plurality of texture maps 17 corresponding to the generated physically based rendering from the terminal device 10 (step S31).

[0044] The information processing device 100 can automatically generate textures by using a diffusion model. Furthermore, the information processing device 100 can generate multiple highly accurate texture maps 17 that support physically based rendering, allowing for efficient generation of highly accurate textures. The information processing device 100 can generate consistent textures by generating each texture map while mutually exchanging feature amounts with the U-Net of each texture map.

[0045] (2-2. Configuration of information processing device according to first embodiment) Next, the configuration of the information processing device 100 according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the information processing device 100 according to the first embodiment.

[0046] As shown in FIG. 4, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0047] The communication unit 110 is connected to the network N via a wired or wireless connection, and transmits and receives information to and from the terminal device 10 and the like via the network N. For example, the communication unit 110 receives first input information and second input information from the terminal device 10, attaches a physically based rendering compatible texture map to a 3D object, and transmits the resulting information to the terminal device 10. The communication unit 110 is realized by, for example, a network interface card (NIC).

[0048] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 4 , the storage unit 120 includes a diffusion model storage unit 121.

[0049] The diffusion model storage unit 121 stores a diffusion model that has been trained to receive inputs of either or both of text information and image information, and mesh data, and to output a physically based rendering correspondence map.

[0050] The control unit 130 is realized, for example, by a central processing unit (CPU), a micro processing unit (MPU), or the like executing a program (for example, an image processing program according to the present disclosure) stored inside the information processing device 100 using a random access memory (RAM) or the like as a working area. The control unit 130 is also a controller, and may be realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0051] Next, a specific configuration example of the control unit 130 will be shown using Fig. 4. As shown in Fig. 4, the control unit 130 includes a reception unit 131, an acquisition unit 132, a generation unit 133, a presentation unit 134, and a learning unit 135. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 4, and other configurations may be used as long as they are capable of performing information processing described below.

[0052] The receiving unit 131 receives, from the user, first input information for adding information about the 3D object to be generated, and second input information indicating the shape of the 3D object to be generated.

[0053] For example, the receiving unit 131 receives text information and image information as the first input information, and receives mesh data as the second input information.

[0054] For example, the receiving unit 131 receives, as text information included in the first input information, either or both of a sentence describing the 3D object to be generated, such as "the surface is glossy and uneven," and a word describing the 3D object, such as "glossy, blue." The sentence and the word may be in any language, and may be abstract sentences or vague words. The acquired text information may be modified to a more detailed description of the 3D object using any language model.

[0055] Furthermore, for example, the receiving unit 131 may receive, as image information included in the first input information, an image that serves as a reference when generating a texture representing a character or mark. The receiving unit 131 may also receive an image of a 3D model onto which a texture to be generated is applied. Here, the receiving unit 131 may use the image of the viewpoint from which the 3D model is captured as is as the texture that constitutes the texture map. The receiving unit 131 may acquire multiple images and may receive different information, such as reference images and conditions for generating textures.

[0056] For example, the receiving unit 131 receives the shape of the 3D model requested by the user as mesh data included in the second input information. The mesh data may be a 3D file in any mesh format, and may or may not be UV-rendered.

[0057] The acquisition unit 132 acquires first input information for adding information about the 3D object to be generated and second input information indicating the shape of the 3D object to be generated. Furthermore, for example, the acquisition unit 132 acquires, as the first input information, at least one of text information describing the 3D object to be generated and image information defining the style of the 3D object to be generated. Furthermore, for example, the acquisition unit 132 acquires, as the second input information, mesh data indicating the shape of the 3D object to be generated. That is, the acquisition unit 132 acquires either or both of the text information and image information received by the reception unit 131, and the mesh data.

[0058] The generation unit 133 generates a plurality of texture maps corresponding to physically based rendering of the 3D object to be generated based on the first input information and the second input information. For example, the generation unit 133 unfolds the surface of the mesh data to generate a UV map, and generates the texture map based on the UV map. Note that the generation unit 133 may generate the texture map corresponding to physically based rendering based on either text information or image information acquired as the first input information.

[0059] For example, the generation unit 133 generates a plurality of texture maps including at least one of a map relating to base color, a map relating to normal direction, a map relating to surface roughness, a map relating to surface reflectance, a map relating to object shadows, and a map relating to light emitting positions.

[0060] A specific example of the plurality of texture maps generated by the generation unit 133 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of a texture map according to the first embodiment. As shown in Fig. 5, the generation unit 133 generates a plurality of texture maps including at least one of a texture map 17a relating to a base color (diffuse), a texture map 17b relating to a normal direction (normal), a texture map 17c relating to surface reflectance (roughness), a texture map 17d relating to surface roughness (metallic), a texture map 17e relating to shadows of an object (ambient occlusion), a texture map 17f relating to emissive portions, and a texture map 17g relating to fine adjustment of a shape (displacement).

[0061] The generation unit 133 may generate multiple texture maps to be applied to the entire 3D object to be generated. Note that the texture maps are not limited to the example shown above, and a map related to specular reflection may be generated instead of the map related to roughness and the map related to metallic, or other texture maps may be generated.

[0062] Furthermore, for example, the generation unit 133 attaches the generated texture maps to a 3D object to generate a 3D model compatible with physically based rendering. A specific example of a 3D model generated by the generation unit 133 will be described with reference to FIG. 6 . FIG. 6 is a diagram showing an example of a 3D model according to the first embodiment. For example, as shown in FIG. 6 , the generation unit 133 attaches the generated texture maps to a 3D object to generate a 3D model compatible with physically based rendering.

[0063] In the following description, an example will be described in which a texture map is generated by inputting first input information and second input information into a trained machine learning model.

[0064] For example, the generation unit 133 generates multiple texture maps by inputting the first input information and the second input information into a trained machine learning model. The trained machine learning model is a diffusion model. Also, for example, the generation unit 133 converts the text information and image information acquired as the first input information into the same format and inputs them into the diffusion model to generate multiple texture maps.

[0065] Although the present disclosure shows an example using a U-Net-based diffusion model, it is not limited to the diffusion model, and a generative model such as a GAN (Generative Adversarial Networks), a VAE (Variational Auto Encoder), or a Flow-based generative model may also be used.

[0066] An example in which the generation unit 133 inputs text information included in the first input information to U-Net will be described using FIG. 7 . FIG. 7 is a diagram illustrating an example of a text information conversion process according to the first embodiment. In the present disclosure, an example in which text information is converted into a text embedding that can be input to U-Net using the CLIPTextEmbeddings model 21 will be described. However, any model may be used as long as it can convert text information into an expression that can be input to U-Net. For example, the CLIPTextEmbeddings model 21 is a trained model that receives text information as input and is trained to output a text embedding 24, which is an expression that can be input to U-Net.

[0067] 7, the generation unit 133 converts the text information acquired by the acquisition unit 132 into a token sequence using a Text Tokenizer 22, and inputs the converted token sequence to a Text Encoder 23 to extract features. For example, the generation unit 133 extracts features from the text information "A picture of a cat." The generation unit 133 then converts the extracted features into a Text embedding 24 and inputs it to U-Net.

[0068] FIG. 8 illustrates an example in which the generation unit 133 inputs image information included in the first input information to the U-Net. FIG. 8 illustrates an example in which text information and image information according to the first embodiment are input to the same U-Net using the IP-Adapter 25. While this disclosure describes an example in which the IP-Adapter is used to convert image information into an image embedding that can be input to the U-Net, any model may be used as long as it can convert the image information into a representation that can be input to the U-Net. For example, as another example of inputting image information to the U-Net, the image information may be input in accordance with the input noise of the diffusion model, and the diffusion model may be constructed so that the distance between the image information and the image information in the semantic space is reduced.

[0069] 8, the generation unit 133 inputs the image information 26 acquired by the acquisition unit 132 to an image encoder 27 to extract features. The generation unit 133 then converts the extracted features into an embedded representation in the same format as text information and inputs it to U-Net.

[0070] Furthermore, for example, the generation unit 133 converts a UV map obtained by expanding mesh data acquired as second input information into three-dimensional coordinates, and inputs the converted data into a diffusion model to generate multiple texture maps. An example in which the generation unit 133 inputs mesh data included in the second input information to U-Net will be described with reference to FIGS. 9 and 10 . FIG. 9 is a diagram illustrating an example of mesh data conversion according to the first embodiment. FIG. 10 is a diagram illustrating an example of inputting three-dimensional coordinates to U-Net according to the first embodiment. While the present disclosure describes an example in which a UV map is converted into three-dimensional coordinates corresponding to each pixel that can be input to U-Net, any conversion may be performed as long as it can be converted into a representation that can be input to U-Net.

[0071] For example, as shown in FIG. 9 , if the shape 50 of a 3D object including the acquired mesh data has not been expanded into a UV map, the generation unit 133 expands the surface of the shape of the acquired 3D object to create a UV map 32. Next, the generation unit 133 converts the expanded UV map into three-dimensional coordinates 33 corresponding to each pixel of the UV map. Note that the generation unit 133 may convert the UV map into normal vectors instead of three-dimensional coordinates. As another example, the generation unit 133 may convert the 3D object including the acquired mesh data into an abstract representation such as an implicit function representation or an SDF (Signed Distance Field).

[0072] Furthermore, the generation unit 133 may convert the generated three-dimensional coordinates or normal vectors into another representation that can be interpreted by U-Net using positional encoding, etc. Then, as shown in Fig. 10, the generation unit 133 inputs mesh data in which the encoded three-dimensional coordinate values ​​are included in each pixel to the Control Net, extracts features, and inputs the extracted features to the U-Net.

[0073] For example, the diffusion model includes multiple networks that generate one texture map using first input information and second input information as input, and the generation unit 133 inputs the first input information and the second input information to each of the multiple networks to generate multiple texture maps from the multiple networks. In other words, the diffusion model includes multiple networks. The diffusion model generates multiple texture maps using a different network for each texture map to be generated. For example, the network refers to U-Net.

[0074] For example, the generation unit 133 has different U-Nets such as a diffuse net 17a that generates a base color (diffuse), a normal net 17b that generates a normal direction (normal), and a roughness net 17c that generates surface roughness (roughness). Note that the generation unit 133 is configured to generate a plurality of texture maps, but may be configured to be divided into different generation units for each texture map to be generated.

[0075] For example, the generation unit 133 uses an attention mechanism that transmits information between multiple networks to calculate the similarity of the features of each network and generates multiple texture maps using the calculated similarity. The feature of each network refers to the intermediate feature output by the network from at least one of the text information and image information input to each network. The similarity is calculated from the intermediate feature between multiple networks included in the diffusion model. For example, the generation unit 133 uses an attention mechanism to calculate the similarity of the intermediate feature output by each U-Net from at least one of the text information and image information input to each U-Net, thereby calculating the dependency of the vectors of each U-Net. In other words, the generation unit 133 exchanges intermediate feature values ​​between multiple diffusion models to generate multiple texture maps from different U-Nets.

[0076] The presentation unit 134 presents to the user the 3D model generated by the generation unit 133. Note that the presentation unit 134 may present texture maps corresponding to a plurality of physically based renderings generated by the generation unit 133.

[0077] The training unit 135 trains the diffusion model using a training set of texture maps for each network so that each network outputs a different texture map from among multiple texture maps. There is a U-Net corresponding to each texture map to be generated. For example, when training a U-Net that outputs a map related to a base color, the training unit 135 trains the diffusion model using a training set of maps related to the base color. Similarly, the training unit 135 trains the diffusion model for each corresponding texture map. A training set is a training data set used to divide an existing 3D model composed of multiple texture maps into each texture map and train them separately.

[0078] The diffusion model adds noise to training data, estimates the inverse process of the process leading to complete noise, and learns to minimize the error between the original training data and the data from which the noise has been removed by the inverse process. For example, the training data may be data related to a texture map, such as a map related to a base color (diffuse).

[0079] The learning process of a diffusion model is divided into a forward process and a reverse process. In the forward process, noise is gradually added to the original data. On the other hand, the reverse process estimates the data generated in the forward process in the reverse direction of the forward process. At this time, the parameters of the diffusion model are optimized so that data close to the original data is obtained. U-Net performs the reverse process.

[0080] For example, the forward process transfers the texture map from the initial point to the X ０ The noise is gradually added as follows. The noise to be added is, for example, Gaussian noise. The process of adding noise starts from the time point X ０ From time X ｔ , time X ｔ－１ , and noise is gradually added until the final point X Ｔ So, this is completely noise data.

[0081] Then, for example, the reverse process is performed at time X ｔ Going back from time X ０ The noise is gradually removed from the data as the time approaches X. ｔ From time X ｔ－１ , time X ０ The noise is gradually removed through ０ This will give you data that restores the original texture map.

[0082] (2-3. Information Processing Procedure According to First Embodiment) Next, the information processing procedure by the information processing device 100 according to the first embodiment will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of the information processing procedure according to the first embodiment.

[0083] 11 , the receiving unit 131 of the terminal device 10 receives text information as first input information received from the user (step S101). The receiving unit 131 receives image information as first input information received from the user (step S102). Note that the receiving unit 131 may receive either or both of the text information and the image information. The receiving unit 131 also receives mesh data as second input information received from the user (step S103).

[0084] Next, the information processing device 100 acquires the text information, image information, and mesh data received by the terminal device 10. The generation unit 133 of the information processing device 100 extracts and converts features of the acquired text information so that it can be input to U-Net (step S104). The generation unit 133 extracts and converts features of the acquired image information so that it can be input to U-Net (step S105). The generation unit 133 also converts the acquired mesh data into three-dimensional coordinates or the like so that it can be input to U-Net (step S106).

[0085] The generation unit 133 then inputs the text information, image information, and 3D coordinates obtained by converting the mesh data from the extracted features to U-Net. U-Net generates texture maps corresponding to multiple physically based renderings (step S107). The information processing device 100 transmits the generated texture maps corresponding to the physically based renderings to the terminal device 10. The presentation unit 134 of the terminal device 10 may then attach the generated texture maps corresponding to the multiple physically based renderings to a 3D object and present them to the user (step S108).

[0086] (2-4. Effects of Information Processing System According to First Embodiment) As described above, the information processing system according to the present disclosure (the information processing device 100 in the first embodiment) includes a reception unit (the reception unit 131 in the first embodiment), an acquisition unit (the acquisition unit 132 in the first embodiment), a generation unit (the generation unit 133 in the first embodiment), a presentation unit (the presentation unit 134 in the first embodiment), and a learning unit (the learning unit 135 in the first embodiment). The acquisition unit acquires first input information for adding information about a 3D object to be generated, and second input information indicating the shape of the 3D object to be generated. The generation unit generates a plurality of texture maps corresponding to physically based rendering of the 3D object to be generated, based on the first input information and the second input information.

[0087] In this way, the information processing device disclosed herein can generate 3D textures corresponding to physically based rendering with a single inference, thereby efficiently generating highly accurate textures. By generating textures corresponding to physically based rendering, the information processing device 100 enables realistic 3D representation according to lighting conditions, etc. Furthermore, by generating textures corresponding to physically based rendering, the information processing device can be seamlessly applied to 3D production.

[0088] The acquisition unit also acquires, as the first input information, at least one of text information that explains the 3D object to be generated and image information that defines the style of the 3D object to be generated.

[0089] In this way, by acquiring text information, the information processing device can generate textures that take the acquired text into consideration. The text information can be input by the user in any language. Furthermore, since the text information can be only single words or abstract language, the user can perform intuitive operations and can also generate textures that are difficult to put into words.

[0090] Furthermore, by acquiring image information, the information processing device can generate a texture taking the acquired image information into consideration. By acquiring image information, the information processing device can generate a texture map even when it is difficult to express as text information. By acquiring image information, the information processing device can generate a texture map according to the user's image. By acquiring a character or mark to be applied as a texture, the information processing device can generate a texture map using the character or mark.

[0091] When the information processing device acquires an image of a viewpoint from which a 3D model is captured as image information, it can use the acquired image parts as they are and generate textures so that the parts for which no image has been acquired are connected to the acquired image.

[0092] Therefore, the information processing device can efficiently generate highly accurate textures by acquiring at least one of text information and image information.

[0093] In addition, the acquisition unit acquires mesh data indicating the shape of the 3D object to be generated as second input information, and the generation unit unfolds the surface of the mesh data to create a UV map and generates a texture map based on the UV map.

[0094] In this way, by inputting mesh data indicating the shape of a 3D object, the information processing device can generate a texture map that takes into account three-dimensional spatial information. The information processing device can generate the entire texture map by taking into account the shape of the object to be generated. Furthermore, by taking into account the shape of the object to be generated, the information processing device can generate a texture that is three-dimensionally consistent. Therefore, the information processing device can efficiently generate high-precision textures.

[0095] The generation unit also generates a plurality of texture maps including at least one of a map relating to base color, a map relating to normal direction, a map relating to surface roughness, a map relating to surface reflectance, a map relating to object shadows, and a map relating to light emitting positions.

[0096] In this way, the information processing device can generate multiple texture maps that correspond to physically based rendering, and therefore can efficiently generate high-precision textures.

[0097] The generation unit also generates a 3D model compatible with physically based rendering by attaching the generated texture maps to the mesh data. The presentation unit presents the 3D model generated by the generation unit to a user.

[0098] In this way, the information processing device can generate a 3D model compatible with physically based rendering by generating a highly accurate texture map.

[0099] The generation unit also generates a plurality of texture maps by inputting the first input information and the second input information into a trained machine learning model.

[0100] In this way, the information processing device can automatically and efficiently generate high-precision textures by using a trained machine learning model, thereby reducing the production costs of 3D production and broadening its reach.

[0101] Furthermore, the machine learning model is a diffusion model.

[0102] In this way, the information processing device can efficiently generate highly accurate textures by using, for example, a diffusion model.

[0103] The generation unit also converts the text information and image information acquired as the first input information into the same format, inputs the converted information into the diffusion model, and generates a plurality of texture maps.

[0104] In this way, the information processing device converts text information and image information so that they can be input to the same U-Net, and can generate texture maps from image information and text information. Therefore, the information processing device can efficiently generate high-precision textures.

[0105] The generation unit also converts a UV map, which is generated by expanding the mesh data acquired as the second input information, into three-dimensional coordinates, and inputs the three-dimensional coordinates into a diffusion model to generate a plurality of texture maps.

[0106] In this way, the information processing device can generate a texture map according to the mesh data requested by the user by converting three-dimensional coordinates into a representation that can be input to U-Net. The information processing device can generate a UV map of the texture because it takes into account the shape information of the object to be generated. The information processing device can generate a three-dimensionally consistent texture because it takes into account the spatial information of the object to be generated. Therefore, the information processing device can efficiently generate high-precision textures.

[0107] The diffusion model also includes a plurality of networks that use first input information and second input information as inputs to generate one texture map, and the generation unit inputs the first input information and the second input information to each of the plurality of networks to generate a plurality of texture maps from the plurality of networks.

[0108] In this way, the information processing device is equipped with multiple U-Nets and can generate separate texture maps from the multiple U-Nets. In other words, since multiple texture maps can be generated, high-precision textures can be generated efficiently.

[0109] The generation unit also uses an attention mechanism that transmits information between multiple networks to calculate the similarity of the features of each network, and generates multiple texture maps using the calculated similarity.

[0110] In this way, the information processing device transmits information using the attention mechanisms of multiple diffusion models, allowing for the generation of textures that are in harmony with each other. Therefore, the information processing device can efficiently generate high-precision textures.

[0111] The learning unit also trains the diffusion model using a learning set of texture maps for each network so that each network outputs a different texture map from among the plurality of texture maps.

[0112] In this way, the information processing device can construct a diffusion model that generates multiple texture maps by performing learning for each network.

[0113] (3. Second Embodiment) (3-1. Overview of Information Processing System According to Second Embodiment) In the first embodiment, an example has been described in which a texture is generated using first input information and second input information, but the processing of the disclosed information processing device 100 is not limited to this. For example, in order to edit the generated texture, the information processing device 100 can generate an edited texture using editing information related to editing a 3D object, etc.

[0114] Therefore, in the second embodiment, an example in which textures are edited based on editing information input by a user will be described. Note that in the second embodiment, descriptions of content common to the first embodiment will be omitted as appropriate.

[0115] 12 is a diagram showing an example of editing textures according to the second embodiment. The configuration of the information processing system 1 is the same as that of the first embodiment, and therefore a description thereof will be omitted.

[0116] 12 , the terminal device 10 receives, from the user U1, at least one of text editing information 51 a, style information 51 b, and fill mask information 51 c as editing information 51. The terminal device 10 also receives, from the user U1, pre-editing texture information 52 a as third input information 52. The terminal device 10 also receives, from the user U1, mesh data 53 a indicating the shape of a 3D object as second input information 53.

[0117] The information processing device 100 receives at least one of text editing information 51a, style information 51b, and fill mask information 51c, pre-edited texture information 52a, and mesh data 53a from the terminal device 10 to edit the texture requested by user U1 (step S51).

[0118] For example, as a method of editing a portion of a texture, the information processing device 100 accepts designation of the portion to be edited and how to edit it based on text editing information 51 a and style information 51 b. Also, as a method of editing a portion of a texture, the information processing device 100 accepts designation of the portion to be edited and how to edit it based on fill mask information 51 c.

[0119] Next, the information processing device 100 generates multiple texture maps 59 corresponding to the edited physically based rendering based on at least one of the text editing information 51a, style information 51b, and fill mask information 51c, the pre-editing texture information 52a, and the mesh data 53a (step S61).

[0120] The process of step S61 will be described in detail below.

[0121] In this disclosure, an example will be described in which a so-called U-Net-based diffusion model is used to generate a texture map, as in the first embodiment. Note that the mesh data 53a is input to the U-net 58 using the control net 57, as in the first embodiment, and therefore will not be described here.

[0122] The information processing device 100 inputs the acquired text editing information 51a to the text encoder 54 (step S62). The text encoder 54 outputs an embedded expression based on the input text editing information 51a.

[0123] The text encoder 54 is a trained model that converts the input text editing information 51a into an embedded representation with extracted features and outputs the embedded representation. The embedded representation is a vector with extracted features so that it can be input to the U-Net 58.

[0124] The information processing device 100 inputs the acquired style information 51b to the image encoder 55 (step S63). The image encoder 55 outputs an embedded expression based on the input style information.

[0125] The image encoder 55 is a trained model that converts the input style information 51b into an embedded representation with extracted features and outputs the embedded representation. The embedded representation is a vector with extracted features so that it can be input to the U-Net 58.

[0126] The information processing apparatus 100 inputs the acquired fill mask information 51c to the texture editing encoder 56 (step S64). The texture editing encoder 56 outputs an embedded representation based on the input fill mask information 61c.

[0127] The information processing apparatus 100 inputs the acquired pre-edit texture information 52a to the texture editing encoder 56 (step S65). The texture editing encoder 56 outputs an embedded representation based on the input pre-edit texture information 52a.

[0128] The texture editing encoder 56 is a trained model that converts the input fill mask information 51c or pre-edited texture information 52a into an embedded representation with extracted features and outputs it. The embedded representation is a vector with extracted features so that it can be input to U-Net. The texture editing encoder 56 may input both fill mask information and pre-edited texture information.

[0129] The information processing apparatus 100 unfolds the surface of the 3D object including the acquired mesh data to generate a UV map, as in the first embodiment, and inputs the UV map to the ControlNet 57 (step S66).

[0130] Next, the information processing device 100 inputs the text editing information 51a from which the feature amounts have been extracted to the U-Net 58 (step S67).The information processing device 100 inputs the style information 51b from which the feature amounts have been extracted to the U-Net 58 (step S68).The information processing device 100 inputs the fill mask information 51c from which the feature amounts have been extracted or the pre-editing texture information 52a to the U-Net 58 (step S69).The information processing device 100 inputs the three-dimensional coordinates obtained by converting the mesh data 53a to the U-Net 58 (step S70).

[0131] U-Net outputs a plurality of texture maps corresponding to physically based rendering that reflect the edits, and the information processing device 100 then provides the plurality of texture maps corresponding to physically based rendering to the user U1 (step S71).

[0132] (3-2. Configuration of Information Processing Apparatus According to Second Embodiment) In the second embodiment, the content common to the first embodiment will be omitted as appropriate.

[0133] The receiving unit 131 receives third input information related to the texture of the 3D object to be edited and editing information related to editing of the 3D object. For example, the texture received as the third input information may be a part or all of the texture of the 3D object to be edited.

[0134] For example, the receiving unit 131 receives pre-editing texture information as third input information related to the texture of the 3D object to be edited. The receiving unit 131 also receives a texture map to be edited as the pre-editing texture information. The receiving unit 131 may receive a texture map that does not support physically based rendering as the texture map to be edited.

[0135] The receiving unit 131 may receive a single texture map, rather than multiple texture maps, as the texture map to be edited. The receiving unit 131 may also receive a texture map that includes only a map related to a base color and does not include a map related to surface roughness. The receiving unit 131 may also receive an incomplete texture map, such as one that is partially missing, as the texture map. The receiving unit 131 may receive a texture map corresponding to physically based rendering generated in the first embodiment, or may receive an already generated texture map.

[0136] The receiving unit 131 receives, as editing information related to editing of a 3D object, at least one of text editing information describing the 3D object to be edited, style information specifying the style conversion of the 3D object to be generated, and fill mask information indicating the portion to be edited.

[0137] For example, the text editing information and the style information may be command statements or directive statements. Specifically, the receiving unit 131 receives, as text editing information, "change the texture of the 3D model to one made up of sweets." Furthermore, the receiving unit 131 receives, as text editing information, "change the texture style to an anime-style." When the receiving unit 131 wants to change the style of the texture, it receives, as style information, an anime-style image or the like.

[0138] For example, the fill mask information is a mask that defines the edited portion of the texture of the 3D model. Specifically, the receiving unit 131 sets a Float value between 0 and 1 for the fill mask, and receives information indicating how much of the pre-edited texture to retain based on the Float value indicating the pixel value in the fill mask received from the user. Note that the size of the fill mask may be any size.

[0139] For example, the receiving unit 131 receives text editing information from the user that explains the location and content of editing, based on third input information related to the texture of the 3D object to be edited, which is acquired by the acquiring unit 132 .

[0140] Specifically, the receiving unit 131 receives, as text editing information, which part of the pre-editing texture is to be edited and how. For example, the receiving unit 131 receives, as text editing information, "edit the upper part of the 3D object to be edited to a slightly brighter color." The receiving unit 131 may also receive style information for changing the style of the 3D object to be edited, or may receive only either the text editing information or the style information.

[0141] Furthermore, for example, the receiving unit 131 receives the fill mask information indicating the editing location and editing content from the user, based on third input information related to the texture of the 3D object to be edited, which is acquired by the acquiring unit 132. In other words, the receiving unit 131 uses the fill mask to receive designation of the location of the texture to be edited before editing.

[0142] For example, the receiving unit 131 receives texture editing as a float value between "0 and 1" using a fill mask. Specifically, if the pre-edited texture is to be completely retained, the receiving unit 131 receives a float value of "0" from the user. Furthermore, if the pre-edited texture is to be retained to an intermediate degree and a texture is to be generated, the receiving unit 131 receives a float value of "0.5" from the user. Furthermore, if the pre-edited texture is not to be retained and the texture is to be regenerated, the receiving unit 131 receives a float value of "1" from the user.

[0143] The receiving unit 131 may accept editing of only some of the texture maps corresponding to physically based rendering. That is, the receiving unit 131 may accept editing of the texture for each texture map. For example, the receiving unit 131 may accept editing of some of the maps related to base colors, without editing the maps related to normal directions, among the texture maps corresponding to physically based rendering.

[0144] For example, the receiving unit 131 may receive an edited portion from an image viewed from a viewpoint on a 3D viewer.

[0145] For example, the acquiring unit 132 acquires third input information related to the texture of the 3D object to be edited and editing information related to editing of the 3D object. The acquiring unit 132 acquires the third input information and editing information accepted by the accepting unit 131.

[0146] For example, the acquiring unit 132 acquires, as the editing information related to the editing of the 3D object, at least one of text editing information that describes the 3D object to be edited, style information that specifies style conversion of the 3D object to be generated, and fill mask information that indicates the edited portion. That is, the acquiring unit 132 acquires the text editing information, style information, and fill mask information as the editing information accepted by the accepting unit 131.

[0147] The following description will discuss an example in which multiple texture maps reflecting editing are generated by inputting editing information, second input information, and third input information. Note that the generation unit 133 described below illustrates an example in which a U-Net-based diffusion model is used, as in the first embodiment. For example, the generation unit 133 may use a trained model that is a conversion of the diffusion model used in the first embodiment to receive editing information, second input information, and third input information as inputs and output multiple texture maps reflecting editing. On the other hand, in order to edit the texture maps, the generation unit 133 may also construct a new trained model that receives editing information, second input information, and third input information as inputs and output multiple texture maps reflecting editing.

[0148] For example, the generation unit 133 generates multiple texture maps reflecting editing based on the second input information, the third input information, and the editing information. The following description explains an example in which multiple texture maps reflecting editing are generated by inputting the second input information, the third input information, and the editing information into a trained machine learning model. Note that text editing information and style information are input to U-Net via a text encoder and an image encoder, respectively, but the processing is the same as in the first embodiment, so a description thereof will be omitted. Furthermore, processing of the second input information is also the same as in the first embodiment, so a description thereof will be omitted.

[0149] The following describes a process for inputting pre-edited texture information and fill mask information to U-Net, which differs from the first embodiment. For example, the generation unit 133 inputs the fill mask information and pre-edited texture information to a texture editing encoder, extracts features, and converts them into embedded representations that can be input to U-Net. Next, the fill mask information and pre-edited texture information converted into embedded representations are input to U-Net. Note that only one of the pre-edited texture information and fill mask information may be input to U-Net.

[0150] U-Net takes as input second input information converted into a representation that can be input to U-Net, third input information converted into a representation that can be input to U-Net, and editing information converted into a representation that can be input to U-Net, and generates multiple texture maps corresponding to physically based rendering that reflects the editing.

[0151] For example, when only one texture map corresponding to physically based rendering is received as the third input information, the generation unit 133 generates multiple texture maps based on the editing information and the second input information.

[0152] For example, the generation unit 133 generates multiple texture maps for the entire 3D model, including the portion where the editing is reflected, based on the second input information, the third input information, and the editing information. For example, when the generation unit 133 receives a texture map with a missing portion as pre-edited texture information, the generation unit 133 generates a texture map for the entire 3D model, including the missing portion, using the second input information and the editing information. Furthermore, when a portion of the texture map is edited, the generation unit 133 generates multiple texture maps for the entire 3D model, taking into account the surroundings of the edited portion, using the second input information and the editing information, so that the edited portion and its surroundings are connected.

[0153] For example, the generation unit 133 attaches a plurality of texture maps reflecting the editing to the 3D object to generate a 3D model compatible with physically based rendering.

[0154] The presentation unit 134 presents the 3D model generated by the generation unit 133 to the user.

[0155] (3-3. Information Processing Procedure According to Second Embodiment) Next, the procedure of information processing by the information processing device 100 according to the second embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the information processing procedure according to the second embodiment.

[0156] 13 , the receiving unit 131 of the terminal device 10 receives text editing information as editing information received from the user (step S201). The receiving unit 131 receives style information as editing information received from the user (step S202). The receiving unit 131 receives fill mask information (step S203). Note that the receiving unit 131 receives at least one of the text editing information, the style information, and the fill mask information.

[0157] The receiving unit 131 receives the pre-edited texture information as the third input information (step S204). The receiving unit 131 receives the mesh data as the second input information (step S205). The information processing device 100 acquires the text editing information, style information, fill mask information, pre-edited texture information, and mesh data received by the terminal device 10.

[0158] Next, the generation unit 133 of the information processing device 100 extracts and converts features of the acquired text editing information so that it can be input to U-Net (step S206). The generation unit 133 extracts and converts features of the acquired style information so that it can be input to U-Net (step S207). The generation unit 133 extracts and converts features of the acquired fill mask information and pre-edited texture information so that it can be input to U-Net (step S208). The generation unit 133 also converts the acquired mesh data into three-dimensional coordinates or the like so that it can be input to U-Net (step S209).

[0159] The generation unit 133 then inputs the text editing information, style information, fill mask information, and 3D coordinates obtained by converting the mesh data from the extracted features to U-Net. U-Net generates texture maps corresponding to the plurality of physically based renderings reflecting the editing (step S210). The terminal device 10 acquires the texture maps corresponding to the physically based renderings reflecting the editing generated from the information processing device 100. The presentation unit 134 of the terminal device 10 may then attach the generated texture maps corresponding to the plurality of physically based renderings to a 3D object and present them to the user (step S211).

[0160] Next, an example of a user interface (UI) of the information processing apparatus 100 according to the second embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the user interface of the information processing apparatus 100 according to the second embodiment.

[0161] 14 , the reception unit 131 of the information processing device 100 determines whether a 3D object has been received from the user (step S301). If the reception unit 131 has received a 3D object from the user (step S301: Yes), the reception unit 131 determines whether at least one of text information and image information has been received (step S302). The reception unit 131 determines whether at least one of text information and image information has been received (step S302: Yes).

[0162] Next, the generation unit 133 generates texture maps corresponding to a plurality of physically based renderings (step S303). Subsequently, the generation unit 133 attaches the generated texture maps corresponding to the plurality of physically based renderings to the mesh data to generate a 3D model (step S304). The generation unit 133 presents the generated 3D model to the user (step S305).

[0163] Then, the receiving unit 131 receives second input information and third input information to receive editing of the generated 3D model from the user (step S306). For example, the second input information is mesh data that indicates the shape of the 3D model to be edited. Furthermore, the third input information is texture information of the 3D model to be edited.

[0164] Next, the receiving unit 131 receives, from the user, instructions to edit the texture of the generated 3D model. The receiving unit 131 receives editing information as information instructing texture editing (step S307). For example, the receiving unit 131 receives text editing information, style information, and fill mask information as the editing information.

[0165] Next, the generation unit 133 generates texture maps corresponding to the plurality of edited physically based renderings (step S308).The generation unit 133 then attaches the generated texture maps corresponding to the plurality of edited physically based renderings to the mesh data to generate a 3D model (step S309).The presentation unit 134 presents the edited 3D model to the user (step S310).

[0166] (3-4. Effects of the information processing system according to the second embodiment) The information processing system according to the present disclosure (information processing device 100 in the second embodiment) includes a reception unit (reception unit 131 in the second embodiment), an acquisition unit (acquisition unit 132 in the second embodiment), a generation unit (generation unit 133 in the second embodiment), and a presentation unit (presentation unit 134 in the second embodiment).

[0167] The acquisition unit acquires third input information related to the texture of the 3D object to be edited and editing information related to editing of the 3D object, and the generation unit generates multiple texture maps reflecting the editing based on the second input information, the third input information, and the editing information.

[0168] In this way, the information processing device can edit the texture of a 3D object and efficiently generate a highly accurate texture that reflects the editing.

[0169] In addition, the acquisition unit acquires at least one of the following editing information related to the editing of the 3D object: text editing information describing the 3D object to be edited, style information specifying the style conversion of the 3D object to be generated, and fill mask information indicating the edited portion.

[0170] In this way, the information processing device can obtain intuitive and local texture editing from the user by obtaining at least one of text editing information, style information, and fill mask information as editing information for a 3D object.

[0171] The receiving unit also receives text editing information from the user that explains the edited portion and the edit content, based on third input information related to the texture of the 3D object to be edited that is acquired by the acquiring unit.

[0172] In this way, the information processing device can receive editing instructions from the user in the form of imperatives or directives as text information, allowing the user to give editing instructions through intuitive operations.

[0173] The receiving unit also receives fill mask information indicating the edited location and edit content from the user based on third input information related to the texture of the 3D object to be edited acquired by the acquiring unit.

[0174] In this way, the information processing device can accept a portion to be locally edited by accepting editing information from the user as fill mask information for the portion of the texture to be edited. The information processing device can accept, as a texture editing instruction, editing of a portion of a plurality of textures that correspond to physically based rendering.

[0175] Furthermore, the information processing device can accept local editing by accepting a fill mask and a Float value.The information processing device can accept editing on a 3D viewer, allowing intuitive texture editing from a user.

[0176] The generator also generates a plurality of texture maps for the entire 3D model, including the portion where the editing is reflected, based on the second input information, the third input information, and the editing information.

[0177] In this way, even if the information processing device receives an incomplete texture map as the third input information, it is possible to generate a plurality of complete texture maps for the entire 3D model.

[0178] The generation unit applies the plurality of texture maps reflecting the editing to the 3D object to generate a 3D model compatible with physically based rendering, and the presentation unit presents the 3D model generated by the generation unit to the user.

[0179] In this way, the information processing device generates a 3D model that corresponds to physically based rendering, thereby enabling realistic 3D representation according to lighting conditions and the like.

[0180] (4. Third Embodiment) In the third embodiment, an example will be described in which a texture for the entire 3D model to be applied to a UV map is generated at once based on first input information and second input information. Note that in the third embodiment, an example will be described in which a U-Net-based diffusion model is used, similar to the first and second embodiments. Furthermore, in the third embodiment, content similar to the first and second embodiments will be omitted.

[0181] In the third embodiment, a generation unit 133 that executes a process different from that in the first embodiment will be described.

[0182] The generation unit 133 generates textures for the entire 3D model to be attached to the UV map at once based on the first input information and the second input information. "At once" means that the textures to be attached to the 3D model are not generated sequentially, for example, as textures for multiple viewpoints, but are generated simultaneously for the entire 3D model. The generation unit 133 also generates the textures for the entire 3D model to be attached to the UV map by attaching them to the 3D object to be generated.

[0183] In other words, by generating the texture for the entire 3D model at once, it is possible to generate texture that does not have unnatural seams in the 3D model.

[0184] (5. Other Embodiments) The processing according to each of the above-described embodiments may be implemented in various different forms other than the above-described embodiments.

[0185] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. Furthermore, the information, including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0186] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0187] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0188] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0189] (6. Hardware Configuration) The information equipment of the terminal device 10 and the information processing device 100 according to each of the above-described embodiments is realized by a computer 1000 having a configuration as shown in FIG. 15, for example. The information processing device 100 according to the embodiment will be described below as an example. FIG. 15 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0190] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0191] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0192] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a conversion program according to the present disclosure, which is an example of program data 1450.

[0193] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0194] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0195] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes an image processing program loaded onto the RAM 1200, thereby realizing the functions of the control unit 130, etc. The information processing program according to the present disclosure and data in the storage unit 120 are stored in the HDD 1400. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0196] The present technology can also be configured as follows. (1) An information processing system including: an acquisition unit that acquires first input information for adding information about a 3D object to be generated and second input information that indicates the shape of the 3D object to be generated; and a generation unit that generates a plurality of texture maps corresponding to physically based rendering of the 3D object to be generated based on the first input information and the second input information. (2) The information processing system described in (1), wherein the acquisition unit acquires, as the first input information, at least one of text information that describes the 3D object to be generated and image information that defines a style of the 3D object to be generated. (3) The information processing system described in (1) or (2), wherein the acquisition unit acquires, as the second input information, mesh data that indicates the shape of the 3D object to be generated, and the generation unit unfolds a surface of the mesh data to form a UV map and generates a texture map based on the UV map. (4) The information processing system according to any one of (1) to (3), wherein the generation unit generates the plurality of texture maps including at least one of a map related to a base color, a map related to a normal direction, a map related to surface roughness, a map related to surface reflectance, a map related to a shadow of an object, and a map related to a light emitting position. (5) The information processing system according to any one of (1) to (4), wherein the generation unit applies the generated plurality of texture maps to the 3D object to generate a 3D model compatible with physically based rendering, and further includes a presentation unit that presents the 3D model generated by the generation unit to a user. (6) The information processing system according to any one of (1) to (5), wherein the generation unit generates the plurality of texture maps by inputting the first input information and the second input information to a trained machine learning model. (7) The information processing system according to (6), wherein the machine learning model is a diffusion model.(8) The information processing system according to (7), wherein the generation unit converts the text information and image information acquired as the first input information into the same format, and inputs the converted information to the diffusion model to generate the multiple texture maps. (9) The information processing system according to (7) or (8), wherein the generation unit converts a UV map obtained by expanding mesh data acquired as the second input information into three-dimensional coordinates, and inputs the converted information to the diffusion model to generate the multiple texture maps. (10) The information processing system according to any one of (7) to (9), wherein the diffusion model includes multiple networks that use the first input information and the second input information as inputs to generate one texture map, and the generation unit inputs the first input information and the second input information to each of the multiple networks to generate multiple texture maps from the multiple networks. (11) The information processing system according to (10), wherein the generation unit calculates similarities between features of each of the multiple networks using an attention mechanism that transmits information between the multiple networks, and generates multiple texture maps using the calculated similarities. (12) The information processing system according to any one of (7) to (11), further comprising a training unit that trains each network with the diffusion model using a training set of texture maps so that each network outputs a different texture map from the plurality of texture maps. (13) The information processing system according to any one of (1) to (12), wherein the acquisition unit acquires third input information related to a texture of a 3D object to be edited and editing information related to editing of the 3D object, and the generation unit generates a plurality of texture maps reflecting the editing based on the second input information, the third input information, and the editing information.(14) The information processing system according to (13), wherein the acquisition unit acquires, as editing information related to editing of the 3D object, at least one of text editing information describing the 3D object to be edited, style information defining style conversion of a 3D object to be generated, and fill mask information indicating an edited portion. (15) The information processing system according to (14), further comprising a reception unit that receives the text editing information describing an edited location and edit content from a user, based on third input information related to a texture of the 3D object to be edited acquired by the acquisition unit. (16) The information processing system according to (14) or (15), further comprising a reception unit that receives the fill mask information indicating the edited location and edit content from a user, based on the third input information related to the texture of the 3D object to be edited acquired by the acquisition unit. (17) The information processing system according to any one of (13) to (16), wherein the generation unit generates the plurality of texture maps of the entire 3D model including a location where editing is reflected, based on the second input information, the third input information, and the editing information. (18) The information processing system according to any one of (1) to (17), wherein the generation unit generates a 3D model compatible with physically based rendering by attaching the plurality of texture maps reflecting the editing to the 3D object, and further comprises a presentation unit that presents the 3D model generated by the generation unit to a user. (19) An information processing method including: a computer acquiring first input information for adding information about the 3D object to be generated and second input information that indicates a structure of the 3D object to be generated, and generating a plurality of texture maps compatible with physically based rendering of the 3D object to be generated based on the first input information and the second input information.(20) An information processing program for causing a computer to function as an information processing device comprising: an acquisition unit that acquires first input information for adding information about a 3D object to be generated and second input information that indicates a structure of the 3D object to be generated, and a generation unit that generates a plurality of texture maps corresponding to physically based rendering of the 3D object to be generated based on the first input information and the second input information. (21) An information processing system comprising: an acquisition unit that acquires first input information for adding information about the 3D object to be generated and second input information that indicates a shape of the 3D object to be generated, and a generation unit that generates a texture for the entire 3D model to be attached to a UV map at once based on the first input information and the second input information. (22) The information processing system according to (21), wherein the generation unit generates the texture for the entire 3D model to be attached to the UV map by attaching it to the 3D object to be generated.

[0197] REFERENCE SIGNS LIST 1 Information processing system 10 Terminal device 11 First input information 12 Second input information 16 U-Net 17 PBR-compatible texture map 51 Editing information 52 Third input information 53 Second input information 58 U-Net 59 PBR-compatible texture map 100 Information processing device 110 Communication unit 120 Storage unit 121 Diffusion model storage unit 130 Control unit 131 Reception unit 132 Acquisition unit 133 Generation unit 134 Presentation unit 135 Learning unit

Claims

1. An information processing system comprising: an acquisition unit that acquires first input information for adding information about a 3D object to be generated and second input information that indicates the shape of the 3D object to be generated; and a generation unit that generates, based on the first input information and the second input information, a plurality of texture maps that correspond to physically based rendering of the 3D object to be generated.

2. The information processing system according to claim 1, wherein the acquisition unit acquires, as the first input information, at least one of text information describing the 3D object to be generated and image information defining the style of the 3D object to be generated.

3. The information processing system according to claim 1, wherein the acquisition unit acquires mesh data indicating the shape of the 3D object to be generated as the second input information, and the generation unit unfolds the surface of the mesh data to create a UV map and generates a texture map based on the UV map.

4. The information processing system of claim 1, wherein the generation unit generates the plurality of texture maps including at least one of a map relating to base color, a map relating to normal direction, a map relating to surface roughness, a map relating to surface reflectance, a map relating to object shadows, and a map relating to light emitting position.

5. The information processing system of claim 1, wherein the generation unit generates a 3D model compatible with physically based rendering by applying the generated texture maps to the 3D object, and further comprises a presentation unit that presents the 3D model generated by the generation unit to a user.

6. The information processing system of claim 1, wherein the generation unit generates the plurality of texture maps by inputting the first input information and the second input information into a trained machine learning model.

7. The information processing system according to claim 6, wherein the machine learning model is a diffusion model.

8. The information processing system according to claim 7, wherein the generation unit converts the text information and image information acquired as the first input information into the same format, inputs the converted information into the diffusion model, and generates the plurality of texture maps.

9. The information processing system according to claim 7, wherein the generation unit converts a UV map generated by expanding the mesh data acquired as the second input information into three-dimensional coordinates and inputs the UV map into the diffusion model to generate the plurality of texture maps.

10. The information processing system of claim 7, wherein the diffusion model comprises a plurality of networks that use the first input information and the second input information as inputs to generate one texture map, and the generation unit inputs the first input information and the second input information to each of the plurality of networks to generate a plurality of texture maps from the plurality of networks.

11. The information processing system of claim 10, wherein the generation unit calculates the similarity of the features of each of the networks using an attention mechanism that transmits information between the multiple networks, and generates multiple texture maps using the calculated similarity.

12. The information processing system according to claim 7, further comprising a training unit that trains the diffusion model using a training set of texture maps for each network, so that each network outputs a different one of the plurality of texture maps.

13. The information processing system according to claim 1, wherein the acquisition unit acquires third input information related to the texture of the 3D object to be edited and editing information related to editing of the 3D object, and the generation unit generates a plurality of texture maps reflecting the editing based on the second input information, the third input information, and the editing information.

14. The information processing system according to claim 13, wherein the acquisition unit acquires, as the editing information related to the editing of the 3D object, at least one of text editing information describing the 3D object to be edited, style information defining style conversion of the 3D object to be generated, and fill mask information indicating the edited portion.

15. The information processing system according to claim 14, further comprising a receiving unit that receives the text editing information explaining the editing location and editing content from a user based on third input information regarding the texture of the 3D object to be edited acquired by the acquisition unit.

16. The information processing system according to claim 14, further comprising a receiving unit that receives the fill mask information indicating the editing location and editing content from a user based on third input information related to the texture of the 3D object to be edited acquired by the acquisition unit.

17. The information processing system according to claim 13, wherein the generation unit generates the plurality of texture maps of the entire 3D model including the areas where the editing is reflected, based on the second input information, the third input information, and the editing information.

18. The information processing system according to claim 17, wherein the generation unit generates a 3D model compatible with physically based rendering by attaching the plurality of texture maps reflecting the editing to the 3D object, and further comprises a presentation unit that presents the 3D model generated by the generation unit to a user.

19. An information processing method including: a computer acquiring first input information for adding information about a 3D object to be generated and second input information indicating the structure of the 3D object to be generated; and generating, based on the first input information and the second input information, a plurality of texture maps corresponding to physically based rendering of the 3D object to be generated.

20. An information processing program for causing a computer to function as an information processing device comprising: an acquisition unit that acquires first input information for adding information about a 3D object to be generated and second input information that indicates the structure of the 3D object to be generated; and a generation unit that generates multiple texture maps corresponding to physically based rendering of the 3D object to be generated based on the first input information and the second input information.

Images