Information processing system, information processing method, and program

Abstract

The present technology relates to an information processing system, an information processing method, and a program, that are capable of facilitating construction of desired 3D scenes. An information processing system according to the present technology includes a determination unit that determines, on the basis of text data relating to an appearance of a 3D scene in which a 3D object is disposed, a 3D asset disposed as a 3D object in the 3D scene to be constructed, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset, and a data generation unit that generates scene data of the 3D scene to be constructed on the basis of the 3D asset, layout, and texture determination results by the determination unit. The present technology can be applied to, for example, a 3D scene construction support system.

Description

Information processing system, information processing method, and program

[0001] The present technology relates to an information processing system, an information processing method, and a program, and more particularly to an information processing system, an information processing method, and a program that enable a desired 3D scene to be easily constructed.

[0002] In recent years, there has been an increasing demand in the fields of video and game production for creating virtual spaces filled with 3D objects (hereinafter referred to as 3D scenes). However, creating 3D scenes requires specialized knowledge of design and knowledge of how to operate digital content creation tools (DCC tools), making it difficult for users without prior knowledge to create 3D scenes.

[0003] For example, Patent Document 1 describes a system that acquires an image of a floor plan and generates a 3D scene in which furniture is arranged according to the floor plan.

[0004] International Publication No. 2022 / 049777

[0005] Generally, in systems that support the construction of 3D scenes, 3D scene data is generated as a single mesh data. In this case, it is not possible to separate meshes for each 3D object, so users cannot edit the 3D scene, such as adjusting the placement position of 3D objects.

[0006] Regarding support for 3D scene construction, research has been conducted on AI (Artificial Intelligence) models that generate 3D scenes based on text, but these AI models only allow users to specify the size and type of room (3D scene). Therefore, users cannot specify the 3D objects to be placed in the 3D scene or its appearance, making it difficult to easily construct the desired 3D scene.

[0007] This technology was developed in light of these circumstances, and makes it possible to easily construct desired 3D scenes.

[0008] An information processing system according to one aspect of the present technology includes a determination unit that determines, based on text data regarding the appearance of a 3D scene in which a 3D object is to be placed, a 3D asset to be placed in the 3D scene to be constructed as the 3D object, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset, and a data generation unit that generates scene data for the 3D scene to be constructed based on the determination results of the 3D asset, the layout, and the texture by the determination unit.

[0009] An information processing method according to one aspect of the present technology includes determining, based on text data regarding the appearance of a 3D scene in which a 3D object is to be placed, a 3D asset to be placed as the 3D object in the 3D scene to be constructed, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset, and generating scene data for the 3D scene to be constructed based on the results of determining the 3D asset, the layout, and the texture.

[0010] A program according to one aspect of the present technology causes a computer to perform processing including determining, based on text data regarding the appearance of a 3D scene in which a 3D object is to be placed, a 3D asset to be placed in the 3D scene to be constructed as the 3D object, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset, and generating scene data for the 3D scene to be constructed based on the results of determining the 3D asset, the layout, and the texture.

[0011] In one aspect of the present technology, a 3D asset to be placed as the 3D object in the 3D scene to be constructed, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset are determined based on text data regarding the appearance of the 3D scene in which the 3D object will be placed, and scene data for the 3D scene to be constructed is generated based on the results of determining the 3D asset, the layout, and the texture.

[0012] 1 is a diagram illustrating a 3D scene construction support system to which the present technology is applied. FIG. 1 is a block diagram illustrating an example of the configuration of the 3D scene construction support system. FIG. 2 is a diagram illustrating an example of a scene appearance description. FIG. 3 is a diagram illustrating an example of a 3D scene constructed by a server. FIG. 4 is a diagram illustrating an example of an asset list. FIG. 5 is a diagram illustrating an example of a method for searching for a 3D asset by a 3D asset search unit. FIG. 6 is a diagram illustrating an example of layout information generated by a layout generation unit. FIG. 7 is a diagram illustrating an example of a method for searching for a layout by a layout search unit. FIG. 8 is a diagram illustrating an example of conversion of a scene appearance description. FIG. 9 is a diagram illustrating an example of use of 3D scene data. FIG. 10 is a diagram illustrating an example of generation of a caption. FIG. 11 is a flowchart illustrating processing performed by the 3D scene construction support system. FIG. 12 is a block diagram illustrating a first modified example of the configuration of the 3D scene construction support system. FIG. 13 is a block diagram illustrating a second modified example of the configuration of the 3D scene construction support system. FIG. 14 is a diagram illustrating an example of use of the 3D scene construction support system in previs. FIG. 15 is a diagram illustrating another example of use of the 3D scene construction support system in previs. FIG. 16 is a diagram illustrating an example of use of the 3D scene construction support system as a plug-in for a DCC tool. FIG. 17 is a block diagram illustrating an example of the configuration of a computer hardware.

[0013] Hereinafter, embodiments of the present technology will be described in the following order: 1. Configuration and operation of 3D scene construction support system 2. Modification 3. Application example

[0014] 1. Configuration and Operation of 3D Scene Construction Support System FIG. 1 is a diagram illustrating a 3D scene construction support system 1 to which the present technology is applied.

[0015] The 3D scene construction support system 1 of the present technology is an information processing system that accepts input of text written in a natural language and generates a virtual space (hereinafter also referred to as a 3D scene) in which 3D objects are placed based on the input text.

[0016] In the example of Figure 1, text T1, "Living Room: Gorgeous diamond room," is input to the 3D scene construction support system 1. The 3D scene construction support system 1 understands the meaning of the text T1 and constructs a 3D scene that reflects the meaning of the text T1. Specifically, the 3D scene construction support system 1 determines, based on the text T1, 3D assets to be placed as 3D objects in the 3D scene to be constructed, the layout within the 3D scene to be constructed, and the textures to be applied to each 3D asset.

[0017] The 3D scene construction support system 1 outputs 3D scene data, which is data on the constructed 3D scene. For example, 3D scene data SD1 corresponding to a 3D scene in which gorgeous textures are applied to the floor and walls and furniture with gorgeous textures applied thereto is generated based on the text T1 and output.

[0018] FIG. 2 is a block diagram showing an example of the configuration of the 3D scene construction support system 1. As shown in FIG.

[0019] As shown in FIG. 2, the 3D scene construction support system 1 is configured by connecting a user terminal 11 and a server 12 via wired or wireless communication.

[0020] The user terminal 11 is configured as an information processing device such as a smartphone, a tablet terminal, or a PC. The user terminal 11 accepts input of text written in a natural language. The text is, for example, a scene appearance description statement regarding the appearance of a 3D scene written in any natural language.

[0021] FIG. 3 is a diagram showing an example of a scene appearance description sentence.

[0022] In Figure 3, examples of scene appearance descriptions include "a five-year-old child's room with a movie theme," "a gorgeous room made of diamonds," "a luxurious Chinese-style room," "a modern room with a white theme," and "a room made of rocks."

[0023] Thus, the scene appearance description sentence includes sentences that describe the appearance of the 3D scene, sentences that indicate 3D objects included in the 3D scene, and the like.

[0024] The user terminal 11 supplies text data to the server 12. The text data may be, for example, text data itself including a scene appearance description sentence, or may be data in which various preprocessing steps (e.g., vectorization) have been performed on the text.

[0025] The user terminal 11 displays a 3D scene corresponding to the 3D scene data generated by the server 12 on a display unit such as a display.

[0026] The server 12 is configured as, for example, a web server or a server device (information processing device) on the cloud. The server 12 acquires text data supplied from the user terminal 11, constructs a 3D scene based on the text data, and generates 3D scene data corresponding to the 3D scene.

[0027] FIG. 4 is a diagram showing an example of a 3D scene constructed by the server 12.

[0028] As shown in the upper part of Figure 4, when the text T21 "Vincent van Gogh-style room" is input, the server 12 constructs a 3D scene of a room in which, for example, the texture of a painting by Vincent van Gogh is applied to 3D furniture assets.

[0029] 4, when the text T22 "A room consisting of four tables and six chairs" is input, the server 12 constructs a 3D scene of a room in which, for example, four 3D assets of tables and six 3D assets of chairs are placed. In this way, the user can control the type and number of 3D assets placed in the 3D scene by inputting text containing specific instructions.

[0030] The size of the 3D scene (room) may be specified by the user. The user can specify the size of the 3D scene in a format such as "3.3m x 3.3m" or "5.4m x 2m." When the size of the 3D scene is specified, a 3D scene of a room with a size close to the size specified by the user is constructed.

[0031] Returning to FIG. 2, the user terminal 11 is made up of an input unit 31 , a drawing unit 32 , and a display unit 33 .

[0032] The input unit 31 accepts text input by the user and supplies the text data to the server 12 .

[0033] The drawing unit 32 executes the game engine and DCC tools installed on the user terminal 11 , renders the 3D scene data supplied from the server 12 to generate an image, and supplies the image to the display unit 33 .

[0034] The display unit 33 displays the image supplied from the drawing unit 32. The user can visually check the 3D scene generated by the server 12 using the image displayed on the display unit 33, and edit the 3D scene, for example, by adjusting the placement position of 3D assets.

[0035] The server 12 includes a 3D scene construction unit 51 and a database management unit 52 .

[0036] The 3D scene construction unit 51 includes an asset list generation unit 71, a 3D asset determination unit 72, a layout determination unit 73, an appearance description sentence conversion unit 74, a 3D asset texture generation unit 75, and a 3D scene data generation unit .

[0037] The text data supplied from the user terminal 11 is supplied to an asset list generating unit 71 , a layout determining unit 73 , and an appearance description sentence converting unit 74 .

[0038] The asset list generation unit 71 generates an asset list, which is a list of 3D assets to be placed in the 3D scene to be constructed, based on the text data. For example, the asset list generation unit 71 has a plurality of predefined asset lists. The asset list generation unit 71 searches for the asset list that is most similar to the scene appearance description corresponding to the text data, based on the similarity between each asset list and the scene appearance description.

[0039] The asset list generation unit 71 can also generate an asset list using a large-scale language model such as a Generative Pretrained Transformer (GPT) that receives text data as input and outputs an asset list. The asset list generation unit 71 can also generate an asset list using an AI model that receives text data and the size of a 3D scene specified by a user as input and outputs an asset list.

[0040] FIG. 5 is a diagram showing an example of an asset list.

[0041] 5, when text T21 of "Vincent van Gogh-style room" is input, the asset list generation unit 71 generates an asset list Li1 based on the text T21. The asset list includes text labels indicating the names of the 3D assets.

[0042] In the example of FIG. 5, the asset list Li1 includes the text labels "bookshelf," "ceiling," "coffee_table," "corner_side_table," "dining_chair," "dining_table," and "multi_seat_sofa."

[0043] The asset list generation unit 71 supplies the generated asset list to the 3D asset determination unit 72 and the appearance description conversion unit 74 shown in FIG.

[0044] The 3D asset determination unit 72 has a 3D asset search unit 81. Based on the asset list supplied from the asset list generation unit 71, the 3D asset search unit 81 searches a 3D asset DB (Database) 91 of the database management unit 52 for candidates for 3D assets to be placed in the 3D scene to be constructed.

[0045] The 3D asset DB 91 (first database) stores asset data sets including 3D assets that have already been generated.

[0046] FIG. 6 is a diagram illustrating an example of a method for searching for a 3D asset by the 3D asset search unit 81. In FIG.

[0047] As shown on the right side of Figure 6, for example, in an asset data set AsS1 that includes a 3D asset named "modern_chair," the generated 3D asset body, a rendered image P1 in which the 3D asset is rendered, and a text label Lab1 indicating the name of the 3D asset are linked.

[0048] A 3D asset is composed of, for example, 3D shape data that represents a 3D shape using a 3D point cloud or polygon mesh, and a texture image that is data of a texture that is applied to the 3D point cloud or polygon mesh.

[0049] When searching for a 3D asset "plastic_chair" indicated by the text label Lab11 in the asset list, the 3D asset search unit 81 refers to the text labels included in each asset data set, as indicated by arrow #1, and searches for a text label that exactly matches the text label Lab11. The 3D asset search unit 81 obtains the body of the 3D asset linked to the searched text label from the 3D asset DB 91.

[0050] In the example of FIG. 6, the text label Lab1 and the text label Lab11 do not completely match, so the main body of the 3D asset included in the asset data set other than the asset data set AsS1 is obtained from the 3D asset DB 91.

[0051] The 3D asset search unit 81 can also calculate the similarity between the text label Lab11 and the text labels included in each asset data set using an AI model such as CLIP, and based on this similarity, search for the text label that is most similar to the text label Lab11. In this case, the 3D asset search unit 81 obtains the main body of the 3D asset linked to the searched text label from the 3D asset DB 91.

[0052] Even if the 3D asset search unit 81 cannot find a text label that exactly matches the text label Lab11, it can search the 3D asset DB 91 for a 3D model that has a name that corresponds to the name indicated by the text label Lab11 by comparing the text labels using an AI model.

[0053] Furthermore, as indicated by arrow #2, the 3D asset search unit 81 can also calculate the similarity between the text label Lab and the rendering image included in each asset data set using an AI model such as CLIP, and search for the rendering image that is most similar to the text label Lab 11. In this case, the 3D asset search unit 81 obtains the body of the 3D asset linked to the searched rendering image from the 3D asset DB 91.

[0054] Even if the 3D asset search unit 81 cannot find a text label that exactly matches the text label Lab11, it can use an AI model to compare the text label with the rendered image and search the 3D asset DB 91 for a 3D asset that has an appearance that corresponds to the name indicated by the text label Lab11.

[0055] The 3D asset determination unit 72 determines the 3D asset candidates searched for by the 3D asset search unit 81 as 3D assets to be placed in the 3D scene to be constructed. The 3D asset determination unit 72 associates the text labels in the asset list with the 3D asset bodies that the 3D asset search unit 81 retrieves from the 3D asset DB 91.

[0056] The 3D asset determination unit 72 supplies the asset list and the 3D assets linked to each text label in the asset list to the layout determination unit 73 in FIG.

[0057] The layout determination unit 73 includes a layout generation unit 82 and a layout search unit 83 .

[0058] The layout generation unit 82 designs (generates) new layout candidates within the 3D scene to be constructed based on the asset list supplied from the 3D asset determination unit 72, and generates layout information for each 3D asset that indicates the placement position of the 3D asset within the 3D scene to be constructed. Specifically, the layout generation unit 82 generates layout information for each 3D asset using a large-scale language model that receives the asset list as input and outputs layout information for each 3D asset.

[0059] FIG. 7 is a diagram showing an example of layout information generated by the layout generating unit 82. As shown in FIG.

[0060] As shown in Figure 7, for example, when an asset list Li1 including text labels "dining_table" and "dining_chair" is input, the layout generation unit 82 generates layout information Lay1 for "dining_table" and layout information Lay2 for "dining_chair".

[0061] For example, the layout information indicates the scale of a 3D asset, the translation of the 3D asset within the 3D scene, and the orientation of the 3D asset within the 3D scene. In the example of Figure 7, the scale of the 3D asset is expressed by width, length, and height (size in the x-, y-, and z-directions), the position of the 3D asset is expressed by three-dimensional coordinates (x, y, z), and the orientation of the 3D asset is expressed by the amount of rotation in the yaw direction. Note that the orientation of the 3D asset may also be expressed by the amount of rotation in the roll, pitch, and yaw directions.

[0062] Although the layout design in the 3D scene by the layout generation unit 82 is not necessarily an appropriate design, it is believed that an appropriate layout can be obtained by repeatedly trying to design the layout in the 3D scene. It is expected that the layout generation unit 82 will design a creative layout.

[0063] The layout search unit 83 searches the layout DB 92 of the database management unit 52 for layout candidates within the 3D scene to be constructed based on the text data, the size of the 3D scene specified by the user, and the asset list supplied from the 3D asset search unit 81.

[0064] A layout DB 92 (second database) stores layout data sets relating to layouts of 3D scenes that have already been constructed.

[0065] FIG. 8 is a diagram for explaining an example of a layout search method performed by the layout search unit 83. In FIG.

[0066] As shown on the right side of Figure 8, a layout data set LayS1 relating to the layout of a constructed 3D scene links, for example, 3D scene data corresponding to the constructed 3D scene, an asset list for the constructed 3D scene, a rendering image P11 in which the constructed 3D scene is rendered, a caption C1, and the size of the 3D scene.

[0067] Layout information about the 3D assets is written in the shape data of the 3D assets included in the 3D scene data corresponding to the constructed 3D scene. The caption C1 is a description of the rendered image P11. One layout data set includes, for example, one to five rendered images in which the 3D scene is rendered from one to five viewpoints, and captions corresponding to each rendered image.

[0068] When a scene appearance description of a "Vincent van Gogh-style room" and a 3D scene size of "6m x 6m" are input, the layout search unit 83 calculates the similarity between the scene appearance description and the rendering images included in each layout data set using an AI model such as CLIP, as indicated by arrow #11, and searches for the rendering image that is most similar to the scene appearance description based on this similarity. The layout search unit 83 then obtains layout information for each 3D asset associated with the retrieved rendering image from the layout DB 92.

[0069] As indicated by arrow #12, the layout search unit 83 can calculate the similarity between the scene appearance description and the captions included in each asset data set using an AI model such as CLIP, and search for the caption that is most similar to the scene appearance description based on this similarity. In this case, the layout search unit 83 obtains layout information for each 3D asset linked to the searched caption from the layout DB 92.

[0070] Furthermore, as indicated by arrow #13, the layout search unit 83 can also refer to the size of the 3D scene included in each layout data set and search for a constructed 3D scene whose size is closest to the size specified by the user. In this case, the layout search unit 83 obtains, from the layout DB 92, layout information for each 3D asset included in the 3D scene data corresponding to the searched constructed 3D scene.

[0071] Furthermore, the layout search unit 83 can also refer to the asset list included in each layout data set and search for the asset list that is most similar to the asset list for the 3D scene to be constructed. In this case, the layout search unit 83 obtains layout information for each 3D asset linked to the searched asset list from the layout DB 92.

[0072] The layout search unit 83 can search the layout DB 92 for layout candidates within the 3D scene to be constructed based on the similarity between the scene appearance description and the rendered image, the similarity between the scene appearance description and the caption, the size of the 3D scene specified by the user, and the asset list for the 3D scene to be constructed.

[0073] The layout search unit 83 searches by combining a plurality of pieces of information, thereby being able to search for a layout that is more similar to the layout desired by the user.

[0074] The layout determination unit 73 determines one of the layout candidates newly generated by the layout generation unit 82 as described above and the layout candidates searched for by the layout search unit 83 as the layout within the 3D scene to be constructed. The layout determination unit 73 writes layout information into the shape data of each 3D asset supplied from the 3D asset determination unit 72.

[0075] As described above, the layout data set also includes the 3D asset itself. When the layout determination unit 73 determines a layout candidate searched by the layout search unit 83 as a layout in the 3D scene to be constructed, the layout determination unit 73 may update the 3D assets linked to each text label in the asset list to the 3D assets included in the layout data set.

[0076] The layout determination unit 73 supplies the asset list and the 3D assets with the layout information written therein to the 3D asset texture generation unit 75 shown in FIG.

[0077] The scene appearance description entered by the user may not contain enough information or may not be in the correct format for the server 12 to determine the texture of the 3D asset to be placed in the 3D scene to be constructed. Therefore, the appearance description conversion unit 74 converts the scene appearance description entered by the user into a format that is easy for the subsequent 3D asset texture generation unit 75 to handle.

[0078] FIG. 9 is a diagram showing an example of conversion of a scene appearance description sentence.

[0079] 9, the appearance description conversion unit 74 converts the scene appearance description T41 of "Modern room based on white" based on the asset list Li11 supplied from the asset list generation unit 71, and generates an asset appearance description T42 that describes the appearance of each 3D asset to be placed in the 3D scene to be constructed. For example, the appearance description conversion unit 74 generates the asset appearance description T42 using an AI model that receives the scene appearance description and the asset list as input and outputs an asset appearance description.

[0080] The asset appearance description T42 in FIG. 9 describes the appearance of each of four 3D assets "armchair," "dining_table," "coffee_table," and "dining_chair" included in the asset list Li11.

[0081] The appearance description conversion unit 74 supplies the asset appearance description to the 3D asset texture generation unit 75 shown in FIG.

[0082] The 3D asset texture generation unit 75 generates texture candidates to be applied to each 3D asset supplied from the layout determination unit 73, based on the asset appearance description supplied from the appearance description conversion unit 74. For example, the 3D asset texture generation unit 75 receives as input one 3D asset and an asset appearance description corresponding to the 3D asset, and generates texture candidates to be applied to each 3D asset using an AI model such as Text2Tex that outputs a texture for the 3D asset.

[0083] Since the asset appearance description describes the appearance of each of the 3D assets to be placed in the 3D scene to be constructed, the 3D asset texture generation unit 75 can generate textures for all of the 3D assets to be placed in the 3D scene to be constructed based on the asset appearance description.

[0084] Here, the 3D asset texture generation unit 75 functions as a determination unit that determines the generated texture candidates as textures to be applied to 3D assets to be placed in the 3D scene to be constructed. The 3D asset texture generation unit 75 updates the texture images included in each 3D asset supplied from the layout determination unit 73 based on the generated textures. Note that the 3D asset texture generation unit 75 does not necessarily need to update the texture images included in each 3D asset.

[0085] The 3D asset texture generator 75 supplies the asset list and the 3D assets with updated textures to the 3D scene data generator 76 of FIG.

[0086] In addition, the server 12 can generate multiple patterns of textures to be applied to each 3D asset without changing the layout within the 3D scene to be constructed, by simply re-executing the texture generation by the 3D asset texture generation unit 75.

[0087] The 3D scene data generation unit 76 combines the 3D assets supplied from the 3D asset texture generation unit 75 to generate 3D scene data, and supplies the 3D scene data to the user terminal 11, while also supplying the 3D scene data and the asset list to the database management unit 52. It is desirable that the 3D scene data be generated in a file format such as the fbx format or OpenUSD format that can be used with a variety of DCC tools.

[0088] Each 3D asset included in the 3D scene data is determined by the 3D asset determination unit 72. Layout information corresponding to the layout determined by the layout determination unit 73 is written to each 3D asset, and a texture determined by the 3D asset texture generation unit 75 is applied to each 3D asset. Therefore, it can be said that the 3D scene data generation unit 76 generates 3D scene data based on the 3D asset determination results by the 3D asset determination unit 72, the layout determination results by the layout determination unit 73, and the texture determination results by the 3D asset texture generation unit 75.

[0089] As shown in Figure 10, the 3D scene data SD11 generated by the 3D scene data generation unit 76 is rendered on the user terminal 11, and an image corresponding to the 3D scene constructed by the 3D scene construction unit 51 is generated and presented to the user.

[0090] 2 can also execute the processing of only a part of the components provided in the 3D scene construction unit 51. For example, the 3D scene construction unit 51 can change only the layout in the 3D scene or only the texture of the 3D asset based on newly input text data.

[0091] The database management unit 52 is made up of a 3D asset DB 91 , a layout DB 92 , and a layout caption generation unit 93 .

[0092] The layout caption generation unit 93 generates a rendered image by rendering the 3D scene data supplied from the 3D scene data generation unit 76. The layout caption generation unit 93 generates a caption corresponding to the rendered image, for example, using an AI model that receives the rendered image as input and outputs a caption.

[0093] FIG. 11 is a diagram showing an example of caption generation.

[0094] For example, the layout caption generation unit 93 generates a caption C21 by inputting a rendering image Pi21 corresponding to a 3D scene in which a bed, a side table, a pendant light, etc. are arranged, as shown in the upper part of Figure 11, into an AI model such as GPT-4V.

[0095] Also, for example, the layout caption generation unit 93 generates a caption C22 by inputting a rendering image Pi22 corresponding to a 3D scene in which a bed, a side table, etc. are arranged, as shown in the lower part of Figure 11, into an AI model such as GPT-4V.

[0096] The layout caption generating unit 93 registers a layout data set, which is a collection of 3D scene data, rendering images, captions, and asset lists, in the layout DB 92 .

[0097] It should be noted that instead of registering all of the 3D scene data generated by the 3D scene construction unit 51 in the layout DB 92, excellent 3D scene data may be selected and registered in the layout DB 92. As excellent 3D scene data is accumulated in the layout DB 92, the quality of the layout DB 92 improves, and the layout search unit 83 can stably acquire high-quality layouts as candidates for layouts in the 3D scene to be constructed.

[0098] The user may input a caption for the rendering image generated by the layout caption generation unit 93. When constructing a 3D scene, the scene appearance description input by the user may be registered as a caption in the layout DB 92.

[0099] Next, the processing performed by the 3D scene construction support system 1 having the above configuration will be described with reference to the flowchart of FIG.

[0100] In step S1, the input unit 31 of the user terminal 11 receives a scene appearance description input by the user, and also receives a designation of the size of the 3D scene to be constructed by the user.

[0101] In step S2, the asset list generating unit 71 of the server 12 generates an asset list based on the scene appearance description.

[0102] In step S3, the 3D asset determination unit 72 of the server 12 determines, based on the asset list, a 3D model of each 3D asset to be placed in the 3D scene to be constructed.

[0103] In step S4, the layout determination unit 73 of the server 12 determines the layout within the 3D scene to be constructed based on at least one of the scene appearance description, the size of the 3D scene specified by the user, and the asset list.

[0104] In step S5, the layout search unit 83 writes layout information for each 3D asset to be placed in the 3D scene to be constructed into the shape data of each 3D asset.

[0105] The processes of steps S6 and S7 are carried out in parallel with the processes of steps S4 and S5.

[0106] In step S6, the appearance description conversion unit 74 of the server 12 converts the scene appearance description into an asset appearance description.

[0107] In step S7, the 3D asset texture generation unit 75 of the server 12 determines, based on the asset appearance description, a texture to be applied to each 3D asset to be placed in the 3D scene to be constructed.

[0108] After the processes of steps S5 and S7 are performed, in step S8, the 3D scene data generating unit 76 of the server 12 compiles the 3D assets to generate 3D scene data.

[0109] In step S9, the drawing unit 32 of the user terminal 11 renders the 3D scene data to generate an image corresponding to the 3D scene.

[0110] In step S10, the display unit 33 of the user terminal 11 displays an image corresponding to the 3D scene.

[0111] As described above, in the 3D scene construction support system 1 of the present technology, 3D assets to be placed as 3D objects in the 3D scene to be constructed, layouts within the 3D scene to be constructed, and textures to be applied to the 3D assets are determined based on text data related to the appearance of the 3D scene in which the 3D objects are placed, and scene data for the 3D scene to be constructed is generated based on the results of determining the 3D assets, layouts, and textures. A user can easily construct a desired 3D scene simply by inputting text including scene appearance descriptions, even if they do not have specialized design knowledge or knowledge of operating DCC tools.

[0112] 13 is a block diagram showing a first modified example of the configuration of the 3D scene construction support system 1. In Fig. 13, the same components as those in Fig. 2 are denoted by the same reference numerals. Duplicate explanations will be omitted as appropriate.

[0113] The 3D scene construction support system 1 in Fig. 13 differs from the 3D scene construction support system 1 in Fig. 2 in that a 3D asset generation unit 101 is provided in the 3D asset determination unit 72 of the server 12, and in that a scene texture generation unit 102 is provided in the layout determination unit 73. The 3D scene construction support system 1 in Fig. 13 also differs from the 3D scene construction support system 1 in Fig. 2 in that a 3D asset caption generation unit 121 is provided in the database management unit 52.

[0114] The 3D asset generation unit 101 generates new candidates for each 3D asset to be placed in the 3D scene to be constructed, based on the asset list supplied from the asset list generation unit 71. For example, the 3D asset generation unit 101 uses an AI model such as DreamFusion, which takes text labels as input and 3D assets as output, to generate 3D assets corresponding to the names indicated by the text labels included in the asset list.

[0115] The 3D asset determination unit 72 determines one of the 3D asset candidates newly generated by the 3D asset generation unit 101 and the 3D asset candidates searched for by the 3D asset search unit 81 as the 3D asset to be placed in the 3D scene to be constructed.

[0116] Since the 3D asset generation unit 101 generates new 3D asset candidates, the 3D asset determination unit 72 can adopt (determine) a 3D asset that is not registered in the 3D asset DB 91 as the 3D asset to be placed in the 3D scene to be constructed.

[0117] The appearance description sentence conversion unit 74 receives the asset list and the scene appearance description sentence entered by the user as input, and uses an AI model that outputs a new scene appearance description sentence to generate a new scene appearance description sentence that describes the specific appearance of the entire 3D scene to be constructed. The appearance description sentence conversion unit 74 supplies the converted scene appearance description sentence to the scene texture generation unit 102.

[0118] The scene texture generation unit 102 generates texture candidates to be applied to each 3D asset supplied from the layout determination unit 73 in a batch based on the scene appearance description supplied from the appearance description conversion unit 74. For example, the scene texture generation unit 102 receives as input the scene appearance description, all 3D assets to be placed in the 3D scene to be constructed, and layout information written in each 3D asset, and generates texture candidates to be applied to each 3D asset in a batch using an AI model such as SceneTex that outputs textures to be applied to each 3D asset.

[0119] Like the 3D asset texture generation unit 75, the scene texture generation unit 102 functions as a determination unit that determines the generated texture candidates as textures to be applied to 3D assets to be placed in the 3D scene to be constructed. The scene texture generation unit 102 updates the texture images included in each 3D asset supplied from the layout determination unit 73 based on the generated textures. Note that the scene texture generation unit 102 does not necessarily need to update the texture images included in each 3D asset.

[0120] The scene texture generator 102 supplies the asset list and the 3D assets with updated textures to the 3D scene data generator 76 .

[0121] In addition, the server 12 can also generate multiple patterns of texture to be applied to each 3D asset without changing the layout within the 3D scene to be constructed, by simply re-executing the texture generation by the scene texture generation unit 102.

[0122] In the 3D scene construction unit 51 of FIG. 13, one of the 3D asset texture generation unit 75 and the scene texture generation unit 102 generates textures for each 3D asset included in the 3D scene to be constructed.

[0123] The 3D scene data generation unit 76 generates 3D scene data by combining the 3D assets supplied from the 3D asset texture generation unit 75 or the scene texture generation unit 102. The 3D scene data generation unit 76 supplies the generated 3D scene data to the 3D asset caption generation unit 121.

[0124] The 3D asset caption generation unit 121 renders each 3D asset included in the 3D scene data supplied from the 3D scene data generation unit 76 to generate a rendered image, and generates a caption corresponding to the rendered image by inputting the rendered image to an AI model such as GPT-4V. The 3D asset caption generation unit 121 registers an asset data set that compiles the 3D asset body, the rendered image, and the caption in the 3D asset DB 91.

[0125] The 3D asset caption generation unit 121 registers, for example, only the best 3D assets among the 3D assets generated by the 3D asset generation unit 101 in the 3D asset DB 91 .

[0126] As described above, in the 3D scene construction support system 1 of the present technology, a 3D model of a 3D asset may be newly generated.

[0127] Fig. 14 is a block diagram showing a second modified example of the configuration of the 3D scene construction support system 1. In Fig. 14, the same components as those in Fig. 13 are denoted by the same reference numerals. Duplicate explanations will be omitted as appropriate.

[0128] The 3D scene construction support system 1 in FIG. 14 differs from the 3D scene construction support system 1 in FIG. 13 in that a UI (User Interface) unit 151 is provided in the database management unit 52 of the server 12 .

[0129] The UI unit 151 presents an image corresponding to the 3D scene constructed by the 3D scene construction unit 51 to the administrator of the server 12 and accepts operation inputs by the administrator. The administrator can view the image presented by the UI unit 151, check the 3D scene constructed by the 3D scene construction unit 51, and designate an excellent layout or 3D asset within the 3D scene as a layout or 3D asset to be registered in the database.

[0130] The UI unit 151 supplies 3D scene data corresponding to a 3D scene having a layout specified by an administrator to the layout caption generation unit 93, and supplies 3D assets specified by the administrator to the 3D asset caption generation unit 121.

[0131] As described above, the administrator of the server 12 may select the layouts and 3D models to be registered in the database.

[0132] The asset data set registered in the 3D asset DB 91 may be used, for example, to train an AI model used to generate 3D assets in the 3D asset generation unit 101. Also, the layout data set registered in the layout DB 92 may be used, for example, to train an AI model used to generate layouts in the layout generation unit 82.

[0133] <3. Application Examples> - Example of use in previsualization in film production In film production, it is important for the director and cinematographer to share ideas based on the script, such as what kind of setting will be created and what angles will be used for filming. In film production, a process called previs (previsualization), which visualizes a scene from a film, is carried out so that the director and cinematographer can share ideas and so that people involved in the film can estimate the number of man-hours required for filming and consider the feasibility of a scene in the film.

[0134] The 3D scene construction support system 1 of the present technology can be applied to this previsualization support tool.

[0135] FIG. 15 is a diagram illustrating an example of using the 3D scene construction support system 1 in previsualization.

[0136] The 3D scene construction support system 1 can search for 3D scenes that are close to the contents of a movie script from among constructed 3D scenes based on the movie script, and present the searched 3D scenes to people involved in the movie.

[0137] The layout DB 92 stores 3D scene data corresponding to constructed 3D scenes, and a rendered image P41 obtained by rendering the 3D scene data and a caption C41 explaining the rendered image P41 are prepared in advance.

[0138] The layout search unit 83 acquires the movie script Sc1 as a scene appearance description sentence, and searches for 3D scenes that are similar to the content of the movie script Sc1 based on the similarity between the movie script Sc1 and the rendered image P41, for example, as shown by arrow #41. The layout search unit 83 can also search for 3D scenes that are similar to the content of the movie script Sc1 based on the similarity between the movie script Sc1 and the caption C41, for example, as shown by arrow #41.

[0139] FIG. 16 is a diagram illustrating another example of use of the 3D scene construction support system 1 in previs.

[0140] The 3D scene construction support system 1 can construct a new 3D scene that reflects the contents of a movie script based on the movie script, and present the constructed 3D scene to people involved in the movie.

[0141] 16, the 3D scene construction unit 51 acquires the movie script Sc1 as a scene appearance description sentence, and generates an asset list Li21 based on the movie script Sc1. The 3D scene construction unit 51 constructs a new 3D scene based on the asset list Li21, and outputs 3D scene data corresponding to the constructed 3D scene.

[0142] During the previsualization process, film professionals can easily create 3D scenes that reflect the contents of the movie script simply by inputting the movie script into the 3D scene construction support system 1.

[0143] - Example of use in game production and in-game map creation The 3D scene construction support system 1 of the present technology can be applied as a plug-in for game engines and DDC tools.

[0144] FIG. 17 is a diagram illustrating an example of using the 3D scene construction support system 1 as a plug-in for a DCC tool.

[0145] 17, an input UI 201 for inputting text related to a 3D scene is provided in the DCC tool executed on the user terminal 11. The input UI 201 includes, for example, a text box for inputting a theme for the entire 3D scene and a text box for inputting text including a scene appearance description.

[0146] When the user inputs text into each text box and then presses the "Generate" button located in the input UI 201, the server 12 starts constructing a 3D scene.

[0147] For example, first, as shown in #51 in Fig. 17, 3D assets to be placed in the 3D scene to be constructed are determined from asset data sets prepared in advance. Next, as shown in #52 in Fig. 17, textures to be applied to each 3D asset are determined. Finally, as shown in #53 in Fig. 17, the layout of the 3D scene to be constructed is determined from a group of layout data sets prepared in advance, thereby constructing a 3D scene that reflects the content of the text entered by the user.

[0148] The DCC tool is provided with an editing UI 202 that allows the user to edit the 3D scene. The editing UI 202 displays, for example, an image corresponding to the 3D scene constructed by the server 12. The user can edit the 3D scene while viewing the displayed image.

[0149] The 3D scenes constructed by the server 12 and the 3D scenes edited by the user are used, for example, as in-game maps. Game creators can create in-game maps simply by inputting text, which is expected to simplify game creation.

[0150] This technology can also be applied to a function that allows game players to create new in-game maps. Players can create in-game maps simply by entering text and experience the maps through the characters they control. Players can also fine-tune the positions of 3D assets placed on the in-game maps.

[0151] With this technology, 3D assets having polygon meshes and the like are placed in a 3D scene, making it easy to implement collision detection for the 3D assets, making this technology highly compatible with game applications.

[0152] - Example of computer configuration The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program that constitutes the software is installed from a program recording medium into a computer built into dedicated hardware or a general-purpose personal computer.

[0153] FIG. 18 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0154] A CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0155] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.

[0156] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0157] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0158] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0159] ·others

[0160] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0161] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0162] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0163] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0164] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0165] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0166] Example of Combination of Configurations The present technology can also be configured as follows.

[0167] (1) An information processing system comprising: a determination unit that determines, based on text data regarding the appearance of a 3D scene in which a 3D object is to be placed, a 3D asset to be placed as the 3D object in the 3D scene to be constructed, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset, and a data generation unit that generates scene data for the 3D scene to be constructed based on the determination results of the 3D asset, the layout, and the texture by the determination unit. (2) The information processing system described in (1), further comprising: a list generation unit that generates a list of the 3D assets based on the text data, wherein the determination unit generates at least one of the 3D asset, the layout, and the texture based on the list. (3) The information processing system described in (2), wherein the determination unit determines, as the 3D asset to be placed in the 3D scene to be constructed, one of 3D asset candidates searched, based on the list, from a first database in which already generated 3D assets are registered, or a 3D asset candidate newly generated based on the list. (4) The information processing system according to (3), wherein the determination unit searches for candidates for 3D assets to be placed in the 3D scene to be constructed based on the names of the 3D assets described in the list. (5) The information processing system according to (4), wherein the determination unit searches for candidates for 3D assets to be placed in the 3D scene to be constructed based on the similarity between the names of the 3D assets described in the list and the names of the generated 3D assets. (6) The information processing system according to (4), wherein the determination unit searches for candidates for 3D assets to be placed in the 3D scene to be constructed based on the similarity between the names of the 3D assets described in the list and a rendered image in which the generated 3D assets are rendered. (7) The information processing system according to any of (3) to (6), wherein the determination unit newly generates candidates for 3D assets to be placed in the 3D scene to be constructed using an AI model that receives the names of the 3D assets described in the list as input and outputs the 3D assets.(8) The information processing system according to any one of (3) to (7), further comprising a database management unit that registers the 3D assets included in the scene data generated by the data generation unit in the first database. (9) The information processing system according to any one of (2) to (8), wherein the determination unit determines, as the layout in the 3D scene to be constructed, one of the layout candidates searched for based on the text data from a second database in which layouts in the already constructed 3D scene are registered, and the layout candidate newly generated based on the list. (10) The information processing system according to (9), wherein the determination unit searches for the layout candidate in the 3D scene to be constructed based on the similarity between the text data and a rendered image in which the already constructed 3D scene is rendered. (11) The information processing system according to (9) or (10), wherein the determination unit searches for the layout candidate in the 3D scene to be constructed based on the similarity between the text data and a caption in a rendered image in which the already constructed 3D scene is rendered. (12) The information processing system according to any one of (9) to (11), wherein the determination unit searches for layout candidates in the 3D scene to be constructed based on the list. (13) The information processing system according to any one of (9) to (12), wherein the determination unit newly generates layout candidates in the 3D scene to be constructed using an AI model that receives the list as input and outputs information indicating the placement positions of the 3D assets in the 3D scene to be constructed. (14) The information processing system according to any one of (9) to (13), further comprising a database management unit that registers the layout in the 3D scene corresponding to the scene data generated by the data generation unit in the second database. (15) The information processing system according to any one of (2) to (14), wherein the list generation unit generates the list using an AI model that receives the text data as input and outputs the list.(16) The information processing system according to any one of (1) to (15), further comprising a conversion unit that converts the text data related to the appearance of the 3D scene to be constructed into text data related to the appearance of each of the 3D assets, wherein the determination unit determines, as the texture to be applied to the 3D asset, a texture candidate newly generated using an AI model that receives the converted text data as an input and outputs the texture. (17) The information processing system according to any one of (1) to (16), wherein the determination unit receives the text data related to the appearance of the 3D scene to be constructed as an input and uses an AI model that outputs the texture as an output, and determines, as the texture to be applied to the 3D asset. (18) The information processing system according to any one of (1) to (17), wherein the determination unit determines the 3D asset, the layout, and the texture based on the text data input by a user and the size of the 3D scene specified by the user. (19) An information processing method comprising: determining, based on text data regarding the appearance of a 3D scene in which a 3D object is to be placed, a 3D asset to be placed as the 3D object in the 3D scene to be constructed, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset, and generating scene data for the 3D scene to be constructed based on the determination results of the 3D asset, the layout, and the texture. (20) A program for causing a computer to execute processes including: determining, based on text data regarding the appearance of a 3D scene in which a 3D object is to be placed, a 3D asset to be placed as the 3D object in the 3D scene to be constructed, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset, and generating scene data for the 3D scene to be constructed based on the determination results of the 3D asset, the layout, and the texture.

[0168] 1 3D scene construction support system, 11 user terminal, 12 server, 31 input unit, 32 drawing unit, 33 display unit, 51 3D scene construction unit, 52 database management unit, 71 asset list generation unit, 72 3D asset list determination unit, 73 layout determination unit, 74 appearance description sentence conversion unit, 75 3D asset texture generation unit, 76 3D scene data generation unit, 81 3D asset search unit, 82 layout generation unit, 83 layout search unit, 91 3D asset DB, 92 layout DB, 93 layout caption generation unit, 101 3D asset generation unit, 102 scene texture generation unit, 121 3D asset caption generation unit, 161 UI unit

Claims

1. An information processing system comprising: a determination unit that determines, based on text data regarding the appearance of a 3D scene in which a 3D object is to be placed, a 3D asset to be placed in the 3D scene to be constructed as the 3D object, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset; and a data generation unit that generates scene data for the 3D scene to be constructed based on the determination results of the 3D asset, the layout, and the texture by the determination unit.

2. The information processing system of claim 1, further comprising a list generation unit that generates a list of the 3D assets based on the text data, wherein the determination unit generates at least one of the 3D assets, the layout, and the texture based on the list.

3. The information processing system of claim 2, wherein the determination unit determines one of the 3D asset candidates searched based on the list from a first database in which the already generated 3D assets are registered, and the 3D asset candidates newly generated based on the list, as the 3D asset to be placed in the 3D scene to be constructed.

4. The information processing system according to claim 3, wherein the determination unit searches for candidates for the 3D assets to be placed in the 3D scene to be constructed based on the names of the 3D assets described in the list.

5. The information processing system of claim 4, wherein the determination unit searches for candidates for the 3D asset to be placed in the 3D scene to be constructed based on the similarity between the name of the 3D asset described in the list and the name of the 3D asset that has already been generated.

6. The information processing system of claim 4, wherein the determination unit searches for candidates for the 3D asset to be placed in the 3D scene to be constructed based on the similarity between the name of the 3D asset described in the list and a rendered image in which the generated 3D asset is rendered.

7. The information processing system of claim 3, wherein the determination unit uses an AI model that takes the names of the 3D assets described in the list as input and outputs the 3D assets to generate new candidates for the 3D assets to be placed in the 3D scene to be constructed.

8. The information processing system according to claim 3, further comprising a database management unit that registers the 3D assets included in the scene data generated by the data generation unit in the first database.

9. The information processing system of claim 2, wherein the determination unit determines one of the layout candidates searched based on the text data from a second database in which layouts in the 3D scenes that have already been constructed are registered, and the layout candidates newly generated based on the list, as the layout in the 3D scene to be constructed.

10. An information processing system as described in claim 9, wherein the determination unit searches for candidates for the layout within the 3D scene to be constructed based on the similarity between the text data and a rendering image of the constructed 3D scene.

11. The information processing system of claim 9, wherein the determination unit searches for layout candidates within the 3D scene to be constructed based on the similarity between the text data and a caption of a rendered image in which the constructed 3D scene is rendered.

12. The information processing system according to claim 9, wherein the determination unit searches for candidates for the layout within the 3D scene to be constructed based on the list.

13. The information processing system of claim 9, wherein the determination unit uses an AI model that receives the list as input and outputs information indicating the placement positions of the 3D assets within the 3D scene to be constructed, to newly generate candidates for the layout within the 3D scene to be constructed.

14. The information processing system according to claim 9, further comprising a database management unit that registers the layout in the 3D scene corresponding to the scene data generated by the data generation unit in the second database.

15. An information processing system according to claim 2, wherein the list generation unit generates the list using an AI model that receives the text data as input and outputs the list.

16. The information processing system of claim 1, further comprising a conversion unit that converts the text data relating to the appearance of the 3D scene to be constructed into text data relating to the appearance of each of the 3D assets, wherein the determination unit determines, as the texture to be applied to the 3D asset, a newly generated texture candidate using an AI model that uses the converted text data as input and outputs the texture.

17. The information processing system of claim 1, wherein the determination unit determines, as the texture to be applied to the 3D asset, a newly generated texture candidate using an AI model that receives as input the text data relating to the appearance of the 3D scene to be constructed and outputs the texture.

18. An information processing system as described in claim 1, wherein the determination unit determines the 3D assets, the layout, and the texture based on the text data input by the user and the size of the 3D scene specified by the user.

19. An information processing method comprising: determining, based on text data regarding the appearance of a 3D scene in which a 3D object is to be placed, a 3D asset to be placed in the 3D scene to be constructed as the 3D object, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset; and generating scene data for the 3D scene to be constructed based on the results of determining the 3D asset, the layout, and the texture.

20. A program for causing a computer to execute processes including: determining, based on text data regarding the appearance of a 3D scene in which a 3D object is to be placed, a 3D asset to be placed in the 3D scene to be constructed as the 3D object, a layout within the 3D scene to be constructed, and a texture to be applied to the 3D asset; and generating scene data for the 3D scene to be constructed based on the results of determining the 3D asset, the layout, and the texture.

Images