Generation program, generation method, and information processing device

Abstract

An information processing device (100) receives an input from a user and generates a task that can be executed on the basis of the input with respect to a 3D space generated on the basis of the input. The information processing device (100) evaluates the 3D space on the basis of the result of executing the generated task in the 3D space and the input, and corrects the 3D space on the basis of the evaluation result.

Description

GENERATION PROGRAM, GENERATION METHOD, AND INFORMATION PROCESSING APPARATUS

[0001] The present invention relates to a generation program and the like.

[0002] It is expected that technologies related to metaverse, digital twins, robotics research, etc. will be applied to areas where interaction in 3D space is important. Interaction refers to interactions based on the physicality of subjects (humans, robots, etc.) active in 3D space. Interactions include moving, manipulating, viewing, and interacting in 3D space.

[0003] For example, conventional technologies for creating 3D spaces include Holodeck, 3D-GPT, and LI3D.

[0004] Holodeck accepts an input prompt and uses Large Language Models (LLMs) to infer the objects and layout of the space. Based on procedural 3D spatial algorithms, Holodeck incrementally constructs a 3D indoor space where interactions such as predefined actions, manipulations, and observations are possible.

[0005] 3D-GPT generates 3D spaces using an agent system in which a Task Dispatch agent, a Conceptualization agent, and a Modeling agent work together. The Task Dispatch agent selects the API (Application Programming Interface) required for space generation. The Conceptualization agent creates text that describes the space expected by the user. The Modeling agent generates Python code that can be executed by the space generator.

[0006] LI3D uses a learning-based space generator to generate a 3D space from the input text. LI3D also uses large multimodal models (LMMs) to evaluate the 3D space based on observed images of the overall appearance and structure of the 3D space and the input text. If the LMMs determine that the 3D space needs improvement, they provide feedback prompts to adjust the 3D space.

[0007] Chunyi Sun et al. “3D-GPT: PROCEDURAL 3D MODELING WITH LARGE LANGUAGE MODELS” arXiv:2310.12945v1 [cs.CV] 19 Oct 2023

[0008] However, the above-mentioned conventional techniques do not easily allow for the generation of a 3D space that enables the user to perform the intended interaction.

[0009] For example, in order to obtain a 3D space that allows a user to "interact within the 3D space" as intended using conventional technology, the user must evaluate whether or not such interaction is possible and edit the 3D space. This work not only places a heavy burden on the user, but also makes the quality of the 3D space subject to personal preference.

[0010] In one aspect, the present invention aims to provide a generation program, a generation method, and an information processing device that can easily generate a 3D space that enables interactions intended by a user.

[0011] In the first proposal, a computer performs the following process: the computer accepts user input, generates a task that can be executed based on the input for a 3D space generated based on the input, evaluates the 3D space based on the input and the results of executing the generated task in the 3D space, and modifies the 3D space based on the evaluation result.

[0012] It is possible to easily create a 3D space that allows the user to have the desired interaction.

[0013] FIG. 1 is a diagram illustrating the overall processing flow of an information processing device according to this embodiment. FIG. 2 is a diagram illustrating processing for estimating a user's intention. FIG. 3 is a diagram illustrating processing for generating a 3D space. FIG. 4 is a diagram (1) illustrating an example of a 3D space generated by a space generation unit. FIG. 5 is a diagram illustrating processing for generating and breaking down a requirements definition. FIG. 6 is a diagram illustrating processing for performing evaluation and feedback. FIG. 7 is a diagram illustrating processing for iterating autonomous evaluation and improvement. FIG. 8 is a diagram (2) illustrating an example of a 3D space generated by a space generation unit. FIG. 9 is a diagram illustrating an example of autonomous 3D space generation through agent collaboration. FIG. 10 is a diagram (3) illustrating an example of a 3D space generated by a space generation unit. FIG. 11 is a diagram illustrating an example of an observed image. FIG. 12 is a diagram illustrating evaluation results by a space evaluation unit. FIG. 13 is a functional block diagram illustrating the configuration of an information processing device according to this embodiment. FIG. 14 is a flowchart illustrating the processing procedure of information processing according to this embodiment. FIG. 15 is a diagram illustrating an example of the hardware configuration of a computer that realizes functions similar to those of the information processing device according to the embodiment.

[0014] Hereinafter, embodiments of the generating program, generating method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0015] The processing of an information processing device 100 according to this embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram for explaining the overall processing flow of the information processing device according to this embodiment. A user 10 operates an input device to give an instruction 11 in natural language to the information processing device 100. The instruction 11 in natural language is an ambiguous instruction such as "Please create a private office suitable for research and development work." Upon receiving the instruction 11 in natural language, the information processing device 100 automatically generates a 3D space that enables the interaction intended by the user 10.

[0016] For example, when generating a 3D space, the information processing device 100 uses the agent 50, the embodiment agent 60, the external knowledge information 141, the space generation unit 151, the task generation unit 152, and the space evaluation unit 153.

[0017] The external knowledge information 141 includes domain knowledge information 141a, space generation API information 141b, task generation API information 141c, and space evaluation API information 141d.

[0018] The domain knowledge information 141a is information related to the user's work, information on the user's attributes, preferences, etc. For example, the information related to the work may include text information on what has been said in the past by the user's superior or a person equivalent to a member of the management team, text information on the current business policy (work style, etc.) of the department to which the user belongs, text information on the short-term, medium-term, or long-term plans of the department to which the user belongs, etc.

[0019] The domain knowledge information 141a also includes the layout of the convenience store, the attributes and preferences of the persona, business knowledge, etc. The domain knowledge information 141a is not limited to the above information, and may include information on other domain knowledge related to the user.

[0020] The space generation API information 141b includes a plurality of APIs (hereinafter referred to as space generation APIs) used to generate a 3D space. For example, the space generation APIs include an API for generating a layout of the 3D space, an API for adjusting a color scheme, an API for rendering, and the like.

[0021] The task generation API information 141c has a plurality of APIs (hereinafter, task generation APIs) for causing the embodiment agent 60 to execute tasks. For example, the APIs for executing tasks include an API for observing an image of a 3D space, a route movement API for moving along a route specified in the 3D space, and the like.

[0022] The space evaluation API information 141d includes a plurality of APIs for evaluating a 3D space (hereinafter referred to as space evaluation APIs). For example, the APIs for evaluating a 3D space include an API for evaluating a pseudo-sensation, an API for evaluating movement efficiency, and the like.

[0023] Here, the agent 50 performs the following processing using a large-scale language model (hereinafter referred to as LLM). When the agent 50 receives an instruction 11 in natural language from the user 10, it estimates the user's intention 20e based on external knowledge information 141. The agent 50 outputs a space generation instruction 21 generated based on the user's intention 20e to the space generation unit 151. The space generation unit 151 generates a 3D space 21a based on the space generation instruction 21.

[0024] The agent 50 generates a space requirement definition 22 based on the user's intention 20e, and generates a task generation instruction 23 and a space evaluation instruction 24 based on the space requirement definition 22. The agent 50 outputs the task generation instruction 23 to a task generation unit 152, and outputs the space evaluation instruction 23 to a space evaluation unit 153. The task generation unit 152 generates a task 23a that can be executed by the embodiment agent 60 based on the task generation instruction 23.

[0025] The realization agent 60 attempts to execute executable tasks 23a in the 3D space using the 3D space simulator 40. The realization agent 60 outputs task execution results 25 to the space evaluation unit 153. The space evaluation unit 153 outputs space evaluation results 26 to the agent 50 based on the task execution results 25 and space evaluation instructions 24.

[0026] The agent 50 obtains the space evaluation result 26, and if the 3D space needs to be improved, modifies the space generation instructions 21 and outputs the modified space generation instructions 21 to the space generation unit 151 again.

[0027] The information processing device 100 repeatedly executes the above process until a predetermined termination condition is met.

[0028] The above has described the overall processing flow of the information processing device 100. Here, the processes executed by the information processing device 100 include a process for estimating a user's intention, a process for generating a 3D space, a process for generating and breaking down a requirements definition, a process for conducting evaluation and providing feedback, and a process for repeating autonomous evaluation and improvement. Each process will be described in more detail below.

[0029] First, the process of estimating a user's intention will be described. FIG. 2 is a diagram for explaining the process of estimating a user's intention. For example, in the example shown in FIG. 2, a user 10 operates an input device or the like to input a natural language instruction, such as "Please create a private office suitable for research and development work," to the information processing device 100. In addition to the natural language instruction, the user 10 may also input a user ID that uniquely identifies the user 10.

[0030] When the agent 50 receives an instruction in natural language from the user 10, it generates domain knowledge 20a using the LLM. For example, the agent 50 acquires the domain knowledge 20a from the LLM by inputting a prompt that instructs the generation of domain knowledge, the prompt including "instruction in natural language," "external knowledge information 141," and "user ID" into the LLM. In the example shown in FIG. 2, the domain knowledge 20a includes research and development business data, user attributes / preferences, a business policy recommending telework, etc.

[0031] When the agent 50 generates the domain knowledge 20a, it uses the LLM to generate a space generation section specification 20b, a task generation section specification 20c, and a space evaluation section specification 20d that correspond to the domain knowledge 20a.

[0032] The agent 50 inputs a prompt that sets the domain knowledge 20a and the space generation API information 141b and instructs the LLM to acquire a space generation API corresponding to the domain knowledge 20a. The agent 50 acquires the space generation API corresponding to the domain knowledge 20a from the LLM and sets the acquired space generation API as the "space generation unit specifications 20b." In the example shown in FIG. 2, the space generation unit specifications 20b include a layout generation API, a color adjustment API, etc. The LLM is assumed to have learned the correspondence between the domain knowledge and the space generation API.

[0033] The agent 50 inputs a prompt that sets the domain knowledge 20a and task generation API information 141c and instructs the LLM to acquire a task generation API corresponding to the domain knowledge 20a. The agent 50 acquires the task generation API corresponding to the domain knowledge 20a from the LLM and defines the acquired task generation API as the "task generation unit specification 20c." In the example shown in FIG. 2, the task generation unit specification 20c includes an image observation API, a route movement API, etc. The LLM is assumed to have learned the correspondence between the domain knowledge and the task generation API.

[0034] The agent 50 inputs a prompt that sets the domain knowledge 20a and the spatial evaluation API information 141d and instructs the LLM to acquire the spatial evaluation API corresponding to the domain knowledge 20a. The agent 50 acquires the spatial evaluation API corresponding to the domain knowledge 20a from the LLM and sets the acquired spatial evaluation API as the "spatial evaluation unit specifications 20d." In the example shown in FIG. 2, the spatial evaluation unit specifications 20d include a pseudo-perceptual evaluation API, a movement efficiency evaluation API, and the like. The LLM is assumed to have learned the correspondence between the domain knowledge and the spatial evaluation API.

[0035] The agent 50 generates the domain knowledge 20a, the space generation specification 20b, the task generation specification 20c, and the space evaluation specification 20d, and then generates the user's intention 20e. For example, the agent 50 inputs a prompt that sets the domain knowledge 20a and instructs the generation of the user's intention (spatial configuration / appearance / interaction) that can be inferred from the domain knowledge 20a into the LLM, and generates the user's intention 20e.

[0036] 2, the user's intention 20e includes "a personal room equipped with facilities for conducting research and development work through telework" and "looking around the room from my desk gives me inspiration and makes me feel energized." The user's intention 20e also includes "good access to office equipment, allowing me to print and answer phone calls without stress."

[0037] Next, the process of generating a 3D space will be described. Fig. 3 is a diagram for explaining the process of generating a 3D space. The agent 50 generates a space generation instruction 21 based on domain knowledge 20a, a space generation specification 20b, a task generation specification 20c, a space evaluation specification 20d, and a user's intention 20e.

[0038] For example, the agent 50 generates a prompt that sets domain knowledge 20a, space generation section specifications 20b, task generation section specifications 20c, space evaluation section specifications 20d, and user intention 20e, and that instructs the generation of a "space generation instruction." The agent 50 inputs the generated prompt into the LLM to obtain a space generation instruction 21 from the LLM.

[0039] In the example shown in Figure 3, the space generation instructions 21 include "Generate a 4x5 (m) room," "Generate a layout for teleworking," "Search / install / adjust appearance of 3D assets," etc.

[0040] The agent 50 inputs a space generation instruction 21 to the space generation unit 151. The space generation unit 151 generates a 3D space 21a based on the space generation instruction 21. FIG. 4 is a diagram (1) showing an example of a 3D space generated by the space generation unit. The 3D space 21a corresponds to "generate a 4x5 (m) room," "generate a layout for teleworking," and "search / install / adjust the appearance of a 3D asset."

[0041] Next, an example of a process for generating and breaking down a requirements definition will be described. FIG. 5 is a diagram for explaining the process for generating and breaking down a requirements definition. The agent 50 generates a space requirements definition 22 based on a user's intention 20e, a task generation section specification 20c, and a space evaluation section specification 20d. The space requirements definition 22 is a textual definition of the requirements that the 3D space must satisfy in relation to the user's intention.

[0042] For example, the agent 50 generates a prompt that sets the user's intention 20e and instructs the generation of a "spatial requirement definition." The agent 50 inputs the generated prompt into the LLM to obtain the spatial requirement definition 22 from the LLM.

[0043] 5, the spatial requirement definition 22 includes (requirement 1), (requirement 2), (requirement 3), etc. For example, the contents of (requirement 1), (requirement 2), and (requirement 3) are as follows:

[0044] (Requirement 1) The equipment necessary for work (PC, desk, chair, office equipment, etc.) is provided. (Requirement 2) The elements of the view when looking around the room from the desk are composed of colors and objects that contribute to "inspiration and vitality." (Requirement 3) Efficiency when moving back and forth from the desk to various office equipment is good.

[0045] The agent 50 may define requirements taking into consideration multiple personas by setting multiple personas as prompts. Requirements definition taking into consideration multiple personas may include observation from an adult's perspective, observation from a child's perspective, heading straight to the destination, making a detour to the destination, etc.

[0046] The agent 50 generates a task generation instruction 23 based on the space requirement definition 22 and the task generation unit specification 20c. When the task generation instruction 23 is input to the task generation unit 152, a task to be attempted in the space by the embodiment agent 60 to evaluate the conformance of the requirements is generated.

[0047] For example, the agent 50 generates a prompt that sets the spatial requirement definition 22 and the task generation unit specification 20c, and instructs the generation of a "task generation instruction." The agent 50 inputs the generated prompt into the LLM, thereby obtaining a task generation instruction 23 from the LLM.

[0048] 5, the task generation instruction 23 includes (task 1), (task 2), (task 3), etc. For example, the contents of (task 1), (task 2), and (task 3) are as follows:

[0049] (Task 1) Try to move to all the equipment necessary for work. (Task 2) Observe the surroundings 360 degrees from the desk position. (Task 3) Move from the desk to various office equipment.

[0050] For example, (Task 1) is obtained from the LLM by a prompt that sets (Requirement 1) of the spatial requirement definition 22 and the task generation unit specification 20c. (Task 2) is obtained from the LLM by a prompt that sets (Requirement 2) of the spatial requirement definition 22 and the task generation unit specification 20c. (Task 3) is obtained from the LLM by a prompt that sets (Requirement 3) of the spatial requirement definition 22 and the task generation unit specification 20c.

[0051] The agent 50 generates a space evaluation instruction 24 based on the space requirement definition 22, the task generation instruction 23, and the specifications of the space evaluation unit 153. The space evaluation instruction 24 is input to the space evaluation unit 153 and is used when performing space evaluation.

[0052] For example, the agent 50 generates a prompt that sets the spatial requirement definition 22, task generation instructions 23, and spatial evaluation section specifications 20d, and instructs the generation of a "spatial evaluation instruction." The agent 50 inputs the generated prompt into the LLM, thereby obtaining a spatial evaluation instruction 24 from the LLM.

[0053] 5, the space evaluation instructions 24 include (Evaluation 1), (Evaluation 2), (Evaluation 3), etc. For example, the contents of (Evaluation 1), (Evaluation 2), and (Evaluation 3) are as follows:

[0054] (Evaluation 1) Evaluate whether all equipment is accessible. (Evaluation 2) Evaluate whether the observed image meets the requirements. (Evaluation 3) Evaluate the path length from the desk to the office equipment.

[0055] For example, (Rating 1) is obtained from the LLM by a prompt that sets (Requirement 1) of the spatial requirement definition 22, (Task 1) of the task generation instruction 23, and the spatial evaluation unit specification 20d. (Rating 2) is obtained from the LLM by a prompt that sets (Requirement 2) of the spatial requirement definition 22, (Task 2) of the task generation instruction 23, and the spatial evaluation unit specification 20d. (Rating 3) is obtained from the LLM by a prompt that sets (Requirement 3) of the spatial requirement definition 22, (Task 3) of the task generation instruction 23, and the spatial evaluation unit specification 20d.

[0056] Next, the process of performing evaluation and feedback will be described. FIG. 6 is a diagram for explaining the process of performing evaluation and feedback. The agent 50 inputs a task generation instruction 23 and the 3D space 21a to the task generation unit 152. The task generation unit 152 generates a task 23a that can be executed by the embodiment agent 60 in the 3D space 21a based on the task generation instruction 23 and the 3D space 21a. For example, the executable tasks 23a include (executable task 1), (executable task 2), and (executable task 3). The contents of (executable task 1), (executable task 2), and (executable task 3) are as follows:

[0057] (Executable task 1) Access facility A, access facility B, ... (Executable task 2) Take an observation image X1 in the X direction from the desk position, take an observation image Y1 in the Y direction from the desk position, ... (Executable task 3) Measure the path length from the desk to facility A, measure the path length from the desk to facility B, ...

[0058] For example, the task generation unit 152 generates (executable task 1) based on (task 1) in the task generation instruction 23 and the 3D space 21 a. The task generation unit 152 generates (executable task 2) based on (task 2) in the task generation instruction 23 and the 3D space 21 a. The task generation unit 152 generates (executable task 3) based on (task 3) in the task generation instruction 23 and the 3D space 21 a.

[0059] The task generation unit 152 may generate executable tasks based on predefined rules, or may generate executable tasks by inputting a prompt that sets the task content and 3D space into the LLM.

[0060] Here, the realization agent 60 uses the 3D space simulator 40 to execute the executable tasks 23a in the 3D space 21a, respectively, and generates task execution results 25. The following describes the processing when the realization agent 60 executes (executable task 1), (executable task 2), and (executable task 3).

[0061] The embodiment agent 60 executes (executable task 1) and determines whether facility A present in the 3D space 21a is accessible, and whether facility B present in the 3D space 21a is accessible. For example, if facility A is accessible but facility B is not accessible, the embodiment agent 60 generates (output 1) as shown below.

[0062] (Output 1) Facility A is accessible, facility B is inaccessible, ...

[0063] The materialization agent 60 executes (executable task 2), captures an observation image X1 in the X direction from the position of the desk in the 3D space 21a, and also captures an observation image Y1 in the Y direction from the position of the desk in the 3D space 21a. As a result, the materialization agent 60 generates (output 2) as shown below.

[0064] (Output 2) Photographed image X1, photographed image Y1, ...

[0065] The embodiment agent 60 executes (executable task 3) and measures the path length from the desk in the 3D space 21a to device C, and also measures the path length from the desk in the 3D space 21a to device D. As a result, the embodiment agent 60 generates (output 3) as shown below.

[0066] (Output 3) Travel distance to device C, travel distance to device D, ...

[0067] The materialization agent 60 outputs the generated (output 1), (output 2), and (output 3) to the space evaluation unit 153 as task execution results 25. As described above, the materialization agent 60 uses the 3D space simulator 40 to execute each executable task in the 3D space 21 a, which corresponds to estimating the "interaction in the 3D space" intended by the user 10.

[0068] When the space evaluation unit 153 acquires the task execution result 25, it evaluates the task execution result 25 based on the space evaluation instruction 24, and generates a space evaluation result 26.

[0069] For example, a case will be described in which the space evaluation unit 153 evaluates (output 1) of the task execution result 25. The space evaluation unit 153 evaluates (output 1) using (evaluation 1) of the space evaluation instruction 24. If (output 1) satisfies the requirements of (evaluation 1) (all facilities are accessible), the space evaluation unit 153 evaluates the result as "Excellent". If (output 1) partially satisfies the requirements of (evaluation 1) (some facilities are accessible), the space evaluation unit 153 evaluates the result as "Good". If (output 1) does not satisfy the requirements of (evaluation 1) (all facilities are inaccessible), the space evaluation unit 153 evaluates the result as "Poor".

[0070] In the example shown in FIG. 6, the spatial evaluation unit 153 generates the following (Result 1) because (Output 1) partially satisfies the requirements of (Evaluation 1).

[0071] (Result 1) △<Good>: Some facilities cannot be accessed

[0072] The following describes a case where the spatial evaluation unit 153 evaluates (output 2) of the task execution result 25. The spatial evaluation unit 153 evaluates (output 2) using (evaluation 2) of the spatial evaluation instruction 24. If (output 2) satisfies the requirements of (evaluation 2) (all observed images satisfy the image characteristics set in advance), the spatial evaluation unit 153 evaluates the result as "Excellent (Excellent)". If (output 2) partially satisfies the requirements of (evaluation 2) (some observed images satisfy the image characteristics set in advance), the spatial evaluation unit 153 evaluates the result as "Good (Good)". If (output 2) does not satisfy the requirements of (evaluation 2) (all observed images do not satisfy the image characteristics set in advance), the spatial evaluation unit 153 evaluates the result as "Poor (Poor)".

[0073] In the example shown in FIG. 6, the space evaluation unit 153 determines that (output 2) does not satisfy the requirements of (evaluation 2), and generates the following (result 2).

[0074] (Result 2) × <Poor>: Appearance does not meet the requirements

[0075] The following describes a case where the spatial evaluation unit 153 evaluates (output 3) of the task execution result 25. The spatial evaluation unit 153 evaluates (output 3) using (evaluation 3) of the spatial evaluation instruction 24. If (output 3) satisfies the requirements of (evaluation 3) (all movement distances are shorter than the predetermined path length), the spatial evaluation unit 153 evaluates the result as "Excellent". If (output 3) partially satisfies the requirements of (evaluation 3) (some movement distances are shorter than the predetermined path length), the spatial evaluation unit 153 evaluates the result as "Good". If (output 3) does not satisfy the requirements of (evaluation 3) (all movement distances are longer than the predetermined path length), the spatial evaluation unit 153 evaluates the result as "Poor".

[0076] In the example shown in FIG. 6, the space evaluation unit 153 determines that (output 3) satisfies the requirements of (evaluation 3), and generates the following (result 3).

[0077] (Result 3) Excellent: Easy access to office equipment from desk

[0078] The space evaluation unit 153 outputs the space evaluation results 26 including (Result 1), (Result 2), and (Result 3) to the agent 50.

[0079] Next, a process of repeating autonomous evaluation and improvement will be described. FIG. 7 is a diagram for explaining the process of repeating autonomous evaluation and improvement. The agent 50 determines whether or not to improve the 3D space 21a based on the space evaluation results 26. For example, the agent 50 determines to make an improvement if at least one of all results included in the space evaluation results 26 includes "x <Poor>". On the other hand, the agent 50 determines not to make an improvement if none of all results included in the space evaluation results 26 includes "x <Poor>".

[0080] If the agent 50 determines that an improvement is to be made, it modifies the space generation instructions 21 and outputs the modified space generation instructions 21 to the space generation unit 151. For example, the agent 50 may obtain the modified space generation instructions 21 from the LLM by inputting a prompt that sets the space evaluation result 26 and the previous space generation instructions 21, etc., to the LLM, and that requests a space generation instruction that improves the space evaluation result 26.

[0081] The agent 50 fixes the contents of the space requirement definition 22, task generation instruction 23, and space evaluation instruction 24. As a result, the processes described with reference to Figures 3 and 6 are executed again, and the agent 50 again obtains the space evaluation result 26. If the information processing device 100 determines that the agent 50 should make improvements, it repeats the above processes.

[0082] On the other hand, if the agent 50 determines not to make any improvements, it presents the 3D space 21 a generated by the space generation unit 151 to the user 10 and waits until an instruction is input by the user 10 .

[0083] The above has described the process executed by the information processing device 100 to estimate user intentions, generate a 3D space, generate and break down requirements definitions, conduct evaluation and provide feedback, and repeat autonomous evaluation and improvement.

[0084] In this way, the information processing device 100 according to this embodiment generates executable tasks 23a for the 3D space 21a generated based on instructions 11 in natural language input by the user 10. The information processing device 100 generates a space evaluation result 26 based on a task execution result 25 in which the embodiment agent 60 executes the executable tasks 23a in the 3D space 21a. The information processing device 100 modifies the 3D space 21a based on the space evaluation result 26.

[0085] This makes it easy to create a 3D space that allows the user to have the desired interaction.

[0086] In the above example, the instruction in natural language input by the user 10 to the information processing device 100 was "Please create a private office suitable for research and development work." Below, an example of processing by the information processing device 100 when an instruction in another natural language is input to the information processing device 100 will be described.

[0087] First, a process performed by the information processing device 100 when the user 10 inputs a command in natural language to the information processing device 100, such as "Please create a modern style living room."

[0088] When the agent 50 receives an instruction in natural language from the user 10, it uses the LLM to generate domain knowledge 20a, a space generation specification 20b, a task generation specification 20c, and a space generation specification 20d. The agent 50 also generates a user's intention 20e.

[0089] For example, the user's intention 20e may be "The user is involved in research and development work. He / she would like a living room where he / she can feel 'improved motivation, creativity, and productivity' just by spending time there."

[0090] The process by which the agent 50 generates the domain knowledge 20a, the space generation specification 20b, the task generation specification 20c, the space generation specification 20d, and the user's intention 20e is the same as the process described in FIG.

[0091] Next, the agent 50 generates space generation instructions 21 based on the domain knowledge 20a, the space generation unit specifications 20b, the task generation unit specifications 20c, the space generation unit specifications 20d, and the user's intention 20e. The process by which the agent 50 generates the space generation instructions 21 is the same as the process described in FIG. 3.

[0092] The agent 50 inputs a space generation instruction 21 to the space generation unit 151. The space generation unit 151 generates a 3D space 21a based on the space generation instruction 21. FIG. 8 is a diagram (2) showing an example of a 3D space generated by the space generation unit. For example, the 3D space 21a shown in FIG. 8 is the 3D space that is initially generated when a natural language instruction such as "Please generate a modern-style living room" is input.

[0093] Next, the agent 50 generates a space requirement definition 22 based on the user's intention 20e, the task generation section specification 20c, and the space evaluation section specification 20d. For example, the space requirement definition 22 includes the following (requirements):

[0094] (Requirement) The visual elements when looking around the room are composed of color schemes and objects that contribute to "improving motivation, creativity, and productivity."

[0095] The agent 50 generates a task generation instruction 23 based on the spatial requirement definition 22 and the task generation unit specification 20c. For example, the task generation instruction 23 includes the following (task):

[0096] (Task) Observe the surroundings from the center of the room

[0097] The agent 50 generates a space evaluation instruction 24 based on the space requirement definition 22, the task generation instruction 23, and the space evaluation unit specification 20d. For example, the space evaluation instruction 24 includes the following (evaluation):

[0098] (Evaluation) LMM is used to evaluate whether the observed image has a visual appearance that "contributes to improving motivation, creativity, and productivity."

[0099] The process in which the agent 50 generates the spatial requirement definition 22, the task generation instruction 23, and the space evaluation instruction 24 is the same as the process described with reference to FIG.

[0100] Next, the information processing device 100 performs a process of performing evaluation and feedback using the embodiment agent 60. The process of performing evaluation and feedback executed by the information processing device 100 is similar to the process described with reference to FIG.

[0101] Next, the information processing device 100 executes a process of repeating autonomous evaluation and improvement. The process of repeating autonomous evaluation and improvement executed by the information processing device 100 is similar to the process described with reference to FIG.

[0102] Next, a description will be given of how the user 10 autonomously generates a 3D space in cooperation with the agent 50. Fig. 9 is a diagram showing an example of autonomous generation of a 3D space through agent cooperation.

[0103] 9, the user 10 gives a natural language instruction 11-1, "Please create a modern style living room," to the agent 50 of the information processing device 100. The agent 50 generates an instruction for task creation and the like based on the natural language instruction 11-1, and generates a 3D space 21 a-1.

[0104] The realization agent 60 of the information processing device 100 uses a 3D space simulator to perform executable tasks on the 3D space 21 a-1 and generate task execution results. The agent 50 determines whether to make improvements based on the results of evaluating the space using the task execution results. For example, if the agent 50 determines that improvements should be made, it generates a task generation instruction 21 and generates the 3D space 21 a-2.

[0105] For example, "add small items in an accent color" and "move the floor lamp and add a potted plant" are set as task generation instructions 21. The task generation instructions 21 are modified and updated by the agent 50 in order to improve motivation and creativity and space efficiency.

[0106] The realization agent 60 uses a 3D space simulator to perform executable tasks on the 3D space 21 a-2 and generate task execution results. The agent 50 determines whether to make improvements based on the space evaluation results using the task execution results. For example, if the agent 50 determines not to make improvements to the 3D space 21 a-2, it presents the 3D space 21 a-2 to the user 10 and waits until an instruction is input by the user 10.

[0107] The user 10 refers to the 3D space 21 a-2 and inputs additional natural language instructions 11-2, such as "Change the color scheme to something cute," "Make the cafe table bigger," and "Add a window."

[0108] The agent 50 generates instructions for task creation and the like based on the instructions 11-2 in natural language, and generates a 3D space 21a-3.

[0109] The realization agent 60 uses a 3D space simulator to perform executable tasks on the 3D space 21 a-3 and generate task execution results. The agent 50 determines whether to make improvements based on the results of evaluating the space using the task execution results.

[0110] As described with reference to FIG. 9, the user 10 cooperates with the agent 50 to generate an autonomous 3D space.

[0111] Next, we will explain the processing of the information processing device 100 when the user 10 inputs a natural language instruction to the information processing device 100 saying, "Please create a convenience store interior with shelves that have a lineup of products that will appeal to people of all ages and genders."

[0112] When the agent 50 receives an instruction in natural language from the user 10, it uses the LLM to generate domain knowledge 20a, a space generation specification 20b, a task generation specification 20c, and a space generation specification 20d. The agent 50 also generates a user's intention 20e.

[0113] For example, the user's intention 20e may be "I want to have product shelves with lineups for adults at the higher positions and lineups for children at the lower positions."

[0114] The process by which the agent 50 generates the domain knowledge 20a, the space generation specification 20b, the task generation specification 20c, the space generation specification 20d, and the user's intention 20e is the same as the process described in FIG.

[0115] Next, the agent 50 generates space generation instructions 21 based on the domain knowledge 20a, the space generation unit specifications 20b, the task generation unit specifications 20c, the space generation unit specifications 20d, and the user's intention 20e. The process by which the agent 50 generates the space generation instructions 21 is the same as the process described in FIG. 3.

[0116] The agent 50 inputs a space generation instruction 21 to the space generation unit 151. The space generation unit 151 generates a 3D space 21a based on the space generation instruction 21. FIG. 10 is a diagram (3) showing an example of a 3D space generated by the space generation unit. For example, the 3D space 21a shown in FIG. 10 is a 3D space that is initially generated when a natural language instruction, "Please create the interior of a convenience store with shelves stocked with products that will appeal to people of all ages and genders," is input.

[0117] When generating the 3D space 21a of FIG. 10, the space generation unit 151 may further use constraints on the floor layout of the convenience store, product data, and product display.

[0118] Next, the agent 50 generates a space requirement definition 22 based on the user's intention 20e, the task generation section specification 20c, and the space evaluation section specification 20d. For example, the space requirement definition 22 includes the following (requirement 1), (requirement 2), (requirement 3), and (requirement 4).

[0119] (Requirement 1) When observing shelves with attributes corresponding to the attributes of persona A, the desire to purchase is strongly induced. (Requirement 2) When observing shelves with attributes corresponding to the attributes of persona B, the desire to purchase is strongly induced. (Requirement 3) When observing shelves with attributes corresponding to the attributes of persona C, the desire to purchase is strongly induced. (Requirement 4) When observing shelves with attributes corresponding to the attributes of persona D, the desire to purchase is strongly induced.

[0120] Personas A to D are, for example, as follows:

[0121] Persona A: Female, in her 20s, health conscious Persona B: Male, in his 30s, wants to buy lunch Persona C: Male, 10 years old, wants to buy sweets Persona D: Male, 5 years old, visits the store with his mother

[0122] The agent 50 generates a task generation instruction 23 based on the spatial requirement definition 22 and the task generation unit specification 20c. For example, the task generation instruction 23 includes (task 1), (task 2), (task 3), and (task 4) as shown below.

[0123] (Task 1) Observe product shelf A with the physical attributes of persona A. (Task 2) Observe product shelf A with the physical attributes of persona B. (Task 3) Observe product shelf A with the physical attributes of persona C. (Task 4) Observe product shelf A with the physical attributes of persona D.

[0124] The agent 50 generates space evaluation instructions 24 based on the space requirement definition 22, task generation instructions 23, and space evaluation unit specifications 20b. For example, the space evaluation instructions 24 include (Evaluation 1), (Evaluation 2), (Evaluation 3), and (Evaluation 4) as shown below.

[0125] (Evaluation 1) Evaluate using LLM whether the observed image has an appearance that will motivate persona A to purchase. (Evaluation 2) Evaluate using LLM whether the observed image has an appearance that will motivate persona B to purchase. (Evaluation 3) Evaluate using LLM whether the observed image has an appearance that will motivate persona C to purchase. (Evaluation 4) Evaluate using LLM whether the observed image has an appearance that will motivate persona D to purchase.

[0126] The process in which the agent 50 generates the spatial requirement definition 22, the task generation instruction 23, and the space evaluation instruction 24 is the same as the process described with reference to FIG.

[0127] Subsequently, the information processing device 100 performs a process of performing evaluation and feedback using the embodiment agent 60. For example, the embodiment agent 60 executes executable tasks in the 3D space 21 a shown in Fig. 10 , captures observed images Z1 and Z2 as shown in Fig. 11 , and outputs them to the space evaluation unit 153 as task execution results 25.

[0128] 11 is a diagram showing an example of observed images. For example, observed image Z1 is an image captured from an adult's perspective (the perspective of personas A and B). Observed image Z2 is an image captured from a child's perspective (the perspective of personas C and D).

[0129] For example, in observed image Z1, the top shelf contains bottled water, energy drinks, and soft drinks. The middle shelf contains boxes of herbal tea and instant soup. The bottom shelf contains boxes of crackers of various flavors, granola bars, and cereal boxes.

[0130] In image Z2, the top shelf contains more boxes of crackers and granola bars, the middle shelf contains bags of potato chips and popcorn, and the bottom shelf contains boxes of gummy candy and colorful cereal boxes with cartoon characters on them.

[0131] For example, the space evaluation unit 153 evaluates whether the appearance of the observed images Z1 and Z2 is likely to attract purchasing interest for each persona using LLM, based on the observed images Z1 and Z2.

[0132] The spatial evaluation unit 153 obtains the attraction scores for the observed images Z1 and Z2 by inputting a prompt that sets persona A and observed images Z1 and Z2 and instructs the LLM to calculate attraction scores (0 to 100 points) for the observed images Z1 and Z2. The spatial evaluation unit 153 similarly generates prompts for personas B to D and obtains the scores for the observed images Z1 and Z2.

[0133] 12 is a diagram illustrating the evaluation results of the space evaluation unit. For example, the attraction score of observed image Z1 for persona A is "80 points," and the attraction score of observed image Z2 is "20 points." Observation image Z1 has healthy drinks and foods, so the attraction score of observed image Z1 for persona A is high. Observation image Z2 has few healthy products, so the attraction score of observed image Z2 for persona A is low.

[0134] The attraction score of observed image Z1 for persona B is "60 points," and the attraction score of observed image Z2 is "40 points." Observed image Z1 has instant foods and drinks that are suitable for lunch, so the attraction score of observed image Z1 for persona B is high. Observed image Z2 has many sweets that are not suitable for lunch, so the attraction score of observed image Z2 for persona B is low.

[0135] The attraction score of observed image Z1 for persona C is "30 points," and the attraction score of observed image Z2 is "80 points." Observed image Z1 has several attractive sweets, but observed image Z2 has more children's sweets and colorful cereals compared to observed image Z1. For this reason, the attraction score of observed image Z1 for persona C is low, and the attraction score of observed image Z2 for persona C is high.

[0136] The attraction score of observed image Z1 for persona D is "20 points," and the attraction score of observed image Z2 is "90 points." Observed image Z1 has few sweets aimed at small children, so the attraction score of observed image Z1 for persona D is low. Observed image Z2 includes products that are attractive to small children, such as colorful cereal, so the attraction score of observed image Z2 for persona D is high.

[0137] The space evaluation unit 153 outputs the space evaluation result 26 including the evaluation result shown in Fig. 12 to the agent 50. Other explanations of the evaluation and feedback process executed by the information processing device 100 are the same as those of the process explained in Fig. 6 .

[0138] Next, the information processing device 100 executes a process of repeating autonomous evaluation and improvement. For example, the agent 50 determines to make an improvement if the number of attraction points of the observed image Z1 of persona A and persona B is not 50 or more, or if the number of attraction points of the observed image Z2 of persona A and persona B is not 50 or more. If the agent 50 determines to make an improvement, it modifies the space generation instruction 21 and outputs the modified space generation instruction 21 to the space generation unit 151.

[0139] On the other hand, the agent 50 determines not to make an improvement if the number of attraction points in the observation image Z1 of persona A and persona B is 50 or more and the number of attraction points in the observation image Z2 of persona A and persona B is 50 or more. If the agent 50 determines not to make an improvement, it presents the 3D space 21a generated by the space generation unit 151 to the user 10 and waits until an instruction is input by the user 10.

[0140] The other processes performed by the information processing device 100 to repeat autonomous evaluation and improvement are similar to those described with reference to FIG.

[0141] Next, an example of the configuration of an information processing device 100 that executes the above-mentioned processing will be described. Fig. 13 is a functional block diagram showing the configuration of an information processing device according to this embodiment. As shown in Fig. 13, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0142] The communication unit 110 executes data communication with an external device or the like via a network. The communication unit 110 is a network interface card (NIC) or the like. For example, the communication unit 110 receives external knowledge information 141 or the like from the external device.

[0143] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the information processing device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc. The user 10 operates the input unit 120 to input instructions 11 in natural language.

[0144] The display unit 130 is a display device that displays information output from the control unit 150. For example, the display unit 130 displays the 3D space 21a and the like.

[0145] The storage unit 140 has external knowledge information 141. The storage unit 140 is a storage device such as a memory.

[0146] The external knowledge information 141 includes domain knowledge information 141 a, space generation API information 141 b, task generation API information 141 c, and space evaluation API information 141 d. The domain knowledge information 141 a, space generation API information 141 b, task generation API information 141 c, and space evaluation API information 141 d are the same as those described in FIG. 1.

[0147] The control unit 150 includes a space generation unit 151, a task generation unit 152, a space evaluation unit 153, an agent execution unit 154, and an embodied agent execution unit 155. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0148] The space generation unit 151 generates a 3D space 21a based on a space generation instruction 21 generated by the agent 50. The space generation unit 151 may use an external tool such as Holddeck or 3D-GPT when generating the 3D space 21a. Other explanations regarding the space generation unit 151 are the same as those regarding the space generation unit 151 described above.

[0149] The task generation unit 152 generates a task 23a that can be executed in the 3D space 21a by the embodiment agent 60 based on the task generation instruction 23 and the 3D space 21a. Other details regarding the task generation unit 152 are the same as those described above regarding the task generation unit 152.

[0150] When the space evaluation unit 153 acquires the task execution result 25 from the embodiment agent 60, it evaluates the task execution result 25 based on the space evaluation instruction 24, and generates a space evaluation result 26. Other explanations regarding the space evaluation unit 153 are the same as those regarding the space evaluation unit 153 described above.

[0151] The agent execution unit 154 operates as the agent 50 described above. The agent 50 may be a Task Dispatch agent, a Conceptualization agent, or a Modeling agent described in 3D-GPT. The description of the agent 50 is the same as the description of the agent 50 described above.

[0152] The embodied agent execution unit 155 operates as the above-mentioned embodied agent 60. The embodied agent is an embodied agent, etc. The explanation regarding the embodied agent 60 is the same as the explanation regarding the embodied agent 60 described above.

[0153] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 14 is a flowchart showing the processing procedure of the information processing procedure according to this embodiment. As shown in Fig. 14, the agent 50 of the information processing device 100 receives an instruction in natural language from the user 10 (step S101).

[0154] The agent 50 generates domain knowledge 20a, space generation unit specifications 20b, task generation unit specifications 20c, space evaluation unit specifications 20d, and user intention 20e using the LLM (step S102). The agent 50 generates space generation instructions 21 and inputs the space generation instructions 21 to the space generation unit 151 to generate a 3D space 21a (step S103).

[0155] The agent 50 generates a space requirement definition 22 based on the user's intention 20e, the task generation section specification 20c, and the space evaluation section specification 20d (step S104).

[0156] The agent 50 generates a task generation instruction 23 based on the spatial requirement definition 22 and the task generation unit specification 20c (step S105). The agent 50 generates a space evaluation instruction 24 based on the spatial requirement definition 22, the task generation instruction 23, and the space evaluation unit specification 20d (step S106).

[0157] The agent 50 inputs the task generation instruction 23 and the 3D space 21a to the task generation unit 152, and generates an executable task 23a (step S107).

[0158] The realization agent 60 uses the 3D space simulator 40 to execute each executable task 23a in the 3D space 21a, and generates a task execution result 25 (step S108). Upon receiving the task execution result 25, the space evaluation unit 153 evaluates the task execution result 25 based on the space evaluation instruction 24, and generates a space evaluation result 26 (step S109).

[0159] The agent 50 determines whether to improve the 3D space 21a (step S110) based on the space evaluation result 26. If the agent 50 does not improve the 3D space 21a (step S110, No), the agent 50 displays the 3D space on the display unit 130 (step S111).

[0160] If the agent 50 determines to make an improvement (Yes at step S110), the process proceeds to step S112. The agent 50 generates a new space generation instruction 21 and inputs the space generation instruction 21 to the space generation unit 151, thereby generating a 3D space 21 a (step S112).

[0161] The agent 50 reuses the previously generated requirement definition, task generation instruction, and space evaluation instruction (step S113), and proceeds to step S107.

[0162] The information processing device 100 repeatedly executes the process shown in FIG. 14 every time it receives an instruction in natural language from the user 10 .

[0163] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 receives user input, generates tasks that can be executed based on the input for a 3D space generated based on the input, evaluates the 3D space based on the input and the results of executing the generated tasks in the 3D space, and modifies the 3D space based on the evaluation results. This makes it possible to easily generate a 3D space that enables the user to perform the interactions intended.

[0164] For example, the agent 50 can infer the "interaction in 3D space" that the user intends based on vague instructions from the user, and autonomously evaluate and improve the 3D space so that a 3D space that allows that interaction can be obtained. This makes it easy to obtain a 3D space that allows the interaction that the user intends.

[0165] When the agent 50 of the information processing device 100 receives a process 11 in natural language, the agent 50 generates a space requirement definition 22 based on external knowledge information 141, and generates a task generation instruction 23 based on the space requirement definition 22. The agent 50 inputs the task generation instruction 23 to the task generation unit 152, thereby efficiently generating a task that the embodiment agent 60 should attempt in the space in order to evaluate the suitability of the requirements.

[0166] The information processing device 100 generates a space evaluation instruction 24 based on a space requirement definition 22 and a task generation instruction 23. This makes it possible to evaluate whether or not the user can execute the intended interaction.

[0167] Next, a description will be given of an example of the hardware configuration of a computer that realizes the same functions as the above-described information processing device 100. Fig. 15 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0168] 15, computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. Computer 200 also has a communication device 204 that exchanges data with external devices via a wired or wireless network, and an interface device 205. Computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. Each of devices 201 to 207 is connected to a bus 208.

[0169] The hard disk drive 207 stores a space generation program 207 a, a task generation program 207 b, a space evaluation program 207 c, an agent execution program 207 d, and an embodied agent execution program 207 e. The CPU 201 reads out each of the programs 207 a to 207 e and loads them into the RAM 206.

[0170] The space generation program 207a functions as a space generation process 206a. The task generation program 207b functions as a task generation process 206b. The space evaluation program 207c functions as a space evaluation process 206c. The agent execution program 207d functions as an agent execution process 206d. The embodied agent execution program 207e functions as an embodied agent execution process 206e.

[0171] The processing of the space generation process 206a corresponds to the processing of the space generation unit 151. The processing of the task generation process 206b corresponds to the processing of the task generation unit 152. The processing of the space evaluation process 206c corresponds to the processing of the space evaluation unit 153. The processing of the agent execution process 206d corresponds to the processing of the agent execution unit 154. The processing of the materialized agent execution process 206e corresponds to the processing of the materialized agent execution unit 155.

[0172] It should be noted that each of the programs 207a to 207e does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207e.

[0173] 50 Agent 60 Materialized agent 100 Information processing device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 External knowledge information 141a Domain knowledge information 141b Space generation API information 141c Task generation API information 141d Space evaluation API information 150 Control unit 151 Space generation unit 152 Task generation unit 153 Space evaluation unit 154 Agent execution unit 155 Materialized agent execution unit

Claims

1. A generation program that causes a computer to perform the following processes: accept user input; generate tasks that can be executed based on the input for a 3D space generated based on the input; evaluate the 3D space based on the input and the results of executing the generated tasks in the 3D space; and modify the 3D space based on the evaluation results.

2. The generation program according to claim 1, characterized in that an agent runs on the computer, the agent generates instructions to generate the task based on requirements for the 3D space obtained from the input, and the task generation process generates the task to be executed by an embodied agent in the 3D space based on the instructions to generate the task.

3. The generation program described in claim 2, characterized in that the agent further generates the contents of an evaluation of the 3D space based on the requirements of the 3D space obtained from the input and the instructions for generating the task, and the evaluation process further uses the contents of the evaluation of the 3D space to evaluate the 3D space.

4. The generation program according to claim 2, wherein the agent further executes a process of inferring the user's intention based on the input and external knowledge information related to the user.

5. The generation program according to claim 4, wherein the agent further generates requirements for the 3D space to satisfy the user's intention, and generates instructions for generating the task based on the requirements for the 3D space.

6. A generation method characterized by a computer executing the following processes: accepting user input; generating a task that can be executed based on the input for a 3D space generated based on the input; evaluating the 3D space based on the input and the results of executing the generated task in the 3D space; and modifying the 3D space based on the evaluation results.

7. The generation method according to claim 6, wherein an agent operates on the computer, the agent generates instructions for generating the task based on requirements for the 3D space obtained from the input, and the task generation process generates the task to be executed by an embodied agent in the 3D space based on the instructions for generating the task.

8. The generation method described in claim 7, wherein the agent further generates content for an evaluation of the 3D space based on requirements for the 3D space obtained from the input and instructions for generating the task, and the evaluation process further uses the content for the evaluation of the 3D space to evaluate the 3D space.

9. The generation method according to claim 7, wherein the agent further executes a process of inferring the user's intention based on the input and external knowledge information related to the user.

10. The generation method according to claim 9, wherein the agent further generates requirements for the 3D space to satisfy the user's intention, and generates instructions for generating the task based on the requirements for the 3D space.

11. An information processing device having a control unit that executes the following processes: accepts user input; generates a task that can be executed based on the input for a 3D space generated based on the input; evaluates the 3D space based on the input and the results of executing the generated task in the 3D space; and modifies the 3D space based on the evaluation results.

12. The information processing device according to claim 11, characterized in that the control unit operates an agent, the agent generates instructions to generate the task based on requirements for the 3D space obtained from the input, and the task generation process generates the task to be executed by the embodied agent in the 3D space based on the instructions to generate the task.

13. The information processing device described in claim 12, characterized in that the agent further generates the contents of an evaluation of the 3D space based on the requirements of the 3D space obtained from the input and the instructions for generating the task, and the evaluation process further uses the contents of the evaluation of the 3D space to evaluate the 3D space.

14. The information processing apparatus according to claim 12, wherein said agent further executes a process of inferring the intention of said user based on said input and external knowledge information related to said user.

15. The information processing device according to claim 14, wherein the agent further generates requirements for the 3D space to satisfy the user's intention, and generates instructions for generating the task based on the requirements for the 3D space.

Images