Material generating device, material generating method, and program

Abstract

This material generating device comprises a language input unit that accepts input of a query indicating material to be generated, an information refining unit that infers a document among materials indicated by the query, by causing two dialog-based document generation models to carry out a dialog by repeatedly performing reciprocal input and output, with the query as an initial input, and an image generating unit that generates an image on the basis of the document resulting from the inference by the information refining unit, wherein preference information including preferences or thought patterns of a specific individual is added to the dialog between the two dialog-based document generation models.

Description

Material generation device, material generation method, and program

[0001] The present invention relates to a material generation device, a material generation method, and a program.

[0002] In recent years, generative artificial intelligence (AI) using deep learning has been developed, such as chatGPT (registered trademark) by OpenAI (registered trademark) and Stable Diffusion by the CompVis group at the University of Munich. Patent Literature 1 discloses an information processing device that, when responding to a given question with an answer derived by deep learning, generates an answer based on multiple pieces of information including text and images related to the question.

[0003] Japanese Patent Application Publication No. 2022-112541

[0004] However, the information processing device described in Patent Document 1 has a problem in that it may not be possible to obtain materials that correspond to the characteristics of a person.

[0005] The present disclosure has been made in consideration of the above circumstances, and provides a material generation device, a material generation method, and a program that can generate materials according to the characteristics of a person.

[0006] This disclosure has been made to solve the above-mentioned problems, and one aspect of the present disclosure is a material generation device that includes an input unit that accepts input of a query that indicates the material to be generated, and a dialogue management unit that infers the material indicated by the query by having a dialogue between multiple AI (Artificial Intelligence) agents based on the query, and in which the dialogue between the multiple AI agents includes information indicating the characteristics of a person corresponding to at least one of the multiple AI agents.

[0007] Another aspect of this disclosure is the above-mentioned document generation device, wherein the plurality of AI agents are a plurality of dialogue-attached document generation models, the input unit is a language input unit that accepts input of the query, the dialogue management unit is an information refining unit that uses the query as an initial input and causes the plurality of dialogue-attached document generation models to repeatedly exchange input and output with each other to infer a document from the document indicated by the query, and the device is further equipped with an image generation unit that generates an image based on the document resulting from the inference by the information refining unit, and adds preference information including the preferences or thought patterns of a specific individual to the dialogue generated by the plurality of dialogue-attached document generation models.

[0008] Another aspect of this disclosure is the above-mentioned document generation device, further comprising: an image input unit that accepts input of an image that serves as an example of the document to be generated; and an image recognition unit that inputs the image input to the image input unit into a trained model to infer features of the image input to the image input unit and use them as image-derived context features, and the image generation unit generates an image based on the image-derived context features in addition to the document of the inference result by the information refinement unit.

[0009] Another aspect of the present disclosure is the above-mentioned material generation device, further comprising: a natural language processing unit that inputs the query to a trained model, infers features of the query as language-derived context features, and generates a generated query by combining at least a portion of the query, the language-derived context features, the image-derived context features, the preference information, and a document inferred as a result of the inference by the information refining unit; and a document generation unit that inputs the generated query to a dialogue-attached document generation model, infers a document requested by the generated query, and sets the inference result as a command query, and the image generation unit inputs the command query to the trained image generation model, infers an image indicated by the command query.

[0010] Another aspect of the present disclosure is the above-mentioned material generation device, wherein the dialogue between the multiple AI agents includes generation of a hypothesis by one AI agent and criticism or critique of the generated hypothesis by the other AI agents.

[0011] Another aspect of this disclosure is the above-mentioned material generation device, wherein the plurality of AI agents correspond one-to-one to a plurality of people, and each of the plurality of AI agents includes in its dialogue personal information indicating the characteristics of the corresponding person among the plurality of people.

[0012] Another aspect of this disclosure is the above-mentioned material generation device, wherein at least one of the plurality of characters is a virtual character, and the person information of the virtual character includes information indicating the role of the virtual character.

[0013] Another aspect of this disclosure is the above-mentioned material generation device, which includes a dialogue strategy memory unit that stores dialogue strategy information indicating a dialogue process and a person to be dialogued, and the dialogue management unit causes the AI ​​agent selected from the plurality of AI agents to dialogue in accordance with the dialogue process indicated by the dialogue strategy information obtained from the dialogue strategy memory unit.

[0014] Another aspect of this disclosure is the above-mentioned material generation device, which includes a memory unit that stores information related to the material, and each of the multiple AI agents acquires reference information related to the dialogue from the memory unit and includes the acquired reference information in the dialogue.

[0015] Another aspect of this disclosure is the above-mentioned material generation device, wherein the input unit accepts input of multimodal data related to the material, the dialogue management unit causes a dialogue between the multiple AI agents based on the query and the multimodal information, and the dialogue between the multiple AI agents is multimodal.

[0016] Another aspect of this disclosure is the above-mentioned material generation device, which includes an output generation unit that generates the material based on the results of the dialogue between the multiple AI agents, and an evaluation and verification unit that evaluates and verifies the material generated by the output generation unit.

[0017] Another aspect of this disclosure is a material generation method having a first step of receiving an input of a query that indicates material to be generated, and a second step of inferring the material indicated by the query by having a plurality of AI agents interact with each other based on the query, and adding information indicating characteristics of a person, including a fictional person, corresponding to at least one of the plurality of AI agents to the interaction between the plurality of AI agents.

[0018] Another aspect of this disclosure is a program for causing a computer to function as an input unit that accepts input of a query that indicates materials to be generated, and a dialogue management unit that infers materials indicated by the query by having multiple AI agents interact with each other based on the query, and the program adds information indicating characteristics of a person, including a virtual person, corresponding to at least one of the multiple AI agents to the interaction between the multiple AI agents.

[0019] The material generation device, material generation method, and program of the present disclosure can generate materials according to the characteristics of a person.

[0020] FIG. 1 is a schematic block diagram showing the configuration of a document generation device 10 according to a first embodiment of the present disclosure. FIG. 2 is a schematic block diagram showing an example configuration of an image recognition unit 110 in the same embodiment. FIG. 3 is a schematic block diagram showing an example configuration of a natural language processing unit 130 in the same embodiment. FIG. 4 is a schematic block diagram showing an example configuration of an information refining unit 140 in the same embodiment. FIG. 5 is a schematic block diagram showing an example configuration of an image generation unit 170 in the same embodiment. FIG. 6 is a flowchart illustrating an example operation of the document generation device 10 in the same embodiment. FIG. 7 is a schematic diagram showing a use case of the document generation device 10 in the same embodiment. FIG. 8 is a schematic block diagram showing the configuration of a document generation device 20 according to a second embodiment of the present disclosure. FIG. 9 is a schematic block diagram showing the configuration of a hypothesis generation unit 208 in the same embodiment. FIG. 10 is an explanatory diagram illustrating the hardware configuration of the document generation device 10 according to each embodiment.

[0021] First Embodiment A first embodiment of the present disclosure will now be described with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of a material generation device 10 according to a first embodiment of the present disclosure. The material generation device 10 is a device that generates materials consisting of text and images in response to a user request. The material generation device 10 may be realized by one or more computers reading and executing a program. The multiple computers may be connected to each other so as to be able to communicate with each other via a general-purpose or dedicated network. The material generation device 10 includes an image input unit 100, an image recognition unit 110, a language input unit 120 (input unit), a natural language processing unit 130, an information refining unit 140 (dialogue management unit), a memory unit 150, a document generation unit 160, an image generation unit 170, a display processing unit 180, and a display unit 190.

[0022] The image input unit 100 accepts user input of an image D1 serving as an example of the material to be generated. The image input unit 100 may accept user input using an input device such as a keyboard or a mouse, or may accept the image D1 input by receiving it from another device. The image recognition unit 110 inputs the image D1 input to the image input unit 100 into a trained model and infers its features to generate image-derived context features D2. The image-derived context features D2 may be identified by identifying features with significant weights from the features of the inference results. The trained model may be, for example, a hierarchical network such as the Image Encoder of OpenAI (registered trademark)'s CLIP (Contrastive Language-Image Pre-training) or ViT (Vision Transformer). The trained model may be provided by the image recognition unit 110 or by another device connected to the material generation device 10 via a network or the like. Identifying features with significant weights may be performed by an attention mechanism such as multi-head attention.

[0023] The language input unit 120 accepts user input of language (query D3) specifying the material to be generated. The language input unit 120 may accept user input using an input device such as a keyboard or a mouse, or may accept input of query D3 by receiving it from another device. This query D3 is text data and may include text data that serves as an example of the material to be generated. The natural language processing unit 130 inputs the query D3 input to the language input unit 120 into a trained model to infer its features and use them as language-derived context features. The language-derived context features may be features identified as having important weights from the features of the inference result. Furthermore, the natural language processing unit 130 generates a generated query D4, which is language (text data) that combines at least a portion of the query D3, the language-derived context features, the image-derived context features D2, complementary information D5, which is preference information described below, and a summary sentence D6, which is a document resulting from inference by the information refining unit 140. The generated query D4 may be a prompt that causes the document generating unit 160 to infer the document indicated by the generated query D4.

[0024] The trained model of the natural language processing unit 130 may be, for example, a hierarchical network such as a CLIP text encoder. The trained model may be provided by the natural language processing unit 130 itself, or by another device connected to the material generation device 10 via a network or the like. Furthermore, the identification of features with significant weights may be performed by an attention mechanism. The trained model of the natural language processing unit 130 is paired with the trained model of the image recognition unit 110, and is trained to infer, as natural language features describing an image, features similar to those inferred from the image by the trained model of the image recognition unit 110.

[0025] The information refining unit 140 uses the query D3 as an initial input and causes two dialogue-based document generation models to repeatedly exchange input and output to infer a document from the materials specified by the query D3. The information refining unit 140 adds preference information, including a specific individual's preferences or thought patterns, to the dialogue between these two dialogue-based document generation models. These two dialogue-based document generation models may be, for example, a dialogue-based document generation network such as chatGPT (registered trademark) by OpenAI (registered trademark). The information refining unit 140 summarizes the inferred document and inputs the result as a summary sentence D6 to the natural language processing unit 130. The information refining unit 140 also inputs the result of the inferred document summary as a linguistic answer D9 to the display processing unit 180.

[0026] Note that one of these two dialogue-with document generation models may be required to infer detailed documents, and the other may be required to infer concise documents. Furthermore, the input to at least one of the two dialogue-with document generation models may be the output of the other, to which a dialogue / inference history D8 stored in the storage unit 150 has been added. The dialogue / inference history D8 may include an inference history D7 that the information refining unit 140 has stored in the storage unit 150 as the inference results from the two dialogue-with document generation models. Furthermore, these dialogue-with document generation models may be provided in the information refining unit 140, or may be provided in another device that is communicatively connected to the material generation device 10 via a network or the like.

[0027] Furthermore, the dialogue / inference history D8 stored in the storage unit 150 may include preference information (e.g., the individual's posting history to a social network service (SNS)) that includes the preferences or thought patterns of a specific individual. The specific individual may be the user who requested the generation of the material, or a person or group of people who are expected to read the generated material. Furthermore, the information refining unit 140 may input the preference information or thought patterns extracted from the preference information by an encoder, expressed as high-dimensional features, or the decoded high-dimensional features, to the natural language processing unit 130 as complementary information D5.

[0028] The document generation unit 160 inputs the generation query D4 into a dialogue-attached document generation model, infers the document requested by the generation query D4, and sets the inference result as a command query D10. This dialogue-attached document generation model may be, for example, a large-scale language model such as chatGPT (registered trademark) or BERT. Furthermore, this dialogue-attached document generation model may be provided in the document generation unit 160, or may be provided in another device communicatively connected to the material generation device 10 via a network or the like. Note that the command query D10 may be a prompt for causing the image generation unit 170 to infer an image corresponding to the document requested by the generation query D4.

[0029] The image generation unit 170 generates an image based on at least the document inferred by the information refinement unit 140. Furthermore, the generation of this image may be based on image-derived context features. Specifically, the image generation unit 170 inputs the command query D10 generated using this information into a trained image generation model to infer the image indicated by the command query D10, and the inference result is an image answer D11. This trained image generation model may be, for example, an image synthesis model that generates images from text, such as Stable Diffusion developed by the CompVis group at the University of Munich, or a combination of a CLIP text encoder and a corresponding diffusion decoder.

[0030] The display processing unit 180 acquires the verbal answers D9 from the information refining unit 140 and the image answers D11 from the image generating unit 170, and displays them on the display unit 190. The display unit 190 is a display such as a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0031] FIG. 2 is a schematic block diagram illustrating an example configuration of the image recognition unit 110 according to this embodiment. The image recognition unit 110 includes a trained deep feature extraction unit 111 and a context analysis unit 112. The trained deep feature extraction unit 111 inputs an image D1 into a trained model, such as CLIP's Image Encoder or ViT, to infer its features. The context analysis unit 112 identifies features with significant weights from the features inferred by the trained deep feature extraction unit 111 using an attention mechanism, and defines these features as image-derived context features D2. Self-attention and cross-attention are used as the attention mechanisms. Self-attention learns the relationships between elements at different positions within sequence data and assigns weights. In other words, self-attention learns similar locations and features within the same image space and assigns weights. Cross-attention can learn correlations between different inputs; in the case of images, it can learn similar features between different images given as input, and in the case of language, it can obtain global features between multiple texts given as input. Cross-attention can also acquire important features of different modalities, such as language and images. Both self-attention and cross-attention have the advantage that the model itself can automatically acquire relationships between different elements and automatically learn the importance of information.

[0032] FIG. 3 is a schematic block diagram showing an example configuration of the natural language processing unit 130 in this embodiment. The natural language processing unit 130 includes a learned deep feature extraction unit 131, a context analysis unit 132, and a prompt generation unit 133. The learned deep feature extraction unit 131 inputs the query D3 into a learned model such as a CLIP text encoder to infer its features. The context analysis unit 132 identifies features that have important weights according to the attention mechanism from among the features inferred by the learned deep feature extraction unit 131, and sets these as language-derived context features. Self-attention and cross-attention are used as the attention mechanism.

[0033] The prompt generation unit 133 inputs the context features obtained by combining the language-derived context features and the image-derived context features D2 into the trained model to infer a context language, which is text data representing the context features. The combination of the language-derived context features and the image-derived context features D2 may be, for example, element-wise addition or multiplication of the language-derived context features and the image-derived context features D2, or may be concatenation as a combination of different tensors, or may involve learning the relationship between elements of different modalities by cross-attention.

[0034] That is, the elements of the language-derived context feature are T1 ｉ and the element of the image-derived context feature D2 is T2 ｉ where i is an integer between 0 and N-1 (N is the number of elements of each feature), and the synthesized context feature T ｉ = T1 ｉ ×T2 ｉ or T ｉ = T1 ｉ +T2 ｉThe trained model may be, for example, a hierarchical network such as a text decoder compatible with CLIP. The prompt generation unit 133 combines the query D3, the supplementary information D5, the summary D6, and the context language to generate a generated query D4, which is text data instructing the document generation unit 160 to generate an instruction query D10 from the material indicated by the query D3, with reference to the supplementary information D5, the summary D6, and the context language. Specifically, the prompt generation unit 133 decodes the query D3, the supplementary information D5, the summary D6, and the context language, if they are high-dimensional features, to generate text information, and combines the text to generate the generated query D4. The query D3, the supplementary information D5, the summary D6, and the context language are converted into linguistic information and then combined into text, which can be input to the dialogue-assisted document generation model of the document generation unit 160. For example, if the image recognition unit 110 converts an image into high-dimensional features using a transformer encoder such as ViT, the image is decoded here to restore the linguistic information, thereby aligning the linguistic information to be understandable by the dialogue-assisted document generation model.

[0035] FIG. 4 is a schematic block diagram showing an example configuration of the information refining unit 140 in this embodiment. The information refining unit 140 includes an information control unit 141, dialogue-assisted document generation inference units 142 and 143, an information complementation unit 144, and an information summary unit 145. The information control unit 141 repeatedly exchanges input and output between the dialogue-assisted document generation inference units 142 and 143, causing them to interact with each other. At this time, the information control unit 141 uses query D3 as an initial input, requests the dialogue-assisted document generation inference unit 142 to infer a detailed document, and requests the dialogue-assisted document generation inference unit 143 to infer a concise document. When requesting the dialogue-assisted document generation inference unit 142 to infer a detailed document, the information control unit 141 may use language (text) such as, for example, "Please tell me more about the following sentence," "Please provide specific details about the following text," "Please provide the following information in this context," "Please explain this in more detail," or "Please provide information supplementing the above content." In addition, when the information control unit 141 requests the dialogue-assisted document generation inference unit 142 to infer a concise document, it may use language such as "Please summarize the following sentence briefly," "Please tell me the main points of the following text," "Please provide a summary of this context," "Please explain this briefly," or "Please tell me the main points of the above content."

[0036] The information control unit 141 may terminate the repetition of the dialogue after a predetermined number of times, or may terminate it when it is determined that no change in the inference result occurs even after the repetition. The information control unit 141 may extract preference information from the dialogue content by the dialogue-attached document generation type inference unit 142 or the dialogue-attached document generation type inference unit 143 and from the storage unit 150. Specifically, when requesting document inference from the dialogue-attached document generation type inference unit 143 or 142, the information control unit 141 adds the following to the text output by the dialogue-attached document generation type inference unit 142 or 143: A. "In order to extract the user's preference information, please tell us the following elements from posts on the SNS: [topics of interest, favorite activities, favorite music and movie genres, etc.]", B. "Please tell us the user's preference information on the SNS. Specifically, please tell us about things you like, topics you are interested in, etc.", C. "Please tell us the user's preference information on the SNS. Specifically, please tell us about things you like, topics you are interested in, etc." By adding language (text) such as "In order to extract user preference information from social media, please tell us about your likes, topics of interest, and content you often post," D "Describe your [hobbies and interests] and how they influence your daily life," E "Tell us about your favorite book, movie, or music and tell us why it resonates with you," F "Explain your decision-making process when choosing a product or service and what factors are most important," G "Describe a challenge you recently faced and tell us how you dealt with it based on your personal values," H "Imagine an ideal day or a perfect experience. Tell us what it was like and why it was important to you," and I "Tell us about a recent achievement or accomplishment, and tell us what you learned from it," preference information that contains specific individual preferences or thought patterns can be included in the inference results.

[0037] Furthermore, when requesting document inference from the dialogue-assisted document generation inference unit 143 or 142, the information control unit 141 adds pairs of questions, such as J. "Please tell us about your recent experiences and thoughts related to [a specific topic]," K. "Please tell us your opinions and feelings about [recent news or events]," L. "Please explain how to address challenges in [a specific area or domain]," M. "What is the most motivating factor for you when using ChatGPT (registered trademark) or other AI systems?", and N. "Imagine an ideal conversation with ChatGPT (registered trademark). What topics would you discuss, and what do you expect from the interaction?", along with the user's answers, so that preference information containing a specific individual's thought patterns is included in the inference result. Note that the text in [ ] may be stored in advance or may be extracted by the information control unit 141, the dialogue-assisted document generation inference unit 143, or 142 from the content of posts to SNS stored in the storage unit 150.

[0038] In addition, the preference patterns of specific individuals can also be obtained through back-end history analysis. For example, the following methods can be used to obtain thought patterns: 1) Chat history analysis: By analyzing a specific individual's past chat history, it is possible to understand their interests and preferences in specific topics, their unique expressions and phrases, and their tendencies. 2) Natural language processing technology: Natural language processing (NLP) technology can be used to extract emotions, attitudes, interests, etc. from a specific individual's writing. 3) Personalization: Providing individualized responses to specific topics or information based on a specific individual's answers makes it easier to understand a specific individual's thought patterns. 4) Clustering: Using clustering technology to group individuals with similar answers, it is possible to find common characteristics and tendencies. 5) Context understanding: Understanding the context of a specific individual's conversation and generating responses based on that understanding enables more appropriate dialogue.

[0039] Furthermore, preference information may be extracted based on the language extracted from query D3. For example, the language included in the parentheses of A and D above may be the language extracted from query D3. When an image or the like is input to the image input unit 100, the information control unit 141 may predict proper nouns of objects in the image using object recognition or VQA (visual question answering), and may add a document describing the name and context to query D3, with the name being written in parentheses of A above. This allows the information control unit 141 to automatically specify a preference target. Furthermore, when query D3 includes a proper noun, the proper noun or sentence context containing preference information may be written in parentheses of A above. This allows a preference target to be given, and preference information to be automatically acquired.

[0040] Here, preferences include, for example, topics of interest, favorite activities, favorite music genres, favorite movie and TV drama genres, food preferences, hobbies, special skills, favorite tourist spots or travel destinations, and fashion styles. Thinking patterns are information indicating the thinking tendencies of individuals who are expected to read the generated materials, such as users, and are information indicating, for example, logical thinking, intuitive thinking, concrete thinking, mocking thinking, analytical thinking, and comprehensive thinking. Logical thinkers tend to analyze information logically and make rational decisions, and are characterized by a preference for logical problem-solving and argumentation. Intuitive thinkers tend to process information based on intuition and emotion, make intuitive decisions, and are characterized by a preference for intuitive insights and creative problem-solving. Concrete thinkers are characterized by thinking based on concrete facts and detailed information, preferring concrete approaches, and emphasizing concrete examples and specific procedures. Abstract thinkers are characterized by thinking based on broad perspectives and abstract ideas, and being interested in finding general principles and patterns. Analytical thinkers are characterized by analyzing problems, researching information in detail to understand the details and related factors, and emphasizing logical reasoning and data analysis. Holistic thinkers consider multiple perspectives and factors comprehensively, are interested in grasping the big picture, and seek to integrate information and find comprehensive solutions. SNS can be, for example, Facebook (registered trademark), Instagram (registered trademark), Twitter (registered trademark), LINE (registered trademark), etc.

[0041] The information control unit 141 may instruct the dialogue-attached document generation type inference unit 142 or the dialogue-attached document generation type inference unit 143 to extract preference information from the storage unit 150. Specifically, the information control unit 141 extracts preference information of a specific individual from the content posted to the SNS stored in the storage unit 150 by inputting the above-mentioned texts A, B, C, etc. to the dialogue-attached document generation type inference unit 143 or 142. When instructing the information control unit 141 to extract preference information from the storage unit 150, the information control unit 141 may input the content posted to the SNS containing the phrase extracted from query D3 to the dialogue-attached document generation type inference unit 142 or 143. Furthermore, the information control unit 141 may extract preference information from an external device such as an SNS server, rather than from the storage unit 150.

[0042] Each of the dialogue-attached document generation type inference units 142 and 143 inputs the output of the other into a dialogue-attached document generation model. The dialogue-attached document generation type inference unit 142 and the dialogue-attached document generation type inference unit 143 may use different dialogue-attached document generation models. The information complementing unit 144 uses preference information as complementary information D5. The information complementing unit 144 may combine preference information extracted from the dialogue content with preference information extracted from the storage unit 150 to create complementary information D5. The information summarizing unit 145 creates a summary of the inference results of the dialogue-attached document generation type inference units 142 and 143, and creates a summary sentence D6 and a linguistic answer D9. The summary may be created using a trained model for creating a summary, or by instructing the dialogue-attached document generation model to create a summary.

[0043] FIG. 5 is a schematic block diagram showing an example configuration of the image generation unit 170 in this embodiment. The image generation unit 170 includes a trained deep feature extraction unit 171 and an image decoder unit 172. The trained deep feature extraction unit 171 inputs the command query D10 to a trained model to infer its features. This trained model may be, for example, a hierarchical network such as a CLIP text encoder. The image decoder unit 172 inputs the features of the inference result by the trained deep feature extraction unit 171 to the trained model to infer an image answer D11, which is an image representing the features. This trained model may be, for example, a hierarchical network such as a diffusion decoder compatible with CLIP.

[0044] In addition, the learned deep feature extraction unit 171 may identify features with important weights from the features of the inference result, and the image decoder unit 172 may input the identified features into the learned model to infer an image representing the identified features, and use this as the image answer D11.

[0045] 6 is a flowchart illustrating an example of the operation of the material generation device 10 according to this embodiment. First, the language input unit 120 accepts input of a query D3 by a user (step S1). Next, the image input unit 100 accepts input of an image D1 by the user (step S2).

[0046] Next, the information refining unit 140 uses the query D3 input to the language input unit 120 in step S1 as an initial input, and infers the document requested by the query D3 by repeatedly mutually inputting and outputting between the two dialogue-based document generation models to cause a dialogue, and summarizes the inference results as a language answer D9 and complementary information D5 (step S3). At this time, the information refining unit 140 may input preference information of a specific individual read from the storage unit 150 to one of the two dialogue-based document generation models so that the preference information is reflected in the inference results. The information refining unit 140 also uses the preference information as complementary information D5.

[0047] Next, the image recognition unit 110 inputs the image D1 input to the image input unit 100 in step S2 into the trained model, infers its features, and identifies features with important weights from the features of the inference results, and sets them as image-derived context features D2 (step S4).

[0048] Next, the natural language processing unit 130 inputs the query D3 into the trained model to infer its features, and identifies features (language-derived context features) with important weights from the features of the inference result. Furthermore, the natural language processing unit 130 combines the query D3, the language-derived context features, the image-derived context features D2, the supplementary information D5, and the summary sentence D6 to generate a generated query D4 (step S5).

[0049] Next, the document generation unit 160 inputs the generation query D4 generated in step S5 into a dialogue-attached document generation model, infers the document requested by the generation query D4, and sets the inference result as an instruction query D10 (step S6). Next, the image generation unit 170 inputs the instruction query D10 of step S6 into a trained image generation model, infers the image indicated by the instruction query D10, and sets the inference result as an image answer D11 (step S7).

[0050] Next, the display processing unit 180 acquires the verbal response D9 of step S3 from the information refining unit 140 and the image response D11 of step S7 from the image generating unit 170, and displays them on the display unit 190 (step S8). The display processing unit 180 may output the verbal response D9 and the image response D11 as electronic files or may transmit them to another device.

[0051] 7 is a schematic diagram showing a use case of the material generation device 10 according to this embodiment. In FIG. 7, a user inputs text E3, "I want to get a rabbit image with holding a carrot," as query D3 to the language input unit 120. The user also inputs three images E1 of rabbits holding carrots as images D1 to the image input unit 100. Here, the rabbits in image E1 are gray, pink, and brown.

[0052] The information refining unit 140 extracts text E8, "She loves white rabbits," as preference information from the user's SNS posting history stored in the storage unit 150, and inputs it to one of the dialogue-assisted document generation type inference units 142 and 143. The dialogue-assisted document generation type inference units 142 and 143 carry out a dialogue E12, such as "What does look like a rabbit?" and "Common rabbits are Lepus Timidus." Based on this, the information summarizing unit 145 generates text E9, "A white rabbit like Lepus Timidus with ears up, holding a tip-up carrot and facing forward," as a linguistic answer D9 or summary sentence D6.

[0053] The command query D10 generated by the document generation unit 160 includes the content of the text E9, so the image generation unit 170 generates an image E11 of a white rabbit holding a carrot as the image answer D11.

[0054] Second Embodiment A second embodiment of the present disclosure will now be described with reference to the drawings. In the first embodiment, preference information including the person's preferences or thought patterns is used as information indicating a person's characteristics. In the second embodiment, information indicating a person's role is used as information indicating a person's characteristics. FIG. 8 is a schematic block diagram showing the configuration of a material generation device 20 according to the second embodiment of the present disclosure. The material generation device 10 is a device that generates multimodal materials in response to user requests. The material generation device 10 includes a user interface unit 201, a language analysis unit 202, an image analysis unit 203, a multimodal information integration unit 204, a user information DB (Data Base) unit 205, a provider information DB unit 206, an external information DB unit 207, a hypothesis generation unit 208, an artificial intelligence unit 209, and a display processing unit 212. The material generation device 10 may be realized by one or more computers reading and executing a program, or at least a portion of the material generation device 10 may be located on a so-called cloud.

[0055] The user interface unit 201 includes input / output devices such as a keyboard, a mouse, a touch panel, a microphone, a display, and a speaker, and provides a user interface for inputting data to the material generation device 10. The input data is multimodal data including text data, audio data, image data, document data such as PDF files, and reference data such as URLs. The input data also includes a query that indicates the material to be generated, user information that is stored in advance and referenced when generating the material, and provider information. The user information is information held by a user of the material generation service provided by the material generation device 10, such as know-how held by the user. The provider information is information held by a provider who provides the material generation service using the material generation device 10, such as know-how held by the provider.

[0056] The language analysis unit 202 processes text data input to the user interface unit 201. The text data may be text data included in document data such as a PDF file, or may be text data included in content (such as web page content) indicated by referenced data. The language analysis unit 202 may also perform speech recognition processing on voice data input to the user interface unit 201 to convert it into text data.

[0057] The image analysis unit 203 analyzes image data input to the user interface unit 201 and extracts features. The image data may be image data included in document data such as a PDF file, or may be image data included in content indicated by reference data.

[0058] The multimodal information integration unit 204 integrates the feature quantities of the text data and image data from the language analysis unit 202 and the image analysis unit 203, and converts them into a vector representation. The multimodal information integration unit 204 inputs the query converted into a vector representation to the hypothesis generation unit 208. Furthermore, the multimodal information integration unit 204 stores the user information converted into a vector representation in the user information DB unit 205, and stores the provider information converted into a vector representation in the provider information DB unit 206.

[0059] The user information DB unit 205, the provider information DB unit 206, and the external information DB unit 207 store information related to materials. The user information DB unit 205 is a database that stores user information converted into vector representation. The provider information DB unit 206 is a database that stores provider information converted into vector representation. The external information DB unit 207 is a database that stores a database located externally, such as on the Internet, or data acquired from external sources.

[0060] The hypothesis generation unit 208 generates materials indicated by the query converted into a vector representation input from the multimodal information integration unit 204. The hypothesis generation unit 208 infers materials indicated by the query by having multiple AI agents interact with each other. The hypothesis generation unit 208 includes information indicating the characteristics of a person corresponding to at least one of the multiple AI agents in the interaction between the multiple AI agents. This interaction may be between a hypothesis generation side and a criticism side. For example, the hypothesis generation unit 208 may cause one AI agent to generate a hypothesis and another AI agent to criticize or critique the hypothesis. This allows the hypothesis generation unit 208 to improve the accuracy of the hypothesis and the quality of the generated materials. Furthermore, the hypothesis generation unit 208 may cause the AI ​​agent that generated the hypothesis to revise the hypothesis based on the results of the criticism or critique.

[0061] Furthermore, when generating materials, hypothesis generation unit 208 refers to information stored in user information DB unit 205, provider information DB unit 206, and external information DB unit 207. Furthermore, when generating materials, hypothesis generation unit 208 uses artificial intelligence unit 209 having artificial intelligence functions.

[0062] The artificial intelligence unit 209 refers to an artificial intelligence having intelligent functions such as inference and judgment, and its operating environment. The artificial intelligence unit 209 is composed of a model control unit 210 and a trained model storage unit 211. The artificial intelligence unit 209 is a model configured to output an answer to a prompt when the prompt is input, and its operating environment. When a prompt is input from the hypothesis generation unit 208, the artificial intelligence unit 209 outputs an answer based on the prompt and a trained model described below.

[0063] The trained model storage unit 211 stores trained models.

[0064] The trained model includes model information, which will be described later. The trained model may include model parameters, which are information that defines the behavior of the model, such as constraints, weighting variables, and evaluation functions.

[0065] The models include, for example, NN (Neural Network), CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), VAE (Variational Autoencoder), GAN (Generative Adversarial Networks), Diffusion model, Transformer, LLM (Large Language Model), VLM (Visual Language Model), and BERT (Bidirectional Encoder Representations from The model may be a model called Transformers, GPT (Generative Pre-trained Transformer), or CLIP (Contrastive Language Image Pre-training). Note that the above-mentioned models are not exclusive, and for example, LLM, VLM, BERT, and GPT are included in Transformers. Also, for example, Transformers are included in NN. Furthermore, the learning algorithm and model may be a combination of multiple types. Models also include what are called multimodal models that are learned by combining multiple different types of data.

[0066] When the model control unit 210 receives a prompt, it outputs an answer corresponding to the prompt based on the prompt and the trained model. That is, when the model control unit 210 receives a prompt, it generates and outputs an answer corresponding to the prompt using the model indicated by the trained model.

[0067] The trained model and other information used by the artificial intelligence unit 209 may be prepared in advance, or may be obtained via a network as needed.

[0068] The display processing unit 212 includes a display such as a liquid crystal display or an organic EL (Electro Luminescence) display, and displays the materials generated by the hypothesis generation unit 208. The display included in the display processing unit 212 may be the same as the display included in the user interface unit 201.

[0069] Fig. 9 is a schematic block diagram showing the configuration of the hypothesis generation unit 208 in this embodiment. The hypothesis generation unit 208 includes three AI agent units 301a, 301b, and 301c (AI agents), an AI agent dialogue management unit 302 (dialogue management unit), a dialogue history knowledge DB unit 303, a dialogue strategy DB unit 304 (dialogue strategy storage unit), an output generation unit 305, and an evaluation verification unit 306. Although Fig. 9 shows three AI agent units 301a, 301b, and 301c, the number of AI agent units included in the hypothesis generation unit 208 may be two, or four or more.

[0070] The three AI agent units 301a, 301b, and 301c correspond one-to-one to the three people. Hereinafter, any AI agent unit, such as an AI agent unit selected from the three AI agent units 301a, 301b, and 301c, will also be referred to as the AI ​​agent unit 301. Each of the three AI agent units 301a, 301b, and 301c includes personal information indicating the characteristics of the corresponding person among the three people in the dialogue. At least one of the three people may be a fictitious person. The personal information of the fictitious person may include information indicating the role of the fictitious person. The role of the fictitious person may be, for example, an excellent business analyst, an excellent consultant for a provider promoting digitalization, or a senior executive of the user's X division. Note that if a specific, non-fictitious person is included among the three people, the personal information of the specific person may include information indicating the role of the specific person, or may include a copyrighted work of or related to the specific person.

[0071] Each of the AI ​​agent units 301a, 301b, and 301c acquires reference information related to the dialogue from the user information DB unit 205, the provider information DB unit 206, the external information DB unit 207, and the dialogue history knowledge DB unit 303, and includes the acquired reference information in the dialogue. For example, each of the AI ​​agent units 301a, 301b, and 301c is specified by the AI ​​agent dialogue management unit 302 as reference information and tasks to be acquired from the user information DB unit 205, the provider information DB unit 206, the external information DB unit 207, and the dialogue history knowledge DB unit 303. Each of the AI ​​agent units 301a, 301b, and 301c acquires the specified reference information from the user information DB unit 205, the provider information DB unit 206, the external information DB unit 207, and the dialogue history knowledge DB unit 303. Each of the AI ​​agent units 301a, 301b, and 301c generates a prompt including the acquired reference information, a specified task, and corresponding person information, and inputs the generated prompt to the model control unit 210 of the artificial intelligence unit 209. That is, each of the AI ​​agent units 301a, 301b, and 301c requests the model control unit 210 to provide an answer that would be given if the person indicated by the person information performed the specified task while referring to the acquired reference information. Each of the AI ​​agent units 301a, 301b, and 301c stores the answer from the model control unit 210 in the dialogue history knowledge DB unit 303. The task includes generating a hypothesis, criticizing or commenting on the generated hypothesis, and modifying the hypothesis based on the criticism or comment. For example, the task may be, "Analyze the reference information and create a detailed report," "Evaluate the feasibility of the report output in the previous step," or "Modify the report based on the evaluation of feasibility." The task may also include specifying a specific format for the answer.

[0072] The AI ​​agent dialogue management unit 302 causes the AI ​​agent unit 301 selected from the three AI agent units 301a, 301b, and 301c to dialogue in accordance with the dialogue process indicated by the dialogue strategy information acquired from the dialogue strategy DB unit 304. The AI ​​agent dialogue management unit 302 inputs the dialogue results to the output generation unit 305. The dialogue and the dialogue results may be multimodal data including not only text but also images, diagrams, tables, graphs, audio, and video. The dialogue process may include multiple steps, and each step may include a task, designation of one of the AI ​​agent units 301a, 301b, and 301c, and designation of reference information. Note that the AI ​​agent dialogue management unit 302 may use the model control unit 210 of the artificial intelligence unit 209 to perform a dialogue between the three AI agent units 301a, 301b, and 301c based on the dialogue strategy information. For example, the AI ​​agent dialogue management unit 302 may input a prompt to the model control unit 210 inquiring about the content of the next step to be executed, along with the execution history of the steps up to that point and dialogue strategy information, and cause the AI ​​agent unit 301 to dialogue according to the answer corresponding to the prompt.

[0073] The AI ​​agent dialogue management unit 302 may acquire dialogue strategy information from the dialogue strategy DB unit 304 based on a query (hereinafter also referred to as query information) converted into a vector representation input from the multimodal information integration unit 204. For example, the AI ​​agent dialogue management unit 302 may search the dialogue strategy DB unit 304 using the query information and acquire dialogue strategy information as a search result. Furthermore, each step of the dialogue strategy may include acquiring dialogue strategy information by searching the dialogue strategy DB unit 304 again and specifying information to be used for the search. The information to be used for the search may be, for example, query information or reference information.

[0074] The dialogue history knowledge DB unit 303 stores, as a dialogue history, the answers that the AI ​​agent unit 301 acquires from the model control unit 210. The dialogue history knowledge DB unit 303 may also store prompts input to the model control unit 210.

[0075] The dialogue strategy DB unit 304 stores dialogue strategy information. The dialogue strategy information may include the purpose of the dialogue strategy and the dialogue process. The dialogue strategy information stored in the dialogue strategy DB unit 304 may be, for example, a natural language sentence such as "A proposal for improving a business process is created. The dialogue process consists of three steps. The first step is to have the business analyst's AI agent collect and analyze the user's business process and submit a report of the results. The report format is..." converted into a vector representation.

[0076] The output generation unit 305 generates materials based on the dialogue results input from the AI ​​agent dialogue management unit 302. The format of the materials may be specified by the AI ​​agent dialogue management unit 302 based on the dialogue strategy information. For example, the output generation unit 305 may generate materials by inputting a prompt instructing the generation of materials to the model control unit 210 along with the dialogue results and the format of the materials input from the AI ​​agent dialogue management unit 302, and acquiring the materials as answers corresponding to the prompt. The materials may be multimodal data including images, figures, tables, graphs, and the like in addition to text.

[0077] The output generation unit 305 may also modify the generated materials based on the results of evaluation and verification by the evaluation verification unit 306. For example, the output generation unit 305 may modify the materials by inputting a prompt to the model control unit 210 instructing modification of the materials along with the results of evaluation and verification by the evaluation verification unit 306 for the generated materials, and acquiring the materials as an answer to the prompt. The output generation unit 305 inputs the modified materials to the display processing unit 212. Note that the output generation unit 305 may repeat the evaluation and verification by the evaluation verification unit 306 and modification multiple times before inputting the modified materials to the display processing unit 212.

[0078] The evaluation verification unit 306 evaluates and verifies the materials input from the output generation unit 305. For example, the evaluation verification unit 306 may input a prompt instructing the evaluation and verification of the materials to the model control unit 210 along with the materials input from the output generation unit 305, and obtain the results of the evaluation and verification as an answer to the prompt. The evaluation verification unit 306 may also include personal information indicating, for example, a reviewer in the prompt. The personal information included in the prompt may change each time evaluation and verification is performed.

[0079] In this way, in the material generation device 20, the multiple AI agent units 301 correspond one-to-one to the multiple people, and each of the multiple AI agent units 301 includes personal information indicating the characteristics of the corresponding person among the multiple people in the dialogue. This allows the material generation device 20 to generate materials according to the characteristics of the multiple people.

[0080] FIG. 10 is an explanatory diagram illustrating the hardware configuration of the material generation device 10, 20 according to this embodiment. The material generation device 10, 20 includes an input / output module I, a storage module M, and a control module P. The input / output module I includes some or all of the following: a communication module H11, a connection module H12, a pointing device H21, a keyboard H22, a display H23, a button H3, a microphone H41, a speaker H42, a camera H51, and a sensor H52. The storage module M includes a drive H7. The storage module M may further include some or all of a memory H8. The control module P includes a memory H8 and a processor H9. These hardware components are connected to each other via a bus (Bus) for communication and receive power from a power supply H6.

[0081] The connection module H12 is a digital input / output port such as a USB (Universal Serial Bus). In the case of a portable device, the pointing device H21, keyboard H22, and display H23 are touch panels. The sensor H52 is an acceleration sensor, a gyro sensor, a GPS receiving module, a proximity sensor, or the like. The power supply H6 is a power supply unit that supplies the electricity necessary to operate each device. In the case of a portable device, the power supply H6 is a battery. The drive H7 is an auxiliary storage medium such as a hard disk drive or a solid-state drive. The drive H7 may be a non-volatile memory such as an EEPROM or a flash memory, or a magneto-optical disk drive or a flexible disk drive. Furthermore, the drive H7 is not limited to being built into each device, but may also be an external storage device connected to the connector of the connection module H12. The memory H8 is a main storage medium such as a random access memory. The memory H8 may also be a cache memory. The memory H8 stores instructions when the instructions are executed by one or more processors H9. The processor H9 is a CPU (Central Processing Unit). The processor H9 may be an MPU (Microprocessing Unit) or a GPU (Graphics Processing Unit). The processor H9 reads programs and various data from the drive H7 via the memory H8 and performs calculations to execute instructions stored in one or more memories H8.

[0082] In this specification and the like, the descriptions of the material generation devices 10 and 20 may be replaced with the description of the control module P.

[0083] The following embodiments may also be used: (1) One embodiment of the present disclosure is a material generation device including an input unit that receives an input of a query that indicates a material to be generated, and a dialogue management unit that infers the material indicated by the query by causing a dialogue between a plurality of AI agents based on the query, and in which information indicating characteristics of a person corresponding to at least one of the plurality of AI agents is included in the dialogue between the plurality of AI agents.

[0084] This allows the material generation device to generate materials that match the characteristics of the person.

[0085] (2) Another embodiment of the present disclosure is the document generation device described in (1), wherein the plurality of AI agents are two dialogue-attached document generation models, the input unit is a language input unit that accepts input of the query, the dialogue management unit is an information refining unit that uses the query as an initial input and causes the two dialogue-attached document generation models to repeatedly exchange input and output with each other to infer a document from the document indicated by the query, and the document generation device includes an image generation unit that generates an image based on the document resulting from the inference by the information refining unit, and adds preference information including the preferences or thought patterns of a specific individual to the dialogue between the two dialogue-attached document generation models.

[0086] This allows the material generation device to generate materials that reflect the preferences or thought patterns of a specific individual.

[0087] (3) Another embodiment of the present disclosure is a document generation device as described in (2), further comprising: an image input unit that accepts input of an image that serves as an example of the document to be generated; and an image recognition unit that inputs the image input to the image input unit into a trained model to infer features of the image input to the image input unit and use them as image-derived context features, wherein the image generation unit generates an image based on the image-derived context features in addition to the document of the inference result by the information refinement unit.

[0088] This allows the material generation device to generate an image that reflects an image that serves as an example of the material.

[0089] (4) Another embodiment of the present disclosure is the document generation device described in (3), further comprising: a natural language processing unit that inputs the query to a trained model, infers features of the query as language-derived context features, and generates a generated query by combining at least a portion of the query, the language-derived context features, the image-derived context features, the preference information, and a document inferred as a result of the inference by the information refining unit; and a document generation unit that inputs the generated query to a dialogue-attached document generation model, infers a document requested by the generated query, and sets the inference result as a command query, and the image generation unit inputs the command query to a trained image generation model, infers an image indicated by the command query.

[0090] This allows the material generation device to generate an image that reflects the preferences or thought patterns of a specific individual and an image that serves as a material example.

[0091] (5) Another embodiment of the present disclosure is the data generation device described in (1), wherein the dialogue between the multiple AI agents includes the generation of a hypothesis by one AI agent and criticism or critique of the generated hypothesis by the other AI agents.

[0092] This allows the material generation device to improve the quality of the materials it generates.

[0093] (6) Another embodiment of the present disclosure is a material generation device described in (1) or (5), wherein the plurality of AI agents correspond one-to-one to a plurality of people, and each of the plurality of AI agents includes in its dialogue personal information indicating the characteristics of the corresponding person among the plurality of people.

[0094] This allows the material generation device to generate materials according to the characteristics of multiple people.

[0095] (7) Another embodiment of the present disclosure is a material generation device described in (6), wherein at least one of the plurality of characters is a virtual character, and the person information of the virtual character includes information indicating the role of the virtual character.

[0096] This allows the material generation device to generate materials according to the role of the virtual person.

[0097] (8) Another embodiment of the present disclosure is a material generation device described in any one of (1) to (7), comprising a dialogue strategy memory unit that stores dialogue strategy information indicating a dialogue process and a person to be dialogued, and the dialogue management unit causes the AI ​​agent selected from the plurality of AI agents to dialogue in accordance with the dialogue process indicated by the dialogue strategy information acquired from the dialogue strategy memory unit.

[0098] This allows the material generation device to make the interaction process flexible.

[0099] (9) Another embodiment of the present disclosure is a material generation device described in any one of (1) to (8), comprising a memory unit that stores information related to the material, and each of the plurality of AI agents acquires reference information related to the dialogue from the memory unit and includes the acquired reference information in the dialogue.

[0100] This allows the material generation device to generate materials based on reference information related to the dialogue.

[0101] (10) Another embodiment of the present disclosure is a document generation device described in any one of (1) to (9), wherein the input unit accepts input of multimodal data related to the document, the dialogue management unit causes a dialogue between the multiple AI agents based on the query and the multimodal information, and the dialogue between the multiple AI agents is multimodal.

[0102] This allows the material generation device to generate materials based on multimodal information.

[0103] (11) Another embodiment of the present disclosure is a material generation device described in any one of (1) to (10), comprising an output generation unit that generates the material based on the results of dialogue between the multiple AI agents, and an evaluation and verification unit that evaluates and verifies the material generated by the output generation unit.

[0104] This allows the material generation device to generate evaluated and verified materials.

[0105] (12) Another embodiment of the present disclosure is a material generation method including a first step of receiving an input of a query indicating materials to be generated, and a second step of inferring the materials indicated by the query by having a plurality of AI agents interact with each other based on the query, and adding information indicating characteristics of a person, including a fictional person, corresponding to at least one of the plurality of AI agents to the interaction between the plurality of AI agents.

[0106] (13) Another embodiment of the present disclosure is a program that includes a computer, an input unit that accepts input of a query that indicates materials to be generated, and a program that adds information indicating characteristics of a person, including a virtual person, corresponding to at least one of the plurality of AI agents to a dialogue between the plurality of AI agents in the query.

[0107] 1 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to realize the material generation device 10. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0108] "Computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" also includes devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, and devices that store programs for a fixed period of time, such as volatile memory within the computer systems that serve as servers or clients. The programs may also be programs that implement some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0109] The embodiments of this disclosure have been described in detail above with reference to the drawings, but the specific configuration is not limited to this embodiment, and design changes and the like are also included within the scope that does not deviate from the gist of this disclosure.

[0110] 10, 20 Document generation device 100 Image input unit 110 Image recognition unit 111 Learned deep feature extraction unit 112 Context analysis unit 120 Language input unit 130 Natural language processing unit 131 Learned deep feature extraction unit 132 Context analysis unit 133 Prompt generation unit 140 Information refinement unit 141 Information control unit 142, 143 Dialogue-assisted document generation inference unit 144 Information complementation unit 145 Information summarization unit 150 Memory unit 160 Document generation unit 170 Image generation unit 171 Learned deep feature extraction unit 172 Image decoder unit 180 Display processing unit 190 Display unit 201 User interface unit 202 Language analysis unit 203 Image analysis unit 204 Multimodal information integration unit 205 User information DB unit 206 Provider information DB unit 207 External information DB unit 208 Hypothesis generation unit 209 Artificial intelligence unit 210 Model control unit 211 Learned model storage unit 212 Display processing unit 301a, 301b, 301c AI agent unit 302 AI agent dialogue management unit 303 Dialogue history knowledge DB unit 304 Dialogue strategy DB unit 305 Output generation unit 306 Evaluation verification unit

Claims

1. A material generation device comprising: an input unit that receives input of a query that indicates the material to be generated; and a dialogue management unit that infers the material indicated by the query by having a plurality of AI (Artificial Intelligence) agents interact with each other based on the query, wherein the dialogue between the plurality of AI agents includes information indicating the characteristics of a person corresponding to at least one of the plurality of AI agents.

2. The material generation device according to claim 1, wherein the plurality of AI agents are a plurality of dialogue-attached document generation models, the input unit is a language input unit that accepts input of the query, the dialogue management unit is an information refining unit that uses the query as an initial input and causes the plurality of dialogue-attached document generation models to repeatedly exchange input and output with each other to infer a document from the material indicated by the query, and further comprises an image generation unit that generates an image based on the document resulting from the inference by the information refining unit, and preference information including the preferences or thought patterns of a specific individual is added to the dialogue between the plurality of dialogue-attached document generation models.

3. The document generation device of claim 2, further comprising: an image input unit that accepts input of an image that serves as an example of the document to be generated; and an image recognition unit that inputs the image input unit to a trained model to infer features of the image input unit and use them as image-derived context features, wherein the image generation unit generates an image based on the image-derived context features in addition to the document that is the inference result of the information refinement unit.

4. The document generation device of claim 3 further comprises: a natural language processing unit that inputs the query into a trained model, infers features of the query as language-derived context features, and generates a generated query that combines at least a portion of the query, the language-derived context features, the image-derived context features, the preference information, and a document that is an inference result by the information refining unit; and a document generation unit that inputs the generated query into a dialogue-attached document generation model, infers a document requested by the generated query, and sets the inference result as an instruction query, wherein the image generation unit inputs the instruction query into a trained image generation model, and infers an image indicated by the instruction query.

5. The data generation device of claim 1, wherein the dialogue between the multiple AI agents includes the generation of a hypothesis by one AI agent and criticism or critique of the generated hypothesis by the other AI agents.

6. The material generation device according to claim 1, wherein the plurality of AI agents correspond one-to-one to a plurality of people, and each of the plurality of AI agents includes in its dialogue personal information indicating the characteristics of the corresponding person among the plurality of people.

7. The material generation device according to claim 6, wherein at least one of the plurality of characters is a virtual character, and the character information of the virtual character includes information indicating the role of the virtual character.

8. A material generation device as described in claim 1, comprising a dialogue strategy memory unit that stores dialogue strategy information indicating a dialogue process and a person to be dialogued, and the dialogue management unit causes the AI ​​agent selected from the plurality of AI agents to dialogue in accordance with the dialogue process indicated by the dialogue strategy information obtained from the dialogue strategy memory unit.

9. The material generation device of claim 1, further comprising a memory unit that stores information related to the material, wherein each of the plurality of AI agents acquires reference information related to the dialogue from the memory unit and includes the acquired reference information in the dialogue.

10. The document generation device of claim 1, wherein the input unit accepts input of multimodal data related to the document, the dialogue management unit causes a dialogue between the plurality of AI agents based on the query and the multimodal information, and the dialogue between the plurality of AI agents is multimodal.

11. The material generation device according to claim 1, comprising: an output generation unit that generates the material based on the results of the dialogue between the plurality of AI agents; and an evaluation and verification unit that evaluates and verifies the material generated by the output generation unit.

12. A material generation method comprising: a first step of receiving input of a query that indicates the material to be generated; and a second step of inferring the material indicated by the query by having a plurality of AI agents interact with each other based on the query, wherein information indicating the characteristics of a person, including a fictional person, corresponding to at least one of the plurality of AI agents is added to the interaction between the plurality of AI agents.

13. A program for causing a computer to function as: an input unit that accepts input of a query that indicates the material to be generated; and a dialogue management unit that infers the material indicated by the query by having multiple AI agents interact with each other based on the query; the program adding, to the interaction between the multiple AI agents, information indicating the characteristics of a person, including a fictional person, corresponding to at least one of the multiple AI agents.

Images