Information processing system, information processing device, information processing method, and program

Abstract

To enable the identification of missing information during document generation. [Solution] The information processing system includes: a search unit that searches a first set of document data for document data related to instruction information that includes instructions for an AI to generate a document; an AI control unit that causes the AI ​​to generate a first document based on the searched document data and the instruction information; a determination unit that determines whether or not there is missing information in the first set of document data regarding the generation of a document based on the instruction information, based on a comparison between the first document and a second document generated by the AI ​​based on document data different from the searched document data; and a display control unit that displays the result of the determination by the determination unit.

Description

Technical Field

[0006] , , , , ,

[0001] The present invention relates to an information processing system, an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, a technique called RAG (Retrieval Augmented Generation) has been devised. In RAG, for a message such as a question or instruction given from a user, highly relevant document data is retrieved from a set (database) of pre-prepared document data, and based on the retrieved document data, an AI for generating a response to the message is made to generate a response. By doing so, it is possible to enable the generation of a response based on knowledge that the generation AI has not learned.

[0003] Note that Patent Document 1 discloses a method of providing a domain-specific dialogue using a language model.

Summary of the Invention

Problems to be Solved by the Invention

[0004] When generating a document corresponding to an instruction from a user based on a set of pre-prepared document data, the quality of the generated document is considered to greatly depend on the set of document data.

[0005] The present invention has been made in view of the above points, and an object thereof is to make it possible to grasp the presence or absence of insufficient information in document generation.

Means for Solving the Problems

[0006] To solve the above problems, the information processing system includes: a search unit that searches a first set of document data for document data related to instruction information that includes instructions for the AI ​​to generate a document; an AI control unit that causes the AI ​​to generate a first document based on the searched document data and the instruction information; a determination unit that determines whether or not there is missing information in the first set of document data regarding the generation of a document based on the instruction information, based on a comparison between the first document and a second document generated by the AI ​​based on document data different from the searched document data; and a display control unit that displays the result of the determination by the determination unit. [Effects of the Invention]

[0007] This makes it possible to identify whether or not there is missing information during document generation. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 2] This figure shows an example of the hardware configuration of the information processing device 10 in an embodiment of the present invention. [Figure 3] This figure shows an example of the functional configuration of an information processing system in an embodiment of the present invention. [Figure 4] This figure shows an example configuration of the template storage unit 121. [Figure 5] This is a sequence diagram illustrating an example of a processing procedure performed by the information processing device 10. [Figure 6] This figure shows an example of how the dialogue screen will look. [Figure 7] This figure shows an example of the template selection screen. [Figure 8] This figure shows an example of a dialogue screen displaying the use of the target template. [Figure 9] This figure shows an example of the display of the dialogue screen after entering supplementary information. [Figure 10] This figure shows an example of the configuration of the comparative document storage unit 123. [Figure 11] This figure shows an example of the display of the dialogue screen at the end of the generation of the target document and missing information. [Figure 12] This figure shows an example of the document display screen. [Figure 13] This figure shows an example of a document display screen that includes missing information. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. Figure 1 is a diagram showing an example of the configuration of an information processing system in an embodiment of the present invention. In Figure 1, one or more terminals 20 are connected to the information processing device 10 via a network such as a LAN (Local Area Network) or the Internet.

[0010] The information processing device 10 is one or more computers having software (hereinafter referred to as "agents") that interacts with the user. An agent is an example of a conversational AI and is a personified virtual entity that appears to the user as a conversation partner. Interaction with the conversational AI (conversation partner) means that when the user inputs a message, a response is output in response. In this embodiment, the agent generates a document based on a pre-prepared set of document data in response to instructions from the user. For example, if the set of document data lacks information related to the field of the message indicating the user's instructions, the likelihood of generating a high-quality document is low. However, since the computer that is the conversation partner is a black box for the user, it is difficult to identify the reason why the expected document is not generated. For example, it is difficult to determine whether there is a problem with the message that the user inputted or whether there is missing information in the set of document data used to generate the document. Therefore, the agent evaluates the set of document data used to generate the generated document in relation to the generation of the document in response to the user's instructions. Evaluation of the set of document data means, for example, determining whether there is missing information for generating a document in response to the user's instructions. The agent displays the generated document and the judgment result on terminal 20. In this embodiment, "document" refers to an electronically recorded document, but its data format is not limited to a specific one. The agent may also be called an automated response means, AI agent, digital clone, personalized AI, AI assistant, automated response AI, conversation partner, AI chatbot, companion, concierge, or virtual conversation interface. The agent may also be a virtual human displayed as a conversation partner on the screen of terminal 20 as a 3D avatar modeled after a person.

[0011] Terminal 20 is a device that functions as a user interface for the information processing system. For example, a PC (Personal Computer), smartphone, or tablet device may be used as Terminal 20. Terminal 20 receives input from the user and transmits information corresponding to that input (hereinafter referred to as "input information") to the information processing device 10. Terminal 20 also receives and displays information generated from the information processing device 10 in response to the input information.

[0012] In this embodiment, the information processing system is assumed to be operated within a certain company (hereinafter referred to as "Company X"). Therefore, users who can access the information processing device 10 are employees of Company X or other persons belonging to Company X. However, the services provided by the information processing device 10 may be made publicly available as cloud services.

[0013] Figure 2 is a diagram showing an example of the hardware configuration of the information processing device 10 in an embodiment of the present invention. As shown in Figure 2, the information processing device 10 is built by a computer and includes a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, HD (Hard Disk) 104, HDD (Hard Disk Drive) controller 105, display 106, external device connection I / F (Interface) 108, network I / F 109, data bus 110, keyboard 111, pointing device 112, DVD-RW (Digital Versatile Disk Rewritable) drive 114, and media I / F 116.

[0014] Of these, the CPU 101 controls the operation of the entire information processing unit 10. The ROM 102 stores programs used to drive the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is used as the work area for the CPU 101. The HD 104 stores various data such as programs. The HDD controller 105 controls the reading or writing of various data to the HD 104 according to the control of the CPU 101. The display 106 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 108 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 109 is an interface for data communication using a communication network. The data bus 110 is an address bus and data bus, etc., for electrically connecting each component such as the CPU 101 shown in Figure 2.

[0015] The keyboard 111 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 112 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 114 controls the reading or writing of various data to the DVD-RW 113, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media I / F 116 controls the reading or writing (storage) of data to the recording medium 115, such as flash memory.

[0016] FIG. 3 is a diagram showing a functional configuration example of an information processing system according to an embodiment of the present invention. In FIG. 3, an information processing apparatus 10 includes, as functional units constituting an agent, a reception unit 11, a template management unit 12, a conversion unit 13, a search unit 14, an AI control unit 15, a determination unit 16, and a display control unit 17. Further, the information processing apparatus 10 includes, as storage units constituting an agent, a template storage unit 121, a data storage unit 122, and a comparison document storage unit 123. Furthermore, the information processing apparatus 10 has an AI 150.

[0017] Each of the units as functional units of the agent and the AI 150 is realized by a process in which one or more programs installed in the information processing apparatus 10 cause the CPU 101 to execute. Each of the units as storage units of the agent can be realized, for example, using an HD 104 or a storage device that can be connected to the information processing apparatus 10 via a network.

[0018] The AI 150 is a machine learning model (for example, a neural network) that is learned to generate text (hereinafter referred to as "response") corresponding to the input text (hereinafter referred to as "prompt"). The AI 150 generates, for example, the text with the highest appearance probability as a response based on the learning result for the prompt. For example, a generative AI using a large language model (LLM) may be used as the AI 150. The LLM is a machine learning model that has learned natural language processing using a large amount of text data and is used in many NLP tasks such as generating responses to specific questions, automatically generating articles, summarizing text, translation, sentiment analysis, etc. Also, it can be utilized in various applications such as education, entertainment, customer service, product development, etc. In the present embodiment, the text including the message input by the user becomes the prompt. Note that the information processing apparatus 10 may not have the AI 150. In this case, a generative AI publicly available outside, such as the Internet, may be used as the AI 150.

[0019] Here, machine learning is a technique for enabling computers to acquire human-like learning abilities. It refers to a technique in which a computer autonomously generates algorithms necessary for judgment, such as data identification, from pre-inputted training data, and applies these algorithms to new data to make predictions. The learning method for machine learning can be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, or a combination of these learning methods; the learning method for machine learning is not restricted.

[0020] The template storage unit 121 stores templates (prompts) of text instructions (hereinafter referred to as "prompts") that are input to the AI ​​150 in order to generate a document using the set of document data stored in the data storage unit 122. Templates are prepared (created) in advance for each type of document to be generated (hereinafter referred to as "document type").

[0021] Figure 4 shows an example of the configuration of the template storage unit 121. As shown in Figure 4, the template storage unit 121 stores template ID, document type, items, system prompt, and query information for each template.

[0022] The template ID is the identifier for the template. In Figure 4, the template ID is shown as a string, but values ​​in other data formats, such as numbers, may also be used as the template ID.

[0023] As mentioned above, the document type is the type of document to be generated.

[0024] The items are the components that make up the document to be generated (for example, chapters). System prompts and query information are set for each item in order to have AI150 generate the document for each item (i.e., to generate the target text in parts).

[0025] A system prompt is a template prompt for causing AI150 to generate a document related to the corresponding item, and is an example of pre-prepared instruction information that includes instructions for AI150 to generate the document.

[0026] Query information is a keyword used to search the data storage unit 122 for document data that the AI ​​150 should refer to when generating a document related to a corresponding item. Basically, query information is a string that constitutes the system prompt, and is a string with relatively high semantic importance within the system prompt. Note that the entire system prompt may also be considered query information. In this case, query information does not need to be defined separately.

[0027] For example, by pre-preparing templates for document types that need to be created regularly or frequently in the operations of company X, the burden on each employee regarding document creation can be reduced, and the quality of prompts to AI150 can be standardized.

[0028] The reception unit 11 accepts input from the user. For example, the reception unit 11 accepts the user's selection of a template to be used for generating a document. The selection of a template is equivalent to the selection of the type of document to be generated. The reception unit 11 also accepts instructions from the user to generate a document using the selected template. In this embodiment, supplementary information for the template is input by the user in the document generation instructions. Supplementary information is information that makes the instructions to the AI ​​150 more specific, so that the generated document is in line with the user's intentions. For example, the template with template ID = "Competitive Analysis Template" shown in Figure 4 is a template for generating a document of type "Competitive Analysis Report," but in order to increase the versatility of the template, it does not specify what product or service the competitive analysis is about. Therefore, when using this template, the user needs to specify the product or service to be the target of the competitive analysis as supplementary information.

[0029] The template management unit 12 receives notification from the reception unit 11 regarding which template has been selected by the user, and notifies the relevant units (such as the conversion unit 13 and the determination unit 16) that require the information indicating the selected template.

[0030] The conversion unit 13 converts a string containing supplementary information and query information related to an item (for example, a string concatenating the supplementary information and the query information) for each item of the template selected by the user (hereinafter referred to as the "target template") into a vector that represents the meaning of the string using multidimensional numerical values. A vector representing the meaning of a string (hereinafter referred to as the "semantic vector") can be generated using natural language processing such as BERT. Hereinafter, the semantic vector of a string containing supplementary information and query information will be referred to as the "query vector".

[0031] The search unit 14 searches the first set of document data for document data related to pre-prepared instruction information (system prompts), including instructions for the AI ​​150 to generate a document. In this embodiment, the first set of document data refers to the set of document data stored in the data storage unit 122. More precisely, the search unit 14 searches the data storage unit 122 for document data related to supplementary information and criterion information (important strings within the system prompt related to the item) for each item of the target template. For example, the data storage unit 122 pre-stores document data related to various operations of company X. Each employee of company X may upload document data to the data storage unit 122 at any time. For each such document data, the data storage unit 122 stores the document data and semantic vectors for each chunk of the document data. A chunk refers to a part of the document data obtained by dividing the document data into predetermined units. The units for dividing the document data may be the number of characters, the number of sentences, or semantic units (e.g., paragraphs), and are not limited to predetermined units. Hereinafter, the semantic vector of each chunk will be referred to as a "chunk vector." Note that the data storage unit 122 may store the first set of document data, separated into folders for each field, etc.

[0032] The search unit 14 searches the data storage unit 122 for document data (essentially document data related to system prompts) that are related to the supplementary information and query information represented by the query vector, using the query vector generated by the conversion unit 13 for each item in the target template. In searching for document data based on the query vector, the search unit 14 calculates the similarity between the query vector and the chunk vector of each chunk related to the document data stored in the data storage unit 122, and identifies the chunk related to the chunk vector with the highest similarity to the document data (hereinafter referred to as "similar chunk"). The search unit 14 compares the similarity of the similar chunks for each document data and extracts the top N similar chunks. Therefore, essentially N document data items are extracted. Cosine similarity may be used to evaluate the similarity between vectors, or other indicators may be used.

[0033] The AI ​​control unit 15 instructs the AI ​​150 to generate a first document based on the document data (a collection of chunks) retrieved by the search unit 14 and the instruction information (a system prompt related to the item) for each item in the target template. Specifically, the AI ​​control unit 15 generates a prompt based on supplementary information and a system prompt related to the item for each item in the target template, and sends the prompt to the AI ​​150. The AI ​​control unit 15 receives the response from the AI ​​150 to the prompt (i.e., the first document generated by the AI ​​150 based on the prompt) from the AI ​​150.

[0034] The determination unit 16 determines whether there is any missing information in the data storage unit 122 (the set of first document data), which is the source from which the document data (a set of chunks) used to generate the first document was extracted, regarding the generation of the first document based on the target template. Missing information refers to information that, if stored in the data storage unit 122, would have allowed for the generation of a higher-quality first document. In other words, missing information refers to information that is recommended to be added to the data storage unit 122 (the set of first document data) in order to generate a higher-quality document by referring to the data storage unit 122 based on the target template. The determination unit 16 determines whether there is any missing information by referring to the comparison document storage unit 123.

[0035] The comparison document storage unit 123 stores, for each template, a second set of document data, different from the first set of document data stored in the data storage unit 122, containing document data related to the query information (system prompt) of the template and a second document (hereinafter referred to as the "comparison document") generated by the AI ​​150 based on the system prompt of the template. Here, the second set of document data is a set of document data that contains sufficient information for generating a document based on the template (i.e., no or very little missing information). Therefore, the comparison document can be said to be a document that represents a successful example of template-based document generation, generated based on a set of document data that has no or very little missing information. The comparison document may be provided, for example, by a service vendor that provides an information processing system to company X. In this case, the second set of document data may be prepared in the environment of the service vendor.

[0036] The determination unit 16 controls the determination of whether there is any missing information regarding the generation of the document based on the instruction information, based on a comparison between the first document and the comparison document corresponding to the target template, which is stored in the comparison document storage unit 123 for each template. More precisely, the determination unit 16 controls the AI ​​150 to perform a determination of whether there is any missing information based on a comparison between the first document and the second document. The determination unit 16 controls the determination of whether there is any missing information for each item of the template. Therefore, the information related to the item that is determined to be missing is considered the missing information.

[0037] Furthermore, the second set of document data does not necessarily have to be a set of document data relating to the same field (products or services, etc.) as the first set of document data. The reason why it is possible to determine whether or not there is missing information in the first set of document data (data storage unit 122) by comparing it with a second document that is in a different field from the first document is that the only difference between the generation of the first document and the generation of the comparison document is basically the set of document data referenced in the generation of the document. In other words, if one of two documents generated by referencing different sets of document data based on the same instruction information (system prompt) is of higher quality, it is highly probable that the reason the other is of lower quality is the set of document data referenced in the generation of the other. In this embodiment, the determination of whether or not there is missing information is made based on this idea.

[0038] The display control unit 17 transmits information generated by the agent in response to user input to the terminal 20, thereby displaying various screens on the terminal 20. For example, the display control unit 17 displays the first document generated by the AI ​​control unit 15 to the AI ​​150, or the judgment results from the judgment unit 16, on the terminal 20. In this case, the terminal 20 may display the screen using a browser.

[0039] On the other hand, terminal 20 has a reception unit 21, a communication unit 22, and a display control unit 23. Each of these units is realized by a program installed on terminal 20 that causes the terminal 20's CPU to execute a process.

[0040] The reception unit 21 receives user operations on the terminal 20.

[0041] The communication unit 22 controls communication with the information processing device 10.

[0042] The display control unit 23 controls the display of screens, etc. (for example, the dialogue screen 510 described later, the first document generated by the AI ​​control unit 15 to the AI ​​150, the judgment result by the judgment unit 16, etc.) based on information (display data), etc. received from the information processing device 10.

[0043] The following describes the processing steps performed by the information processing device 10. Figure 5 is a sequence diagram illustrating an example of the processing steps performed by the information processing device 10.

[0044] At the start of the processing procedure shown in Figure 5, the terminal 20 displays a template selection screen through interaction between the user and the agent. The template selection screen is a screen that allows the user to select a template from among the templates stored in the template storage unit 121. The interaction between the user and the agent takes place through the dialogue screen displayed on the terminal 20.

[0045] Figure 6 shows an example of the display of the dialogue screen. The dialogue screen 510 shown in Figure 6 includes a dialogue display area 511 and a message input area 512. The dialogue display area 511 is the area where the content of the dialogue between the agent and the user is displayed. Figure 6 shows a state where a greeting message g1 ("Is there anything I can help you with regarding your work?") prompting the user to enter a message, and a message m1 entered by the user are displayed. Message m1 indicates that the user wishes to use a template. The message input area 512 is an area for receiving message input from the user and includes a send icon 5121. When the send icon 5121 is clicked, the message entered in the message input area 512 is displayed in the dialogue display area 511, and the message is sent to the information processing device 10. Therefore, Figure 6 shows an example of the display of the dialogue screen 510 after message m1 has been entered in the message input area 512 and the send icon 5121 has been clicked. In response to message m1, the display control unit 17 of the information processing device 10 displays the template selection screen on the terminal 20. Specifically, the display control unit 17 generates display data for the template selection screen and transmits this display data to the terminal 20 in order to display the template selection screen.

[0046] Figure 7 shows an example of the template selection screen. The template selection screen 520 shown in Figure 7 includes radio buttons 521 that select any one template ID from all templates stored in the template storage unit 121 (Figure 4). One template ID can be selected using the radio buttons 521.

[0047] When a user selects a template ID via the template selection screen 520, the terminal 20 transmits the selected template ID (hereinafter referred to as the "target template ID") to the information processing device 10 (S101).

[0048] When the reception unit 11 receives the target template ID, it inputs the target template ID to the template management unit 12 (S102). The template management unit 12 sets the system prompts for each item included in the target template (Figure 4) corresponding to the target template ID to the AI ​​control unit 15 (S103). Next, the template management unit 12 inputs the target template ID to the display control unit 17 (S104). The display control unit 17 displays a response on the dialogue screen 510 indicating that the target template will be used (S105). Specifically, the display control unit 17 generates display data for the response and sends the display data to the terminal 20 in order to display the response.

[0049] Figure 8 shows an example of a dialogue screen displaying the use of the target template. In Figure 8, the same parts as in Figure 6 are denoted by the same reference numerals, and their explanations are omitted. In the dialogue display area 511 of the dialogue screen 510 shown in Figure 8, response r1 has been added. Response r1 is the response displayed in step S105.

[0050] Next, when the user enters a message containing supplementary information into the message input area 512 of the dialogue screen 510 and clicks the send icon 5121, the terminal 20 sends the message (supplementary information) to the information processing device 10 (S106). At this time, the display content of the dialogue screen 510 changes as shown in Figure 9.

[0051] Figure 9 shows an example of the display of the dialogue screen after supplementary information has been entered. In Figure 9, the same parts as in Figure 8 are denoted by the same reference numerals, and their explanations are omitted. In the dialogue display area 511 of the dialogue screen 510 shown in Figure 9, message m2 has been added. Message m2 is a message indicating supplementary information. In Figure 9, message m2 is the string "Create a competitive analysis report for surveillance camera solutions." Message m2 clarifies that the subject of the competitive analysis is surveillance camera solutions.

[0052] When the receiving unit 11 of the information processing device 10 receives the supplementary information, it inputs the supplementary information to the template management unit 12 (S107). The following steps S111 to S125 are loop processing that is executed for each item of the target template. The items that are processed in this loop processing are called "target items".

[0053] In step S111, the template management unit 12 inputs the supplementary information and the query information for the target item to the conversion unit 13. The conversion unit 13 generates a query vector by converting the input supplementary information and query information into a single string into a semantic vector (S112). Subsequently, the conversion unit 13 inputs the query vector and the supplementary information to the search unit 14 (S113).

[0054] The search unit 14 compares the input query vector with the chunk vectors stored in the data storage unit 122 for each document data and each chunk, thereby identifying similar chunks for each document data and extracting some similar chunks that have a relatively high similarity to the target item text (S114). The search unit 14 generates search results for each extracted similar chunk, including related document information for that similar chunk. The related document information for a certain similar chunk includes the chunk ID of that similar chunk, and the document ID and document name of the document data to which that similar chunk belongs. Subsequently, the search unit 14 inputs the search results and supplementary information to the AI ​​control unit 15 (S115).

[0055] The AI ​​control unit 15 generates a prompt to send to the AI ​​150 (S116) based on the input search results and supplementary information, and the system prompt corresponding to the target item among the system prompts set in step S103. The content of the prompt is, for example, as follows:

[0056] <Example of a prompt> Based on the instruction {supplementary information}, {system prompt} Please refer to the following document for further information.

[0057] {A set of similar chunks related to the search results} <Example of prompts> Note that {supplementary information} is replaced with supplementary information (in the example in Figure 5, "Create a competitive analysis report for surveillance camera solutions"). {system prompt} is replaced with the system prompt for the target item (hereinafter referred to as the "target system prompt"). {set of similar chunks related to search results} is replaced with the set of similar chunks related to the search results from the search unit 14.

[0058] Next, the AI ​​control unit 15 sends the generated prompt to the AI ​​150 (S117).

[0059] AI150 generates text corresponding to the prompt based on learned parameters and sends the text to AI control unit 15 (S118). This text is the document generated for the target item. Hereinafter, the document for each item will be called an "item document," and the document for the target item will be called a "target item document." When AI control unit 15 receives the target item document, it inputs the target item document into template management unit 12 (S119). Template management unit 12 inputs the target template ID and the target item document into determination unit 16 (S120).

[0060] The determination unit 16 obtains the comparison document corresponding to the target template ID and target item from the comparison document storage unit 123 (S121).

[0061] Figure 10 shows an example of the configuration of the comparison document storage unit 123. As shown in Figure 10, the comparison document storage unit 123 stores comparison documents for each template (for each template ID) and for each item. The comparison document for a certain item of a certain template is a document generated by executing steps S111 to S118 for that item, using a second set of document data instead of the data storage unit 122. The supplementary information used when generating the comparison document can be appropriately specified according to the field corresponding to the second set of document data.

[0062] Hereinafter, the comparison document obtained in step S121 will be referred to as the "target comparison document".

[0063] Next, the determination unit 16 inputs a determination instruction to the AI ​​control unit 15 regarding whether or not there are any deficiencies, including the target item document and the target comparison document (S122).

[0064] In response to the determination instruction, the AI ​​control unit 15 generates a prompt containing instruction information to instruct the AI ​​150 to check for deficiencies (S123). The content of the prompt is, for example, as follows:

[0065] <Example of a prompt> The following two documents, Document 1 and Document 2, are reports generated based on the following instructions.

[0066] {Target System Prompt} Compare Document 1 and Document 2, and evaluate Document 1 in terms of length, amount of information, and diversity of information.

[0067] Please output the following information as the evaluation results.

[0068] Regarding Document 1, is there any deficiency in the set of document data used to generate Document 1 from the perspective described above? • Evaluation score for Document 1 based on the above criteria (scored out of 100 points) • Estimated evaluation score (predicted score out of 100 points) when supplementing the missing data to address the deficiencies in the set of document data used to generate Document 1. ## Document 1 {Target item document} ##Document 2 {Target Comparison Document} <Example of prompts> The target system prompt (the system prompt for the target item) is applied to {target system prompt}. The target item document is applied to {target item document}. The target comparison document is applied to {target comparison document}. The prompts exemplified above determine whether there is missing information in the data storage unit 122 (the first set of document data) based on a comparison between the target item document and the target comparison document in terms of text volume, amount of information, and diversity of information. The comparison criteria between the target item document and the target comparison document may be at least one of text volume, amount of information, and diversity of information, or other criteria may be added. Text volume refers to the amount of text, for example, the number of characters or sentences. Amount of information refers to the amount of information contained in the document, and how this amount is evaluated depends on AI150. Diversity of information refers to the number of different types of information. How different types of information are distinguished depends on AI150.

[0069] Next, the AI ​​control unit 15 sends the generated prompt to the AI ​​150 (S124).

[0070] AI150 generates a response (text) corresponding to the prompt based on the learned parameters and sends the response to AI control unit 15 (S125). This response is text (hereinafter referred to as "item-specific deficiency information") that includes whether or not there are deficiencies in the target item document from the perspectives of text volume, amount of information, and diversity of information compared to the target comparison document, as well as evaluation points and expected evaluation points.

[0071] Next, the AI ​​control unit 15 outputs the item-specific missing information included in the response to the determination unit 16 (S126).

[0072] When steps S111 to S126 are executed for all items in the target template, the determination unit 16 will have obtained item-specific missing information for each item.

[0073] Next, the determination unit 16 inputs the missing information, which is the item-specific missing information for each item of the target template, and the target document, which is the item document for each item connected together, to the display control unit 17 (S131). The display control unit 17 generates display information to display the target document and display information to display the target document and the missing information, and displays a response including link information (e.g., URL) indicating the storage location of each display information on the dialogue screen 510 displayed on the terminal 20 (S132). Specifically, the display control unit 17 generates the display data for the response and transmits the display data to the terminal 20 in order to display the response.

[0074] Figure 11 shows an example of the dialogue screen displayed when the generation of the target document and missing information is complete. In Figure 11, the same parts as in Figure 9 are denoted by the same reference numerals, and their explanations are omitted.

[0075] The dialogue display area 511 of the dialogue screen 510 shown in Figure 11 has a response r2 added to it. Response r2 includes a message indicating that the document requested in message m2 has been generated, and links L1 and L2. Link L1 is a link to which link information to the display information of the target document is attached. Link L2 is a link to which link information to the display information of the target document and the missing information is attached.

[0076] When a user clicks link L1, terminal 20 downloads the display information of the target document based on the link information attached to link L1 and displays the document display screen containing the target document.

[0077] Figure 12 shows an example of a document display screen. The document display screen 530 shown in Figure 12 includes a document display area 531. The document display area 531 is the area where the target document is displayed.

[0078] Furthermore, when the user clicks link L2, terminal 20 downloads the target document and display information for missing information based on the link information assigned to link L2, and displays a document display screen that also includes the missing information.

[0079] Figure 13 shows an example of a document display screen that includes missing information. In Figure 13, the same parts as in Figure 12 are denoted by the same reference numerals, and their descriptions are omitted. The document display screen 530 shown in Figure 13 further includes a missing information display area 532. The missing information display area 532 includes, for each item of the target template, whether or not information about that item is missing from the stored document, and a [Overall Evaluation]. In this embodiment, the stored document refers to the data storage unit 122 (a collection of first document data). In Figure 13, it is indicated that information is missing for all items. Also in Figure 13, an example is shown where the average value of the evaluation score for each item and the average value of the expected evaluation score for each item are displayed as the [Overall Evaluation]. However, the evaluation score and expected evaluation score may be displayed for each item.

[0080] Furthermore, in Figure 13, the document display area 531 is assigned a number corresponding to each item ((1) to (3)). As a result, the parts of the target document corresponding to each item are easier to see.

[0081] As described above, according to this embodiment, the quality of the target document (text length, amount of information, and diversity of information) is evaluated by comparing the target document (first document) and the comparison document (second document), which are generated under the conditions that the reference destination of the document data set is different, but the system prompt input to the AI150 is the same. If there is a problem with the quality of the target document, it is determined that there is a difference between the generation of the target document and the generation of the comparison document, and that there is a deficiency in the first document data set used to generate the target document. Therefore, it is possible to grasp whether or not there is missing information during document generation.

[0082] Furthermore, the information processing device 10 is not limited to a general-purpose server computer, as long as it is a device equipped with communication capabilities. The information processing device 10 may be, for example, an output device such as a PJ (Projector), IWB (Interactive White Board: an electronic whiteboard with the ability to communicate with each other), or digital signage, a HUD (Head Up Display) device, industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet device, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, or a desktop PC.

[0083] Furthermore, each function of this embodiment can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute the functions described above.

[0084] Furthermore, the apparatus in this embodiment represents only one of several computing environments for carrying out the embodiments disclosed herein.

[0085] In one embodiment, the information processing device 10 includes a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein. Similarly, the terminal 20 may include a plurality of computing devices configured to communicate with each other.

[0086] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims.

[0087] Examples of the present invention are as follows: <1> A search unit searches for document data related to instruction information containing instructions for generating documents for the AI ​​from a first set of document data, An AI control unit that causes the AI ​​to generate a first document based on the retrieved document data and the instruction information, A determination unit determines, based on a comparison between the first document and a second document generated by the AI ​​based on document data different from the retrieved document data, whether or not there is missing information in the set of first document data regarding the generation of a document based on the instruction information. A display control unit that displays the result of the determination by the determination unit, An information processing system characterized by having the following features. <2> Document data different from the retrieved document data is document data related to the instruction information in the second set of document data. Characterized by <1> The information processing system described above. <3> The determination unit controls the AI ​​to perform a determination of whether or not there is missing information based on a comparison between the first document and the second document. Characterized by <1> or <2> The information processing system described above. <4> The determination unit causes the AI ​​to score an evaluation score for the first document based on the first document and the second document. The display control unit displays the evaluation score. Characterized by <1> ~ <3> The information processing system described in any of the following. <5> A search unit searches for document data related to instruction information containing instructions for generating documents for the AI ​​from a first set of document data, An AI control unit that causes the AI ​​to generate a first document based on the retrieved document data and the instruction information, A determination unit determines, based on a comparison between the first document and a second document generated by the AI ​​based on document data different from the retrieved document data, whether or not there is missing information in the set of first document data regarding the generation of a document based on the instruction information. A display control unit that displays the result of the determination by the determination unit, An information processing device characterized by having the following features. <6> A search procedure for searching a first set of document data for document data related to instruction information containing instructions for generating documents for AI, and An AI control procedure that causes the AI ​​to generate a first document based on the retrieved document data and the instruction information, A determination procedure for determining whether there is any missing information in the set of first document data regarding the generation of a document based on the instruction information, based on a comparison between the first document and a second document generated by the AI ​​based on document data different from the retrieved document data, A display control procedure for displaying the result of the determination procedure described above, An information processing method characterized by a computer executing the following. <7> A search procedure for searching a first set of document data for document data related to instruction information containing instructions for generating documents for AI, and An AI control procedure that causes the AI ​​to generate a first document based on the retrieved document data and the instruction information, A determination procedure for determining whether there is any missing information in the set of first document data regarding the generation of a document based on the instruction information, based on a comparison between the first document and a second document generated by the AI ​​based on document data different from the retrieved document data, A display control procedure for displaying the result of the determination procedure described above, A program that causes a computer to execute something. [Explanation of Symbols]

[0088] 10 Information Processing Devices 11 Reception Department 12. Template Management Department 13 Conversion section 14 Search Section 15 AI control section 16 Judgment section 17 Display Control Unit 20 devices 21 Reception Department 22 Communications Department 23 Display Control Unit 121 Template Storage Unit 122 Data Storage Unit 123 Comparative Document Storage Unit 150 AI B Bus [Prior art documents] [Patent Documents]

[0089] [Patent Document 1] Japanese Patent Publication No. 2023-76413

Claims

1. A search unit searches for document data related to instruction information containing instructions for generating documents for the AI ​​from a first set of document data, An AI control unit that causes the AI ​​to generate a first document based on the retrieved document data and the instruction information, A determination unit determines, based on a comparison between the first document and a second document generated by the AI ​​based on document data different from the retrieved document data, whether or not there is missing information in the set of first document data regarding the generation of a document based on the instruction information. A display control unit that displays the result of the determination by the determination unit, An information processing system characterized by having the following features.

2. Document data different from the retrieved document data is document data related to the instruction information in the second set of document data. The information processing system according to feature 1.

3. The determination unit controls the AI ​​to perform a determination of whether or not there is missing information based on a comparison between the first document and the second document. The information processing system according to feature 1.

4. The determination unit causes the AI ​​to score an evaluation score for the first document based on the first document and the second document. The display control unit displays the evaluation score. The information processing system according to feature 1.

5. A search unit searches for document data related to instruction information containing instructions for generating documents for the AI ​​from a first set of document data, An AI control unit that causes the AI ​​to generate a first document based on the retrieved document data and the instruction information, A determination unit determines, based on a comparison between the first document and a second document generated by the AI ​​based on document data different from the retrieved document data, whether or not there is missing information in the set of first document data regarding the generation of a document based on the instruction information. A display control unit that displays the result of the determination by the determination unit, An information processing device characterized by having the following features.

6. A search procedure for searching a first set of document data for document data related to instruction information containing instructions for generating documents for AI, An AI control procedure that causes the AI ​​to generate a first document based on the retrieved document data and the instruction information, A determination procedure for determining whether there is any missing information in the set of first document data regarding the generation of a document based on the instruction information, based on a comparison between the first document and a second document generated by the AI ​​based on document data different from the retrieved document data, A display control procedure for displaying the result of the determination procedure described above, An information processing method characterized by a computer executing the following.

7. A search procedure for searching a first set of document data for document data related to instruction information containing instructions for generating documents for AI, An AI control procedure that causes the AI ​​to generate a first document based on the retrieved document data and the instruction information, A determination procedure for determining whether there is any missing information in the set of first document data regarding the generation of a document based on the instruction information, based on a comparison between the first document and a second document generated by the AI ​​based on document data different from the retrieved document data, A display control procedure for displaying the result of the determination procedure described above, A program that causes a computer to execute something.

Images