Document generation device, document generation method, and document generation program

Abstract

This invention provides a document generation device that can easily generate documents suitable for reference purposes. [Solution] The document generation device 10 comprises an extraction unit 12, a classification unit 13, and a generation unit 14. The extraction unit 12 extracts initial queries, which are the queries initially entered from the query group, and desired answers, which are the answers requested by the user, from data that associates a series of input queries with a group of answers to the query group generated by referring to supplementary documents. The classification unit 13 classifies the supplementary documents used to generate the desired answers based on whether or not they are useful in generating the desired answers. The generation unit 14 generates new supplementary documents based on the group of documents classified as useful.

Description

Technical Field

[0001] The present disclosure relates to a document generation device and the like.

Background Art

[0002] Answers generated using a language model for inquiries from users are provided. A user repeats an inquiry to an information processing system using a language model and obtains the desired information by obtaining an answer from the information processing system. In such an information processing system, a document similar to the content of the user's inquiry is extracted from the documents stored in a database, and the language model refers to the extracted document to generate an answer, so that a method of providing an answer that matches the user's inquiry may be used. Such a method is called, for example, retrieval augmented generation. When using the retrieval augmented generation method, it is desirable that a document similar to the content of the inquiry be stored in the database.

[0003] The dialogue management device of Patent Document 1 generates an answer to an inquiry by referring to a document stored in a storage. Further, the dialogue management device of Patent Document 1 generates a reference document based on the history of inquiries and answers.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technology described in Patent Document 1, it may be difficult to generate a document suitable for the reference document.

[0006] This disclosure aims to provide a document generation device, etc., that can easily generate documents suitable for reference in order to solve the above-mentioned problems. [Means for solving the problem]

[0007] To solve the above problems, the document generation device of this disclosure includes an extraction means for extracting initial queries, which are the queries initially entered in the query group, and desired answers, which are the answers requested by the user, from data relating a series of input queries and a group of answers to the query group generated by referencing supplementary documents, which are documents referenced when generating the answers; a classification means for classifying the supplementary documents used in generating the desired answers based on whether or not they are useful in generating the desired answers; and a generation means for generating new supplementary documents based on the group of documents classified as useful.

[0008] The document generation method of this disclosure extracts the initial query, which is the first query entered in the query group, and the desired answer, which is the answer requested by the user, from data that associates a series of input queries with a group of answers to the query group generated by referring to supplementary documents, which are documents that are referenced when generating the answers. The supplementary documents used to generate the desired answer are classified based on whether or not they are useful in generating the desired answer, and new supplementary documents are generated based on the group of documents classified as useful.

[0009] The document generation program of this disclosure causes the computer to perform the following processes from data that associates a set of input queries with a set of answers to the set of queries generated by referencing supplementary documents that are referenced when generating the answers: extracting the initial query, which is the first query entered in the set of queries, and the desired answer, which is the answer requested by the user; classifying the supplementary documents used to generate the desired answer based on whether or not they are useful in generating the desired answer; and generating new supplementary documents based on the set of documents classified as useful. [Effects of the Invention]

[0010] According to this disclosure, documents suitable for reference can be easily generated. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the configuration of an information processing system in an embodiment of the present disclosure. [Figure 2] This figure shows an example of the configuration of a document generation device in an embodiment of the present disclosure. [Figure 3] This figure shows an example of the classification results of useful documents in the embodiments of this disclosure. [Figure 4] This figure schematically illustrates an example of the process for generating an additional document in an embodiment of the present disclosure. [Figure 5] This figure shows an example of a display screen for selecting a document to be used to generate an additional document in an embodiment of the present disclosure. [Figure 6] This figure shows an example of the configuration of a chat processing device in an embodiment of the present disclosure. [Figure 7] This figure shows an example of the operation flow of a document generation device in an embodiment of the present disclosure. [Figure 8] This figure shows an example of the operation flow of the chat processing device in an embodiment of the present disclosure. [Figure 9] This figure shows an example of the hardware configuration in an embodiment of the disclosure. [Modes for carrying out the invention]

[0012] Embodiments of this disclosure will be described in detail with reference to the figures. Figure 1 is a diagram showing an example of the configuration of an information processing system. The information processing system includes, for example, a document generation device 10, a chat processing device 20, a terminal device 30, and a user terminal device 40. The document generation device 10 is connected to the chat processing device 20, for example, via a network. The document generation device 10 is connected to the terminal device 30, for example, via a network. The chat processing device 20 is connected to the user terminal device 40, for example, via a network. There may be multiple chat processing devices 20, terminal devices 30, and user terminal devices 40. The number of chat processing devices 20, terminal devices 30, and user terminal devices 40 can be set as appropriate.

[0013] Information processing systems, for example, use language models to generate answers to queries entered by users. In information processing systems, query input and response output by users are performed, for example, in a chat format. For example, large-scale language models are used as language models. Specific examples of large-scale language models will be explained later. The information processing system inputs supplementary documents along with queries into the language model. The language model then generates answers to queries by referring to the supplementary documents, for example. Supplementary documents are reference documents that the language model refers to when generating answers to queries. In other words, the language model generates answers to queries using reference documents. The method of generating answers by referring to supplementary documents in large-scale language models is also called RAG (Retrieval-Augmented Generation), for example.

[0014] The information processing system, for example, extracts supplementary documents that match a query from a database of supplementary documents. Then, the information processing system generates an answer to the query, for example, using the query and the supplementary documents as input to a language model. The information processing system also stores, for example, the query and the supplementary documents used as input to the language model as historical data for each session. A session refers to, for example, a series of queries on a single theme and the process of generating answers to those queries. The information processing system generates a new supplementary document, for example, using supplementary documents that were useful in generating the desired answer to a query. The desired answer is, for example, an answer that is suitable for the user as an answer to the query. That is, the desired answer is an answer that contains the information the user needs, and it is an answer that is likely to end the session when the answer is obtained. The information processing system saves the generated supplementary document as a new supplementary document in the database of supplementary documents. By saving the generated supplementary document as a new supplementary document in the database in this way, for example, if a similar query is entered again, the information processing system can refer to the new supplementary document and generate an answer. By generating answers by referring to new supplementary documents, the information processing system can, for example, reduce the number of times a user has to repeat the query input, answer generation, and answer output process until they obtain the desired answer.

[0015] Here, a specific example of the configuration of the document generation device 10 will be described. Figure 2 is a diagram showing an example of the configuration of the document generation device 10. The document generation device 10 basically comprises an extraction unit 12, a classification unit 13, and a generation unit 14. The document generation device 10 also further comprises, for example, an acquisition unit 11, an output unit 15, and a storage unit 16.

[0016] The acquisition unit 11 acquires, for example, a series of input query groups and answer groups for the query groups generated by referring to additional documents. The acquisition unit 11 acquires, for example, from the history data storage unit 25 of the chat processing device 20, a series of input query groups by the user and answer groups for the query groups generated by referring to additional documents. The user is, for example, a person who obtains an answer to a query using the chat processing device 20. Also, the user may include a virtual entity. The virtual entity is, for example, an AI (Artificial Intelligence) agent. The virtual entity is not limited to the above.

[0017] The acquisition unit 11 acquires, for example, a query group and an answer group for the query group for each session. The acquisition unit 11 acquires, for example, for a plurality of sessions, a query group and an answer group for the query group for each session. A session is, for example, the input of related queries performed from when the user first inputs a query until a desired answer is obtained, and the process of generating an answer to the query. That is, a session is a series of processes performed to input further queries to get closer to the answer to the desired answer with respect to the answer. Also, when the user starts over from the input of the first query, for example, the next input query is treated as a new session. Also, the sessions to be acquired among the plurality of sessions may be selected by a person in charge, for example. The person in charge is, for example, a person who performs work related to the process of generating additional documents.

[0018] The acquisition unit 11 may acquire the attributes of the user associated with the query group and the response group for the query group for each session. The attributes of the user are, for example, attributes that affect the level required for the response. For example, the attributes of the user are information in one or more items among the user's expertise, proficiency, and job position. For example, when a user who is an administrator of a computer obtains an answer regarding a malfunction in the operation of the computer, the user may require a corresponding method associated with the detailed settings of the computer. On the other hand, for example, when a general user of a computer obtains an answer regarding a malfunction in the operation of the computer, the user may require a corresponding method that can resolve the malfunction without the need for specialized knowledge. In such a case, for example, by associating an attribute regarding the user's expertise with an additional document, it becomes possible to obtain a more suitable answer for the user.

[0019] The attributes of the user may be, for example, attributes that affect the content of the required response. In this case, the attributes of the user are, for example, information in one or more items among the user's affiliation, gender, age, place of residence, family composition, and occupation. The attributes of the user are not limited to the above. For example, when a user obtains information regarding administrative services, the required responses may differ depending on the user's age and place of residence because the administrative services provided differ. In such a case, for example, by associating an attribute regarding the user's age and place of residence with an additional document, it becomes possible to obtain a more suitable answer for the user.

[0020] When the initial query is selected by the person in charge, the acquisition unit 11, for example, acquires the selection result of the initial query. The acquisition unit 11, for example, acquires the selection result of the initial query from the terminal device 30. Also, when the desired response is selected by the person in charge, the acquisition unit 11, for example, acquires the selection result of the desired response. The acquisition unit 11, for example, acquires the selection result of the desired response from the terminal device 30.

[0021] When a new supplemental document is generated using a selected supplemental document from among those used to generate the answer, the acquisition unit 11 acquires, for example, the selection result of the supplemental document to be used to generate the new supplemental document. The acquisition unit 11 acquires, for example, the selection result of the supplemental document to be used to generate the new supplemental document from the terminal device 30. The supplemental documents used to generate the answer are, for example, supplemental documents input into the language model that generates the answer for the process of generating an answer to a query. That is, the supplemental documents used to generate the answer may also include supplemental documents that were input into the language model that generates the answer for a query, but which the language model did not use to generate the answer. Furthermore, if the supplemental document is a fragment of a document, the acquisition unit 11 may acquire not only the supplemental document but also fragments surrounding the document. Being a fragment of a document means, for example, that the supplemental document is part of a document. If the supplemental document is a fragment of a document, the acquisition unit 11 acquires, for example, the supplemental document along with fragments that are located close to the supplemental document in the document.

[0022] The extraction unit 12 extracts the initial query, which is the first query entered in the query group, and the desired answer, which is the answer the user is seeking, from data that associates a series of queries entered by the user with a group of answers to the query group generated by referring to supplementary documents. The series of queries entered by the user are, for example, session-specific queries entered by the user into the chat processing device 20. The extraction unit 12 extracts the initial query and the desired answer from data that associates a group of queries and an answer group included in a single session, for example.

[0023] The extraction unit 12, for example, extracts the first query of the session from among the queries included in the query group as the initial query. The extraction unit 12 may also extract queries in a predetermined order within the session from among the queries included in the query group as the initial query. The predetermined order for setting the extraction target is set, for example, to an order in which queries suitable for the initial query are most likely to be entered. For example, if the first query contains a standard phrase and specific content is entered in the second query and subsequent queries, the predetermined order would be set to second. The criteria for setting the predetermined order are not limited to the above.

[0024] The extraction unit 12 may exclude queries in a predetermined order within a session from the initial query extraction target. For example, the extraction unit 12 may exclude queries from a predetermined order onward from the initial query extraction target. The predetermined order for exclusion from the extraction target may be set so that queries unsuitable as initial queries are excluded from the extraction target. In addition, if the content of the queries included in the query group changes midway through, the extraction unit 12 may exclude the query group whose content has changed midway through, as well as the answer group to that query group, from the extraction target. A change in the content of a query means, for example, that the theme the query is targeting changes. For example, a change in the content of a query means that if the initial query is a question about the settlement method of domestic business travel expenses, the query changes midway through to a question about the transfer system.

[0025] The extraction unit 12 may extract a query selected by the user of the chat processing device 20 as the initial query. The extraction unit 12 extracts the initial query based, for example, on the user's selection result on the initial query selection screen. Alternatively, the initial query may be selected on the chat display screen where queries are entered and answers are output.

[0026] Furthermore, the extraction unit 12 may extract a query selected by the person in charge as the initial query. For example, the extraction unit 12 extracts a query selected from among the queries included in the query group as the initial query. For example, the extraction unit 12 extracts a query selected on a display screen that shows a list of queries included in the query group as the initial query. The person in charge is, for example, a person who performs work related to the generation of new supplementary documents using the document generation device 10. The user and the person in charge may be the same person. The person in charge may also include a virtual entity. That is, when the initial query is selected by the user or the person in charge, the initial query may be selected by a virtual entity. A virtual entity is, for example, an AI agent. The virtual entity is not limited to the above.

[0027] The extraction unit 12 may, for example, extract the last response in the session from among the responses included in the response group as the desired response. The extraction unit 12 may also extract responses in a predetermined order from the end of the session from among the responses included in the response group as the desired response. For example, if the last response in the session is a standard sentence unrelated to the initial query, the extraction unit 12 will select the second-to-last response in the session as the desired response. The predetermined order from the end of the session is set, for example, so that responses unrelated to the response desired by the user are not extracted as the desired response. That is, the predetermined order from the end of the session is set, for example, so that the extracted response is the response desired by the user. The extraction unit 12 may also exclude responses in a predetermined order within the session from the extraction of desired responses. For example, the extraction unit 12 excludes responses prior to a predetermined order from the extraction of desired responses. The predetermined order for exclusion from the extraction target is set, for example, so that responses unsuitable as desired responses are excluded from the extraction target.

[0028] The extraction unit 12 may extract responses selected by the user of the chat processing device 20 as desired responses. For example, the extraction unit 12 extracts desired responses based on the user's selection of desired responses on the desired response selection screen. Alternatively, the selection of desired responses may be performed on the chat display screen where queries are entered and responses are output. For example, the extraction unit 12 may extract responses that have been checked as highly rated as desired responses on the chat display screen.

[0029] Furthermore, the extraction unit 12 may extract the response selected by the person in charge as the desired response. For example, the extraction unit 12 extracts the response selected from among the responses included in the response group as the desired response. For example, the extraction unit 12 extracts the response selected on a display screen that shows a list of responses included in the response group as the desired response.

[0030] The extraction unit 12 may extract initial queries and desired answers for each session group. A session group is, for example, a group of sessions that are similar to each other. The extraction unit 12 groups sessions based on the similarity of at least one of the queries, answers, and supplementary documents. The extraction unit 12 groups sessions based on the similarity of at least one of the queries, answers, and supplementary documents, for example, using a language model.

[0031] For example, Word2vec can be used as a language model to group sessions. Alternatively, GPT-2 (Generative Pre-trained Transformer-2), GPT-3, GPT-3.5, GPT-4, or GPT-4o may be used as the language model for grouping sessions. Furthermore, Claude3, Claude3.5, T5 (Text-to-Text Transfer Transformer), BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly optimized BERT approach), or ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) may also be used as the language model for grouping sessions. The language models used for grouping sessions are not limited to those listed above.

[0032] Furthermore, when extracting initial queries and desired answers for each session group, the extraction unit 12 may select session groups with a number of sessions equal to or greater than a certain threshold as targets for extracting initial queries and desired answers. The threshold for the number of sessions is set, for example, to a number that can suppress the influence of a particular session.

[0033] The extraction unit 12 extracts, for example, the query with the highest similarity to other queries among the queries included in the session group as the initial query. The extraction unit 12 calculates the similarity to other queries for each query included in the session group. Then, the extraction unit 12 extracts, for example, the query with the highest average similarity to other queries as the initial query.

[0034] The extraction unit 12 converts each query included in the session group into a feature vector, for example, using a language model. For example, Word2vec is used as the language model. However, the language model is not limited to the above. The extraction unit 12 calculates the similarity between the feature vectors converted from each query, for example. Then, the extraction unit 12 uses the similarity between the feature vectors corresponding to each query as the similarity between queries. For example, cosine similarity or Euclidean distance can be used to measure the similarity between feature vectors. Other methods may be used to measure the similarity between feature vectors.

[0035] Furthermore, the extraction unit 12 may generate an initial query using queries with an average similarity score to other queries included in the session group, from the highest to a predetermined rank. For example, the extraction unit 12 generates a new query by synthesizing multiple queries up to a predetermined rank using a language model. For example, Word2vec can be used as the language model for generating the new query. The language model for generating the new query may also be a large-scale language model. The language model for generating the new query is not limited to the above. The predetermined rank is set, for example, to a number that leaves the synthesized query with features suitable for an initial query. Remaining features means, for example, that it can be distinguished from other queries in different session groups. The extraction unit 12 then treats the generated new query as the initial query. Furthermore, the extraction unit 12 may generate an initial query using multiple queries with an average similarity score to other queries that is above a predetermined standard. For example, the extraction unit 12 generates an initial query by synthesizing multiple queries with an average similarity score to be above a predetermined standard using a language model. The predetermined standard is set, for example, so that the characteristics of the session group remain in the synthesized query.

[0036] The extraction unit 12 extracts, for example, the response with the highest similarity to other responses from among the responses included in the session group as the desired response. The extraction unit 12 calculates the similarity to other responses for each response included in the session group. Then, the extraction unit 12 extracts, for example, the response with the highest average similarity to other responses as the desired response.

[0037] The extraction unit 12 converts each response included in the session group into a feature vector, for example, using a language model. For example, Word2vec is used as the language model for converting to feature vectors. However, the language model for converting to feature vectors is not limited to the above. The extraction unit 12 calculates the similarity between the feature vectors converted from each response, for example. Then, the extraction unit 12 uses the similarity between the feature vectors corresponding to each response as the similarity between the responses. For example, cosine similarity or Euclidean distance can be used for the similarity between feature vectors. Other methods may be used for the similarity between feature vectors.

[0038] The extraction unit 12 may generate a desired answer using answers with an average similarity score to other answers included in the session group, from the highest to a predetermined rank. For example, the extraction unit 12 generates a new answer by synthesizing multiple answers up to a predetermined rank using a language model. For example, Word2vec can be used as the language model for generating the new answer. A large-scale language model may also be used as the language model for generating the new answer. The language model for generating the new answer is not limited to the above. The predetermined rank is set, for example, to a number that retains features suitable for a desired answer in the synthesized answer. Retaining features means, for example, that it can be distinguished from other answers in different session groups. The extraction unit 12 then treats the generated new answer as the desired answer. The extraction unit 12 may also generate a desired answer using multiple answers with an average similarity score to other answers that is above a predetermined standard. For example, the extraction unit 12 generates a desired answer by synthesizing multiple answers with an average similarity score to be above a predetermined standard using a language model. The predetermined standard is set, for example, so that the features of the session group remain in the synthesized answer.

[0039] The classification unit 13 classifies the supplementary documents used to generate the desired answer based on whether or not they are useful in generating the desired answer. For example, the classification unit 13 classifies whether or not the supplementary documents used to generate the answer are useful documents for generating the desired answer. For example, the classification unit 13 uses a language model to classify whether or not the supplementary documents referenced in generating the desired answer are useful documents for generating the desired answer. For example, the classification unit 13 uses a prompt including an instruction to classify whether or not a document is useful for generating the desired answer, an initial query, the desired answer, and the referenced supplementary documents as input to a language model to classify whether or not the supplementary documents referenced in generating the desired answer are useful documents for generating the desired answer. That is, for example, the classification unit 13 uses a prompt including an instruction to classify whether or not a document is useful for generating the desired answer, an initial query, the desired answer, and the referenced supplementary documents as input to a language model to classify whether or not the supplementary documents referenced in generating the desired answer are useful documents for generating the desired answer.

[0040] For classifying supplementary documents, a large-scale language model may be used. For example, GPT-2, GPT-3, GPT-3.5, GPT-4, or GPT-4o may be used. Alternatively, Claude3, Claude3.5, T5, BERT, RoBERTa, or ELECTRA may be used. The language models used for classifying supplementary documents are not limited to those listed above.

[0041] The classification unit 13 may calculate the usefulness of the supplementary documents used to generate the desired response. The classification unit 13 calculates the usefulness of the supplementary documents used to generate the desired response, for example, using a language model. The classification unit 13 then classifies supplementary documents with a usefulness score above a certain threshold as useful supplementary documents. The usefulness threshold for classifying supplementary documents is set, for example, so that when the usefulness score meets the threshold, the supplementary document becomes a document suitable for generating an answer to the query. The classification unit 13 may classify each referenced supplementary document into a group based on a usefulness score set in multiple stages. For example, the classification unit 13 classifies each referenced supplementary document into a group based on a usefulness score set in three stages. There may be four or more stages of usefulness. Alternatively, the language model may perform the calculation of usefulness and the classification process based on usefulness. In this case, the language model may output the usefulness score of each supplementary document along with the usefulness classification result.

[0042] The classification unit 13 may classify the supplementary documents using a language model that operates on an information processing device outside the information processing system. The classification unit 13 outputs a prompt to an information processing device operating a large-scale language model, for example, that includes an instruction to classify whether a document is useful for generating a desired answer, an initial query, a desired answer, and the referenced supplementary documents. The classification unit 13 then obtains the classification result of the supplementary documents based on their usefulness in generating the desired answer from the information processing device to which the prompt was output, for example.

[0043] The classification unit 13 may classify the supplementary documents using a language model operating in the chat processing unit 20. For example, the classification unit 13 outputs prompts to the chat processing unit 20 that include instructions to classify whether a document is useful for generating a desired response, an initial query, a desired response, and referenced supplementary documents. The classification unit 13 then obtains the classification results of the supplementary documents based on their usefulness in generating the desired response from the chat processing unit 20.

[0044] The classification unit 13 may classify the additional documents at a timing based on the load status of the information processing device on which the language model operates. The classification unit 13 may classify the additional documents using the language model at a timing based on the usage status of the language model that generates answers to queries, for example. For example, the classification unit 13 may classify the additional documents at a timing based on the load status of the chat processing device 20 or an external information processing device of the information processing system on which the language model operates. The classification unit 13 may classify the additional documents when the load status of the information processing device on which the language model operates is below a certain threshold. The threshold for load status is set, for example, so that the classification process does not affect other processes. The classification unit 13 may also classify the additional documents during a predetermined time period. The predetermined time period is set, for example, so that the processing capacity of the information processing device that performs the classification of additional documents is available. Furthermore, if usage fees are set according to the level of processing volume by the language model per unit period, the classification unit 13 may determine the timing of the classification process so that the usage fee does not increase even if the processing is performed, and then perform the classification process. For example, if the contract stipulates that the usage fee increases when the monthly usage of the language model exceeds the processing volume P, the classification unit 13 performs a process to classify additional documents so as not to exceed the processing volume P. The available processing volume in the language model is set, for example, in units of tokens.

[0045] Furthermore, the classification unit 13 may classify the additional documents at a timing based on the usage status of the chat processing device 20 or an external information processing device on which the language model operates. The classification unit 13 may also classify the additional documents at a timing based on the condition that the input / output throughput of the language model falls below a predetermined threshold. Input / output throughput is an indicator, for example, the number of tokens per unit time. The predetermined threshold is set, for example, to a value that does not affect other processes even if processing related to the classification of additional documents is performed. The predetermined threshold may also be determined by the terms of the usage agreement for the information processing device on which the language model operates.

[0046] When a language model that generates an answer to a query outputs supplementary documents that it referenced during answer generation, the classification unit 13 may acquire the supplementary documents referenced by the language model. In this case, the classification unit 13 may, for example, classify the acquired supplementary documents referenced by the language model into useful documents.

[0047] The generation unit 14 generates supplemental documents based on supplemental documents classified as useful. The generation unit 14 generates new supplemental documents by summarizing the supplemental documents classified as useful. For example, suppose that among supplemental documents A, B, C, D, and E, supplemental documents A, B, and E are classified as useful documents. In this case, the generation unit 14, for example, summarizes supplemental documents A, B, and E to generate supplemental document N as a new supplemental document.

[0048] The generation unit 14, for example, uses a language model to summarize supplementary documents classified as useful documents. The generation unit 14, for example, uses supplementary documents classified as useful documents and summarization instructions as input to a language model and generates a summary of the supplementary documents classified as useful documents as a new supplementary document. The generation unit 14, using an initial query, supplementary documents classified as useful documents and summarization instructions as input to a language model and generates a summary of the supplementary documents classified as useful documents as a new supplementary document.

[0049] For example, a large-scale language model may be used as the language model. For example, GPT-2, GPT-3, GPT-3.5, GPT-4, or GPT-4o may be used as the large-scale language model for generating summaries. Alternatively, Claude3, Claude3.5, T5, BERT, RoBERTa, or ELECTRA may also be used as the large-scale language model for generating summaries. The large-scale language models used for the processing of classifying supplementary documents are not limited to those mentioned above. Furthermore, the language model used by the generation unit 14 for summarization and the language model used by the classification unit 13 for classifying whether a document is useful for generating the desired answer may be the same language model, or they may be different language models.

[0050] The generation unit 14 may summarize supplementary documents classified as useful documents at a timing based on the usage status of the chat processing device 20 or an external information processing device on which the language model operates. The generation unit 14 may summarize supplementary documents classified as useful documents at a timing based on, for example, the input / output throughput of the language model, provided that it falls below a predetermined threshold. Input / output throughput is, for example, an indicator represented by the number of tokens per unit time. The predetermined threshold is set, for example, to a value such that processing related to summarizing supplementary documents classified as useful documents does not affect other processing. The predetermined threshold may also be determined by the terms of the usage agreement for the information processing device on which the language model operates.

[0051] When the classification unit 13 calculates the usefulness of each additional document, the generation unit 14 may generate a summary by weighting each additional document based on its usefulness. For example, the generation unit 14 generates a summary weighted based on usefulness by associating the usefulness of each additional document with the language model and inputting it. When the classification unit 13 classifies the additional documents into groups based on usefulness levels set in multiple stages, the generation unit 14 may generate new additional documents by summarizing the additional documents by weighting each group according to the usefulness level. For example, if usefulness is set in three stages, the generation unit 14 generates a new document by summarizing the additional documents with weights corresponding to each of the three stages. The generation unit 14 may also generate a summary weighted based on usefulness by associating each additional document with the weight calculated from the usefulness of each additional document and inputting it into the language model.

[0052] Figure 3 shows an example of classification results based on the usefulness of supplementary documents. In the example of supplementary document classification results in Figure 3, each supplementary document used as input to the language model is classified as either a "useful document" or a "not useful document." For example, in the example of classification results in Figure 3, Document 1 "Domestic Business Trip Manual," Document 2 "Transportation Expense Payment Regulations," and Document 4 "Travel Expense Payment Regulations" are classified as "useful documents." Also, in the example of classification results in Figure 3, for example, Document 3 "Overseas Business Trip Manual" is classified as a "not useful document." The generation unit 14 generates new supplementary documents using, for example, the documents classified as "useful documents" in the example of classification results in Figure 3.

[0053] Figure 4 shows an example of a process for generating a new supplementary document using supplementary documents classified as useful, as in the example in Figure 3. In the example in Figure 4, the generation unit 14 summarizes documents classified as useful, such as Document 1, Document 2, and Document 4, and generates a new document N as a new supplementary document. In this way, by generating a new document from supplementary documents classified as useful, it is possible to generate supplementary documents that are more suitable for generating answers to queries.

[0054] The output unit 15 outputs, for example, the new supplementary document generated by the generation unit 14. The output unit 15 outputs the generated new supplementary document to, for example, the database unit 24 of the chat processing device 20. The output unit 15 may also output the generated new supplementary document with an associated initial query.

[0055] The output unit 15 may associate and save the user's attributes with the newly generated supplementary document. The user's attributes are, for example, the attributes of the person who obtains the response using the chat processing unit 20. The user's attributes are obtained from the chat processing unit 20, for example, along with data that associates the query group used to generate the new supplementary document with the response group to the query group. The user's attributes may also be added to the new supplementary document by the person responsible for generating the supplementary document. The user's attributes may also be attributes estimated from the input query. For example, the user's attributes are estimated by referring to table-formatted data that associates the query with the user's attributes. The user's attributes may also be estimated using a deep learning model generated by learning the relationship between the query and the user's attributes. The user's attributes may also be estimated by inputting a query as a prompt that requests the estimation of the user's attributes to a large-scale language model. The query used to estimate the user's attributes is not limited to the initial query. The estimation of the user's attributes is performed, for example, in the generation unit 14. The estimation of the user's attributes may also be performed outside of the generation unit 14. The user's attributes may be estimated in the chat processing device 20, and the acquisition unit 11 may acquire the estimated user attributes from the chat processing device 20.

[0056] The output unit 15 may output the newly generated supplemental document in association with the supplemental document used to generate the new supplemental document. Furthermore, the output unit 15 may highlight and output text from the supplemental document used to generate the new supplemental document that is related to the text included in the new supplemental document.

[0057] If an initial query is selectable, the output unit 15 outputs, for example, a display screen for selecting an initial query. The output unit 15 outputs, for example, a display screen for selecting an initial query to a terminal device 30. The output unit 15 outputs, for example, a display screen that shows candidate initial queries and a selection field as the display screen for selecting an initial query.

[0058] If a desired answer is selectable, the output unit 15 outputs, for example, a display screen for selecting a desired answer. The output unit 15 outputs, for example, a display screen for selecting a desired answer to the terminal device 30. The output unit 15 outputs, for example, a display screen that shows candidate desired answers and a selection field as the display screen for selecting a desired answer.

[0059] The output unit 15 may output a display screen for selecting an additional document to be used to generate a new additional document from among the additional documents used to generate the desired response. For example, the output unit 15 outputs a display screen for selecting an additional document to be used to generate a new additional document to the terminal device 30.

[0060] Figure 5 shows an example of a display screen for selecting documents to be used to generate new supplemental documents. In the example display screen of Figure 5, the "Supplemental Document List," the "Application Requirement" selection field, and the "Generate Document" button are displayed. In the example display screen of Figure 5, the "Supplemental Document List" field displays a list of supplemental documents used, for example, for input to a language model. The "Application Requirement" selection field is a field for selecting supplemental documents to be used for summarizing to generate new supplemental documents. In the example display screen of Figure 5, for example, a black square indicates that a supplemental document has been selected. Also, in the example display screen of Figure 5, for example, a white square indicates that no supplemental document has been selected. Also, in the example display screen of Figure 5, the "Generate Document" button is a button to start generating a new supplemental document using the selected supplemental document.

[0061] The output unit 15 outputs, for example, a display screen to the terminal device 30 for selecting the additional documents to be used to generate a new additional document, as shown in the example in Figure 5. The acquisition unit 11 acquires the selection result of the additional documents to be used to generate a new additional document from the terminal device 30, which is entered by the operator on the display screen displayed on the display device of the terminal device 30. The generation unit 14 then generates a new additional document by summarizing the additional documents selected in the selection result.

[0062] The storage unit 16 stores, for example, data related to the process of generating supplementary documents. The storage unit 16 stores, for example, a set of queries entered by the user and a set of answers to the queries generated by referring to supplementary documents. The storage unit 16 stores, for example, extracted initial queries and desired answers. The storage unit 16 stores, for example, newly generated supplementary documents. If an initial query is selectable, the storage unit 16 stores, for example, the selection result of the initial query. If a desired answer is selectable, the storage unit 16 stores, for example, the selection result of the desired answer. The storage unit 16 may also store language models used for each process. The data stored by the storage unit 16 is not limited to the above.

[0063] An example of the configuration of the chat processing device 20 will be described. Figure 6 shows an example of the configuration of the chat processing device 20. The chat processing device 20 includes, for example, an interface unit 21, a chat control unit 22, a chat processing unit 23, a database unit 24, and a history data storage unit 25.

[0064] The interface unit 21 acquires queries entered by user operations, for example. The interface unit 21 acquires queries entered by user operations from, for example, the user terminal device 40. The interface unit 21 also outputs answers to queries generated in, for example, the chat processing unit 23. The interface unit 21 outputs answers to queries to, for example, the user terminal device 40.

[0065] The chat control unit 22, for example, obtains a query from the interface unit 21. The chat control unit 22, for example, extracts supplementary documents that match the query. Then, the chat control unit 22 outputs, for example, the query and the extracted supplementary documents to the chat processing unit 23.

[0066] The chat control unit 22 extracts supplementary documents based on the similarity between the feature vector converted from the query and the feature vector converted from the supplementary document. The chat control unit 22 converts the query into a feature vector using a language model, for example. Then, it calculates the similarity between the feature vector converted from the query and the feature vector converted from the supplementary document. The chat control unit 22 calculates the similarity between the feature vector converted from the query and the feature vector converted from the supplementary document using Euclidean distance or cosine similarity, for example. The indicators used to show similarity are not limited to Euclidean distance and cosine similarity. Furthermore, a language model such as Word2vec can be used to convert the query into a feature vector. The language model used to convert the query into a feature vector is not limited to the above.

[0067] The chat control unit 22 may, for example, extract supplementary documents with a similarity score above a predetermined standard as supplementary documents that match the query. The chat control unit 22 may also extract supplementary documents with similarity scores from the top to a predetermined rank as supplementary documents that match the query. The chat control unit 22 may also extract supplementary documents with similarity scores above a predetermined standard and from the top to a predetermined rank as supplementary documents that match the query. The predetermined standard and predetermined rank used for extracting similarity are set so that the extracted supplementary documents become supplementary documents suitable for the query.

[0068] The chat control unit 22 may extract supplementary documents based on user attributes in addition to the similarity between the query and the supplementary documents. User attributes are, for example, attributes that influence at least one of the required level and content of the answer. The chat control unit 22 extracts supplementary documents that match the query based on similarity from among supplementary documents whose associated attributes match the attributes of the user who entered the query. Matching includes similarity. By using supplementary documents extracted based on user attributes to generate answers, it is possible to generate answers that are more suitable for the user, even for similar queries.

[0069] User attributes are stored, for example, in association with user identification information. User identification information is, for example, a user identification number. User identification information is not limited to the above. The chat control unit 22 may also estimate user attributes using a deep learning model generated by learning the relationship between queries and user attributes. Alternatively, the chat control unit 22 may estimate user attributes by inputting a query as a prompt that requests a large-scale language model to estimate user attributes. The query used to estimate user attributes is not limited to the initial query.

[0070] The chat control unit 22 outputs a query and supplementary documents to the chat processing unit 23, for example. The chat control unit 22 also retrieves the answer to the query from the chat processing unit 23, for example. The chat control unit 22 then outputs the retrieved answer to the query to the interface unit 21. The chat control unit 22 also stores the query, supplementary documents, and the answer to the query in association with each other in the history data storage unit 25, for example. The chat control unit 22 stores the query, supplementary documents, and the answer to the query in association with each session in the history data storage unit 25, for example. The chat control unit 22 may also store the query, supplementary documents, and the answer to the query in association with the order of the queries within the session in the history data storage unit 25. The chat control unit 22 may also store the query, supplementary documents, and the answer to the query in association with the order of the queries within the session, along with the user's attributes.

[0071] The chat processing unit 23 generates, for example, a response to a query. The chat processing unit 23 generates a response to a query using, for example, a language model. The language model used is, for example, a large-scale language model that generates documents by referencing supplementary documents. Examples of large-scale language models include GPT-2, GPT-3, GPT-3.5, GPT-4, or GPT-4o. Alternatively, Claude3, Claude3.5, T5, BERT, RoBERTa, or ELECTRA may also be used as large-scale language models.

[0072] The chat processing unit 23 may generate answers to queries based on the user's attributes. For example, the chat processing unit 23 generates answers based on the user's attributes by using a prompt containing information indicating the user's attributes as input to the language model. For example, the chat processing unit 23 generates answers suitable for beginners by using a prompt instructing the language model to generate answers suitable for beginners as input to the language model.

[0073] The chat processing unit 23 may classify the supplementary documents using a language model operating on an external information processing device to the chat processing unit 20. The chat processing unit 23 outputs queries and supplementary documents to, for example, an information processing device operating a large-scale language model. The chat processing unit 23 then retrieves, for example, the answer to the query from the information processing device that received the queries and supplementary documents.

[0074] The database unit 24 stores, for example, supplementary documents. The database unit 24 stores, for example, supplementary documents in a vectorized state. Furthermore, when, for example, a new supplementary document is acquired, the database unit 24 stores the newly acquired document in a vectorized state. In addition, the supplementary documents may be associated with user attributes suitable for generating responses using those supplementary documents.

[0075] The history data storage unit 25 stores, for example, queries, responses to queries, the order of query inputs, and supplementary documents in association with each other. The history data storage unit 25 stores, for example, data that associates queries, responses to queries, the order of query inputs, and supplementary documents, grouped by session.

[0076] The terminal device 30 is an information processing device used by, for example, a person responsible for generating new supplementary documents in an information processing system. The terminal device 30 obtains the new supplementary document from, for example, the output unit 15 of the document generation device 10. The terminal device 30 then outputs the new supplementary document to, for example, a display device (not shown).

[0077] When the initial query is selected by the person in charge, the terminal device 30 obtains, for example, the display screen for selecting the initial query from the output unit 15 of the document generation device 10. The terminal device 30 then outputs the display screen for selecting the initial query to, for example, a display device (not shown). The terminal device 30 also obtains, for example, the selection result of the initial query entered by the person in charge on the display screen for selecting the initial query. The terminal device 30 then outputs the selection result of the initial query to, for example, the acquisition unit 11 of the document generation device 10.

[0078] When the person in charge selects the desired answer, the terminal device 30 obtains, for example, the display screen for selecting the desired answer from the output unit 15 of the document generation device 10. The terminal device 30 then outputs the display screen for selecting the desired answer to, for example, a display device (not shown). The terminal device 30 also obtains, for example, the selection result of the desired answer entered by the person in charge on the display screen for selecting the desired answer. The terminal device 30 then outputs the selection result of the desired answer to, for example, the acquisition unit 11 of the document generation device 10.

[0079] When an employee selects additional documents to be classified as useful, the terminal device 30 obtains, for example, a display screen for selecting additional documents to be classified as useful from the output unit 15 of the document generation device 10. The terminal device 30 then outputs the display screen for selecting additional documents to be classified as useful to, for example, a display device (not shown). The terminal device 30 also obtains, for example, the selection result of additional documents to be classified as useful, which is entered by the employee's operation on the display screen for selecting additional documents to be classified as useful. The terminal device 30 then outputs the selection result of additional documents to be classified as useful to, for example, the acquisition unit 11 of the document generation device 10.

[0080] The terminal device 30 can be, for example, a notebook computer or a desktop computer. Furthermore, the terminal device 30 is not limited to the above.

[0081] The user terminal device 40 is, for example, an information processing device used by a user to obtain answers to queries in an information processing system. The user terminal device 40 obtains queries entered by the user through user operations. The user terminal device 40 then outputs the queries entered by the user to, for example, the interface unit 21 of the chat processing device 20. The user terminal device 40 also obtains answers to queries from, for example, the interface unit 21 of the chat processing device 20. The user terminal device 40 then outputs answers to queries to, for example, a display device (not shown).

[0082] The user terminal device 40 can be, for example, a notebook computer, a desktop computer, a tablet computer, a smartphone, or a smart device. However, the user terminal device 40 is not limited to the above. Furthermore, the terminal device 30 and the user terminal device 40 may be the same information processing device.

[0083] An example of the operation of the document generation device 10 in the process of generating supplementary documents will be described. Figure 7 shows an example of the operation flow in the document generation device 10 in the process of generating supplementary documents.

[0084] The acquisition unit 11 acquires data that associates a set of input queries with a set of answers to those queries generated by referring to supplementary documents (step S11). The acquisition unit 11 acquires data that associates a set of queries with a set of answers to those queries from the chat processing device 20, for example. The acquisition unit 11 may also acquire user attributes associated with the set of queries and the set of answers to those queries.

[0085] The extraction unit 12 extracts the initial query and the desired answer, which is the answer requested by the user, from data that associates the input series of queries with the set of answers to the queries generated by referring to the supplementary documents (step S12). The initial query is the query that was entered first from the set of queries.

[0086] Once the initial query and desired response are extracted, the classification unit 13 classifies the supplementary documents used to generate the desired response based on whether or not they were useful in generating the desired response (step S13).

[0087] Once the supplementary documents are classified, the generation unit 14 generates new supplementary documents based on the group of documents classified as useful (step S14).

[0088] When an additional document is generated, the output unit 15 outputs the newly generated additional document (step S15). The output unit 15 outputs the generated additional document to the database unit 24 of the chat processing device 20, for example. The output unit 15 may also output the newly generated additional document with the user attributes associated with the query group used to generate the additional document and the answer group to the query group.

[0089] This section describes an example of the operation of the chat processing unit 20 in the process of generating a response to a query. Figure 8 shows an example of the operation flow in the chat processing unit 20 in the process of generating a response to a query.

[0090] The interface unit 21, for example, obtains the input query (step S21). The interface unit 21 obtains the query input by the user's operation from the user terminal device 40, for example.

[0091] When a query is obtained, the chat control unit 22 extracts additional documents that match the query (step S22). The chat control unit 22 extracts additional documents that match the query from, for example, the additional documents stored in the database unit 24. The chat control unit 22 may also extract additional documents that match the query based on the user's attributes.

[0092] Once supplementary documents matching the query are extracted, the chat processing unit 23 generates an answer to the query by referring to the supplementary documents, for example (step S23). The chat processing unit 23 may also generate an answer to the query based on the user's attributes.

[0093] When a response to a query is generated, the interface unit 21 outputs the generated response, for example (step S24). The interface unit 21 outputs the generated response to, for example, the user terminal device 40.

[0094] If an additional query is entered for the outputted response (Yes in step S25), the process returns to step S22, and the chat control unit 22 extracts, for example, additional documents that match the entered additional query.

[0095] If the session ends without any additional queries being entered for the outputted response (No in step S25), the chat control unit 22 saves, for example, the query, the response to the query, and any additional documents associated with each session (step S26). The chat control unit 22 may also save, for example, the user's attributes further associated with the query, the response to the query, and any additional documents.

[0096] The document generation device 10 extracts the initial query, which is the first query entered in the query group, and the desired answer, which is the answer requested by the user, from data that associates the input series of queries with the answer group generated by referring to supplementary documents. The document generation device 10 classifies the supplementary documents used to generate the desired answer based on whether or not they were useful in generating the desired answer. Then, the document generation device 10 generates supplementary documents based on the documents classified as useful. By generating documents in this way, documents suitable for reference can be easily generated.

[0097] Furthermore, by generating the supplementary document generated as described above to produce the answer to the query, the number of processing steps required to reach the desired answer can be reduced when a query similar to the initial query is entered. Therefore, generating the supplementary document as described above can reduce the computer resources and power consumption required to generate the answer. For this reason, the answer to the query can be obtained efficiently by generating the supplementary document as described above.

[0098] Furthermore, by extracting initial queries and desired answers for each group of sessions (which are then categorized into groups) and generating supplementary documents, it is possible to generate supplementary documents while suppressing the effects of variations in how users express queries. This allows for the more efficient generation of documents suitable for reference.

[0099] Each process in the document generation device 10 can be realized by executing a computer program on a computer. Figure 9 shows an example of the configuration of a computer 100 that executes a computer program to perform each process in the document generation device 10. The computer 100 includes a CPU (Central Processing Unit) 101, memory 102, storage device 103, input / output interface 104, and communication interface 105.

[0100] The CPU 101 reads and executes computer programs that perform various processes from the storage device 103. The CPU 101 may be composed of a combination of multiple CPUs. Alternatively, the CPU 101 may be composed of a combination of a CPU and another type of processor. For example, the CPU 101 may be composed of a combination of a CPU and a GPU. The memory 102 is composed of DRAM (Dynamic Random Access Memory) or the like, and temporarily stores computer programs executed by the CPU 101 and data being processed. The storage device 103 stores computer programs executed by the CPU 101. The storage device 103 is composed of, for example, a non-volatile semiconductor storage device. Other storage devices such as hard disk drives may be used for the storage device 103. The input / output interface 104 is an interface that receives input from the operator and outputs display data, etc. The communication interface 105 is an interface that sends and receives data between the chat processing device 20, the terminal device 30, and the user terminal device 40. Furthermore, the chat processing unit 20, terminal device 30, and user terminal device 40 may have the same configuration as the computer 100.

[0101] The computer programs used to execute each process can also be stored and distributed on a computer-readable recording medium that non-temporarily stores data. Examples of recording media include magnetic tapes for data recording and magnetic disks such as hard disks. Optical discs such as CD-ROMs (Compact Disc Read Only Memory) can also be used as recording media. Non-volatile semiconductor memory devices may also be used as recording media.

[0102] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0103] [Note 1] An extraction means for extracting initial queries, which are the queries initially entered from the query group, and desired answers, which are the answers requested by the user, from data that associates a set of input queries with a set of answers for the said query group generated by referring to an additional document, which is a document referenced when generating the answers. A classification means for classifying the supplementary documents used in generating the desired response based on whether or not they were useful in generating the desired response, A generation means that generates new supplementary documents based on a group of documents classified as useful documents. A document generation device equipped with the following features.

[0104] [Note 2] The classification means uses a language model to classify each of the supplementary documents used to generate the responses included in the response group. The document generation device described in Appendix 1.

[0105] [Note 3] The classification means classifies the supplementary documents using the language model at a timing based on the usage status of the language model that generates answers to queries. The document generation device described in Appendix 2.

[0106] [Note 4] The generation means generates new supplementary documents by summarizing supplementary documents included in a group of documents classified as useful documents using a language model. The document generation device described in Appendix 1.

[0107] [Note 5] The classification means classifies each of the referenced supplementary documents into groups based on usefulness levels set in at least three stages, The generation means generates the new additional document by assigning weights to each group according to its usefulness level. The document generation device described in Appendix 4.

[0108] [Note 6] The extraction means calculates the average similarity score for each of the multiple queries included in the session group with other queries included in the query group, and extracts the query with the highest calculated average similarity score as the initial query. A document generation device as described in any of the appendices 1 to 5.

[0109] [Note 7] The extraction means generates queries based on multiple queries included in the session group, and extracts the generated queries as the initial queries. A document generation device as described in any of the appendices 1 to 5.

[0110] [Note 8] The extraction means calculates the average similarity of each of the multiple queries included in the session group with other queries included in the session group, and generates queries to be extracted as initial queries based on queries whose calculated average similarity is above a predetermined standard. The document generation device described in Appendix 7.

[0111] [Note 9] The extraction means extracts queries from among a group of queries within a session that are entered in a predetermined order as the queries. A document generation device as described in any of the appendices 1 to 5.

[0112] [Note 10] The extraction means extracts the last generated response from among the group of responses within the session as the desired response. A document generation device as described in any of the appendices 1 through 9.

[0113] [Note 11] The extraction means, when the content of a query included in the query group changes midway through the process, excludes the query group whose content has changed midway through the process, and the set of answers to the query group, from the extraction target. A document generation device as described in any of the appendices 1 through 10.

[0114] [Note 12] The classification means classifies the supplementary documents using a language model different from the language model that generates the response to the query. The document generation device described in Appendix 2.

[0115] [Note 13] The generation means summarizes documents included in a group of documents classified as useful documents using a language model different from the language model that generates the answer to the query. The document generation device described in Appendix 4.

[0116] [Note 14] The information processing device that controls the process of generating answers to queries is further equipped with output means for outputting the newly generated supplementary document to a database used by the device. The document generation device described in Appendix 1 to 13.

[0117] [Note 15] The output means vectorizes the newly generated additional document and outputs it to the database. The document generation device described in Appendix 14.

[0118] [Note 16] The information processing device controls the process of generating answers to queries, and further comprises acquisition means for acquiring data relating the group of queries and the group of answers to the group of queries. The document generation device described in Appendix 1 to 15.

[0119] [Note 17] The output means outputs a display screen for selecting an additional document to be used to generate a new additional document from among the additional documents classified as useful documents. The acquisition means acquires the selection result of the additional document selected on the display screen, The generation means generates the new additional document using the additional document selected in the selection result. The document generation device described in Appendix 1 to 16.

[0120] [Note 18] From data that associates a set of input queries with a set of answers to those queries generated by referring to an additional document, which is a document referenced when generating the answers, the initial query, which is the first query input from the set of queries, and the desired answer, which is the answer requested by the user, are extracted. The supplementary documents used to generate the aforementioned desired response are classified based on whether or not they are useful in generating the aforementioned desired response. Based on a group of documents classified as useful, new supplementary documents are generated. Document generation method.

[0121] [Note 19] A process to extract the initial query, which is the first query entered in the query group, and the desired answer, which is the answer requested by the user, from data that associates a set of input queries with a set of answers to the said queries generated by referring to an additional document, which is a document referenced when generating the answers. A process for classifying the supplementary documents used to generate the desired response based on whether or not they were useful in generating the desired response, A process that generates new supplementary documents based on a group of documents classified as useful. A document generation program that causes a computer to execute a command.

[0122] Furthermore, some or all of the configurations described in Appendices 2 to 17, which are dependent on Appendice 1 above, may also be dependent on Appendices 18 and 19 in the same way as those described in Appendices 2 to 17. Moreover, not limited to Appendices 1, 18, and 19, some or all of the configurations described as appendices may also be dependent on various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.

[0123] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the forms of implementation described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate. [Explanation of Symbols]

[0124] 10 Document generation device 11 Acquisition Department 12 Extraction part 13 Classification section 14 Generation part 15 Output section 16 Memory section 20 Chat Processing Devices 21 Interface section 22 Chat Control Unit 23 Chat Processing Unit 24 Database Department 25 History data storage unit 30 Terminal devices 40. User terminal device 100 Computers 101 CPU 102 memory 103 Storage device 104 Input / Output Interfaces 105 Communication I / F

Claims

1. An extraction means for extracting initial queries, which are the queries initially entered from the query group, and desired answers, which are the answers requested by the user, from data that associates a set of input queries with a set of answers for the said query group generated by referring to an additional document, which is a document referenced when generating the answers. A classification means for classifying the supplementary documents used in generating the desired response based on whether or not they were useful in generating the desired response, A generation means that generates new supplementary documents based on a group of documents classified as useful documents. A document generation device equipped with the following features.

2. The classification means uses a language model to classify each of the supplementary documents used to generate the responses included in the response group. The document generation apparatus according to claim 1.

3. The classification means classifies the supplementary documents using the language model at a timing based on the usage status of the language model that generates answers to queries. The document generation apparatus according to claim 2.

4. The generation means generates new supplementary documents by summarizing supplementary documents included in a group of documents classified as useful documents using a language model. The document generation apparatus according to claim 1.

5. The classification means classifies each of the referenced supplementary documents into groups based on usefulness levels set in at least three stages, The generation means generates the new additional document by assigning weights to each group according to its usefulness level. The document generation apparatus according to claim 4.

6. The extraction means calculates the average similarity score for each of the multiple queries included in the session group with other queries included in the query group, and extracts the query with the highest calculated average similarity score as the initial query. A document generation apparatus according to any one of claims 1 to 5.

7. The extraction means generates queries based on multiple queries included in the session group, and extracts the generated queries as the initial queries. A document generation apparatus according to any one of claims 1 to 5.

8. The extraction means calculates the average similarity of each of the multiple queries included in the session group with other queries included in the session group, and generates queries to be extracted as initial queries based on queries whose calculated average similarity is above a predetermined standard. The document generation apparatus according to claim 7.

9. From data that associates a set of input queries with a set of answers to those queries generated by referring to an additional document, which is a document referenced when generating the answers, the initial query, which is the first query input from the set of queries, and the desired answer, which is the answer requested by the user, are extracted. The supplementary documents used to generate the aforementioned desired response are classified based on whether or not they are useful in generating the aforementioned desired response. Based on a group of documents classified as useful, new supplementary documents are generated. Document generation method.

10. A process to extract the initial query, which is the first query entered in the query group, and the desired answer, which is the answer requested by the user, from data that associates a set of input queries with a set of answers to the said queries generated by referring to an additional document, which is a document referenced when generating the answers. A process for classifying the supplementary documents used to generate the desired response based on whether or not they were useful in generating the desired response, A process that generates new supplementary documents based on a group of documents classified as useful. A document generation program that causes a computer to execute a command.

Images