Method, computer device, and computer program for providing domain-specific dialogue utilizing language models

Abstract

To provide a method, a computer device, and a computer program for providing a dialogue dedicated to a domain by using a language model.SOLUTION: The method includes the steps of: generating at least one piece of query data 302 which can answer in a given document 301 by using a language model, for the document; and using data made of a pair (Document, Query) of the document which belongs to a specific domain and the query data generated for the document, for learning of a search model 320 for a conversation bot.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The following description relates to a technique for providing answers to user questions with reference to appropriate documents. [Background technology]

[0002] A QA (question and answer) system is a dialogue system that, when a user inputs a question, provides an answer while referring to appropriate documents.

[0003] In recent years, the performance of dialogue systems has improved rapidly with the development of language models, and the capabilities of such systems have expanded beyond natural language to include multi-modal domains such as images and speech.

[0004] As an example, Patent Document 1 (published on April 25, 2002) discloses a technique for providing answers to questions by utilizing artificial intelligence technology and natural language processing technology. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Korean Patent Publication No. 10-2002-0030545 Summary of the Invention [Problem to be solved by the invention]

[0006] We provide a technology that uses a large-scale language model to generate a conversational bot specialized for a given domain based on documents in that domain. [Means for solving the problem]

[0007] The present invention provides a search model for operating a computer, which is trained to select documents that may contain answers to input user questions based on data consisting of pairs (Document, Query) of each document belonging to a specific domain and a query generated for that document, and to output answers to the user questions.

[0008] According to one aspect, a search model is provided for operating a computer to perform ranking by utilizing the score of each document for the user's question and to select documents that may contain the answer.

[0009] According to another aspect, the present invention provides a search model, wherein the generated query is generated using a language model that has already been trained based on the document.

[0010] Provided is a method executed on a computing device including at least one processor configured to execute computer-readable instructions contained in a memory, the method including: for a given document, the at least one processor generating at least one query answerable by the document using a language model; and the at least one processor using the document and the query to train a retrieval model for a conversational bot.

[0011] According to one aspect, the generating step may generate the query for a domain-specific search model, the query being targeted at target documents belonging to the domain.

[0012] According to another aspect, the generating step may include constructing a prompt for the target document using sample data consisting of a sample document and a sample query, and using the prompt as an input to the language model to generate a query for the target document according to a pattern in the sample data.

[0013] According to another aspect, the generating step may include constructing a first prompt for the target document using first sample data consisting of a first sample document and a first sample keyword; using the first prompt as an input to the language model and generating keywords for the target document according to a pattern of the first sample data; constructing a second prompt for keywords of the target document using second sample data consisting of a second sample document, a second sample keyword, and a sample query; and using the second prompt as an input to the language model and generating a query for the target document for keywords of the target document according to a pattern of the second sample data.

[0014] According to another aspect, the method may further include the at least one processor providing an answer to a question entered by a user by reference to documents selected by the search model.

[0015] According to yet another aspect, the providing step may include converting the scores of each document calculated by the search model for the question into a form of a probability distribution, and determining a type of answer to the question based on the entropy of the probability distribution.

[0016] There is also provided a computer program recorded on a computer-readable recording medium for causing the computer device to execute the above-mentioned method.

[0017] A computing device is provided that includes at least one processor configured to execute computer-readable instructions contained in a memory, the at least one processor performing, for a given document, a process of generating at least one query that can be answered by the document using a language model, and a process of using the document and the query to train a search model for a conversational bot. Effect of the Invention

[0018] According to an embodiment of the present invention, a dialogue system can be generated quickly and efficiently by using a language model to generate large-scale data specific to a given domain based on documents in that domain.

[0019] According to an embodiment of the present invention, data generated by a large-scale language model is used to learn a retrieval model, and a dialogue system centered on the retrieval model is constructed, thereby making it possible to generate an advanced dialogue bot with a fast inference speed. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 illustrates an example of a network environment in accordance with an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a computing device in accordance with an embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram for explaining a generation system of a dialogue system using a large-scale language model in one embodiment of the present invention. [Figure 4] 1 is a flowchart illustrating an example of a method for generating dialogue data that can be executed by a computer device in an embodiment of the present invention. [Diagram 5] 1 is a flow chart illustrating an example process for generating query data using a prompt in accordance with an embodiment of the present invention. [Figure 6]FIG. 2 is a diagram illustrating an example of a query generation method according to an embodiment of the present invention. [Figure 7] FIG. 2 is a diagram illustrating an example of a query generation method in an embodiment of the present invention. [Figure 8] 11 is a diagram for explaining another example of a query generation method in one embodiment of the present invention. [Figure 9] 11 is a diagram for explaining another example of a query generation method in one embodiment of the present invention. [Figure 10] 4 is a flowchart illustrating an example of a process for providing an answer to a user's question in accordance with an embodiment of the present invention. [Figure 11] FIG. 11 is a diagram for explaining an example of a process for classifying answer types in one embodiment of the present invention. [Figure 12] FIG. 11 is a diagram for explaining an example of a process for classifying answer types in one embodiment of the present invention. [Figure 13] 11A to 11C are diagrams illustrating an example of a process for distinguishing and displaying sentences related to a user's question in an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0022] SUMMARY OF THE DISCLOSURE The present invention relates to a technique for providing answers to user questions while referring to appropriate documents.

[0023] Embodiments including those specifically disclosed in this specification can generate a dialogue system quickly and efficiently by replacing the process of collecting dialogue data required to generate a domain-specific dialogue system with an automated process that utilizes large-scale language models.

[0024] The dialogue data generation system according to the embodiment of the present invention may be realized by at least one computer device, and the dialogue data generation method according to the embodiment of the present invention may be executed by at least one computer device included in the dialogue data generation system. In this case, a computer program according to an embodiment of the present invention may be installed and executed in the computer device, and the computer device may execute the dialogue data generation method according to the embodiment of the present invention according to the control of the executed computer program. The above-mentioned computer program may be recorded in a computer-readable recording medium in combination with the computer device to cause the computer to execute the dialogue data generation method.

[0025] Fig. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in Fig. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Such Fig. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited to those shown in Fig. 1. Furthermore, the network environment in Fig. 1 is merely an example of an environment applicable to this embodiment, and the environment applicable to this embodiment is not limited to the network environment in Fig. 1.

[0026] The electronic devices 110, 120, 130, 140 may be fixed terminals or mobile terminals realized by computer devices. Examples of the electronic devices 110, 120, 130, 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), tablets, etc. As an example, FIG. 1 shows a smartphone as an example of the electronic device 110, but in the embodiment of the present invention, the electronic device 110 may mean one of various physical computer devices that can communicate with other electronic devices 120, 130, 140 and / or servers 150, 160 via a network 170 using a substantially wireless or wired communication method.

[0027] The communication method is not limited, and may include not only a communication method using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 170 can include, but also short-range wireless communication between devices. For example, the network 170 may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. Furthermore, the network 170 may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, and the like.

[0028] Each of the servers 150, 160 may be realized by one or more computing devices that communicate with the multiple electronic devices 110, 120, 130, 140 via the network 170 to provide instructions, code, files, content, services, etc. For example, the server 150 may be a system that provides a service (e.g., a conversational bot service) to the multiple electronic devices 110, 120, 130, 140 connected via the network 170.

[0029] Fig. 2 is a block diagram showing an example of a computer device in an embodiment of the present invention. Each of the above-mentioned electronic devices 110, 120, 130, and 140 and each of the servers 150 and 160 may be realized by the computer device 200 shown in Fig. 2. For example, the dialogue data generation system according to the embodiment of the present invention may be realized by the computer device 200 shown in Fig. 2.

[0030] Such a computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240, as shown in FIG. 2. The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a permanent mass storage device such as a ROM or a disk drive may be included in the computer device 200 as a separate permanent storage device separate from the memory 210. The memory 210 may also store an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 210 through a communication interface 230 that is not a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computing device 200 based on a computer program installed by a file received over the network 170 .

[0031] The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a storage device, such as the memory 210.

[0032] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (for example, the above-mentioned recording device) via the network 170. For example, requests, instructions, data, files, etc. generated by the processor 220 of the computer device 200 according to program codes recorded in a recording device such as the memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc. from other devices may be received by the computer device 200 via the communication interface 230 of the computer device 200 via the network 170. The signals, instructions, data, etc. received via the communication interface 230 may be transmitted to the processor 220 or the memory 210, and the files, etc. may be recorded on a recording medium (the above-mentioned permanent recording device) that the computer device 200 may further include.

[0033] The input / output interface 240 may be a means for interfacing with the input / output device 250. For example, the input device may include devices such as a microphone, a keyboard, or a mouse, and the output device may include devices such as a display and a speaker. As another example, the input / output interface 240 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 250 may be configured as one device together with the computer device 200.

[0034] Also, in other embodiments, computing device 200 may include fewer or more components than those of Figure 2. However, most prior art components need not be explicitly illustrated. For example, computing device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as a transceiver, a database, etc.

[0035] This embodiment can solve the problem of massive data collection in a domain by utilizing a large-scale language model to generate a domain-specific conversational bot.

[0036] The large-scale language model used in this invention refers to a language model that has been trained with a huge amount of data and is a natural language generation model that can properly execute a task when given only few-shot samples.

[0037] In other words, a large-scale language model is an autoregressive model that can perform inference without fine-tuning by using methods such as few-shot learning. It can have more than 10 times the parameters (e.g., more than 100 billion parameters) compared to existing general language models. For example, large-scale language models such as GPT-3 (Generative Pre-trained Transformer 3) and HyperClova (registered trademark) are excellent few-shot learners that can be controlled by natural prompts, and are capable of in-context learning, which is the ability to understand patterns even with a small amount of data through prompts and solve NLP problems.

[0038] In this embodiment, a domain-specific conversational bot is designed with a large-scale language model and a search model as the core. As a document-based conversational system using a language model-based generator and a search model, a highly advanced conversational bot with a high inference speed can be generated.

[0039] FIG. 3 is a diagram for explaining a generation system of a dialogue system using a large-scale language model in one embodiment of the present invention.

[0040] Referring to FIG. 3, in this embodiment, a large-scale language model 310 is used as a backbone for generating query data 302 suitable for the domain as data required for generating a domain-specific conversational bot.

[0041] Given a particular domain document 301, e.g., a web page within a home page, the processor 220 may generate query data 302 indicating questions that are expected from the given document 301 by using the large-scale language model 310 to generate a chatbot capable of question-answering (QA) on the content within the document.

[0042] The processor 220 may use the query data 302 generated by the language model 310 as training data to train a search model 320 for a dialogue bot. By using the query data 302 that matches the domain to train the search model 320, a model specialized for the domain can be constructed.

[0043] When a user inputs a question, the processor 220 may provide an answer by referencing documents corresponding to the user's question through the search model 320.

[0044] FIG. 4 is a flowchart illustrating an example of a method for generating dialogue data that can be executed by a computer device according to an embodiment of the present invention.

[0045] The dialogue data generating method according to the present embodiment may be executed by the above-mentioned computer device 200. In this case, the processor 220 of the computer device 200 may be realized to execute control instructions according to the operating system code and at least one program code included in the memory 210. Here, the processor 220 may control the computer device 200 to execute steps 410 to 430 included in the dialogue data generating method of FIG. 4 according to the control instructions provided by the code recorded in the computer device 200.

[0046] Referring to FIG. 4, in step 410, the processor 220 may generate query data from target documents of a given domain using a large-scale language model. The processor 220 may generate predicted questions for the target documents using prompts that are input sentences for the language model. The processor 220 may generate data for model training using the excellent data generation and in-context learning capabilities of the language model. A pair of the question thus generated and the target document (or a correct answer phrase included in the target document) from which the question was generated may be used as training data for the QA model.

[0047] In step 420, the processor 220 may use the target documents and the query data as training data to train a retrieval model for QA. The processor 220 may build a dialogue model for the domain by using the target documents of the domain and the query data generated by the language model for the target documents to train the retrieval model 320.

[0048] In step 430, when the user inputs a question, the processor 220 may provide an answer by referring to documents corresponding to the user's question using the search model 320. The processor 220 may perform answer formatting or answer generation according to the documents corresponding to the user's question. In other words, the processor 220 may format the documents retrieved as the user's question and provide them as an answer, or may directly generate and provide an answer using the documents.

[0049] FIG. 5 is a flow diagram illustrating an example process for generating query data using a prompt in accordance with an embodiment of the present invention.

[0050] Referring to FIG. 5, in step 501, the processor 220 may select original data to use for the prompt.

[0051] The processor 220 may use at least one or more predefined pairs of documents and queries that can be answered by the documents as original data. In addition to the predefined original sample pairs, another data set may exist, and in some embodiments, an original sample pair consisting of a document and a query may be selected from the data set.

[0052] In this case, the prompt may be composed of at least one document, a sample pair of original questions that can be answered by the document, and a target document. A detailed method of constructing the prompt will be described with reference to FIG.

[0053] In step 502, the processor 220 may configure a prompt corresponding to the input of the language model. The processor 220 may configure an input prompt of the language model using the original data selected in step 501. The processor 220 may create a dedicated prompt template in which the characteristics of a given NLP problem are appropriately reflected, and the prompt template may include a task definition and meta information. In other words, the processor 220 configures a prompt in a natural language form using original data selected from a dataset, and the prompt is created in a format that can be understood by the language model and is provided as an input of the language model. If the original data is labeled data, the processor 220 may design the prompt so that an input sentence is generated together with label information. The format of the prompt itself may be configured in various ways. As an example, the prompt may be configured to include a document (e.g., a web page) and a query that can be answered by the document as a sample.

[0054] For a given task T, the processor 220 may construct an input prompt for the language model by combining a document sample, which is original data sampled from a dataset, with a prompt template that takes into account the characteristics of the given task T.

[0055] In step 503, the processor 220 may input the prompt constructed in step 502 into a language model, and generate query data in a natural language form from the language model. In other words, the processor 220 may input the prompt input sentence into the language model, and then obtain the query data as a language generation result by the generation or completion function of the language model. The processor 220 may input the prompt into the language model, and obtain a new result having the pattern by analyzing a sample natural language pattern included in the prompt in the language model.

[0056] 6 and 7 are diagrams illustrating an example of a query generation method in one embodiment of the present invention.

[0057] Referring to FIG. 6, the processor 220 may construct a prompt input sentence 610 using a sample data pair consisting of a document selected as the original data and at least one query that can be answered based on the contents of the document.

[0058] The processor 220 may take the prompt input sentence 610, which is composed of at least one sample data pair and the target document, as an input to the language model, and cause the language model to generate a question for the target document, i.e., a completion function of the language model may generate at least one query that can be answered by the target document.

[0059] As an example, the processor 220 may generate a predetermined number of queries for the target document by iteratively generating queries using the prompt input sentences 610 composed of sample data pairs. In some embodiments, queries for the target document may be generated until a saturation point is reached where no new queries are generated.

[0060] The processor 220 may use the prompt 610, which is composed of a sample data pair in the form of [sample document + query] and a [target document], to generate a query for the target document according to the pattern of the sample data included in the prompt.

[0061] For example, as shown in FIG. 7, when a target document 701 is given, the processor 220 may input sample data and the target document 701 as a prompt input sentence 610 into a language model, and generate query data 702 predicted from the target document 701 based on the pattern of the sample data.

[0062] In other words, the processor 220 may provide documents and queries derived from the documents as samples and generate queries for the target documents according to the patterns of the samples.

[0063] 8 and 9 are diagrams for explaining another example of a query generation method in one embodiment of the present invention.

[0064] The processor 220 may apply a method for generating queries based on keywords to improve the quality of queries generated by the language model.

[0065] For example, the processor 220 may use two prompts in a stepwise manner, first extracting keywords related to the document from the document in the first step, and then generating queries for the extracted keywords in the second step. When additional keywords are used, more diverse queries can be generated from the document, and the coverage of the questions can be expanded.

[0066] Referring to FIG. 8, the processor 220 may construct a first prompt input sentence 810 using a first sample data pair consisting of a document selected as the original data and a keyword for the document.

[0067] The processor 220 may input a first prompt input sentence 810 consisting of at least one keyword-based first sample data pair and a target document into a language model, and generate at least one keyword for the target document using a generation function of the language model.

[0068] Next, the processor 220 may construct a second prompt input sentence 820 using a second sample data pair consisting of the document selected as the original data, keywords extracted from the document, and a query that can be answered based on the contents of the document.

[0069] At this time, the second prompt input sentence 820 may include keywords for the target document that are generated by a language model using the first prompt input sentence 810.

[0070] The processor 220 may input the at least one keyword-based second sample data pair, the target document, and a second prompt input sentence 820 composed of keywords for the target document into the language model, and generate at least one query that can be answered by the target document through a completion function of the language model.

[0071] The processor 220 may generate keywords for the target document according to a pattern of the primary prompt using a first sample data pair consisting of [sample document + keyword] and a primary prompt consisting of [target document], and then, for each keyword, generate a query for the target document according to a pattern of the secondary prompt using a second sample data pair consisting of [sample document + keyword + query] and a secondary prompt consisting of [target document + keyword].

[0072] For example, as shown in FIG. 9, when a target document 901 is given, the processor 220 may first input the first prompt input sentence 810 into a language model and generate keywords 90 that may appear as a query in the target document 901 according to the patterns of the sample data.

[0073] The processor 220 may then input the second prompt input sentence 820 containing the keywords 90 into the language model and generate predicted query data 902 for the target document 901 based on the keywords 90 in accordance with the patterns of the sample data.

[0074] The processor 220 may extract at least one keyword 90 for the target document 901, and may generate query data 902 corresponding to each keyword 90. In this way, by using keywords, query data related to each keyword and having an intent corresponding to the keyword is uniformly generated. In this way, when various data are used for learning, the performance of the search model will be improved.

[0075] Therefore, the processor 220 can provide a document, keywords extracted from the document, and a query containing the keywords as samples, and generate a query for the target document based on the keywords extracted from the target document according to the pattern of the samples.

[0076] The processor 220 may use the data consisting of the target documents and queries generated for the target documents to train a model for the dialogue bot.

[0077] In the case of the search model 320, for example, a poly encoder structure may be applied to search documents from a query to provide an answer. The search model 320 may include a reranker with a cross encoder structure to rerank documents corresponding to the query to provide an answer.

[0078] A pair (Document, Query) of each document for the corresponding domain and a query generated for the document may be used to train the search model 320. The search model 320 may be generated by fine-tuning a model pre-trained using a labeled dataset or a model pre-trained by an unsupervised search model 320 from an unlabeled corpus of various linguistic features using data generated by an embodiment of the present invention.

[0079] In the process of fine-tuning the search model 320, a data augmentation technique using synonyms and antonyms specific to the domain to which the target document belongs may be used. A certain data augmentation effect can be expected by replacing each word with a synonym or antonym with a certain probability for the questions in the training batch and training the model. For example, "postdoctoral researcher" may be replaced with "post-doc," "post-doctor," or "OCR" may be replaced with "optical character recognition," or "OCR."

[0080] In addition to the data augmentation technique, a rule-based replacement method for words from the inference input can also be used to improve the learning performance of the retrieval model 320.

[0081] FIG. 10 is a flow chart illustrating an example process for providing an answer to a user's question in accordance with an embodiment of the present invention.

[0082] 10, in step 1001, the processor 220 may calculate a score for each document with respect to the user's question using the search model 320. The processor 220 may search for and rank documents that contain answers to the user's question using the search model 320. The processor 220 may calculate a score for each document by measuring the similarity between a sentence corresponding to the user's question and the document.

[0083] In step 1002, the processor 220 may convert the document scores calculated by the retrieval model 320 into a form of probability distribution. As an example, the processor 220 may use a softmax function to convert the document scores of the retrieval model 320 into a form of probability distribution.

[0084] In step 1003, the processor 220 may determine a type of answer to the user's question according to the probability distribution of the document scores. The processor 220 may provide as an answer to the user's question by formatting at least one document selected based on the ranking by the search model 320.

[0085] The number of documents provided as answers may vary depending on the question, and the number of documents may be defined for each type of answer. When document scores are converted into a probability distribution, in cases where a document is definitely the number one relevant to a question, the entropy of the score probability distribution is close to 0, and the more relevant documents there are, the higher the entropy. In addition, questions for which no relevant documents are searched for tend to have a uniform distribution.

[0086] According to such a tendency, a threshold may be set based on the entropy of the score probability distribution to classify the answer type.

[0087] For example, as shown in FIG. 11, when the entropy of the score probability distribution is equal to or less than a certain threshold close to 0 (for example, equal to or less than 0.3), the answer type may be classified into three cases: case 1 where the top 1 is certain, case 2 where the documents in the first to third places are all useful, and case 3 where there is no related document when the entropy is 2 or more. As another example, the threshold for classifying the answer type may be set to 0.1 and 1.75 as a reference. When the entropy (E) of the score probability distribution is 0≦E<0.1, the answer may be classified into case 1, when 0.1≦E<1.75, the answer may be classified into case 2, and when E≧1.75, the answer may be classified into case 3.

[0088] As shown in FIG. 12, when the entropy of the document score probability distribution is indicated as case 2 for a question about "lamp color", the processor 220 may select the top three documents based on the document score to provide an answer. When detailed documents about red, yellow, and green are searched for as documents containing the contents of lamp colors, all three documents may be used as an answer depending on a threshold value. In this case, the titles of the three documents may be displayed as an answer to the user's question, and then, if the user selects a specific document, the detailed contents of the selected document may be displayed. On the other hand, when the entropy of the document score probability distribution is indicated as case 1 for a question about "internship preferential treatment conditions", the processor 220 may select the top one document based on the document score to provide an answer. The highest-scoring document containing the contents of internship preferential treatment conditions may be displayed as an answer to the question.

[0089] The processor 220 may distinguish sentences related to the user's question from other sentences in the document provided as an answer from other sentences. For example, as shown in Fig. 13, which part of the content of the document 1310 provided as an answer corresponds to the answer to the user's question may be displayed by highlighting 1301.

[0090] In order to highlight sentences in a document that correspond to a user's question, a question may be generated for each sentence in the document, with the sentence as the answer, and the generated question may be used as data for training a search model. In other words, the processor 220 may use a prompt consisting of [sample sentence+query] and [target sentence] for each sentence in the target document to generate a query for the target sentence according to the pattern of the prompt, and may use a data pair consisting of [target sentence+query] to train a search model, and may search for and highlight sentences that answer the user's question from within the document.

[0091] In order to generate a query for a target sentence using a sample of the sentence and a query that uses the sentence as an answer for highlighting an answer sentence in a document, the processor 220 may apply a logic for filtering whether the target sentence is an answerable sentence prior to query generation in order to improve the quality of the query generation. For example, the target sentence may be filtered by taking into consideration the title of the target document and other sentences in the document. In other words, sentences that are not included in the title of the document, sentences that are different in format from other sentences in the document, etc. may be excluded from the query generation target.

[0092] In this embodiment, by combining a document-based conversational bot with a Task Oriented Dialog (TOD), it is possible to provide not only active conversations such as questions and queries for necessary information, but also more advanced services such as processing actual user requests through API calls. If the above-mentioned conversational system is combined with a TOD, it is possible to build an active conversational system that can provide appropriate answers to users and at the same time grasp the users' implicit requirements and preemptively ask questions to resolve them.

[0093] In this way, according to the embodiment of the present invention, a dialogue system can be generated quickly and efficiently by generating large-scale data specialized for a given domain using a language model based on documents of the domain. According to the embodiment of the present invention, a dialogue bot with high inference speed can be generated by using the data generated by the large-scale language model to learn a retrieval model and constructing a dialogue system centered on the retrieval model.

[0094] The above-mentioned devices may be realized by hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be realized using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the OS. The processing device may also respond to the execution of the software to access, record, manipulate, process, and generate data. For convenience of understanding, a single processing device may be described as being used, but one skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0095] The software may include computer programs, codes, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired or instruct the processing device, either independently or collectively. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed and stored or executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0096] The method according to the embodiment may be realized in the form of program instructions executable by various computer means and recorded on a computer-readable medium. Here, the medium may be one that continuously records a computer-executable program or one that temporarily records the program for execution or download. The medium may be one of various recording or storage means in the form of a single or multiple hardware devices combined, and may not be limited to a medium directly connected to a certain computer system, but may be one that is distributed over a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROMs, RAMs, flash memories, etc., configured to record program instructions. Other examples of the medium include recording media or storage media managed by application stores that distribute applications, or sites, servers, etc. that supply or distribute various other software.

[0097] Although the embodiments have been described based on the limited embodiments and drawings as above, those skilled in the art will be able to make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be replaced or substituted by other components or equivalents, and still achieve suitable results.

[0098] Therefore, different embodiments are within the scope of the appended claims provided that they are equivalent to the claims. [Explanation of symbols]

[0099] 301: Domain Document 302: Query Data 310: Large-scale language models 320: Search model

Claims

1. 1. A method implemented on a computer device for generating a dialogue system using a language model and a retrieval model, comprising: The method comprises: A processor of the computer device generates queries as expected questions for documents belonging to a particular domain by utilizing the language model; The processor uses training data consisting of pairs of the document and the query to train a search model to construct a QA model for a conversational bot; a step of, when a user's question is input to the search model, the processor selecting documents including an answer to the user's question and providing the answer to the user's question; A method comprising:

2. The step of providing the answer includes the processor searching for documents including the answer content by using the search model, calculating a document score for each document by measuring a similarity between a sentence corresponding to the user's question and the document including the answer content, and selecting a document including the answer content based on the document score. The method according to claim 1 , characterized in that

3. The generating step includes: constructing prompts for target documents using sample data consisting of sample documents and sample queries; and using the prompts as input to the language model to generate queries against the target documents according to patterns in the sample data. The method of claim 2 , comprising:

4. The generating step includes: constructing a first prompt for the target document using first sample data, the first sample document and the first sample keywords; using the first prompt as an input to the language model and generating keywords for the target document according to patterns in the first sample data; constructing second prompts for keywords of the target documents using second sample data, the second sample documents, second sample keywords, and sample queries; and using the second prompt as an input to the language model to generate queries for the target documents according to patterns in the second sample data for keywords of the target documents. The method of claim 2 , comprising:

5. The providing step comprises: converting the document score of each document into a form of a probability distribution; and determining a type of answer to the user's question according to the entropy of the probability distribution; The method of claim 2 , comprising:

6. A computer program product for causing a computer device to carry out the method according to any one of claims 1 to 5.

7. 1. A computer device for generating a dialogue system using a language model and a retrieval model, comprising: At least one processor configured to execute computer-readable instructions contained in the memory. Including, The at least one processor generating queries as anticipated questions for documents belonging to a particular domain by utilizing the language model; training a search model using training data consisting of pairs of the document and the query to construct a QA model for a conversational bot; When a user's question is input to the search model, selecting documents including an answer to the user's question and providing the answer to the user's question; A computer device that executes the above-mentioned.

8. The step of providing the answer includes searching for documents including the answer content using the search model, calculating a document score for each document by measuring a similarity between a sentence corresponding to the user's question and the document including the answer content, and selecting a document including the answer content based on the document score.

8. The computer device of claim 7 .

9. The at least one processor Configure prompts for target documents using sample data consisting of sample documents and sample queries; using the prompts as input to the language model to generate queries against the target documents according to patterns in the sample data; 8. The computer device of claim 7 .

10. The at least one processor constructing a first prompt for the target document using first sample data, the first sample document and the first sample keywords; using the first prompt as an input to the language model, and generating keywords for the target document according to patterns in the first sample data; constructing second prompts for keywords of the target documents using second sample data, the second sample documents, second sample keywords, and sample queries; using the second prompt as an input to the language model to generate a query for the target document according to a pattern of the second sample data for a keyword of the target document; 8. The computer device of claim 7 .

11. The at least one processor Converting the document score of each document into a probability distribution; determining a type of answer to the user's question according to the entropy of the probability distribution; 8. The computer device of claim 7 .

Images