Information processing device, program, and dialogue method for retrieving knowledge data during user utterances.

Abstract

This invention provides an information processing device, etc., that can generate response sentences as quickly as possible for a voice dialogue system that responds to user speech with a voice response. [Solution] The information processing device includes a knowledge database that stores chunks containing the text of knowledge data; a speech recognition means that takes the user's utterances as input and sequentially recognizes the user's utterances; an utterance memory that stores utterances that are a time series of the user's utterances; a knowledge data retrieval means that takes the user's utterances as input and searches the knowledge database for chunks containing the text based on those utterances; a chunk memory that stores the retrieved chunks; a response sentence generation language model that generates a response sentence to the user at a predetermined response timing using the utterances in the speech sentence memory and the chunks in the chunk memory; and a speech conversion means that takes the response sentence as input and converts it into a response voice to the user.

Description

Technical Field

[0001] The present invention relates to the technology of a dialogue system that returns a response voice to a user's spoken voice. In particular, it is suitable for a voice FAQ (Frequently Asked Question) system.

Background Art

[0002] As a customer support FAQ system, there is a technology such as a chatbot incorporated in a website or an app. Such a system can respond to various utterance sentences from an unspecified number of users regardless of the time zone, contributing to cost reduction of the service and improvement of customer satisfaction.

[0003] Conventionally, there is a technology for semantically understanding a user's spoken language (see, for example, Patent Document 1). According to this technology, text is converted from a user's spoken voice using a model, and semantic information is obtained by a language understanding task.

[0004] In addition, there is also a technology of a FAQ system that infers a response sentence using a large language model (LLM (Large Language Models)) for a natural language utterance sentence from a user (see, for example, Patent Document 2). According to this technology, the large language model determines whether the utterance sentence itself is appropriate for inferring a response sentence. When it is determined that it is not appropriate, a response sentence for digging deeper into the utterance sentence is presented to the user.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] In existing dialogue systems, a response is generated after the user's utterance is complete. Often, generating the response requires external knowledge data (e.g., documents). In this case, after the user's utterance is complete, the system searches for the knowledge data and uses it to generate the response. However, there is a challenge in that there is often a delay between the time the user's speech is completed (i.e., after the entire utterance has been acquired) and the time it takes to search the knowledge data and generate a response, which is why a delay is likely to occur before the response is spoken. In systems like voice FAQs, immediate responses are required for the user.

[0007] Therefore, the present invention aims to provide an information processing device, a program, and a dialogue method that can generate a response sentence as quickly as possible when responding to a user's spoken voice. [Means for solving the problem]

[0008] According to the present invention, in an information processing device that responds to a user's spoken voice with a response voice, A knowledge database that stores chunks containing the text of knowledge data, A speech recognition means that takes user speech as input and sequentially recognizes the user's utterances, A speech sentence memory that stores utterances that represent the chronological sequence of user utterances, A knowledge data retrieval means that sequentially inputs user utterances and searches a knowledge database for chunks containing text based on those utterances, Chunk memory that stores the searched chunks, A response sentence generation language model that generates a response sentence to the user at a predetermined response timing using the utterance in the utterance memory and the chunk in the chunk memory, A voice conversion means that takes a response sentence as input and converts it into a voice response for the user. It is characterized by having the following features.

[0009] According to another embodiment of the information processing apparatus of the present invention, The chunk contains embedded vectors based on text, The knowledge database stores text and embedded vectors in a coordinated manner for each chunk. The knowledge data retrieval means converts an utterance phrase into an embedding vector and searches the knowledge database for a predetermined number of top chunks that have embedding vectors similar to the embedding vector of the utterance phrase. That is also desirable.

[0010] According to another embodiment of the information processing apparatus of the present invention, The knowledge data retrieval method is Retrieval-Augmented Generation (RAG), Knowledge data retrieval methods are, A speech phrase vector conversion means that sequentially converts user speech phrases into embedded vectors, A vector search means performs an embedding vector search using a knowledge database to find a predetermined number of embedding vectors from the top that are similar to the embedding vector of an utterance, and outputs chunks that contain the found embedding vectors. It is also desirable to have it.

[0011] According to another embodiment of the information processing apparatus of the present invention, The system further includes a chunk memory search means that, upon completion of a user's utterance, converts the utterance in the utterance memory into an embedding vector, and searches the chunk memory for a predetermined number of upper chunks that have embedding vectors similar to the embedding vector of the utterance. The response sentence generation language model generates a response sentence to the user using the utterances in the utterance memory and the chunks retrieved by the chunk memory search means. That is also desirable.

[0012] According to another embodiment of the information processing apparatus of the present invention, After inputting a prompt for instructing whether to determine the necessity of chunk search for an utterance sentence, the user's utterance sentences are sequentially input, and further having a search determination model for determining the necessity of chunk search, When the knowledge data search means is determined to require retrieval by the search determination model, it searches for chunks for the utterance sentence This is also preferable.

[0013] According to another embodiment of the information processing apparatus of the present invention, The search determination model, as a prompt, further describes an instruction for estimating the required accuracy that will be necessary for the search of knowledge data when determining the necessity of chunk search for an utterance sentence, then sequentially inputs the user's utterance sentences, and outputs the necessity of chunk search and the required accuracy, When the required accuracy is higher than a predetermined threshold, the response sentence generation language model generates a filler response sentence for the user This is also preferable.

[0014] According to another embodiment of the information processing apparatus of the present invention, When it is determined by the search determination model that retrieval is required and the required accuracy is output, and when the required accuracy is not less than a predetermined value, The knowledge data search means An utterance sentence vector conversion means for sequentially converting the user's utterance sentences into embedding vectors, Using the knowledge database, an embedding vector search is executed to search for the top first predetermined number of embedding vectors similar to the embedding vector of the utterance sentence, and a vector search means for outputting the chunks having the embedding vectors, Using the knowledge database, a keyword search is executed to search for the top second predetermined number of chunks similar to the keyword of the utterance sentence, and a keyword search means for outputting the searched chunks, Semantic ranking search means that executes semantic ranking search to arrange, in descending order of semantic relevance to an utterance sentence, a first predetermined number of chunks based on embedded chunk search and a second predetermined number of chunks based on keyword search, and to search for a third predetermined number of chunks from the top, and outputs the searched chunks It is also preferable to have.

[0015] According to another embodiment of the information processing apparatus of the present invention, After inputting a prompt for instructing to infer a topic that is the user's utterance intention from the time series of the user's utterance sentences, a topic inference model that infers a topic for each utterance sentence and notifies the chunk memory of the topic Further has, The chunk memory has different buffers according to topics, stores the chunks retrieved by the knowledge data retrieval means in the buffer corresponding to the topic, The chunk memory retrieval means retrieves chunks similar to the utterance sentence in the utterance memory from the chunks stored in the buffer corresponding to the topic of the current utterance sentence in the chunk memory It is also preferable.

[0016] According to another embodiment of the information processing apparatus of the present invention, Avatar display control means for displaying an avatar whose operation is controlled based on the response voice to the user Further has, When it is determined by the topic inference model that the past topic and the current topic are the same, the avatar display control means controls the avatar to nod as a non-verbal indication of consent It is also preferable.

[0017] According to the present invention, in a program that causes a computer mounted on an information processing apparatus that returns a response voice to a user's uttered voice to function, A knowledge database that stores chunks including the text of knowledge data, A speech recognition means that takes user speech as input and sequentially recognizes the user's utterances, A speech sentence memory that stores utterances that represent the chronological sequence of user utterances, A knowledge data retrieval means that sequentially inputs user utterances and searches a knowledge database for chunks containing text based on those utterances, Chunk memory that stores the searched chunks, A response sentence generation language model that generates a response sentence to the user at a predetermined response timing using the utterance in the utterance memory and the chunk in the chunk memory, A voice conversion means that takes a response sentence as input and converts it into a voice response for the user. It is characterized by enabling the computer to function in this way.

[0018] According to the present invention, in a method of dialogue for a device that responds to a user's spoken voice with a response voice, The device is It has a knowledge database that stores chunks containing the text of knowledge data, The first step involves using a speech recognition unit to sequentially recognize the user's utterances from the user's spoken audio, The second step involves storing the utterances, which represent the chronological sequence of the user's utterances, in the utterance memory. A third step involves sequentially inputting the user's utterances and searching the knowledge database for chunks containing text based on those utterances. The fourth step is to store the searched chunk in chunk memory, A fifth step involves using a pre-trained response sentence generation language model to generate a response sentence to the user at a predetermined response timing, using the utterance in the utterance memory and the chunk in the chunk memory. The sixth step involves using the voice conversion unit to convert the response text into a voice response to the user. It is characterized by performing the following. [Effects of the Invention]

[0019] According to the information processing device, program, and dialogue method of the present invention, when responding to a user's spoken voice, it is possible to generate a response sentence as quickly as possible. [Brief explanation of the drawing]

[0020] [Figure 1] This is a diagram illustrating the basic functional configuration of the information processing device according to the present invention. [Figure 2] This is a functional configuration diagram of the present invention, which includes a chunk memory search unit. [Figure 3] This is an explanatory diagram of the knowledge data retrieval unit and the chunk memory retrieval unit in the present invention. [Figure 4] This is a functional configuration diagram showing the search and determination model according to the present invention. [Figure 5] This is a functional configuration diagram using the required accuracy of the search and judgment model in the present invention. [Figure 6] This is a functional configuration diagram of the knowledge data retrieval unit in the present invention. [Figure 7] This is a functional configuration diagram showing a topic inference model according to the present invention. [Figure 8] This is an explanatory diagram of the topic inference model and chunk memory search unit in the present invention. [Figure 9] This is a first explanatory diagram showing a sequence of utterances in the present invention. [Figure 10] This is a second explanatory diagram showing the sequence of utterances in the present invention. [Figure 11] This is a third explanatory diagram showing the sequence of utterances in the present invention. [Figure 12] This is a functional configuration diagram for building a knowledge database. [Modes for carrying out the invention]

[0021] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0022] Figure 1 is a diagram illustrating the basic functional configuration of the information processing device according to the present invention.

[0023] According to Figure 1, the information processing device 1 of the present invention communicates with the terminal 2, which serves as the user's front end, via a network.

[0024] <Terminal 2> Terminal 2 picks up the user's spoken voice with its microphone and transmits the spoken voice to the information processing device 1. Terminal 2 also receives a response voice from the information processing device 1 and speaks it to the user through its speaker. Here, "user" can refer to an unspecified number of third parties or a specific number of people (for example, employees who use an internal company system). Let's assume that the user is, for example, someone who asks a question in an FAQ and needs an answer based on knowledge data.

[0025] Furthermore, terminal 2 may display an avatar on its screen that personifies the person the user is interacting with. The avatar's movements are synchronized with the response voice received from information processing device 1 and spoken from the speaker.

[0026] <Information Processing Device 1> The information processing device 1 responds to the user's spoken voice with a corresponding voice response. According to Figure 1, the information processing device 1 basically comprises a knowledge database 10, a speech recognition unit 101, a speech memory 11, a knowledge data retrieval unit 12, a chunk memory 13, a response sentence generation language model 15, and a speech conversion unit 102. These functional components are realized by executing a program that makes the computer installed in the information processing device function. Furthermore, the processing flow of these functional components can also be understood as the interaction method of the information processing device.

[0027] [Knowledge Database 10] Knowledge database 10 is a collection of "chunks" containing the text of knowledge data. Here, "knowledge data" refers to text based on documents (external information) that are the subject of FAQ responses. These documents may be in a wide variety of file formats, but the "knowledge data" refers to the text extracted from them. Furthermore, a "chunk" is a division of text into predetermined units. These units may be based on a predetermined number of characters (e.g., 256 characters), pages, or paragraphs.

[0028] [Voice recognition unit 101] The speech recognition unit 101 takes the user's "spoken voice" as input and sequentially recognizes the user's "spoken phrases". Spoken voice: "By the way, I want to buy telecommunications stocks, but how should I go about it?" Speech phrase: "By the way," "I want to buy telecommunications stocks," "but what should I do?" Specifically, the speech recognition unit 101 could be Vosk®, an open-source library that is lightweight and can be used offline. Alternatively, Faster-Whisper® or Julius® could also be used. By using these libraries, it is possible to output utterance phrases sequentially at high speed from spoken audio. Furthermore, while this invention uses the term "utterance phrase," it may of course be defined as a word or morpheme. Also, breaks, pauses, and silences in the user's utterances may be treated as sentences. However, it should be noted that the finer the length of the utterance phrase, the greater the processing load, while the longer the length, the less responsive the system becomes.

[0029] [Speech Memory 11] The speech memory 11 stores "speech sentences," which are the chronological sequence of the user's utterances. The utterance memory 11 stores the user's utterance until the user has finished speaking (until the response is possible). When the user has finished speaking, it outputs the entire utterance to the response sentence generation language model 15 and also notifies the chunk memory 13 of this fact.

[0030] [Knowledge Data Retrieval Section 12] The knowledge data retrieval unit 12 sequentially inputs the user's "utterances" and searches the knowledge database 10 for "chunks" containing text based on those utterances (search queries). In other words, the chunk search is performed proactively (in the background) even while the user is speaking. Furthermore, the search is not limited to just one chunk; multiple chunks may be searched in order of priority.

[0031] [Chunk memory 13] The chunk memory 13 stores the "chunks" retrieved by the knowledge data retrieval unit 12. When the chunk memory 13 receives notification from the utterance memory 11 that the user's utterance is complete, it outputs the chunks that were searched in advance during the user's utterance to the response sentence generation language model 15.

[0032] [Response Sentence Generation Language Model 15] The response sentence generation language model 15 generates a response sentence to the user at a predetermined response timing, using the "utterances" in the utterance memory 11 and the "chunks" in the chunk memory 13. Furthermore, the system may also consider past responses it has given itself when generating responses to the user. For example, it may consider the past 10 or so turns, including utterances in the utterance memory and the responses given to them.

[0033] The response sentence generation language model 15 may communicate with a Generative Artificial Intelligence (Generative Artificial Intelligence) server via a network as a backend for creating response sentences. The response generation language model 15 is based on a pre-trained large-scale language model (LLM). A large-scale language model is a type of deep learning model that has been trained on a large amount of text data. Based on neural network architectures such as Transformer, it can understand context and generate natural language sentences. These are pre-trained with general language knowledge from a large text corpus. Specifically, examples include OpenAI's ChatGPT®, Anthropic's Claude®, Google's Gemini®, and Meta's Llama®. These are pre-trained using general linguistic knowledge derived from large text corpora.

[0034] [Voice conversion unit 102] The speech conversion unit 102 receives a "response sentence" from the response sentence generation language model 15 and converts it into a "response voice" for the user. Specifically, there are open-source libraries such as VoiceVox® and Piper®, which are lightweight and can be used offline. By using these libraries, it is possible to output response voices quickly from response text. The information processing device 1 then transmits the converted response voice to the terminal 2 and speaks it to the user through its speaker.

[0035] Generally, human interaction in the real world is full-duplex (simultaneous, bidirectional, full-duplex). That is, while listening to the other person's speech, a person is simultaneously selecting and filtering various pieces of memory information in their mind. Then, after the other person has finished speaking, they themselves utter a response. As shown in Figure 1, the information processing device 1 also proactively searches for chunks of text based on knowledge data, even while the user is speaking. As a result, the search for chunks of knowledge data is completed by the time the user finishes speaking, and the generation of a response sentence can begin immediately. In other words, according to the present invention, the search stage and the response sentence generation stage are executed separately. The knowledge data search unit 12 continuously searches for knowledge data for the utterance phrase even while the user is speaking. As a result, when the predetermined timing for response sentence generation is reached, the knowledge data (chunks) have already been acquired, and the response sentence generation stage can be executed immediately. In other words, the response sentence can be generated as quickly as possible.

[0036] Figure 2 is a functional configuration diagram of the chunk memory search unit in the present invention.

[0037] According to Figure 2, a chunk is a unit that includes both "text" and an "embedding vector" in a corresponding manner. The knowledge database 10 has pre-stored the association between "text" and "embedded vectors" for each chunk. Embedding vectors are created by transforming text-based chunks into high-dimensional vectors that numerically represent meaning and features using the Application Programming Interface (API) of a Large-Scale Language Model (LLM). This allows the semantic features of words and sentences to be represented as numerical arrays, enabling quantitative analysis of similarities and relationships between texts. For example, it could be OpenAI's Embedding API, ada®, which is pre-trained with general linguistic knowledge from a large text corpus.

[0038] Furthermore, the knowledge database 10 may store chunks and embedded vectors based on an index. Chunks can be searched quickly by index search from the knowledge data retrieval unit 12.

[0039] According to Figure 2, there are differences compared to Figure 1 in the following respects. (1) The knowledge data retrieval unit 12 performs an "embedded vector search". (2) It further has a "chunk memory search unit 14" that searches the chunk memory 13.

[0040] Figure 3 is an explanatory diagram of the knowledge data retrieval unit and the chunk memory retrieval unit in the present invention.

[0041] According to Figure 3(a), the knowledge data retrieval unit 12 represents a system based on Retrieval-Augmented Generation (RAG). Specifically, the knowledge data retrieval unit 12 includes an utterance phrase vector conversion unit and an embedded vector retrieval unit.

[0042] (Speech phrase vector conversion unit) The utterance vector conversion unit sequentially converts the user's utterances into embedded vectors. The utterance vector conversion unit also uses the API of the large-scale language model mentioned above. (Embedded vector search unit) The embedded vector search unit performs an "embedded vector search" using the knowledge database 10 to search for a predetermined number of embedded vectors that are similar to the embedded vector of the utterance phrase, starting from the top. The chunks containing the found embedded vectors are output to the chunk memory 13. Specifically, the similarity between the embedding vector of the utterance phrase and the embedding vector of the chunk text may be calculated using cosine similarity or Euclidean distance. For example, using cosine similarity or Euclidean distance, the system searches for, for example, the top 5 chunks (a predetermined number) with the highest similarity.

[0043] [Chunk memory search unit 14] The chunk memory search unit 14 searches the chunks stored in the chunk memory 13 for chunks that are similar to the entire utterance in the utterance memory 11. Furthermore, the search is not limited to one chunk; multiple chunks may be searched in order of similarity.

[0044] According to Figure 3(b), the chunk memory search unit 14 includes an utterance vector conversion unit and an embedded vector search unit. (Speech vector conversion unit) The speech vector conversion unit converts the entire speech sentence in the speech memory into an embedded vector when the user's speech is completed. (Embedded vector search unit) The embedded vector search unit searches the chunk memory 13 for a predetermined number of top chunks that have embedded vectors similar to the embedded vector of the utterance. The searched chunks are output to the response sentence generation language model 15. The embedded vector search unit of the chunk memory search unit 14 can perform high-speed processing because it only needs to search the chunk memory 13. On the other hand, the embedded vector search unit of the knowledge data search unit 12 requires computational resources and processing time because it performs a RAG search.

[0045] The response sentence generation language model 15 then uses the "utterances" in the utterance memory 11 and the "chunks" retrieved by the chunk memory search unit 14 to generate a response sentence for the user.

[0046] Figure 4 is a functional configuration diagram showing the search and determination model in the present invention.

[0047] According to Figure 4, the information processing device 1 further includes a search determination model 16 to determine whether or not the knowledge data retrieval unit 12 is activated.

[0048] [Search Judgment Model 16] The search decision model 16 is a lightweight language model that makes decisions based on the prompt. The search determination model 16 receives a prompt instructing it to determine whether a chunk search is necessary for a given utterance. It then takes user utterances sequentially and determines whether a chunk search is necessary. As a result, the knowledge data retrieval unit 12 only needs to search for chunks for an utterance if the search determination model 16 determines that a search is necessary.

[0049] In reality, the knowledge data retrieval unit 12 requires significant computing resources when functioning as a RAG. Therefore, by incorporating the retrieval decision model 16, the knowledge data retrieval unit 12 is activated only when knowledge data retrieval is necessary. This minimizes the frequency of knowledge data retrieval, thereby reducing computing resource consumption.

[0050] The search decision model 16 may use a lightweight language model, such as GPT-4.1-nano(registered trademark). This allows for real-time search activation determination for each utterance with minimal computational resources.

[0051] Specifically, the prompts input to the search judgment model 16 are, for example, as follows: -------------------------------------------------------------------------------- You are the search activation engine for voice assistants. Your task is to determine whether a knowledge search from the database is necessary for each user utterance. If the question concerns facts that require external knowledge, such as referring to specific documents, data, or historical information, please activate the trigger. Conversely, do not activate the trigger in the following cases: (1) When the query is simply a conversation (e.g., "hello", "thank you", "OK") (2) When the question can be answered using only common sense (3) When the utterance is unclear or incomprehensible --------------------------------------------------------------------------------

[0052] Figure 5 is a functional configuration diagram using the required accuracy of the search and judgment model in the present invention.

[0053] As shown in Figure 5, compared to Figure 4, the search decision model 16 further describes the instruction to the prompt to "estimate the 'necessary precision' that will be required to retrieve knowledge data when determining whether a chunk needs to be retrieved for an utterance." -------------------------------------------------------------------------------- ········································ At this time, please also output the required precision [0-1] for the knowledge retrieval. ········································ -------------------------------------------------------------------------------- As a result, the search judgment model 16 sequentially inputs the user's utterances and outputs the "necessity of search" and "required accuracy" for each chunk. The required accuracy is output to the knowledge data retrieval unit 12 and also to the response sentence generation language model 15.

[0054] The required precision may be, for example, in units of 0.1, and could be a numerical value between 0.0 and 1.0. For example, if the value is 0.6 or higher, it may be determined that a search of knowledge data is required, and a trigger may be activated. If the required precision is high (0.6 or higher): Knowledge data retrieval is required. If the required precision is lower than 0.6: Knowledge data retrieval is not required.

[0055] <Generating filler response sentences> In another embodiment, the response statement generation language model 15 may generate a filler response statement for the user if the required accuracy is higher than a predetermined threshold. A filler refers to meaningless utterances in dialogue, such as "um," "uh," or "well," that are uttered unconsciously to fill the gap when one is stuck for words or searching for the next word. In this context, it may also involve repeating the other person's words to create time before formulating a response. This means that the longer the time required to retrieve knowledge data (the higher the required accuracy of the search), the less silent the other person can be. This also leads to a smoother dialogue experience for the user.

[0056] If the required precision output by the search judgment model 16 is, for example, 0.8 or higher, it may be determined that searching for knowledge data will take time (a high-precision search is required). If the required precision is high (0.8 or higher): A filler response is generated because searching for knowledge data takes time. If the required precision is lower than 0.8: Do not generate a filler response.

[0057] Here, the response sentence generation language model 15 receives a high required accuracy (0.8 or higher) from the search judgment model 16, and if the chunk memory search unit 14 cannot find the user's utterance even though it is complete in the utterance memory 11, it outputs a filler response sentence, assuming that the search process in the knowledge data search unit 12 is taking too long.

[0058] Figure 6 is a functional configuration diagram of the knowledge data retrieval unit in the present invention.

[0059] According to Figure 6, the knowledge data retrieval unit 12 includes an utterance phrase vector conversion unit, an embedded vector retrieval unit, a keyword retrieval unit, and a semantic ranking retrieval unit. This means that when the search determination model 16 determines that a search is necessary and the required precision is output, if that "required precision" is above a predetermined value, a triple search (embedded vector search unit, keyword search unit, semantic ranking search unit) is executed to improve the accuracy of the knowledge data search (which takes longer). Furthermore, the utterance vector conversion unit functions in the same way as shown in Figure 3(a) above.

[0060] Here, the knowledge data retrieval unit selects whether to execute only the embedded vector retrieval unit or to perform a triple search, depending on the required accuracy. If the required precision is high (0.8 or higher): Perform a triple search. If the required precision is lower than 0.8: Only the embedded vector search unit will be executed. If high precision is required, it is necessary to perform the search in a way that finds high-precision chunks, even if it means longer search times for knowledge data and higher computational resources.

[0061] (Embedded vector search unit) The embedded vector search unit uses the knowledge database 10 to perform an embedded vector search, searching for a predetermined number of embedded vectors from the top that are similar to the embedded vector of the utterance phrase, and outputs chunks that contain those embedded vectors. This is the same embedded vector search unit as shown in Figure 3(a) above.

[0062] <Triple Search> (Keyword Search Section) The keyword search unit uses the knowledge database 10 to perform a keyword search, searching for a second predetermined number of chunks that are similar to the keyword (search query) of the utterance phrase, and outputs the searched chunks. Specifically, this evaluates the degree of match between the keywords in the utterance and the chunks in the knowledge database 10. For example, it searches for the top 5 (or a second predetermined number) chunks, starting with those with the highest keyword match degrees.

[0063] The keyword search unit may use a statistical algorithm such as BM25 (Okapi BM (Best Matching) 25). This allows for ranking the text chunks for each utterance in order of their score.

[0064] BM25 is an improved algorithm based on TF-IDF (Term Frequency - Inverse Document Frequency). It considers word frequency (TF), determining that words appearing more frequently in a text are more important. Conversely, it also considers inverse document frequency (IDF), determining that words in a search query that are common words appearing in many texts are less important. On the other hand, words that appear only in specific texts are considered more important. A distinctive feature of BM25 is its consideration of text length. Generally, longer texts tend to contain more words, and therefore have a higher word frequency. BM25 compensates for this effect by considering text length. BM25 combines these factors to score the importance of the text. A higher score indicates greater relevance to the search query, resulting in a higher ranking in search results. Of course, the keyword search unit is not limited to BM25; it may be based on existing keyword search methods or other algorithms.

[0065] (Semantic Ranking Search Department) The semantic ranking search unit sorts a predetermined number of chunks based on embedded chunk search and a predetermined number of chunks based on keyword search in descending order of semantic relevance to the utterance phrase, then performs a semantic ranking search to find a predetermined number of chunks from the top, and outputs the searched chunks.

[0066] The semantic ranking unit sorts the text of chunks found by the embedded vector search unit (e.g., 5 items) and the text of chunks found by the keyword search unit (e.g., 5 items) in descending order of semantic relevance to the utterance phrase, and ultimately selects a third predetermined number of chunks from the top. For example, it may select 3 items (the third predetermined number) from 10 items (= 5 items + 5 items) of text.

[0067] Semantic ranking, unlike keyword search, uses techniques such as natural language processing and machine learning to understand the meaning, context, and intent of a search query, and then ranks the results in order of relevance. In particular, it inherits the advantages of fuzzy search and, through syntactic and semantic analysis, can rank search results higher if they are closer to the meaning of the search query. Specific examples include Microsoft's Azure OpenAI (registered trademark) and Google's BERT (Bidirectional Encoder Representations from Transformers).

[0068] As mentioned above, the knowledge data retrieval unit 12 switches between using only the embedded vector retrieval unit or performing a triple search, depending on the required accuracy determined by the search judgment model 16.

[0069] Figure 7 is a functional configuration diagram showing the topic inference model in the present invention.

[0070] According to Figure 7, the information processing device 1 further includes a topic inference model 17 and an avatar display control unit 18, in addition to those shown in Figure 2. These functional components are also realized by executing a program that enables the computer installed in the information processing device to function. Furthermore, the processing flow of these functional components can also be understood as a method of interaction for the information processing device.

[0071] [Topic Inference Model 17] Topic inference model 17 is a lightweight language model that infers a "topic" in response to a prompt. The topic inference model 17 receives prompts instructing it to infer the "topic" representing the user's intent from the time series of the user's utterances. It then infers a "topic" for each utterance and notifies the chunk memory 13 of that topic. Specifically, the topic inference model 17, like the search judgment model 16, may use a lightweight language model, such as GPT-4.1-nano(registered trademark). This allows for real-time determination of user intent for each utterance.

[0072] Here, chunk memory 13 has different buffers depending on the topic. The chunk memory 13 stores the chunks retrieved by the knowledge data retrieval unit 12 in buffers corresponding to their topics. This groups the chunks by topic, making it possible to search even past chunks.

[0073] The chunk memory search unit 14 then searches the chunks stored in the buffer corresponding to the topic of the current utterance in the chunk memory 13 for chunks similar to the utterance in the utterance memory 11. This allows the system to use the knowledge data (chunks) accumulated in the chunk memory 13 for the initial topic A, even if the user's utterance intent switches from topic A to topic B and back to topic A.

[0074] [Avatar display control unit 18] The avatar display control unit 18 displays a personified avatar in response to the user's interaction. The avatar operates in sync with the response voice of the response sentence. The avatar display control unit 18 displays an avatar whose actions are controlled based on the response voice to the user. The avatar display control unit 18 may also control the avatar to nod as a nonverbal expression of agreement when the topic inference model 17 determines that the past topic and the current topic are the same. This allows the user to visually recognize that the intent of their speech has been conveyed.

[0075] Figure 8 is an explanatory diagram of the topic inference model and chunk memory search unit in the present invention.

[0076] According to Figure 8, the topic inference model 17 specifically inputs the following as prompts to be input to the search decision model 16. -------------------------------------------------------------------------------- You are a topic inference engine for voice assistants. Your task is to respond to the user's #utterances. Infer the topic based on the user's intended utterance. -------------------------------------------------------------------------------- The topic inference model 17 then sequentially inputs the user's utterances and notifies the chunk memory 13 of the inferred topic. The chunk memory 13 then instructs the chunk memory search unit 14 to search only the chunks stored in the buffer corresponding to the current topic. Ultimately, the response sentence generation language model 15 generates a response sentence to the user using the utterances in the utterance memory and the chunks in the chunk memory that are stored based on the current utterance intent.

[0077] Figure 9 is a first explanatory diagram showing a sequence of utterances in the present invention. Figure 10 is a second explanatory diagram showing the sequence of utterances in the present invention. Figure 11 is a third explanatory diagram showing the sequence of utterances in the present invention. According to Figures 9-11, the interaction with the user is as follows: <s1> ~ <s6>This is represented by each step.

[0078] <s1> (User's utterance) "What is 5G technology?" (Speech phrase) "What is 5G technology?" (Topic Inference Model 17) Topic A and Inference (Knowledge Data Retrieval Unit 12) Search for the chunk "5G technology" (Chunk memory 13) Remember the chunk "5G technology" as topic A. (Response sentence generation language model 15) Generate the following response: ------------------------------------------------------------------------------- 5G (Fifth Generation Mobile Communication System) refers to the "fifth generation" of new communication standards. This technology has evolved from "communication for watching videos on smartphones" up to 4G, and is designed as "a foundation for connecting all kinds of things (IoT) and social infrastructure to the internet." -------------------------------------------------------------------------------

[0079] <s2> (User's utterance) "I understand, I see..." (Speech phrase) "I understand." "I see." (Topic Inference Model 17) Topic A, the same as in the past. (Knowledge Data Retrieval Unit 12) No search found according to search determination model 16. (Avatar display control unit 18) The avatar nods.

[0080] <s3> (User's utterance) "Is it now available in rural areas as well?" (Speech phrase) Has it become usable in rural areas as well? (Topic Inference Model 17) Topic A and Inference (Knowledge Data Retrieval Unit 12) Search for the "local" chunk (Chunk memory 13) Remember the chunk "Repurposed 5G" as Topic A. (Response sentence generation language model 15) Generate the following response: ------------------------------------------------------------------------------- It's now usable in many places, even in rural areas. However, just like in urban areas, places where you can experience "blazing speed" are still limited. This is because in order to quickly cover a wide area, many places are "repurposed 4G radio waves that have been renamed and used as 5G (repurposed 5G)." -------------------------------------------------------------------------------

[0081] <s4> (User's utterance) "By the way, I want to buy telecommunications stocks, but how do I go about it?" (Speech phrase) "By the way," "I want to buy telecommunications stocks," "but what should I do?" (Topic Inference Model 17) Topic B and Inference (Knowledge Data Retrieval Unit 12) Search for the chunk "telecommunications stocks" (Chunk memory 13) Remember the "telecommunications stocks" chunk as topic B. (Response sentence generation language model 15) Generate the following response: ------------------------------------------------------------------------------- Because major telecommunications companies are in an industry that is less affected by economic fluctuations, many of their stocks offer higher dividends (often yields of around 3-4%) than money deposited in a bank. -------------------------------------------------------------------------------

[0082] <s5> (User's utterance) Which industries are strong in 5G technology? (Speech phrase) Which industries are strong in 5G technology? (Topic Inference Model 17) Topic A and Inference (Knowledge Data Retrieval Unit 12) Search for the "industry" chunk (Chunk memory 13) Remember the "industry" chunk as topic A. (Response sentence generation language model 15) Generate the following response: ------------------------------------------------------------------------------- 5G technology will have a significant impact on many industries, including telecommunications carriers, telecommunications infrastructure providers, semiconductor companies, and measuring instrument companies. -------------------------------------------------------------------------------

[0083] <s6> (User's utterance) "Can you tell me about the communication protocols for the 5G core network?" (Speech phrase) "Can you tell me about the communication protocols for the 5G core network?" (Topic Inference Model 17) Topic A and Inference (Search decision model 16) Required precision for knowledge data retrieval = 0.8 or higher: High (Knowledge Data Retrieval Unit 12) ····· (Response sentence generation language model 15) Generate the following response (filler): ------------------------------------------------------------------------------- That's the communication protocol for the 5G core network. Please wait a moment. -------------------------------------------------------------------------------

[0084] Figure 12 is a functional configuration diagram for building a knowledge database.

[0085] According to Figure 12, the information processing device 1 is equipped with a web interface base that acts as a server for a website and communicates with the operator terminal 3 on which a browser or application is installed. The operator terminal 3 manages "documents" that constitute knowledge data in the response voice, which are transmitted to the information processing device 1 via the user interface base. The operator may be an information operations manager of a specific organization, such as a company, organization, or business operator. The operator is responsible for managing the "documents of knowledge data to be used as responses" to the user, and for publishing FAQs based on those documents.

[0086] <Storage in document database 19> As shown in Figure 12, the operator can specify the file path of the local storage to the information processing device 1. The operator can specify a file path to upload multiple documents under that path to the information processing device 1 and store them in the document database 19. Alternatively, the operator can display the index of multiple files under a specific folder on the operator terminal 3 and select some or all of the indexes of those files. For example, multiple files can be uploaded and registered to the information processing device 1 simultaneously using "cut and paste" or "drag and drop".

[0087] Alternatively, the operator may specify the URL of a website on the internet (Uniform Resource Locator) as the location of the document to the information processing device 1. According to Figure 12, the information processing device 1 has a scraping unit. When a URL or file path is specified by the operator, the scraping unit accesses a website on the internet. It then downloads the document from the URL or file path and stores it in the document database 19.

[0088] Furthermore, the operator can specify documents by including the wildcard "*" in the URL or file path. In this case, the scraping unit downloads multiple documents simultaneously and stores them in the document database 19.

[0089] Furthermore, if a URL or file path is specified by the operator, it is preferable to periodically crawl that URL or file path, download updated documents (site crawling), and update the document database 19.

[0090] <Construction of Knowledge Database 10> According to Figure 12, the information processing device 1 constructs a knowledge database 10 from the document database 19. Here, the information processing device 1 includes a text recognition unit, a chunk division unit, and an embedded vector conversion unit.

[0091] (Text recognition unit) The text recognition unit recognizes text for each document (Character Recognition / Reader). The recognized text is output to the chunking unit. Specifically, character recognition software can be open-source PaddleOCR® or Tesseract OCR®, or cloud-based APIs (Application Programming Interfaces) from Google®, Amazon®, etc. Of course, it can also recognize text from document objects (images, tables, etc.).

[0092] (Chunk division section) The chunking unit divides, for example, text recognized from a document into predetermined chunks. The divided chunks are stored in the knowledge database 10 and also output to the vector conversion unit. The chunking unit may divide the text into chunks of a predetermined unit, for example, every 256 Japanese characters. Here, the number of characters is not limited; for example, if N=2 (the character unit in N-grams), the text may be divided into chunks of 512 tokens. An N-gram is a text analysis algorithm that decodes parts of speech by treating a group of N consecutive words or characters as one token.

[0093] In another embodiment, the chunk division unit may divide the text recognized from each page image into chunks based on layout units, as a predetermined unit. If it is plain text, it may be divided not only by the number of characters but also by paragraphs. If it is HTML or XML, it may be divided not only by the number of characters but also by headers.

[0094] (Embedded vector transformation unit) The embedded vector conversion unit converts each chunk into an embedded vector. As mentioned earlier in Figure 2, embedded vectors are created by converting text-based chunks into high-dimensional vectors that numerically represent meaning and features using the API of a Large-Scale Language Model (LLM). Finally, the transformed embedding vectors are associated with chunks and stored in the knowledge database 10.

[0095] As described in detail above, the information processing device, program, and dialogue method of the present invention enable a voice dialogue system that responds to a user's spoken voice with a response voice to generate a response sentence as quickly as possible.

[0096] Various changes, modifications, and omissions to the scope of the technical concept and viewpoint of the present invention can be readily made by those skilled in the art with respect to the various embodiments of the present invention described above. The above description is merely illustrative and is not intended to limit the present invention in any way. The present invention is limited only to what is limited by the claims and their equivalents. [Explanation of Symbols]

[0097] 1. Information Processing Device 10 Knowledge Databases 101 Voice Recognition Unit 102 Voice Conversion Unit 11. Speech Memory 12 Knowledge Data Retrieval Department 13 chunk memory 14. Chunk memory search unit 15. Response Sentence Generation Language Models 16 Search Decision Model 17. Topic Inference Models 18 Avatar Display Control Unit 19 Document Databases 2 terminals 3 Operator Operation Terminal < / s1>

Claims

1. In an information processing device that responds to a user's spoken voice with a response voice, A knowledge database that stores chunks containing the text of knowledge data, A speech recognition means that takes user speech as input and sequentially recognizes the user's utterances, A speech sentence memory that stores utterances that represent the chronological sequence of user utterances, A knowledge data retrieval means that sequentially inputs user utterances and searches a knowledge database for chunks containing text based on those utterances, Chunk memory that stores the searched chunks, A response sentence generation language model that generates a response sentence to the user at a predetermined response timing using the utterance in the utterance memory and the chunk in the chunk memory, A voice conversion means that takes a response sentence as input and converts it into a voice response for the user. An information processing device characterized by having the following features.

2. The chunk contains embedded vectors based on text, The knowledge database stores text and embedded vectors in a coordinated manner for each chunk. The knowledge data retrieval means converts an utterance phrase into an embedding vector and searches the knowledge database for a predetermined number of top chunks that have embedding vectors similar to the embedding vector of the utterance phrase. The information processing apparatus according to feature 1.

3. The knowledge data retrieval method is Retrieval-Augmented Generation (RAG), Knowledge data retrieval methods are, A speech phrase vector conversion means that sequentially converts user speech phrases into embedded vectors, A vector search means performs an embedding vector search using a knowledge database to find a predetermined number of embedding vectors from the top that are similar to the embedding vector of an utterance phrase, and outputs chunks that contain the found embedding vectors. The information processing apparatus according to claim 2, characterized by having the following features.

4. The system further includes a chunk memory search means that, upon completion of a user's utterance, converts the utterance in the utterance memory into an embedding vector, and searches the chunk memory for a predetermined number of upper chunks that have embedding vectors similar to the embedding vector of the utterance. The response sentence generation language model generates a response sentence to the user using the utterances in the utterance memory and the chunks retrieved by the chunk memory search means. The information processing apparatus according to feature 2.

5. The system further includes a search determination model that takes user utterances sequentially, after inputting a prompt instructing the system to determine whether a chunk search is necessary for each utterance, and then determines whether a chunk search is necessary for each utterance. The knowledge data retrieval means searches for chunks of the utterance phrase when it is determined by the retrieval judgment model to be a searchable item. The information processing apparatus according to feature 1.

6. The search decision model, as a prompt, further describes instructions to estimate the necessary accuracy required for retrieving knowledge data when determining whether a chunk needs to be searched for a given utterance. It then sequentially inputs user utterances and outputs whether a chunk search is necessary and the required accuracy. The response generation language model generates filler responses for the user if the required accuracy is higher than a predetermined threshold. The information processing apparatus according to feature 4.

7. When a search is determined to be necessary by the search determination model and the required precision is output, if the required precision is equal to or greater than a predetermined value, Knowledge data retrieval methods are, A speech phrase vector conversion means that sequentially converts user speech phrases into embedded vectors, A vector search means that uses a knowledge database to perform an embedded vector search, which searches for a first predetermined number of embedded vectors that are similar to the embedded vector of an utterance, and outputs chunks that have said embedded vectors. A keyword search means that uses a knowledge database to perform a keyword search, finding a second predetermined number of chunks from the top that are similar to the keyword of the utterance, and outputs the searched chunks. A semantic ranking search means that sorts a predetermined number of first chunks based on embedded chunk search and a predetermined number of second chunks based on keyword search in order of semantic relevance to the utterance phrase, searches for a predetermined number of third chunks from the top, and outputs the searched chunks. The information processing apparatus according to claim 6, characterized by having the following features.

8. A topic inference model that, after receiving prompts to infer the topic representing the user's utterance intent from the time series of user utterances, infers a topic for each utterance and notifies the corresponding topic to chunk memory. Furthermore, The chunk memory has different buffers depending on the topic, and stores the chunk retrieved by the knowledge data retrieval means in the buffer corresponding to that topic. The chunk memory search means searches for chunks similar to the utterance in the utterance memory from among the chunks stored in the buffer corresponding to the topic of the current utterance phrase in the chunk memory. The information processing apparatus according to claim 1 or 2.

9. An avatar display control means that displays an avatar whose actions are controlled based on the response voice to the user. Furthermore, The avatar display control means controls the avatar to nod as a nonverbal expression of agreement when the topic inference model determines that the past topic and the current topic are the same. The information processing apparatus according to feature 8.

10. In a program that controls a computer installed in an information processing device that responds to a user's spoken voice with a voice response, A knowledge database that stores chunks containing the text of knowledge data, A speech recognition means that takes user speech as input and sequentially recognizes the user's utterances, A speech sentence memory that stores utterances that represent the chronological sequence of user utterances, A knowledge data retrieval means that sequentially inputs user utterances and searches a knowledge database for chunks containing text based on those utterances, Chunk memory that stores the searched chunks, A response sentence generation language model that generates a response sentence to the user at a predetermined response timing using the utterance in the utterance memory and the chunk in the chunk memory, A voice conversion means that takes a response sentence as input and converts it into a voice response for the user. A program characterized by its ability to make a computer function.

11. In a dialogue method for a device that responds to a user's spoken voice with a response voice, The device is It has a knowledge database that stores chunks containing the text of knowledge data, The first step involves using a speech recognition unit to sequentially recognize the user's utterances from the user's spoken audio, The second step involves storing the utterances, which represent the chronological sequence of the user's utterances, in the utterance memory. A third step involves sequentially inputting the user's utterances and searching the knowledge database for chunks containing text based on those utterances. The fourth step is to store the searched chunk in chunk memory, A fifth step involves using a pre-trained response sentence generation language model to generate a response sentence to the user at a predetermined response timing, using the utterance in the utterance memory and the chunk in the chunk memory. The sixth step involves using the voice conversion unit to convert the response text into a voice response to the user. A method for interacting with a device, characterized by performing the following actions.

Images