Information processing apparatus

Abstract

To improve the search accuracy of a knowledge base. An information processing apparatus includes a generation unit that generates a feature amount space based on a feature amount of one or more words having a high frequency of occurrence among a plurality of words included in text data registered in a database, and a conversion unit that converts one word to another word in a case where one feature amount of one word included in one text data newly registered in the database deviates from other feature amounts in the feature amount space.

Description

Technical Field

[0001] This invention relates to the technical field of an information processing device. Background Technology

[0002] As such a device, for example, a device has been proposed that enables a language model to generate document-based query data and uses the pairing of documents and query data for learning the search model of a chatbot (see Patent Document 1).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2023-076413 Summary of the Invention

[0004] As a chatbot, a proposed approach combines large-scale language models (LLMs) with searches using specific information sources (hereinafter, appropriately referred to as "knowledge bases"), employing a mechanism unique to LLMs (Retrieval-Augmented Generation, RAG). However, the terminology in the text data registered in the knowledge base is often inconsistent. For example, searching the knowledge base using the keyword "turn signal" might yield results for text containing either "turn signal" or "direction indicator." The search score for text containing "direction indicator" is typically lower than that for text containing "turn signal." This could potentially decrease the search accuracy of the knowledge base. Furthermore, large-scale language models refer to language models constructed using extremely large datasets and deep learning techniques.

[0005] The present invention was made in view of the above circumstances, and its object is to provide an information processing device that can improve the search accuracy of a knowledge base.

[0006] An information processing apparatus according to one aspect of the present invention comprises: a generation unit that generates a feature quantity space based on the feature quantity of one or more words with relatively high frequency of occurrence among a plurality of words contained in text data registered in a database; and a conversion unit that converts a word into other words when a feature quantity of a word contained in a newly registered text data in the database deviates from other feature quantities in the feature quantity space. Attached Figure Description

[0007] Figure 1 This is a diagram showing the structure of the information processing system involved in the implementation method.

[0008] Figure 2 This is a block diagram illustrating an example of the structure of the computing device involved in the implementation. Detailed Implementation

[0009] refer to Figure 1 and Figure 2 The implementation methods related to the information processing device will be described. Figure 1 In this system, information processing system 1 includes information processing device 10, server 20, and knowledge base 30. Information processing device 10, server 20, and knowledge base 30 are configured to communicate with each other via network NW. Server 20 is a server used for applying Large Scale Language Model (LLM). Therefore, server 20 can be called an LLM server. Alternatively, server 20 can be a cloud server.

[0010] (Chatbot)

[0011] Server 20 and knowledge base 30 can provide chatbot services using RAG. For example, user U can utilize the chatbot service via terminal device 50. In this case, user U can operate terminal device 50 to launch an application for utilizing the chatbot service. User U can operate terminal device 50 to enter a question in the input field of the chat application. Here, "question" is not limited to interrogative sentences. For example, "question" can be a sentence containing requests, instructions, commands, etc., such as "Please tell me about ****" or "Please answer about ****". Therefore, "question" is not limited to interrogative sentences, but can include sentences containing requests, instructions, commands, etc. That is, "question" can refer to a sentence requesting an answer from the other party.

[0012] Terminal device 50 can search knowledge base 30 based on the input query. Terminal device 50 can send first information, including the input query and text data representing the search results from knowledge base 30, to server 20. Server 20 can input the query and text data included in the first information as prompts into a large-scale language model. Server 20 can obtain an answer to the query output from the large-scale language model. Server 20 can send second information representing the answer to terminal device 50. Upon receiving the second information, terminal device 50 can display the answer represented by the second information on a screen related to the chat application. Furthermore, terminal device 50 can be a personal computer, tablet, or smartphone.

[0013] (Information processing device 10)

[0014] exist Figure 1The information processing device 10 includes a computing unit 11, a storage unit 12, a communication unit 13, an input unit 14, and an output unit 15. The computing unit 11, storage unit 12, communication unit 13, input unit 14, and output unit 15 are connected via a data bus 16. Furthermore, the information processing device 10 can be a personal computer, a tablet terminal, or a smartphone.

[0015] The arithmetic unit 11 may have a processor. Furthermore, the arithmetic unit 11 may have a single processor or multiple processors. That is, the arithmetic unit 11 may have more than one processor. Additionally, the processor may be a multi-core processor. In the case where the arithmetic unit 11 has a single processor that functions as a multi-core processor, it can be said that the arithmetic unit 11 logically has multiple processors.

[0016] The processor may be at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), and a Tensor Processing Unit (TPU).

[0017] Storage device 12 may be at least one of random access memory (RAM), read-only memory (ROM), hard disk drive, magneto-optical disk drive, solid-state drive (SSD), and optical disk array. That is, storage device 12 may be implemented by a single device or by multiple devices.

[0018] The communication device 13 can communicate with external devices of the information processing device 10. In addition, the communication device 13 can perform wired communication or wireless communication.

[0019] Input device 14 is a device capable of accepting information input from an external source to information processing device 10. Input device 14 may include user-operable devices (e.g., keyboard, mouse, touch panel, etc.) of information processing device 10. Input device 14 may include, for example, a recording medium reading device capable of reading information recorded on a recording medium removable from information processing device 10, such as a USB (Universal Serial Bus) memory. Furthermore, when information is input to information processing device 10 via communication device 13 (in other words, when information processing device 10 obtains information via communication device 13), communication device 13 can function as an input device.

[0020] Output device 15 is a device capable of outputting information to the outside of information processing device 10. Output device 15 may include a display device capable of outputting visual information such as characters and images. Additionally, output device 15 may include a speaker capable of outputting auditory information such as sound. Output device 15 may include a vibration motor capable of outputting tactile information such as vibration. Output device 15 may include a printer. Output device 15 may output information to a recording medium removable from information processing device 10, such as a USB memory. Furthermore, when information processing device 10 outputs information via communication device 13, communication device 13 may function as an output device.

[0021] Storage device 12 is capable of storing desired data. The computer program CP executed by the arithmetic unit 11 can be stored in storage device 12. When the arithmetic unit 11 executes the computer program CP, storage device 12 can temporarily store data temporarily used by the arithmetic unit 11.

[0022] Furthermore, the computer program CP can be recorded on a computer-readable and non-temporary recording medium. In this case, the computer program CP can be stored in the storage device 12 by reading the recording medium using a recording medium reading device (not shown) included in the information processing device 10. Additionally, at least one of optical discs, magnetic media, magneto-optical discs, semiconductor memory, and any medium capable of storing other programs can be used as the recording medium. Furthermore, the computer program CP can also be obtained from an external (not shown) device outside the information processing device 10 via the communication device 13. In other words, the computer program CP can be downloaded from an external device to the storage device 12 of the information processing device 10.

[0023] The arithmetic unit 11 (e.g., a processor) can perform the processing to be performed by the information processing unit 10 together with the storage device 12 storing the computer program CP (in other words, together with the storage device 12 and the computer program CP stored in the storage device 12). For example, the arithmetic unit 11 can implement logical function blocks for performing the processing to be performed by the information processing unit 10 within the arithmetic unit 11 (e.g., within the processor) by executing the computer program CP.

[0024] (Knowledge Base 30)

[0025] Multiple text data entries can be registered in knowledge base 30. These multiple text data entries can be fragments of text contained in a document. Such fragments can be called "blocks". Furthermore, methods for segmenting the text contained in a document can include, for example, segmenting with a constant length (in other words, fixed length), segmenting by sentence based on sentence delimiters, and segmenting based on structures such as Markdown. Each of the multiple text data entries can be vectorized text data. That is, knowledge base 30 can be a vector database / vector storage.

[0026] The expressions (e.g., words, phrasing) used in the various text data registered in knowledge base 30 are almost not standardized. On the other hand, by improving the accuracy of meaning search or keyword search, it is possible to correspond to words that are different from the word used for the search but have similar meanings. For example, when searching knowledge base 30 using the keyword "turn signal," the search score for text data containing "direction indicator" is generally lower than the search score for text data containing "turn signal." As a result, the search accuracy of knowledge base 30 may decrease.

[0027] Therefore, the information processing apparatus 10 according to this embodiment performs conversion processing on the text data newly registered in the knowledge base 30. For example... Figure 2 As shown, the arithmetic unit 11 of the information processing apparatus 10 includes a generation unit 111 and a conversion unit 112 for performing the aforementioned conversion process. The generation unit 111 and the conversion unit 112 can be implemented as the aforementioned logic function blocks. Alternatively, at least one of the generation unit 111 and the conversion unit 112 can be implemented as a physical processing circuit. Or, at least one of the generation unit 111 and the conversion unit 112 can be implemented in a manner where logic function blocks and physical processing circuits coexist.

[0028] The generation unit 111 generates a feature space based on the feature values ​​of one or more words with relatively high frequency of occurrence among the multiple words contained in the text data registered in the knowledge base 30. Furthermore, various existing methods can be applied to the method of counting word frequency occurrences. Therefore, a detailed description of the method for counting word frequency occurrences is omitted.

[0029] The conversion unit 112 can extract a feature value of a word contained in a newly registered text data in the knowledge base 30. The conversion unit 112 can determine whether the feature value deviates from other features in the aforementioned feature value space (i.e., the feature values ​​of words contained in the text data already registered in the knowledge base 30). Here, in the case where a feature value deviates from other features, a word refers to a word that is not used or is almost never used in the text data already registered in the knowledge base 30.

[0030] When one feature deviates from the others, the transformation unit 112 converts one word into another word. For example, the transformation unit 112 can convert one word into another word that is a synonym of that word. Here, the other words are words that are frequently used in the text data registered in the knowledge base 30.

[0031] (Technical effect)

[0032] The information processing apparatus 10 according to this embodiment can convert words contained in newly registered text data in the knowledge base 30 into words frequently used in text data already registered in the knowledge base 30. Therefore, according to the information processing apparatus 10, expressions used in text data already registered in the knowledge base 30 can be standardized. As a result, according to the information processing apparatus 10, the search accuracy of the knowledge base can be improved.

[0033] Hereinafter, various aspects of the invention derived from the embodiments described above will be described.

[0034] One aspect of the invention relates to an information processing apparatus comprising: a generation unit that generates a feature space based on the feature values ​​of one or more words with relatively high frequency of occurrence among a plurality of words contained in text data registered in a database; and a conversion unit that converts a word into another word when a feature value of a newly registered word in text data in the database deviates from other feature values ​​in the feature space. In the above embodiment, "knowledge base 30" is an example equivalent to "database", "generation unit 111" is an example equivalent to "generation unit", and "conversion unit 112" is an example equivalent to "conversion unit".

[0035] In the information processing apparatus described above, the other words may be synonyms of the first word. In the information processing apparatus described above, the database may be a database generated for search expansion.

[0036] This invention is not limited to the embodiments described above, and appropriate modifications can be made without departing from the spirit or concept of the invention as read in its entirety from the claims and description. Information processing apparatuses that accompany such modifications are also included within the technical scope of this invention.

[0037] Symbol Explanation

[0038] 1-Information processing system, 10-Information processing device, 20-Server, 30-Knowledge base, 111-Generation unit, 112-Conversion unit.

Claims

1. An information processing device, characterized in that, have: The generation unit generates a feature space based on the feature quantity of one or more words with relatively high frequency of occurrence among multiple words contained in the text data registered in the database. and A conversion unit that converts a word into another word when a feature of a word contained in newly registered text data in the database deviates from other feature values ​​in the feature value space.

2. The information processing device according to claim 1, characterized in that, The other words are synonyms of the one word.

3. The information processing device according to claim 1, characterized in that, The database is the one generated for search expansion.

Images