Information processing apparatus

Abstract

An object of the present application is to reduce the burden on an administrator of a knowledge base. An information processing apparatus includes: a presumption unit that presumes preprocessing to be performed on a piece of data newly registered in a database using a learning model that learns a relationship between a category of the data registered in the database and a category of preprocessing performed on the data; and a proposal unit that proposes the presumed preprocessing.

Description

Technical Field

[0001] This invention relates to the technical field of an information processing device. Background Technology

[0002] As such a device, for example, the following device has been proposed: a language model generates document-based query data, and the pairing of documents and query data is used for learning the retrieval model of a chatbot (see Patent Document 1).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2023-076413 Summary of the Invention

[0004] As a conversational robot, a chatbot is proposed that utilizes a mechanism (Retrieval-Augmented Generation: RAG) to endow the large-scale language model with a unique information source by combining retrieval with a specific information source (hereinafter, appropriately referred to as the "knowledge base"). Data subject to prescribed processing is registered in the knowledge base. Prescribed processing may include preprocessing of the raw data. The content of this preprocessing is typically determined by the knowledge base administrator. Furthermore, a large-scale language model refers to a language model constructed using a very large dataset and deep learning techniques.

[0005] The present invention was made in view of the above circumstances, and its object is to provide an information processing device that can reduce the burden on knowledge base administrators.

[0006] One aspect of the present invention relates to an information processing apparatus comprising: an estimation unit that uses a learning model that learns the relationship between the categories of data registered in a database and the categories of preprocessing performed on the data to estimate a preprocessing to be performed on a newly registered piece of data in the database; and a proposal unit that proposes the estimated preprocessing.

[0007] Another aspect of the present invention relates to an information processing apparatus comprising: an estimation unit that uses a learning model that learns the relationship between the categories of data registered in a database and the categories of preprocessing performed on the data to estimate preprocessing to be performed on a newly registered piece of data in the database; and a processing unit that applies the estimated preprocessing to the piece of data. Attached Figure Description

[0008] Figure 1 This is a diagram showing the structure of the information processing system involved in the implementation method.

[0009] Figure 2This is a diagram illustrating an example of processing performed on data registered in a knowledge base.

[0010] Figure 3 This is a block diagram illustrating an example of the structure of the computing device involved in the implementation.

[0011] Figure 4 This is a block diagram illustrating another example of the structure of the computing device involved in the implementation. Detailed Implementation

[0012] <First Embodiment>

[0013] refer to Figures 1 to 3 The first embodiment of the information processing apparatus will be described. Figure 1 In this system, information processing system 1 includes information processing device 10, server 20, and knowledge base 30. Information processing device 10, server 20, and knowledge base 30 are configured to communicate with each other via network NW. Server 20 is a server used for applying Large Scale Language Model (LLM). Therefore, server 20 can be called an LLM server. Alternatively, server 20 can be a cloud server.

[0014] (Chatbot)

[0015] Server 20 and knowledge base 30 can provide chatbot services using RAG. For example, user U can utilize the chatbot service via terminal device 50. In this case, user U can operate terminal device 50 to launch an application for utilizing the chatbot service. User U can operate terminal device 50 to enter a question in the input field of the chat application. Here, "question" is not limited to interrogative sentences. For example, "question" can be a sentence containing requests, instructions, commands, etc., such as "Please tell me about ****" or "Please answer about ****". Therefore, "question" is not limited to interrogative sentences; it can include sentences containing requests, instructions, commands, etc. That is, "question" can refer to a sentence requesting an answer from the other party.

[0016] Terminal device 50 can retrieve knowledge base 30 based on the input query. Terminal device 50 can send first information, including the input query and text data as the retrieval result of knowledge base 30, to server 20. Server 20 can input the query and text data included in the first information as prompts into a large-scale language model. Server 20 can obtain the answer to the query output from the large-scale language model. Server 20 can send second information representing the answer to terminal device 50. Upon receiving the second information, terminal device 50 can display the answer represented by the second information on a screen related to the chat application. Additionally, terminal device 50 can be a personal computer, tablet, or smartphone.

[0017] (Knowledge Base 30)

[0018] Multiple text data can be registered in knowledge base 30. Each of these text data can be vectorized text data. That is, knowledge base 30 can be a vector database / vector store. See here for reference. Figure 2 An example of how knowledge base 30 is constructed is provided.

[0019] exist Figure 2 In this context, the data source may include raw data (e.g., documents, images, etc.). When a piece of raw data included in the data source is registered in knowledge base 30, the administrator M of knowledge base 30 can specify the preprocessing to be performed on that piece of raw data. For example, if the raw data is a document (in other words, text data), the preprocessing to be performed on that raw data could be synonym conversion, summarization, etc. Similarly, if the raw data is image data, the preprocessing to be performed on that raw data could be image documentation (e.g., Optical Character Recognition (OCR)).

[0020] In this embodiment, text data is generated as a result of preprocessing the original data. The text data generated through preprocessing can be segmented. Through this segmentation, the text data can be divided into multiple data segments (i.e., chunks). Specific examples of segmentation include methods that segment by a predetermined length (in other words, fixed length), methods that segment by sentence based on sentence delimiters, and methods that segment based on structures such as Markdown.

[0021] Subsequently, each of the multiple blocks is converted into a numerical vector. That is, an embedding is generated based on each of the multiple blocks. Moreover, the blocks that have been converted into numerical vectors (in other words, the embeddings) are registered in knowledge base 30.

[0022] (Information processing device 10)

[0023] exist Figure 1 The information processing device 10 includes a computing unit 11, a storage unit 12, a communication unit 13, an input unit 14, and an output unit 15. The computing unit 11, storage unit 12, communication unit 13, input unit 14, and output unit 15 are connected via a data bus 16. Furthermore, the information processing device 10 can be a personal computer, a tablet terminal, or a smartphone.

[0024] The arithmetic unit 11 may have a processor. Furthermore, the arithmetic unit 11 may have a single processor or multiple processors. That is, the arithmetic unit 11 may have more than one processor. Additionally, the processor may be a multi-core processor. In the case where the arithmetic unit 11 has a single processor that functions as a multi-core processor, it can be said that the arithmetic unit 11 logically has multiple processors.

[0025] The processor may be at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), and a Tensor Processing Unit (TPU).

[0026] Storage device 12 may be at least one of random access memory (RAM), read-only memory (ROM), hard disk drive, magneto-optical disk drive, solid-state drive (SSD), and optical disk array. That is, storage device 12 may be implemented by a single device or by multiple devices.

[0027] The communication device 13 can communicate with external devices of the information processing device 10. In addition, the communication device 13 can perform wired communication or wireless communication.

[0028] Input device 14 is a device capable of accepting information input from an external source to information processing device 10. Input device 14 may include user-operable devices (e.g., keyboard, mouse, touch panel, etc.) of information processing device 10. Input device 14 may include, for example, a recording medium reading device capable of reading information recorded on a recording medium removable from information processing device 10, such as a Universal Serial Bus (USB) memory. Furthermore, when information is input to information processing device 10 via communication device 13 (in other words, when information processing device 10 obtains information via communication device 13), communication device 13 may function as an input device.

[0029] Output device 15 is a device capable of outputting information to the outside of information processing device 10. Output device 15 may include a display device capable of outputting visual information such as characters and images. Additionally, output device 15 may include a speaker capable of outputting auditory information such as sound. Output device 15 may include a vibration motor capable of outputting tactile information such as vibration. Output device 15 may include a printer. Output device 15 may output information to a recording medium removable from information processing device 10, such as a USB memory. Furthermore, when information processing device 10 outputs information via communication device 13, communication device 13 may function as an output device.

[0030] Storage device 12 is capable of storing desired data. The computer program CP executed by the arithmetic unit 11 can be stored in storage device 12. When the arithmetic unit 11 executes the computer program CP, storage device 12 can temporarily store data temporarily used by the arithmetic unit 11.

[0031] Furthermore, the computer program CP can be recorded on a computer-readable and non-temporary recording medium. In this case, the computer program CP can be stored in the storage device 12 by reading the recording medium using a recording medium reading device (not shown) included in the information processing device 10. Additionally, at least one of optical discs, magnetic media, magneto-optical discs, semiconductor memory, and any medium capable of storing other programs can be used as the recording medium. Furthermore, the computer program CP can also be obtained from an external (not shown) device outside the information processing device 10 via the communication device 13. In other words, the computer program CP can be downloaded from an external device to the storage device 12 of the information processing device 10.

[0032] The arithmetic unit 11 (e.g., a processor) can perform the processing to be performed by the information processing unit 10 together with the storage device 12 storing the computer program CP (in other words, together with the storage device 12 and the computer program CP stored in the storage device 12). For example, the arithmetic unit 11 can implement logical function blocks for performing the processing to be performed by the information processing unit 10 within the arithmetic unit 11 (e.g., within the processor) by executing the computer program CP.

[0033] As described above, when the administrator M specifies the preprocessing to be performed on the raw data, the burden on the administrator M is not small. Therefore, the information processing apparatus 10 according to this embodiment uses a learning model that learns the relationship between the categories of raw data registered in the knowledge base 30 and the categories of preprocessing to be performed on the raw data to make an estimate of the preprocessing to be performed on a piece of raw data newly registered in the knowledge base 30.

[0034] Furthermore, the aforementioned learning model can be a rule-based model based on the categories of the original data and the categories of the preprocessing applied to that original data. Alternatively, the aforementioned learning model can be a model constructed using machine learning to learn data by using a combination of categories representing the categories of the original data and the categories of the preprocessing applied to that original data.

[0035] like Figure 3 As shown, in order to perform the above estimation, the arithmetic unit 11 of the information processing apparatus 10 has an estimation unit 111 and a proposal unit 112. The estimation unit 111 and the proposal unit 112 can be implemented as the above-described logic function blocks. In addition, at least one of the estimation unit 111 and the proposal unit 112 can be implemented as a physical processing circuit. Alternatively, at least one of the estimation unit 111 and the proposal unit 112 can be implemented in a manner where logic function blocks and physical processing circuits coexist.

[0036] For example, the estimation unit 111 can input a piece of raw data included in the data source into the learning model. The learning model, having received a piece of raw data, can output information indicating the preprocessing to be performed on that piece of raw data. The estimation unit 111 can estimate the preprocessing to be performed on a piece of raw data based on the information output from the learning model. The proposal unit 112 can control the output device 15 to propose information indicating the preprocessing estimated by the estimation unit 111 to the administrator of the knowledge base 30 (e.g., administrator M).

[0037] (Technical effect)

[0038] The information processing apparatus 10 according to this embodiment presupposes the preprocessing to be performed on the raw data registered in the knowledge base 30 and proposes the presupposed preprocessing. If configured in this way, for example, the burden on the administrator of the knowledge base 30 (e.g., administrator M) to study the preprocessing to be performed on the raw data can be reduced. Therefore, according to the information processing apparatus 10, the burden on the administrator of the knowledge base 30 can be reduced.

[0039] <Second Implementation>

[0040] refer to Figure 1 and Figure 4 The second embodiment of the information processing apparatus will now be described. In the second embodiment, the structure of the information processing apparatus differs in only one part; otherwise, it is the same as the first embodiment described above. Therefore, descriptions that are repeated in the first embodiment will be omitted regarding the second embodiment.

[0041] like Figure 4 As shown, in order to perform the above estimation, the arithmetic unit 11 of the information processing apparatus 10 according to the second embodiment has an estimation unit 111 and a processing unit 113. The estimation unit 111 and the processing unit 113 can be implemented as the above-described logic function blocks. In addition, at least one of the estimation unit 111 and the processing unit 113 can be implemented as a physical processing circuit. Alternatively, at least one of the estimation unit 111 and the processing unit 113 can be implemented in a manner where logic function blocks and physical processing circuits coexist.

[0042] For example, the estimation unit 111 can input a piece of raw data included in the data source into the learning model. The learning model, having received a piece of raw data, can output information indicating the preprocessing to be performed on that piece of raw data. The estimation unit 111 can estimate the preprocessing to be performed on a piece of raw data based on the information output from the learning model. The processing unit 113 can apply the preprocessing estimated by the estimation unit 111 to the aforementioned piece of raw data.

[0043] (Technical effect)

[0044] The information processing apparatus 10 according to this embodiment presupposes that preprocessing be performed on the raw data registered in the knowledge base 30, and applies the presupposed preprocessing to the raw data. If configured in this way, preprocessing of the raw data is performed automatically. Therefore, according to the information processing apparatus 10, the burden on the administrator of the knowledge base 30 can be reduced.

[0045] Hereinafter, various aspects of the invention derived from the embodiments described above will be described.

[0046] One aspect of the invention relates to an information processing apparatus comprising: an estimation unit that uses a learning model that learns the relationship between the categories of data registered in a database and the categories of preprocessing performed on said data, to estimate the preprocessing to be performed on a newly registered piece of data in the database; and a proposal unit that proposes the estimated preprocessing. In the above embodiment, "knowledge base 30" is an example equivalent to "database", "estimation unit 111" is an example equivalent to "estimation unit", and "proposal unit 112" is an example equivalent to "proposal unit".

[0047] Another aspect of the invention relates to an information processing apparatus comprising: an estimation unit that uses a learning model that learns the relationship between the categories of data registered in a database and the categories of preprocessing performed on said data, to estimate the preprocessing to be performed on a newly registered piece of data in the database; and a processing unit that applies the estimated preprocessing to said piece of data. In the above embodiment, "processing unit 113" is an example of "processing unit".

[0048] In the information processing apparatus described above, the database may be a database used for retrieval enhancement generation.

[0049] This invention is not limited to the embodiments described above, and appropriate modifications can be made without departing from the spirit or concept of the invention as read in its entirety from the claims and description. Information processing apparatuses that accompany such modifications are also included within the technical scope of this invention.

[0050] Symbol Explanation

[0051] 1-Information processing system, 10-Information processing device, 20-Server, 30-Knowledge base, 111-Presumption department, 112-Proposal department, 113-Processing department.

Claims

1. An information processing device, characterized in that, have: An estimation unit, using a learning model that has learned the relationship between the categories of data registered in the database and the categories of preprocessing performed on said data, estimates the preprocessing to be performed on a newly registered piece of data in the database; and The proposal unit proposes the presumed preprocessing.

2. An information processing device, characterized in that, have: The estimation unit uses a learning model that learns the relationship between the categories of data registered in the database and the categories of preprocessing performed on the data to estimate the preprocessing to be performed on a newly registered piece of data in the database. and The processing unit applies the estimated preprocessing to the data.

3. The information processing apparatus according to claim 1 or 2, characterized in that, The database in question is a database generated for retrieval enhancement.

Images