Information processing device, information processing method, and information processing program

Abstract

This information processing device acquires first text, acquires second text as a search result related to the first text, extracts a phrase to be used for text generation by a language model from the first text, inspects the validity of the second text related to the extracted phrase, and determines an input text for the language model on the basis of the first text, the second text, and the validity inspection result.

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program.

[0002] In recent years, text generation using language models has been performed. For example, Japanese Patent Application Laid-Open No. 2021-068007 discloses a technique for searching for legal information related to chemicals, in which a keyword group is generated by inputting an input question into a language model, a search query is created using the generated keyword group, and a search is performed based on the created search query.

[0003] Recently, a technology called Retrieval-Augmented Generation (RAG) has become known as a search technology using a language model. In this technology, in order to improve the accuracy of the output from the language model, in addition to the input text entered by a user, text that complements the input text is also input to the language model. However, in this technology, if the text that complements the input text is inappropriate, the accuracy of the output from the language model may be reduced.

[0004] The present disclosure has been made in consideration of the above circumstances, and aims to provide an information processing device, an information processing method, and an information processing program that can suppress a decrease in the accuracy of output from a language model.

[0005] A first aspect of the information processing device is an information processing device having at least one processor, which acquires a first text, acquires a second text as a search result related to the first text, extracts phrases from the first text to be used for text generation using a language model, verifies the validity of the second text related to the extracted phrases, and determines input text for the language model based on the first text, the second text, and the validity verification results.

[0006] In the information processing device of the second aspect, in the information processing device of the first aspect, the processor controls whether or not the input text includes the second text in addition to the first text, based on the verification result.

[0007] A third aspect of the information processing device is the information processing device of the second aspect, wherein the verification result indicates whether it is positive or negative, and the processor includes both the first text and the second text in the input text if the verification result indicates that it is positive.

[0008] An information processing device of a fourth aspect is an information processing device of the second or third aspect, in which the verification result indicates whether it is positive or negative, and if the verification result indicates that it is negative, the processor includes only the first text of the first text and the second text in the input text.

[0009] An information processing device of a fifth aspect is the information processing device of any one of the first to fourth aspects, wherein the processor acquires the second text by vector search or full-text search.

[0010] An information processing device of a sixth aspect is an information processing device of any one of the first to fifth aspects, in which the processor extracts words and phrases by inputting a first text into an extraction model obtained by machine learning.

[0011] An information processing device of a seventh aspect is the information processing device of any one of the first to fifth aspects, wherein the processor extracts words and phrases from the first text using dictionary data.

[0012] An information processing device of an eighth aspect is an information processing device of any one of the first to seventh aspects, in which the processor verifies the validity of the second text related to the phrase by inputting the first text, the second text, and the phrase into a verification model obtained by machine learning.

[0013] An information processing device of a ninth aspect is an information processing device of any one of the first to eighth aspects, in which the processor outputs text in which a verification result is added to the text output from the language model by inputting an input text into the language model as a third text corresponding to the first text.

[0014] An information processing device of a tenth aspect is the sixth information processing device, wherein the processor trains the extraction model using loss between output data of the extraction model obtained by inputting a first text for training into the extraction model and correct data corresponding to the first text for training.

[0015] An information processing device of an eleventh aspect is the information processing device of the eighth aspect, wherein the processor trains the verification model using output data of the verification model obtained by inputting a first text, a second text, and phrases for training into the verification model, and losses between the output data of the verification model and correct answer data corresponding to the first text, the second text, and phrases for training.

[0016] In a twelfth aspect of the information processing method, a processor of an information processing device having at least one processor acquires a first text, acquires a second text as a search result related to the first text, extracts from the first text words to be used for text generation using a language model, verifies the validity of the second text related to the extracted words, and performs a process of determining an input text for the language model based on the first text, the second text, and the verification result of the validity.

[0017] An information processing program of a thirteenth aspect causes a processor of an information processing device having at least one processor to acquire a first text, acquire a second text as a search result related to the first text, extract phrases from the first text to be used for text generation using a language model, verify the validity of the second text related to the extracted phrases, and determine an input text for the language model based on the first text, the second text, and the verification result of the validity.

[0018] According to the present disclosure, it is possible to suppress a decrease in accuracy of the output from a language model.

[0019] 1 is a block diagram showing an example of a configuration of an information processing system. 2 is a block diagram showing an example of a hardware configuration of an information processing device. 3 is a diagram showing an example of an extraction model. 4 is a diagram showing an example of learning data. 5 is a diagram showing an example of a search DB. 6 is a diagram showing an example of a verification model. 7 is a diagram showing an example of learning data. 8 is a block diagram showing an example of a functional configuration of an information processing device. 9 is a diagram for explaining a learning process of an extraction model. 10 is a diagram for explaining a learning process of a verification model. 11 is a diagram for explaining a process of determining input text for a language model. 12 is a diagram for explaining a process of determining input text for a language model. 13 is a diagram showing an example of a third text. 14 is a flowchart showing an example of a learning process. 15 is a flowchart showing an example of a text generation process.

[0020] Hereinafter, examples of embodiments for carrying out the technology of the present disclosure will be described in detail with reference to the drawings.

[0021] First, the configuration of an information processing system 10 according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the information processing system 10 includes a user terminal 12 used by a user and an information processing device 14. The user terminal 12 and the information processing device 14 are communicatively connected via a network. Examples of the user terminal 12 include a tablet computer and a personal computer. Examples of the information processing device 14 include a server computer and a cloud server.

[0022] The information processing device 14 is also communicatively connected to a language model 16 via a network. The language model 16 is a natural language processing model trained using a large amount of text data and is also called a large language model (LLM). The language model 16 according to this embodiment is a general-purpose model, not one that has been additionally trained for a specific field. Examples of the language model 16 include ChatGPT, Bard, and T5. The language model 16 may be stored in a storage unit 22 of the information processing device 14, which will be described later.

[0023] Next, the hardware configuration of the information processing device 14 according to this embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the information processing device 14 includes a CPU (Central Processing Unit) 20, a memory 21 as a temporary storage area, and a non-volatile storage unit 22. The information processing device 14 also includes a display 23 such as a liquid crystal display, an input device 24 such as a keyboard and a mouse, and a network I / F (Interface) 25 connected to a network. The CPU 20, the memory 21, the storage unit 22, the display 23, the input device 24, and the network I / F 25 are connected to a bus 27. The CPU 20 is an example of a processor according to the disclosed technology.

[0024] The storage unit 22 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 22 serving as a storage medium stores an information processing program 30. The CPU 20 reads the information processing program 30 from the storage unit 22, loads it into the memory 21, and executes the loaded information processing program 30.

[0025] The storage unit 22 also stores an extraction model 31 , learning data 32 , a search DB (DataBase) 33 , a verification model 34 , and learning data 35 .

[0026] The extraction model 31 is a model obtained by machine learning in a learning phase, which will be described later. As shown in FIG. 3 as an example, when text is input, the extraction model 31 extracts, from the input text, words to be used for text generation by the language model 16. The extraction model 31 is configured by, for example, a transformer, a long short-term memory (LSTM), or a convolutional neural network (CNN). Note that, in this embodiment, an example will be described in which words are used as words extracted by the extraction model 31; however, the words extracted by the extraction model 31 may be a collection of multiple meaningful words, such as a compound word.

[0027] The training data 32 is data used for training the extraction model 31. As an example, as shown in Fig. 4 , the training data 32 includes a plurality of pairs of text input to the extraction model 31 and correct phrases (hereinafter referred to as "correct phrases") extracted from the text as phrases to be used in text generation by the language model 16.

[0028] 5, the search DB 33 includes a plurality of texts obtained as search results. The search DB 33 includes texts that complement the input text for the general-purpose language model 16, such as knowledge in a specific field or rules of a specific organization.

[0029] The verification model 34 is a model obtained by machine learning in a learning phase, which will be described later. As shown in FIG. 6 as an example, when a first text, a second text, and a phrase extracted from the first text are input, the verification model 34 verifies the validity of the second text related to the input phrase and outputs a verification result. In this embodiment, the verification model 34 outputs information indicating whether the verification result is positive or negative. The verification model 34 is configured by, for example, a Transformer, an LSTM, a CNN, or the like.

[0030] The training data 35 is data used for training the verification model 34. As shown in FIG. 7 as an example, the training data 35 includes a plurality of sets of a first text, a second text, phrases extracted from the first text, and correct labels. The correct labels are labels that represent the correct answers of the verification results of the validity of the first text, the second text, and the second text corresponding to the phrases extracted from the first text. The correct labels represent whether they are positive or negative, similar to the output of the verification model 34. Details of the first text and the second text will be described later.

[0031] Next, the functional configuration of the information processing device 14 will be described with reference to Fig. 8. As shown in Fig. 8, the information processing device 14 includes an acquisition unit 40, a learning unit 42, an acquisition unit 50, an extraction unit 52, a search unit 54, a verification unit 56, a determination unit 58, an input unit 60, and an output unit 62. When the CPU 20 executes the information processing program 30, the information processing device 14 functions as the acquisition unit 40, the learning unit 42, the acquisition unit 50, the extraction unit 52, the search unit 54, the verification unit 56, the determination unit 58, the input unit 60, and the output unit 62. The acquisition unit 40 and the learning unit 42 are functional units that function in the learning phase, and the acquisition unit 50, the extraction unit 52, the search unit 54, the verification unit 56, the determination unit 58, the input unit 60, and the output unit 62 are functional units that function in the operation phase.

[0032] The acquisition unit 40 acquires the training data 32 and the training data 35 from the storage unit 22. The training unit 42 trains the extraction model 31 using the training data 32. A specific example of the training process of the extraction model 31 by the training unit 42 will be described below.

[0033] The learning unit 42 inputs text included in the training data 32 into the extraction model 31. As an example, as shown in FIG. 9 , the extraction model 31 outputs the generation probability of each word included in the input text. On the other hand, the correct answer data in this case has a probability distribution in which the generation probability of the correct word is 1.0. The learning unit 42 derives cross entropy as the loss between the output data of the extraction model 31 and the correct answer data according to the following equation (1). Note that in equation (1), n ​​represents the position of the word, and c represents the type of word.

[0034]

[0035] The learning unit 42 then trains the extraction model 31 so as to minimize the derived loss. That is, the learning unit 42 trains the extraction model 31 using the loss between the output data of the extraction model 31 obtained by inputting the first text for training to the extraction model 31 and the correct answer data corresponding to the first text for training.

[0036] The words extracted from the first text by the extraction model 31 are words used in text generation by the language model 16. For example, in the training data 32, words that are useful for text generation by the language model 16, i.e., important words, are set as correct words. For example, the developer may specify the correct words, or the CPU 20 may specify words used in dictionary entries as correct words.

[0037] The learning unit 42 also uses the learning data 35 to train the verification model 34. A specific example of the learning process of the verification model 34 by the learning unit 42 will be described below.

[0038] The learning unit 42 inputs the first text, the second text, and phrases extracted from the first text contained in the training data 35 to the verification model 34. As shown in FIG. 10 as an example, the verification model 34 outputs a positive or negative identification probability representing the verification result of the validity of the second text related to the phrase. On the other hand, the correct answer data in this case has a probability distribution in which the identification probability of the verification result represented by the correct answer label is 1.0. The learning unit 42 derives cross entropy according to the above formula (1) as the loss between the output data of the verification model 34 and the correct answer data.

[0039] The training unit 42 then trains the verification model 34 so as to minimize the derived loss. That is, the training unit 42 trains the verification model 34 using the loss between the output data of the verification model 34 obtained by inputting the first text, the second text, and phrases extracted from the first text for training into the verification model 34, and the correct answer data corresponding to the first text, the second text, and the phrases extracted from the first text for training.

[0040] The acquisition unit 50 acquires the first text transmitted from the user terminal 12 via the network I / F 25. Examples of the first text include a search statement and an instruction statement input by the user of the user terminal 12.

[0041] The extraction unit 52 extracts words and phrases to be used for text generation by the language model 16 from the first text acquired by the acquisition unit 50. Specifically, the extraction unit 52 extracts words and phrases from the first text by inputting the first text to the extraction model 31.

[0042] The extraction unit 52 may use dictionary data to extract words and phrases to be used for text generation by the language model 16 from the first text acquired by the acquisition unit 50. For example, the extraction unit 52 may refer to the dictionary data and extract words and phrases used as dictionary headings from among the words and phrases included in the first text.

[0043] The search unit 54 acquires second text as a search result related to the first text. Specifically, the search unit 54 acquires second text that is most similar to the first text from the search DB 33 by vector search. Techniques such as BERT (Bidirectional Encoder Representations from Transformers), one-hot vector, and tf-idf vector can be used to vectorize the text. Furthermore, cosine similarity or the like can be used as the similarity between the vectorized texts.

[0044] The extraction unit 52 may also acquire, from the search DB 33, the second text in which the word included in the first text appears most frequently by performing a full-text search.

[0045] The verification unit 56 verifies the validity of the second text related to the phrase extracted by the extraction unit 52. Specifically, the verification unit 56 verifies the validity of the second text related to the phrase by inputting the first text acquired by the acquisition unit 50, the second text acquired by the search unit 54, and the phrase extracted by the extraction unit 52 into the verification model 34. Note that the verification unit 56 may cause an external server such as a cloud server to execute the process of verifying the validity of the second text. In this case, the verification unit 56 acquires the verification result from the external server via the network I / F 25.

[0046] The determination unit 58 determines the input text for the language model 16 based on the first text, the second text, and the verification result of validity by the verification unit 56. Specifically, the determination unit 58 controls whether or not to include the second text in addition to the first text in the input text based on the verification result.

[0047] 11 , when the verification result is positive, the determination unit 58 includes both the first text and the second text in the input text. In this case, the second text is used as evidence for the search statement represented by the first text. The input text is input to a prompt for the language model 16.

[0048] On the other hand, as shown in FIG. 12 as an example, when the verification result indicates a negative result, the determining unit 58 includes only the first text of the first text and the second text in the input text.

[0049] The input unit 60 inputs the input text determined by the determination unit 58 to the language model 16 .

[0050] The output unit 62 acquires the answer text output from the language model 16 when the input unit 60 inputs the input text to the language model 16. Then, as shown in Fig. 13 as an example, the output unit 62 outputs, as a third text corresponding to the first text, text in which the verification result of the validity by the verification unit 56 has been added to the acquired answer text, to the user terminal 12. Note that the output unit 62 may also output, as the third text, the answer text to the user terminal 12 without adding the verification result of the validity.

[0051] Next, the operation of the information processing device 14 in the learning phase will be described with reference to Fig. 14. The learning process shown in Fig. 14 is performed by the CPU 20 executing the information processing program 30. The learning process is performed, for example, when the user of the information processing device 14 inputs an instruction to start execution via the input device 24.

[0052] 14, the acquisition unit 40 acquires the training data 32 and the training data 35 from the storage unit 22. In step S12, the training unit 42 trains the extraction model 31 using the training data 32 acquired in step S10, as described above. In step S14, the training unit 42 trains the verification model 34 using the training data 35 acquired in step S10, as described above. When the processing of step S14 ends, the training process ends.

[0053] Next, the operation of the information processing device 14 in the operation phase will be described with reference to Fig. 15. The CPU 20 executes the information processing program 30, thereby executing the text generation process shown in Fig. 15. The text generation process is executed, for example, when the information processing device 14 receives a first text transmitted from the user terminal 12.

[0054] 15 , the acquisition unit 50 acquires the first text transmitted from the user terminal 12 via the network I / F 25. In step S22, the extraction unit 52, as described above, extracts words and phrases to be used for text generation by the language model 16 from the first text acquired in step S20. In step S24, the search unit 54, as described above, acquires the second text as a search result related to the first text acquired in step S20.

[0055] In step S26, the verification unit 56 verifies the validity of the second text related to the phrase extracted in step S24, as described above. In step S28, the determination unit 58 determines whether the verification result from the processing of step S26 indicates a positive result. If this determination is a positive determination, the processing proceeds to step S30.

[0056] In step S30, the determination unit 58 includes both the first text and the second text in the input text for the language model 16. When the process of step S30 ends, the process proceeds to step S34.

[0057] On the other hand, if the verification result from the processing in step S26 indicates a negative result, the determination in step S28 is negative, and the processing proceeds to step S32. In step S32, the determination unit 58 includes only the first text of the first text and the second text in the input text for the language model 16. When the processing in step S32 ends, the processing proceeds to step S34.

[0058] In step S34, the input unit 60 inputs the input text determined in step S30 or step S32 to the language model 16. The language model 16 outputs an answer text corresponding to the input text.

[0059] In step S36, the output unit 62 acquires the answer text output from the language model 16 as a result of the input of the input text to the language model 16 in step S34. Then, the output unit 62 outputs to the user terminal 12 the text obtained by adding the verification result of validity obtained in step S26 to the acquired answer text as a third text corresponding to the first text acquired in step S20. When the processing of step S36 ends, the text generation processing ends.

[0060] As described above, according to this embodiment, based on the verification result of the validity of the second text, it is possible to switch between including and not including the second text in the input text to the language model 16, in addition to the first text. Therefore, it is possible to suppress a decrease in the accuracy of the output from the language model 16.

[0061] In the above embodiment, the following various processors can be used as the hardware structure of a processing unit that executes various processes, such as each functional unit of the information processing device 14. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically for executing specific processes, and a GPU (Graphics Processing Unit).

[0062] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0063] Examples of configuring multiple processing units with a single processor include: first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server computers, and this processor functions as multiple processing units; second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs); in this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0064] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0065] In the above embodiment, the information processing program 30 is pre-stored (installed) in the storage unit 22, but the disclosed technology is not limited to this. The information processing program 30 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program 30 may also be downloaded from an external device via a network.

[0066] The disclosure of Japanese Patent Application No. 2024-084967, filed on May 24, 2024, is incorporated herein by reference in its entirety. In addition, all documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An information processing device having at least one processor, wherein the processor: acquires a first text; acquires a second text as a search result related to the first text; extracts from the first text words and phrases to be used for text generation using a language model; verifies the validity of the second text related to the extracted words and phrases; and determines input text for the language model based on the first text, the second text, and the verification result of the validity.

2. The information processing device according to claim 1, wherein the processor controls whether or not the second text is to be included in the input text in addition to the first text based on the verification result.

3. The information processing device according to claim 2, wherein the verification result indicates whether it is positive or negative, and the processor includes both the first text and the second text in the input text if the verification result indicates that it is positive.

4. The information processing device according to claim 2, wherein the verification result indicates whether it is positive or negative, and the processor includes only the first text of the first text and the second text in the input text if the verification result indicates that it is negative.

5. The information processing device according to any one of claims 1 to 4, wherein the processor acquires the second text by vector search or full-text search.

6. An information processing device according to any one of claims 1 to 4, wherein the processor extracts the phrase by inputting the first text into an extraction model obtained by machine learning.

7. The information processing device according to any one of claims 1 to 4, wherein the processor extracts the words and phrases from the first text using dictionary data.

8. An information processing device according to any one of claims 1 to 4, wherein the processor verifies the validity of the second text related to the phrase by inputting the first text, the second text, and the phrase into a verification model obtained by machine learning.

9. An information processing device as claimed in any one of claims 1 to 4, wherein the processor outputs text in which the verification result is added to text output from the language model by inputting the input text into the language model as a third text corresponding to the first text.

10. The information processing device described in claim 6, wherein the processor trains the extraction model using loss between output data of the extraction model obtained by inputting the first text for training into the extraction model and correct data corresponding to the first text for training.

11. The information processing device described in claim 8, wherein the processor trains the verification model using the output data of the verification model obtained by inputting the first text, the second text, and the phrases for training into the verification model, and the loss between the output data of the verification model and the correct answer data corresponding to the first text, the second text, and the phrases for training.

12. An information processing method, in which the processor of an information processing device having at least one processor executes the following processes: acquire a first text; acquire a second text as a search result related to the first text; extract from the first text words and phrases to be used for text generation using a language model; verify the validity of the second text related to the extracted words and phrases; and determine input text for the language model based on the first text, the second text, and the verification result of the validity.

13. An information processing program causing an information processing device having at least one processor to execute the following processes: acquire a first text; acquire a second text as a search result related to the first text; extract from the first text words and phrases to be used in text generation using a language model; verify the validity of the second text related to the extracted words and phrases; and determine input text for the language model based on the first text, the second text, and the verification result of the validity.

Images