Natural language processing system and natural language processing method

Abstract

To enable reliable natural language processing even when using computers with specifications below a certain level. [Solution] The system calculates a first similarity between the question input to each of the multiple primary candidate information processing devices and the reference data that the LLM refers to when generating the answer. The primary candidate information processing device selected so that its calculated first similarity does not approximate any of the other primary candidate information processing devices is designated as the secondary candidate information processing device. The answers generated by each of the secondary candidate information processing devices are classified into multiple groups based on the similarity of their content. Each of the secondary candidate information processing devices that generated the answer belonging to the group with the most responses is selected as the tertiary candidate information processing device. The system is configured to select the final answer from among the answers generated by the tertiary candidate information processing devices based on the answer evaluation value calculated for each tertiary candidate information processing device.

Description

Technical Field

[0001] The present invention relates to a natural language processing system and a natural language processing method.

Background Art

[0002] There is known a technique that receives a question specified by a user and creates an answer by an LLM.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, there may be a case where it is desired to operate an LLM with specifications of a general PC or the like. In this case, since it is difficult to operate a large-scale LLM with a large number of parameters, a small-scale LLM such as a quantized LLM or an LLM with a small number of parameters is used. However, hallucination is likely to occur in such a small-scale LLM.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to enable highly reliable natural language processing while using a computer with specifications below a certain level.

Means for Solving the Problems

[0006] One aspect of the present invention that solves the above-mentioned problems includes: a question receiving unit that receives an input question; an answer requesting unit that requests an answer to the question by inputting the question received by the question receiving unit to a plurality of primary candidate information processing devices; and each of the plurality of primary candidate information processing devices that processes the answer to the input question using LLM (Large Language). This natural language processing system comprises: an answer generation unit that generates answers using Models; a first selection unit that calculates a first similarity between the input question and reference data that the LLM refers to when generating answers for each of the multiple primary candidate information processing units, and selects a primary candidate information processing unit that does not approximate any of the other primary candidate information processing units as a secondary candidate information processing unit; a second selection unit that classifies the answers generated by each of the secondary candidate information processing units selected by the first selection unit into multiple groups based on the similarity of content, and selects each of the secondary candidate information processing units that generated answers belonging to the group with the most answers as a tertiary candidate information processing unit; and a third selection unit that calculates an answer evaluation value indicating the reliability of the answers from among the answers generated by the tertiary candidate information processing units selected by the second selection unit, and selects the final answer to the question received by the question receiving unit based on the calculated answer evaluation value.

[0007] One aspect of the present invention is a natural language processing method in a natural language processing system, comprising: a question receiving step in which a question receiving unit receives an input question; an answer request step in which an answer request unit requests an answer to the question by inputting the question received by the question receiving step to a plurality of primary candidate information processing devices; and an answer generation unit which generates an answer to the input question in each of the plurality of primary candidate information processing devices using LLM (Large Language). This natural language processing method includes: an answer generation step generated by Models; a first selection step in which a first selection unit calculates a first similarity between the input question and reference data that the LLM refers to when generating the answer for each of the plurality of primary candidate information processing devices, and selects a primary candidate information processing device that does not approximate any of the other primary candidate information processing devices as a secondary candidate information processing device; a second selection step in which a second selection unit classifies the answers generated by each of the secondary candidate information processing devices selected in the first selection step into a plurality of groups based on the similarity of content, and selects each of the secondary candidate information processing devices that generated answers belonging to the group with the most answers as a tertiary candidate information processing device; and a third selection step in which a third selection unit calculates an answer evaluation value indicating the reliability of the answer from among the answers generated by the tertiary candidate information processing devices selected in the second selection step, and selects the final answer to the question received by the question receiving step based on the calculated answer evaluation value. [Effects of the Invention]

[0008] As explained above, the present invention provides the effect of enabling highly reliable natural language processing even when using a computer with specifications below a certain level. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the overall configuration of the natural language processing system in this embodiment. [Figure 2] This figure shows an example of the hardware configuration of the information processing device in this embodiment. [Figure 3] This figure shows an example of the functional configuration of the information processing device in this embodiment. [Figure 4] This figure shows an example of a processing procedure executed by the information processing device in this embodiment. [Figure 5] This figure illustrates a specific example of selecting a secondary candidate information processing device in this embodiment. [Figure 6] This figure illustrates a specific example of selecting a third-order candidate information processing device in this embodiment. [Figure 7] This figure illustrates a specific example of the selection of the final answer in this embodiment. [Modes for carrying out the invention]

[0010] Figure 1 shows an example of the overall configuration of the natural language processing system of this embodiment. The natural language processing system of this embodiment is constructed by connecting multiple information processing devices via a network so that they can communicate with each other. In the figure, as one specific example, an example is shown in which four information processing devices 10, namely the first information processing device 10-1, the second information processing device 10-2, the third information processing device 10-3, and the fourth information processing device 10-4, are connected via a network so that they can communicate with each other. The network connecting the multiple information processing devices 10 may be a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0011] In this embodiment, the information processing device 10 is configured to generate answers in response to input questions (prompts) using LLMs (Large Language Models). The information processing device 10 in this embodiment may be, for example, a general-purpose personal computer, or a computer with performance equivalent to that of a general-purpose personal computer. In this case, since it is difficult for the information processing device 10 to run a large-scale LLM, it may use a small-scale LLM of about 1 MB, or a quantized LLM. The natural language processing system of this embodiment is configured to output an answer corresponding to an input question through the cooperation of multiple information processing devices 10.

[0012] Referring to Figure 2, an example of the hardware configuration of the information processing device 10 will be described. This figure is a block diagram showing an example of the hardware configuration of the information processing device 10.

[0013] The information processing device 10 includes a CPU 11, main memory 12, video subsystem 13, display unit 14, chipset 21, BIOS memory 22, storage medium 23, audio system 24, WLAN card 25, USB connector 26, embedded controller 31, input unit 32, power supply circuit 33, and battery 34.

[0014] The CPU 11 controls the entire information processing unit 10 by executing various arithmetic operations under program control. For example, the CPU 11 executes processes based on programs in the OS (Operating System) and BIOS.

[0015] Main memory 12 is writable memory used as a reading area for the CPU 11's executable program, or as a work area for writing processing data for the executable program. Main memory 12 is composed of, for example, multiple DRAM (Dynamic Random Access Memory) chips. This executable program includes the OS, various drivers for hardware operation of peripheral devices, various services / utilities, application programs, etc.

[0016] The video subsystem 13 is a subsystem for implementing functions related to image display and includes a video controller. The video controller processes drawing commands from the CPU 11, writes the processed drawing information to video memory, reads this drawing information from video memory, and outputs it to the display unit 14 as drawing data (display data).

[0017] The display unit 14 is, for example, a liquid crystal display or an organic EL display, and displays a display screen based on the drawing data (display data) output from the video subsystem 13.

[0018] The chipset 21 includes controllers such as USB (Universal Serial Bus), Serial ATA (AT Attachment), SPI (Serial Peripheral Interface) bus, PCI (Peripheral Component Interconnect) bus, PCI-Express bus, and LPC (Low Pin Count) bus, and a plurality of devices are connected. For example, as the plurality of devices, a BIOS memory 22, a storage medium 23, an audio system 24, a WLAN card 25, a USB connector 26, and an embedded controller 31, which will be described later, are included.

[0019] The BIOS memory 22 is composed of an electrically rewritable non-volatile memory such as EEPROM (Electrically Erasable Programmable Read Only Memory) or flash ROM. The BIOS memory 22 stores the BIOS and system firmware for controlling the embedded controller 31 and the like.

[0020] The storage medium 23 includes an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. For example, the storage medium 23 stores an OS, various drivers, various services / utilities, application programs, and various data.

[0021] The audio system 24 has a microphone and a speaker (not shown) connected thereto, and records, plays back, and outputs audio data. As an example, the microphone and the speaker are built in the information processing apparatus 10.

[0022] The WLAN (Wireless Local Area Network) card 25 connects to the network via wireless LAN and performs data communication. For example, when the WLAN card 25 receives data from the network, it generates an event trigger indicating that data has been received. USB connector 26 is a connector for connecting peripheral devices that use USB.

[0023] The input unit 32 collectively represents the input devices provided in the information processing device 10.

[0024] The power supply circuit 33 includes, for example, a DC / DC converter, a charge / discharge unit, and an AC / DC adapter. For example, the power supply circuit 33 converts a DC voltage supplied from an external power source such as an AC adapter (not shown) or a battery 34 into multiple voltages necessary to operate the information processing device 10. The power supply circuit 33 also supplies power to each part of the information processing device 10 based on control from the embedded controller 31.

[0025] The battery 34 is a secondary battery, such as a lithium-ion battery. When the information processing device 10 is powered by an external power source, the battery 34 is charged via the power supply circuit 33, and when the information processing device 10 is not powered by an external power source, the battery 34 outputs the stored power as the operating power of the information processing device 10 via the power supply circuit 33.

[0026] The embedded controller 31 is a one-chip microcomputer that monitors and controls various devices (peripheral devices, sensors, etc.) regardless of the system state of the information processing device 10. The embedded controller 31 includes a CPU (not shown), ROM, RAM, multiple A / D input terminals, D / A output terminals, a timer, and digital input / output terminals. For example, an input unit 32 and a power supply circuit 33 are connected to the digital input / output terminals of the embedded controller 31, and the embedded controller 31 is capable of controlling the operation of these. Furthermore, the embedded controller 31 in the same figure is capable of controlling the CPU 11, such as changing its clock frequency, via the chipset 21.

[0027] Figure 3 illustrates an example of the functional configuration of the information processing device 10. This figure is a block diagram showing an example of the hardware configuration of the information processing device 10. The information processing device 10 in this figure comprises a communication unit 101, an operation unit 102, a display unit 103, a control unit 104, and a storage unit 105.

[0028] The communication unit 101 is connected to other information processing devices 10 via a network so as to be able to communicate with them.

[0029] The operation unit 102 includes operators provided by the information processing device 10 and input devices connected to the information processing device 10, and accepts operations performed by the user. The input unit 32 in Figure 2 may be included in the operation unit 102.

[0030] The display unit 103 corresponds to the display unit 14 in Figure 2 and displays an image.

[0031] The control unit 104 performs various controls in the information processing device 10. The control unit 104 includes a question receiving unit 141, an answer request unit 142, an answer processing unit 143, a first selection unit 144, a second selection unit 145, and a third selection unit 146.

[0032] The question reception unit 141 receives questions (prompts) entered by the user, for example, via the operation unit 102.

[0033] The answer request unit 142 requests an answer to the question received by the question reception unit 141 from a predetermined number of primary candidate information processing devices 10. Specifically, the answer request unit 142 sends an answer request, including the question received by the question reception unit 141, to the primary candidate information processing devices 10. The primary candidate information processing devices 10 are information processing devices 10 selected to output candidate answers that will be adopted as answers to be returned to the user in response to the question received by the question reception unit 141. In the following explanation, we will take the example where the primary candidate information processing devices 10 are all of the four information processing devices 10 (10-1, 10-2, 10-3, 10-4) that constitute the natural language processing system in Figure 1.

[0034] The answer processing unit 143 generates an answer to the question received by the question receiving unit 141. The answer processing unit 143 includes an LLM. The LLM generates an answer to the input question. In this embodiment, the LLM refers to reference data retrieved from the reference data group stored in the reference data storage unit 151 based on the question content, and generates an answer that reflects the referenced content.

[0035] The first selection unit 144 selects a secondary candidate information processing device from among several primary candidate information processing devices. The reference data stored in the reference data storage unit 151 of the information processing device 10 may correspond to common items, but for example, the method of description or the structure of the document may differ. Therefore, the reference data that the LLM refers to in response to a common question will differ for each information processing device 10. In this case, the degree of similarity between the question and the reference data that the LLM refers to when generating the answer will also differ for each information processing device 10. In this case, the LLM of the information processing device 10, where the similarity between the question and the retrieved reference data is similar, tends to generate an answer with similar content. In cases where there are multiple information processing devices 10 that could potentially produce similar answers, it is not necessary to have all of these multiple information processing devices 10 generate an answer. Instead, unnecessary processing can be avoided by having, for example, one representative information processing device 10 generate the answer. Therefore, if there are two or more primary candidate information processing devices 10 among the multiple primary candidate information processing devices 10 whose similarity (reference data similarity: an example of primary similarity) between the question received by the question receiving unit 141 and the retrieved reference data is approximated within a certain range, the first selection unit 144 excludes from the candidates the primary candidate information processing devices 10 other than the one selected as representative based on the scale of the LLM. The first selection unit 144 selects the primary candidate information processing devices 10 that remain without being excluded as secondary candidate information processing devices. The secondary candidate information processing devices selected in this way are the information processing device 10 selected as representative among the multiple primary candidate information processing devices 10 whose reference data similarity is approximated, and the primary candidate information processing device whose reference data similarity is not approximated by the other primary candidate information processing devices 10.

[0036] The second selection unit 145 selects a tertiary candidate information processing device 10 from among the secondary candidate information processing devices 10 selected by the first selection unit 144. The third-order candidate information processing device 10 becomes the second-order candidate information processing device 10 that generated the response belonging to the group with the most responses when the generated responses are classified into groups based on similarity. The more similar responses generated by multiple information processing devices 10 are, the higher the probability that those similar responses are correct. Therefore, the second selection unit 145 selects the information processing device 10 that generated the response with the most similar responses as the second candidate information processing device 10.

[0037] The third selection unit 146 selects the final answer to the question received by the question receiving unit 141 from among the answers generated by each of the third candidate information processing devices 10. The third selection unit 146 calculates an evaluation value (answer evaluation value) for each answer according to the scale of the LLL possessed by the third candidate information processing devices 10, and selects the answer as the final answer based on the calculated answer evaluation value.

[0038] The storage unit 105 stores various types of information in the information processing device 10. The storage unit 105 includes a reference data storage unit 151. The reference data storage unit 151 stores reference data. As mentioned above, even if the reference data stored in the reference data storage unit 151 corresponds to common items, the method of description and the structure of the data may differ for each information processing device 10. The reference data may be stored, for example, on a network or in the cloud, and the information processing device 10 may retrieve it via the network.

[0039] Referring to the sequence diagram in Figure 4, an example of a processing procedure executed by the first information processing device 10-1, the second information processing device 10-2, the third information processing device 10-3, and the fourth information processing device 10-4 in the natural language processing system of this embodiment will be described. In explaining the figure, we will use as an example the case where, at the stage when processing begins, the first information processing device 10-1, the second information processing device 10-2, the third information processing device 10-3, and the fourth information processing device 10-4 are each designated as primary candidate information processing devices. Furthermore, in explaining the figure, we will use as an example the case where, among the four primary candidate information processing devices 10 described above, the first information processing device 10-1 acts as the master, handling input of user questions and output of answers to the user, while the second information processing device 10-2, the third information processing device 10-3, and the fourth information processing device 10-4 act as slaves, executing processing in response to requests from the first information processing device 10-1.

[0040] Step S100: The user inputs a question by operating the operation unit 102 of the first information processing device 10-1. The question receiving unit 141 in the first information processing device 10-1 receives the question input by the user.

[0041] Step S102: The response request unit 142 of the first information processing device 10-1 sends a response request to the second information processing device 10-2. The response request includes the question received by the question receiving unit 141 in step S100.

[0042] Step S104: The response request unit 142 of the first information processing device 10-1 also sends a response request to the third information processing device 10-3.

[0043] Step S106: The response request unit 142 of the first information processing device 10-1 also sends a response request to the fourth information processing device 10-4.

[0044] Step S108: The answer processing unit 143 of the first information processing device 10-1 retrieves reference data from the reference data storage unit 151 of the first information processing device 10-1 that corresponds to the content of the question received in step S100. The answer processing unit 143 of the first information processing device 10-1 calculates the similarity (reference data similarity) between the content of the question and the retrieved reference data. The similarity calculated in this way may be, for example, cosine similarity.

[0045] Step S110: The answer processing unit 143 of the first information processing device 10-1 generates an answer to the question received in step S100. In this case, the answer processing unit 143 of the first information processing device 10-1 may generate the answer by referring to the reference data retrieved in step S108.

[0046] Step S112: The answer processing unit 143 of the second information processing device 10-2 calculates the similarity of reference data in response to the answer request transmitted in step S102. At this time, the answer processing unit 143 of the second information processing device 10-2 searches the reference data storage unit 151 of the second information processing device 10-2 for reference data corresponding to the content of the question included in the answer request transmitted in step S102. The answer processing unit 143 of the second information processing device 10-2 calculates the similarity of reference data between the content of the question and the retrieved reference data.

[0047] Step S114: The response processing unit 143 of the second information processing device 10-2 generates an answer to the question included in the response request sent in step S102. In this case, the response processing unit 143 of the second information processing device 10-2 may generate the answer by referring to the reference data retrieved in step S112.

[0048] Step S116: The response processing unit 143 of the second information processing device 10-2 transmits the response processing result for the response request sent in step S102 to the first information processing device 10-1. The response processing result includes the reference data similarity calculated in step S112, the response generated in step S114, and information indicating the scale of the LLM possessed by the second information processing device 10-2 (LLM scale information). In this case, the LLM scale information may include, for example, the number of parameters of the LLM possessed by the second information processing device 10-2 and the capacity of the reference data storage unit 151 in the second information processing device 10-2 (reference data capacity).

[0049] Step S118: The answer processing unit 143 of the third information processing device 10-3 calculates the similarity of reference data in response to the answer request transmitted in step S104. At this time, the answer processing unit 143 of the third information processing device 10-3 searches the reference data storage unit 151 of the third information processing device 10-3 for reference data corresponding to the content of the question included in the answer request transmitted in step S104. The answer processing unit 143 of the third information processing device 10-3 calculates the similarity of reference data between the content of the question and the retrieved reference data.

[0050] Step S120: The response processing unit 143 of the third information processing device 10-3 generates an answer to the question included in the response request sent in step S104. In this case, the response processing unit 143 of the third information processing device 10-3 may generate the answer by referring to the reference data retrieved in step S118.

[0051] Step S122: The response processing unit 143 of the third information processing device 10-3 transmits the response processing result for the response request sent in step S104 to the first information processing device 10-1. The response processing result includes the reference data similarity calculated in step S118, the response generated in step S120, and information indicating the scale of the LLM held by the third information processing device 10-3 (LLM scale information). In this case, the LLM scale information may include, for example, the number of parameters of the LLM held by the third information processing device 10-3 and the reference data capacity of the reference data storage unit 151 in the third information processing device 10-3.

[0052] Step S124: The answer processing unit 143 of the fourth information processing device 10-4 calculates the similarity of reference data in response to the answer request transmitted in step S106. At this time, the answer processing unit 143 of the fourth information processing device 10-4 searches for reference data from the reference data storage unit 151 of the fourth information processing device 10-4 that corresponds to the content of the question included in the answer request transmitted in step S106. The answer processing unit 143 of the fourth information processing device 10-4 calculates the similarity of reference data between the content of the question and the retrieved reference data.

[0053] Step S126: The response processing unit 143 of the fourth information processing device 10-4 generates an answer to the question included in the response request sent in step S106. In this case, the response processing unit 143 of the fourth information processing device 10-4 may generate the answer by referring to the reference data retrieved in step S124.

[0054] Step S128: The response processing unit 143 of the fourth information processing device 10-4 transmits the response processing result for the response request sent in step S106 to the first information processing device 10-1. The response processing result includes the reference data similarity calculated in step S124, the response generated in step S126, and information indicating the scale of the LLM held by the fourth information processing device 10-4 (LLM scale information). In this case, the LLM scale information may include, for example, the number of parameters of the LLM held by the fourth information processing device 10-4 and the reference data capacity of the reference data storage unit 151 in the fourth information processing device 10-4.

[0055] Step S130: The first selection unit 144 of the first information processing device 10-1 selects a secondary candidate information processing device from among the four primary candidates, the first information processing device 10-1, the second information processing device 10-2, the third information processing device 10-3, and the fourth information processing device 10-4.

[0056] Referring to Figure 5, a specific example of the selection of a secondary candidate information processing device in step S130 will be explained. The first selection unit 144 of the first information processing device 10-1 first acquires the reference data similarity scores for the first information processing device 10-1, the second information processing device 10-2, the third information processing device 10-3, and the fourth information processing device 10-4. The first selection unit 144 of the first information processing device 10-1 identifies, from among the four acquired reference data similarity scores, the reference data similarity scores that do not approximate each other and the reference data similarity scores that approximate each other.

[0057] Figure 5 shows the reference data similarity values ​​obtained for the first information processing device 10-1, the second information processing device 10-2, the third information processing device 10-3, and the fourth information processing device 10-4, respectively. In this figure, the reference data similarity of the first information processing device 10-1 is identified as not being similar to any of the others. Similarly, the reference data similarity of the second information processing device 10-2 is identified as not being similar to any of the others. Furthermore, the reference data similarity of the third information processing device 10-3 and the response similarity of the fourth information processing device 10-4 are identified as being similar to each other.

[0058] In this case, first, the first information processing device 10-1 and the second information processing device 10-2, which have reference data similarity that does not approximate any of the others, are selected as secondary candidate information processing devices. On the other hand, for the third information processing device 10-3 and the fourth information processing device 10-4, whose reference data similarity is similar to each other, one will be selected as a representative. For this purpose, the first selection unit 144 of the first information processing device 10-1 selects the information processing device 10 with the larger LLM among the third information processing device 10-3 and the fourth information processing device 10-4 as the secondary candidate information processing device. In this case, the first selection unit 144 of the first information processing device 10-1 obtains the LLM size information included in the response processing result transmitted from the third information processing device 10-3 in step S122 and the LLM size information included in the response processing result transmitted from the fourth information processing device 10-4 in step S128. The LLM size information is information indicating the size of the LLM and may include the number of parameters and the reference data capacity.

[0059] Specifically, Figure 5 shows the number of parameters and reference data capacity included in the LLM scale information of the third information processing device 10-3 and the fourth information processing device 10-4, respectively. The first selection unit 144 of the first information processing device 10-1 calculates an evaluation value (LLM scale evaluation value) indicating the scale of the LLM of the third information processing device 10-3 and the fourth information processing device 10-4, respectively. Specifically, the first selection unit 144 may calculate the LLM scale evaluation value by multiplying the number of parameters by the reference data capacity. In Figure 5, it is shown that the LLM scale evaluation value of the third information processing device 10-3 is 35.0 and the LLM scale evaluation value of the fourth information processing device 10-4 is 5.0. The LLM size evaluation value may be determined based on either the number of parameters or the reference data capacity, or it may be determined based on predetermined information regarding the size of the LLM other than the number of parameters and the reference data capacity.

[0060] The first selection unit 144 of the first information processing device 10-1 selects the information processing device 10 with the largest LLM scale evaluation value among the third information processing device 10-3 and the fourth information processing device 10-4 as the secondary candidate, and excludes the information processing device 10 that is not selected from the secondary candidates. In this case, since the LLM scale evaluation value of the third information processing device 10-3 is larger than that of the fourth information processing device 10-4, the third information processing device 10-3 is selected as the secondary candidate, and the fourth information processing device 10-4 is excluded from the secondary candidates. In this way, in the example shown in Figure 5, in step S130, the first information processing device 10-1, the second information processing device 10-2, and the third information processing device 10-3 are selected as secondary candidate information processing devices 10.

[0061] Step S132: Next, the second selection unit 145 of the first information processing device 10-1 selects a tertiary candidate information processing device 10 from among the secondary candidate information processing devices 10 selected in step S130. The tertiary candidate information processing device 10 corresponds to the secondary candidate information processing device 10 that generated the response belonging to the group with the most responses when the generated responses are grouped based on similarity, as described above.

[0062] Referring to Figure 6, a specific example of the selection of the third-order candidate information processing device 10 in step S132 will be explained. Figure 6 shows examples of the correctness of the answers when the first information processing device 10-1, the second information processing device 10-2, and the third information processing device 10-3 are selected as the second-order candidate information processing device 10, as illustrated in Figure 5, and each of the first information processing device 10-1, the second information processing device 10-2, and the third information processing device 10-3 is asked to answer five questions, from question 1 to question 5. For the explanation of this figure, the correct content for each of the answers to questions 1 to question 5 is known.

[0063] The answers to Question 1 were correct for the first information processing device 10-1, correct for the second information processing device 10-2, and incorrect for the third information processing device 10-3. In this case, the second selection unit 145 classifies the answers from the first information processing device 10-1, the second information processing device 10-2, and the third information processing device 10-3 into two groups: a group with two correct answers and a group with one incorrect answer. The second selection unit 145 selects the first information processing device 10-1 and the second information processing device 10-2, which generated answers belonging to the group with the most correct answers, as the third candidate information processing device 10.

[0064] Furthermore, the answers to question 2 were incorrect for the first information processing device 10-1, correct for the second information processing device 10-2, and correct for the third information processing device 10-3. In this case, the second selection unit 145 classifies the answers from the first information processing device 10-1, the second information processing device 10-2, and the third information processing device 10-3 into two groups: a group with one incorrect answer and a group with two correct answers. The second selection unit 145 selects the second information processing device 10-2 and the third information processing device 10-3, which generated answers belonging to the group with the most correct answers, as the third candidate information processing device 10.

[0065] The answers to question 3 were correct for the first information processing device 10-1, correct for the second information processing device 10-2, and incorrect for the third information processing device 10-3. In this case, the second selection unit 145 classifies the answers from the first information processing device 10-1, the second information processing device 10-2, and the third information processing device 10-3 into two groups: a group with two correct answers and a group with one incorrect answer. The second selection unit 145 selects the first information processing device 10-1 and the second information processing device 10-2, which generated answers belonging to the group with the most correct answers, as the third candidate information processing device 10.

[0066] Furthermore, the answers to question 4 were incorrect for the first information processing device 10-1, incorrect for the second information processing device 10-2, and correct for the third information processing device 10-3. In this case, the second selection unit 145 classifies the answers from the first information processing device 10-1, the second information processing device 10-2, and the third information processing device 10-3 into two groups: a group with two incorrect answers and a group with one correct answer. The second selection unit 145 selects the first information processing device 10-1 and the second information processing device 10-2, which generated answers belonging to the group with the most incorrect answers, as the third candidate information processing device 10.

[0067] Furthermore, the answers to question 5 were correct for the first information processing device 10-1, incorrect for the second information processing device 10-2, and correct for the third information processing device 10-3. In this case, the second selection unit 145 classifies the answers from the first information processing device 10-1, the second information processing device 10-2, and the third information processing device 10-3 into two groups: a group with two correct answers and a group with one incorrect answer. The second selection unit 145 selects the first information processing device 10-1 and the third information processing device 10-3, which generated answers belonging to the group with the most correct answers, as the third candidate information processing device 10.

[0068] In the example shown in Figure 6, the user ultimately receives the correct answers to questions 1 through 3 and question 5, but the user ultimately receives the incorrect answer to question 4. In this case, the accuracy rate when five questions are input to the natural language processing system of this embodiment is 80% (=4 / 5). In contrast, the accuracy rates for questions 1 through 5 of the first information processing device 10-1, the second information processing device 10-2, and the third information processing device 10-3 are 60% (=3 / 5), respectively. In other words, the example in Figure 6 shows that obtaining the final answer based on a majority vote of responses from multiple information processing devices 10 results in a higher accuracy rate than obtaining the answer based on the response from a single information processing device 10.

[0069] Step S134: The third selection unit 146 of the first information processing device 10-1 selects the final answer from among the answers generated by the third candidate information processing device 10 selected in step S132.

[0070] Referring to Figure 7, a specific example of the selection of the final answer in step S134 will be explained. Figure 7 shows the reference data similarity, number of parameters, reference data capacity, answer, answer similarity (an example of second similarity), and answer evaluation value for each of the first information processing device 10-1, second information processing device 10-2, and third information processing device 10-3 selected as secondary candidates in step S130. The reference data similarity, number of parameters, and reference data capacity are the same as in Figure 5.

[0071] The response items in Figure 7 show the response content generated by the first information processing device 10-1, the second information processing device 10-2, and the third information processing device 10-3 in steps S110, S114, and S120, respectively. The response content of the first information processing device 10-1 is "Response D", the response content of the second information processing device 10-2 is "Response B1", and the response content of the third information processing device 10-3 is "Response B2". This indicates that the response content of the second information processing device 10-2 and the third information processing device 10-3 are similar, while "Response D" of the first information processing device 10-1 is not similar to either of the other responses from the second information processing device 10-2 and the third information processing device 10-3 ("Response B1" and "Response B2"). In this case, according to step S132 above, the second information processing device 10-2 and the third information processing device 10-3 are selected as the third candidate information processing device 10, and the first information processing device 10-1 is excluded from the third candidate information processing device 10.

[0072] Furthermore, the third selection unit 146 of the first information processing device 10-1 uses the answer of the first information processing device 10-1, which is the master, as a reference to determine the similarity of the answers of each of the other third-order candidate information processing devices 10 to the answer of the said reference. The method of determining the answer similarity is not particularly limited. Figure 7 shows an example in which the answer similarity is determined as cosine similarity.

[0073] The third selection unit 146 of the first information processing device 10-1 calculates a response evaluation value that evaluates the reliability of the responses from the second information processing device 10-2 and the third information processing device 10-3, which are third-order candidate information processing devices 10. The third selection unit 146 of the first information processing device 10-1 may calculate a response evaluation value based, for example, on the LLM size and response similarity of the third candidate information processing device 10. Specifically, the third selection unit 146 of the first information processing device 10-1 may use as the response evaluation value a value obtained by multiplying the product of the number of parameters and the reference data capacity by the absolute value of the response similarity for each third candidate information processing device 10. Figure 7 shows an example in which the response evaluation value of the second information processing device 10-2 is calculated to be 9.0 (=1.0 × 9 × |-1|) and the response evaluation value of the third information processing device 10-3 is calculated to be 35.0 (=7.0 × 5 × |-1|). The response evaluation value calculated in this way reflects the reliability of the response based on the LLM size of the third candidate information processing device 10 and the reliability of the response based on the degree of deviation of the content from the standard response. The third selection unit 146 selects the answer with the highest calculated response evaluation value as the final answer. In the case of Figure 7, the third selection unit 146 selects "Answer B2" generated by the third information processing device 10-3 as the final answer.

[0074] Step S136: The answer processing unit 143 of the first information processing device 10-1 outputs the final answer selected in step S134 to the user. Specifically, the answer processing unit 143 may output the answer by displaying it as text on the display unit 103, for example. In this case, output by reading the answer aloud or by printing may also be possible.

[0075] According to the natural language processing system of this embodiment with the above configuration, even if the individual information processing devices 10 have specifications below a certain level, such as those of a typical personal computer, it is possible to obtain highly reliable answers.

[0076] In the natural language processing system of this embodiment, the master information processing device 10 and the information processing device 10 into which the user inputs a question may be different.

[0077] In the natural language processing system of this embodiment, the functions of the question receiving unit 141, the answer requesting unit 142, the answer processing unit 143, the first selection unit 144, the second selection unit 145, and the third selection unit 146 may be appropriately assigned to each of the predetermined information processing devices 10, and the final answer to a question input to one of the information processing devices 10 may be obtained by communicating with each other as appropriate.

[0078] The third selection unit 146 may calculate the response evaluation value by performing calculations using, for example, a portion of the number of parameters, the reference data capacity, and the response similarity.

[0079] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configurations are not limited to the embodiments described above, and include designs and the like that do not depart from the spirit of this invention. The configurations described in the above embodiments can be combined arbitrarily as long as they do not contradict each other. [Explanation of symbols]

[0080] 10 Information Processing Devices (10-1 First Information Processing Device, 10-2 Second Information Processing Device, 10-3 Third Information Processing Device, 10-4 Fourth Information Processing Device), 101 Communication Unit, 102 Operation Unit, 103 Display Unit, 104 Control Unit, 105 Storage Unit, 141 Question Reception Unit, 142 Answer Request Unit, 143 Answer Processing Unit, 144 First Selection Unit, 145 Second Selection Unit, 146 Third Selection Unit, 151 Reference Data Storage Unit

Claims

1. A question reception section that receives submitted questions, The question receiving unit inputs the question received by the question receiving unit into a plurality of primary candidate information processing devices to request an answer to the question, Each of the aforementioned multiple primary candidate information processing devices includes an answer generation unit that generates an answer to an input question using LLM (Large Language Models), A first selection unit calculates a first similarity between the input question and the reference data that the LLM refers to when generating the answer for each of the multiple primary candidate information processing devices, and selects a primary candidate information processing device that does not approximate any of the other primary candidate information processing devices as the secondary candidate information processing device. A second selection unit classifies the responses generated by each of the secondary candidate information processing devices selected by the first selection unit into multiple groups based on the similarity of their content, and selects each of the secondary candidate information processing devices that generated the responses belonging to the group with the most responses as a tertiary candidate information processing device. A third selection unit calculates an answer evaluation value indicating the reliability of an answer from among the answers generated by the information processing device of the tertiary candidates selected by the second selection unit, and selects the final answer to the question received by the question receiving unit based on the calculated answer evaluation value. A natural language processing system equipped with the following features.

2. The first selection unit selects as the secondary candidate information processing device a primary candidate information processing device whose first similarity is not similar to others, and a primary candidate information processing device selected from among the primary candidate information processing devices whose first similarity is similar to each other, based on the scale of the LLM. The natural language processing system according to claim 1.

3. The third selection unit calculates the response evaluation value based on the scale of the LLM for each of the third candidate information processing devices. The natural language processing system according to claim 1 or 2.

4. The size of the LLM is determined by at least one of the following: the number of parameters in the LLM and the volume of the reference data. The natural language processing system according to claim 3.

5. The third selection unit calculates the response evaluation value based on the similarity between each of the responses generated by the third candidate information processing device and a standard response defined as a criterion among the responses generated by the plurality of primary candidate information processing devices. The natural language processing system according to claim 1 or 2.

6. A natural language processing method in a natural language processing system, The question reception department receives the submitted questions in a question reception step, The response request unit requests an answer to the question by inputting the question received by the question reception step into a plurality of primary candidate information processing devices, The answer generation unit performs an answer generation step in which it generates an answer to the input question using LLM (Large Language Models) in each of the plurality of primary candidate information processing devices, The first selection unit calculates a first similarity between the input question and the reference data that the LLM refers to when generating the answer for each of the plurality of primary candidate information processing devices, and selects a primary candidate information processing device that does not approximate any of the other primary candidate information processing devices as the secondary candidate information processing device. The second selection unit classifies the responses generated by each of the secondary candidate information processing devices selected in the first selection step into multiple groups based on the similarity of their content, and selects each of the secondary candidate information processing devices that generated the responses belonging to the group with the most responses as a tertiary candidate information processing device. The third selection unit calculates an answer evaluation value indicating the reliability of an answer from among the answers generated by the information processing device of the tertiary candidates selected in the second selection step, and selects the final answer to the question received by the question receiving step based on the calculated answer evaluation value. A natural language processing method that includes [this].

Images