Information processing programs, information processing systems, and information processing methods

Abstract

The goal is to efficiently switch between LLM processing using the first processor and LLM processing using the second processor. [Solution] If the frequency of occurrence of a term in the question is equal to or greater than a first threshold, that term is determined to be a technical term. If the number of technical terms in the question is equal to or greater than a second threshold, that question is determined to be a technical question. If the question is a technical question, the second LLM2b is made to generate an answer to the question.

Description

Technical Field

[0001] The present invention relates to an information processing program, an information processing system, and an information processing method.

Background Art

[0002] In recent years, large language models (LLMs) have been put into practical use. LLMs can handle various natural language processes such as conversations that are close to natural human interactions without a sense of strangeness, text generation, text summarization, and QA (Question-Answering).

[0003] In LLMs, compared with conventional natural language models, the amount of work, data volume, and number of parameters processed by the computer have increased.

[0004] Also, it is known that by using a GPU (Graphics Processing Unit) instead of a CPU (Central Processing Unit) for the inference processing of LLMs, the processing performance can be improved. For example, a user may select either a CPU or a GPU provided by a cloud provider as computing resources and use it for the inference processing of LLMs.

[0005] Conventionally, a user selects whether to use a CPU or a GPU for inference processing using a language model in consideration of performance and cost in advance. For example, when using simple questions or small amounts of data processing, a high-performance GPU is not necessary, and a low-cost CPU is selected. On the other hand, when a user performs a large amount of data analysis or complex question processing, a GPU is selected. In this case, compared with the case of using a CPU, costs such as cloud costs and power consumption increase.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] Traditionally, it has been difficult to efficiently utilize both CPUs and GPUs, and using GPUs for inference processing with LLMs has resulted in increased costs.

[0008] In one aspect, the present invention aims to enable efficient differentiation between LLM processing using a first processor and LLM processing using a second processor. [Means for solving the problem]

[0009] This information processing program causes a computer connected to a first processor that runs a first LLM (Large Language Model) and a second processor that runs a second LLM with a larger model size than the first LLM and has higher processing performance than the first processor to execute a process that determines a term is a technical term if its frequency of appearance in the question sentence is equal to or greater than a first threshold, determines the question sentence is a technical question sentence if the number of technical terms in the question sentence is equal to or greater than a second threshold, and if the question sentence is a technical question sentence, causes the second LLM to generate an answer to the question sentence. [Effects of the Invention]

[0010] According to one embodiment, it is possible to efficiently switch between LLM processing using the first processor and LLM processing using the second processor. [Brief explanation of the drawing]

[0011] [Figure 1] This diagram illustrates the configuration of an information processing system according to one embodiment. [Figure 2]This figure illustrates a term frequency table in an information processing system according to one embodiment. [Figure 3] This figure illustrates the transition of a term frequency table in an information processing system according to one embodiment. [Figure 4] This figure illustrates the hardware configuration of an information processing system according to one embodiment. [Figure 5] This is a diagram illustrating a computer's hardware configuration. [Figure 6] This is a flowchart illustrating the processing in an information processing system according to one embodiment. [Figure 7] This is a flowchart illustrating the details of the processing performed by the expertise determination unit in an information processing system according to one embodiment. [Modes for carrying out the invention]

[0012] The embodiments of this information processing program, information processing system, and information processing method will be described below with reference to the drawings. However, the embodiments shown below are merely illustrative, and there is no intention to exclude various modifications or applications of technologies not explicitly shown in the embodiments. In other words, these embodiments can be implemented with various modifications without departing from their spirit. Furthermore, each figure is not intended to represent only the components shown in the figure, but may include other functions, etc.

[0013] (A) Configuration (A-1) Example of Functional Configuration Figure 1 illustrates the configuration of an information processing system 1 according to one embodiment.

[0014] The information processing system 1 illustrated in Figure 1 has the functions of a first LLM2a, a second LLM2b, an intermediary processing unit 11, a data conversion processing unit 12, a similarity search execution unit 13, a specialization determination unit 14, and a usage determination unit 15.

[0015] The information processing system 1 generates an answer to a question sentence input by a user using the first LLM 2a or the second LLM 2b and responds to the user.

[0016] The first LLM 2a is executed on one or more CPUs, and the second LLM 2b is executed on one or more GPUs. The second LLM 2b has a larger model size than the first LLM 2a. The model size may be, for example, the data size of the model. Also, the model size may be the number of layers or the number of parameters that make up the model.

[0017] For example, the LLM with a model size less than a predetermined threshold may be the first LLM 2a, and the LLM with a model size greater than or equal to the predetermined threshold may be the second LLM 2b.

[0018] Regarding the model size of the LLM, for example, it can be said that the model size of the LLM that responds to a highly specialized question sentence (specialized question sentence) is larger than the model size of the LLM that responds to a general question sentence (general question sentence).

[0019] Here, the specialized question sentence may be, for example, a question sentence that includes terms used in a relatively small community (specific community) such as a research field or a society. Terms used in a small community may be referred to as specialized terms. The specialized question sentence is used less frequently in daily life. Also, questions other than the specialized question sentence may be referred to as general question sentences.

[0020] The LLM that responds to a general question sentence may be referred to as a general-purpose LLM. Also, the LLM that responds to a specialized question sentence may be referred to as a specialized LLM. The knowledge possessed by the general-purpose LLM is general knowledge, and the knowledge possessed by the specialized LLM is assumed to include general knowledge and specialized knowledge (general knowledge + specialized knowledge).

[0021] In this information processing system 1, the first LLM 2a is assumed to be an LLM (general-purpose LLM) that responds to a general question sentence, and the second LLM 2b is assumed to be an LLM (specialized LLM) that responds to a specialized question sentence.

[0022] LLM12a is an example of the first LLM. LLM2b is an example of the second LLM, with a larger model size than LLM12a (the first LLM).

[0023] Furthermore, the CPU running the first LLM2a and the GPU running the second LLM2b may be provided, for example, by a cloud provider. Also, in this information processing system 1, the first LLM2a and the second LLM2b may be provided as SaaS (Software as a Service). The first LLM2a has a lower cost to use (usage cost) than the second LLM2b, and the second LLM2b has a higher cost to use than the first LLM2a. In addition, these first LLM2a and second LLM2b may be used on a per-question-sentence basis.

[0024] The intermediary processing unit 11 mediates the exchange of data, etc., between the data conversion processing unit 12, the similarity search execution unit 13, the expertise determination unit 14, and the usage determination unit 15.

[0025] The intermediary processing unit 11 receives the question text created by the user. For example, the user may input the question text via a terminal device (not shown), and this input question text may be transferred to the intermediary processing unit 11.

[0026] The question created by the user can be called the user question. The intermediary processing unit 11 receives this user question and transmits (transfers) it to the data conversion processing unit 12. The intermediary processing unit 11 also receives the results of the similarity search (knowledge data) performed by the similarity search execution unit 13 (described later) and transmits them to the usage distinction determination unit 15 (described later). Furthermore, the intermediary processing unit 11 receives the determination results from the expertise determination unit 14 (described later) and transmits them to the usage distinction determination unit 15.

[0027] Furthermore, the mediation processing unit 11 receives the response generated by the first LLM2a or the second LLM2b and presents it to the user.

[0028] The intermediary processing unit 11 may have functions similar to, for example, LangChain in Python (registered trademark).

[0029] The data transformation processing unit 12 vectorizes the user's question text. The vectorized question text can be called a vectorized question text. The functions of the data transformation processing unit 12 may be realized, for example, by using a known embedding model. The vectorized question text generated by the data transformation processing unit 12 is input to the similarity search execution unit 13 and the expertise determination unit 14, respectively.

[0030] The similarity search execution unit 13 uses the vectorized query as a search keyword and performs a similarity search on the information stored in the vector DB 131. The vector DB 131 stores various types of information (for example, public documents and internal company documents). Similarity search is a known search method that can search even if the search keyword is not an exact match, by absorbing variations in spelling and synonyms, and by flexibly interpreting similar sentences. Similarity search can be called fuzzy search.

[0031] The similarity search execution unit 13 outputs multiple knowledge data with the highest similarity scores as search results, for example. For example, the similarity search execution unit 13 may output the top four knowledge data with the highest similarity scores as search results based on the similarity search results. The similarity search execution unit 13 transmits the similarity search results to the intermediary processing unit 11.

[0032] The expertise determination unit 14 determines whether the user question is a general question or a specialized question. This determination of whether a user question is a general question or a specialized question can be called expertise determination of the user question.

[0033] The expertise determination unit 14 receives the vectorized question text as input. The expertise determination unit 14 determines the expertise level of the user question text based on the input vectorized question text.

[0034] The expertise determination unit 14 may, for example, determine that a user question is an expert question if it contains more than a predetermined threshold (second threshold: details described later) of expert terms, and determine that a user question is a general question if the number of expert terms is less than the second threshold.

[0035] Therefore, the expertise determination unit 14 first determines whether a term of a specific part of speech (a noun in this embodiment) included in the user question is a technical term or a general term. Terms other than technical terms may be called general terms. Determining whether a term included in the user question is a technical term may be called term expertise determination. The expertise determination unit 14 performs term expertise determination in order to determine the expertise of the user question.

[0036] The expertise determination unit 14 may, for example, determine the expertise of a term based on its frequency of occurrence, and may determine terms with a low frequency of occurrence as specialized terms.

[0037] Therefore, the expertise determination unit 14 may manage the frequency of occurrence of words included in the user's question using a term frequency table 141.

[0038] Figure 2 is a diagram illustrating a term frequency table 141 in an information processing system 1 according to one embodiment.

[0039] In the term frequency table 141 illustrated in Figure 2, nouns are associated with their frequency of appearance and category. Nouns are extracted from user questions. The frequency of appearance is the number of times that noun has appeared in user questions in the past. The category indicates whether the noun is a technical term or a general term.

[0040] The expertise determination unit 14 performs morphological analysis (part-of-speech analysis) of the user question text based on the vectorized question text, and extracts nouns from the analysis results.

[0041] The expertise determination unit 14, if the extracted noun is not registered in the term frequency table 141, registers the noun as a new entry (row) in the term frequency table 141 and then increments its count (+1). If the extracted noun is already registered in the term frequency table 141, it increments its count (+1). The count value represents the frequency of occurrence of that term.

[0042] The specialization determination unit 14 determines the category by comparing the number of times each term is invoked in the term frequency table 141 with a first threshold. In other words, the specialization determination unit 14 determines whether a noun registered in the term frequency table 141 is a technical term or a general term by comparing the number of invoked terms with a first threshold.

[0043] The first threshold may be, for example, the average number of times a noun is invoked for multiple (e.g., all) nouns registered in the term frequency table 141. However, the first threshold is not limited to the average number of times a noun is invoked for multiple (e.g., all) nouns registered in the term frequency table 141. For example, the first threshold may be the median number of times a noun is invoked for multiple (e.g., all) nouns registered in the term frequency table 141, or a value calculated by some function using these invoke counts, and can be implemented in various forms.

[0044] Thus, by using the average or median of the number of times multiple nouns registered in the term frequency table 141 are invoked, or a value calculated using some function that utilizes these invocation counts, the first threshold dynamically changes in response to changes in the invocation count values ​​in the term frequency table 141. In other words, the first threshold can be said to be a dynamic threshold that changes dynamically according to the frequency of occurrence of multiple terms included in the question sentence.

[0045] The specialization determination unit 14 determines that a noun is a technical term if its number of occurrences is less than the first threshold, and sets the corresponding category in the term frequency table 141 to "specialized". In the example shown in Figure 2, the first threshold is 2, and the category "specialized" is set for each noun whose number of occurrences is less than 2.

[0046] On the other hand, if the number of times a noun is called is equal to or greater than the first threshold, the specialization determination unit 14 determines that the noun is a general term and sets the corresponding category in the term frequency table 141 to "general". In the example shown in Figure 2, the first threshold is "2", and "general" is set for "Japan", which is a noun with a number of calls of 2 or more.

[0047] The expertise determination unit 14 is an example of a first determination unit that determines a term to be a technical term when the frequency of occurrence of the term in the question text is equal to or greater than a first threshold.

[0048] Figure 3 illustrates the transition of the term frequency table 141 in an information processing system 1 according to one embodiment.

[0049] In Figure 3, code A represents the term frequency table 141 at the time the nouns "MR" and "meal" were newly registered. Code B represents the term frequency table 141 at a later point in time than code A. The first threshold is assumed to be 2.

[0050] If "MR" and "meal" are not registered in the term frequency table 141, the expertise determination unit 14 registers "MR" and "meal" as new entries in the term frequency table 141. The expertise determination unit 14 then sets the call count for each of the newly registered "MR" and "meal" in the term frequency table 141 to "1" (see symbol A).

[0051] Furthermore, since the call count of "1" is less than the first threshold of "2", the expertise determination unit 14 sets "expert" for each of these "MR" and "meal" categories.

[0052] Subsequently, after one or more user questions have been processed, the number of times "meals" has been called is set to "3" (see symbol B). In this case, the expertise determination unit 14 determines that the number of calls "3" is greater than or equal to the first threshold "2", and therefore sets the category of "meals" to "expert".

[0053] In this way, terms (nouns) included in the user's question that are not yet registered (new) in the term frequency table 141 are registered in the term frequency table 141 by the expertise determination unit 14. In the term frequency table 141, newly registered terms are initially determined to be "specialized" by the expertise determination unit 14, but if they are truly general terms, their usage frequency will increase over time, and they will be determined to be general terms.

[0054] The expertise determination unit 14 performs an expertise determination on the user question based on the results of the expertise determination of the terms. The expertise determination unit 14 may, for example, perform an expertise determination on the user question by comparing the number of technical terms contained in the user question with a second threshold. The expertise determination unit 14 also sets an expertise determination flag according to the result of the expertise determination on the user question.

[0055] The expertise determination unit 14 determines a user question to be an expert question if, for example, the number of technical terms contained in the user question is equal to or greater than the second threshold, and sets the expertise determination flag to "True" for that user question. On the other hand, if the number of technical terms contained in the user question is less than the second threshold, the expertise determination unit 14 determines that the user question is a general question, and sets the expertise determination flag to "False" for that user question.

[0056] The second threshold may be, for example, 1. That is, the expertise determination unit 14 may determine a user question to be an expert question if it contains at least one expert term. In this case, the expertise determination unit 14 determines a user question to be a general question if it does not contain any expert terms.

[0057] The expertise determination unit 14 is an example of a second determination unit that determines a question to be an expert question if the number of expert terms contained in the question is equal to or greater than a second threshold. The expertise determination unit 14 (second determination unit) determines a question to be a general question if the number of expert terms contained in the question is less than the second threshold.

[0058] The expertise determination unit 14 transmits the expertise determination result for the user question, i.e., the value of the expertise determination flag (True / False), to the mediation processing unit 11.

[0059] The usage determination unit 15 determines which LLM to use to generate the answer to the user question from among the first LLM 2a and the second LLM 2b, based on the value of the expertise determination flag set by the expertise determination unit 14. The usage determination unit 15 then causes the determined LLM to generate the answer to the user question.

[0060] The usage determination unit 15 receives the knowledge data generated by the similarity search execution unit 13, the vectorized question, and the value of the expertise determination flag as input.

[0061] The usage determination unit 15 determines the first LLM 2a as the LLM to generate the answer to the user question if the value of the expertise determination flag is False. The usage determination unit 15 sends the vectorized question and knowledge data to this first LLM 2a to generate the answer to the user question. If the value of the expertise determination flag is True, the usage determination unit 15 determines the second LLM 2b as the LLM to generate the answer to the user question. The usage determination unit 15 sends the vectorized question and knowledge data to this second LLM 2b to generate the answer to the user question.

[0062] The usage determination unit 15 is an example of an answer generation control unit that causes the second LLM 2b (second LLM) to generate an answer to a question if the question is a specialized question. The usage determination unit 15 (answer generation control unit) causes the first LLM 2a (first LLM) to generate an answer to a question if the question is a general question.

[0063] Furthermore, the usage determination unit 15 transmits the responses generated in the first LLM2a and the second LLM2b to the mediation processing unit 11.

[0064] (A-2) Hardware configuration example Figure 4 is a diagram illustrating the hardware configuration of an information processing system 1 according to one embodiment.

[0065] As illustrated in Figure 4, the information processing system 1 may include multiple computers 10a to 10g (seven in the example shown in Figure 4). These multiple computers 10a to 10g may be connected to each other via a network 3 so that they can communicate with one another.

[0066] For example, computer 10a may perform the function of the first LLM2a as described above, and computer 10b may perform the function of the second LLM2b. Also, computer 10c may perform the function of the intermediary processing unit 11, and computer 10d may perform the function of the data conversion processing unit 12. Furthermore, computer 10e may perform the function of the similarity search execution unit 13, computer 10f may perform the function of the expertise determination unit 14, and computer 10g may perform the function of the usage determination unit 15.

[0067] These computers 10a to 10g may have similar configurations. Hereafter, unless otherwise specified, computers 10a to 10g will be referred to as computer 10.

[0068] Figure 5 is a diagram illustrating the hardware configuration of computer 10. As shown in Figure 5, the computer 10 may, as an example, have a hardware configuration comprising one or more CPUs 101 (one in the example shown in Figure 5), one or more GPUs 102 (one in the example shown in Figure 2), memory 103, storage unit 104, IF (Interface) unit 105, IO (Input / Output) unit 106, and read unit 107.

[0069] The CPU 101 is an example of an arithmetic processing unit that performs various control and calculations, and is a control unit that executes various processes. The CPU 101 may be connected to each block in the computer 10 via a bus 110 so as to be able to communicate with each other. The CPU 101 may be a multiprocessor that includes multiple processors, a multicore processor that has multiple processor cores, or a configuration that has multiple multicore processors.

[0070] In computer 10a, which performs the function of the first LLM2a, the LLM is executed on the CPU 101. The CPU 101 is an example of a first processor that executes the first LLM2a (the first LLM).

[0071] The GPU 102 may be, for example, an accelerator such as a GPGPU (General Purpose computing on Graphics Processing Unit). Furthermore, the GPU 102 may be used to control screen display for output devices such as monitors, which are part of the I / O unit 106.

[0072] In computer 10b, which performs the function of the second LLM2b, the LLM is executed on GPU 102. When performing the function of the LLM, GPU 102 can be said to have higher processing performance than CPU 101. GPU 102 is an example of a second processor with higher processing performance than the CPU (first processor), which executes the second LLM2b (second LLM), which has a larger model size than the first LLM2a (first LLM). These CPU 101 and GPU 102 are examples of computing resources.

[0073] Memory 103 is an example of hardware that stores various data and program information. Examples of memory 103 include volatile memory such as DRAM (Dynamic Random Access Memory) and non-volatile memory such as PM (Persistent Memory), or both.

[0074] The memory unit 104 is an example of hardware that stores various data and program information. Examples of memory units 104 include magnetic disk devices such as HDDs (Hard Disk Drives), semiconductor drive devices such as SSDs (Solid State Drives), and various storage devices such as non-volatile memory. Examples of non-volatile memory include flash memory, SCM (Storage Class Memory), and ROM (Read Only Memory).

[0075] The memory unit 104 may store a program 108 (information processing program) that implements all or part of the various functions of the computer 10.

[0076] For example, the CPU 101 of the computer 10f, which performs the function of the expertise determination unit 14, can realize the function of the expertise determination unit 14 described above by loading the program 108 stored in the storage unit 104 into the memory 103 and executing it. In addition, the storage unit 104 of the computer 10f may store information that constitutes the term frequency table 141.

[0077] Furthermore, the CPU 101 of the computer 10g, which performs the function of the usage determination unit 15, can realize the above-mentioned function of the usage determination unit 15 by loading the program 108 stored in the storage unit 104 into the memory 103 and executing it.

[0078] Furthermore, the CPU 101 of the computer 10e, which performs the function of the similarity search execution unit 13, can realize the function of the similarity search execution unit 13 described above by expanding the program 108 stored in the storage unit 104 into the memory 103 and executing it. In addition, the storage unit 104 of the computer 10e, which performs the function of the similarity search execution unit 13, may store information that constitutes the vector DB 131.

[0079] Furthermore, the CPU 101 of computer 10c, which performs the function of the intermediary processing unit 11, can realize the above-mentioned function of the intermediary processing unit 11 by expanding the program 108 stored in the storage unit 104 into the memory 103 and executing it. In addition, the CPU 101 of computer 10d, which performs the function of the data conversion processing unit 12, can realize the above-mentioned function of the data conversion processing unit 12 by expanding the program 108 stored in the storage unit 104 into the memory 103 and executing it.

[0080] The IF unit 105 is an example of a communication interface that controls the connection and communication between the computer 10 on which it is installed and other computers 10. For example, the IF unit 105 may include an adapter compliant with LAN (Local Area Network) such as Ethernet®, or optical communication such as FC (Fibre Channel). The adapter may support wireless and wired communication methods, or both. The computer 10 may be connected to network 3 via the IF unit 105 in a communicative manner.

[0081] Furthermore, terminal devices (not shown) may be connected to network 3, and users may use these terminal devices to input user questions. User questions input from the terminal devices may be transmitted via network 3 to computer 10c, which performs the function of the mediation processing unit 11. In addition, answers to user questions may be transmitted from the mediation processing unit 11 to the terminal devices via network 3 and provided to the user through these terminal devices.

[0082] Furthermore, if a user who has received a response via a terminal device determines that they are dissatisfied with the response, they may provide feedback input from the terminal device indicating their dissatisfaction. This feedback input is transmitted from the terminal device to the mediation processing unit 11 via the network 3.

[0083] The program 108 may be downloaded from the network to the computer 10 via the communication interface and stored in the storage unit 104.

[0084] The I / O unit 106 may include either or both an input device and / or an output device. Examples of input devices include a keyboard, mouse, and touch panel. The I / O unit 106 provided in the computer 10c that performs the function of the intermediary processing unit 11 may be used by the user to input user questions.

[0085] Examples of output devices include monitors, projectors, and printers. The I / O unit 106 may also include a touch panel or similar device that integrates input and output functions. The output device may be connected to the GPU 102.

[0086] The reading unit 107 is an example of a reader that reads data and program information recorded on the recording medium 109. The reading unit 107 may include a connection terminal or device to which the recording medium 109 can be connected or inserted. Examples of the reading unit 107 include an adapter compliant with USB (Universal Serial Bus), a drive device that accesses a recording disk, and a card reader that accesses flash memory such as an SD card. The recording medium 109 may store a program 108, and the reading unit 107 may read the program 108 from the recording medium 109 and store it in the storage unit 104.

[0087] Examples of recording media 109 include non-temporary computer-readable recording media such as magnetic / optical discs and flash memory. Examples of magnetic / optical discs include flexible discs, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray discs, and HVDs (Holographic Versatile Discs). Examples of flash memory include semiconductor memory such as USB memory and SD cards.

[0088] The hardware configuration of computer 10 described above is illustrative. Therefore, the addition or deletion of hardware within computer 10 (for example, adding or deleting arbitrary blocks), division, integration in any combination, or addition or deletion of buses may be performed as appropriate.

[0089] (B) Operation The processing in the information processing system 1 according to one embodiment configured as described above will be explained according to the flowchart (steps A1 to A16) shown in Figure 6. This processing is performed each time the user inputs a user question.

[0090] In step A1, the user question entered by the user is transferred to the intermediary processing unit 11. The intermediary processing unit 11 sends the user question to the data conversion processing unit 12. In step A2, the data conversion processing unit 12 vectorizes the user question to generate a vectorized question. The generated vectorized question is then sent to the similarity search execution unit 13 and the expertise determination unit 14, respectively.

[0091] In step A3, the similarity search execution unit 13 performs a similarity search on the information stored in the vector DB 131 using the vectorized query as the search keyword. In step A4, the similarity search execution unit 13 outputs, for example, multiple knowledge data with the highest similarity (e.g., the top 4) as search results based on the similarity search results. For example, the similarity search execution unit 13 may output the top 4 knowledge data with the highest similarity as search results.

[0092] Furthermore, in step A5, the expertise determination unit 14 performs an expertise determination on the terms included in the user question based on the vectorized question. In step A6, the expertise determination unit 14 performs an expertise determination on the user question based on the results of the term expertise determination. In step A7, the expertise determination unit 14 sets an expertise determination flag according to the result of the expertise determination on the user question and outputs the value (True / False) of this expertise determination flag.

[0093] The details of steps A5 to A7 will be described later using the flowchart shown in Figure 9.

[0094] In step A8, the similarity search results (knowledge data) output from the similarity search execution unit 13 and the expert judgment flag value output from the expert judgment unit 14 are transmitted (transferred) to the mediation processing unit 11. The mediation processing unit 11 uses the expert judgment flag value, the similarity search results (knowledge data), and the vectorized question text to determine which values ​​to transmit to the judgment unit 15.

[0095] In step A9, the usage determination unit 15 checks the value of the expertise determination flag. If the value of the expertise determination flag is True (see the "True" route in step A9), the process proceeds to step A11.

[0096] In step A11, the selection determination unit 15 determines the second LLM 2b as the LLM that will generate the answer to the user question, and sends the vectorized question and knowledge data to this second LLM 2b to generate the answer to the user question.

[0097] In step A12, the second LLM2b generates an answer to the user's question. In step A13, the answer (answer text) generated by the second LLM2b is output to the user from the mediation processing unit 11.

[0098] Furthermore, if the result of the verification in step A9 is that the value of the expertise judgment flag is False (see the "False" route in step A9), the process proceeds to step A10. In step A10, the usage determination unit 15 determines the first LLM 2a as the LLM that will generate the answer to the user question, and sends the vectorized question and knowledge data to this first LLM 2a to generate the answer to the user question.

[0099] Subsequently, in step A12, the first LLM2a generates an answer to the user question. In step A13, the answer (response text) generated by the first LLM2a is output to the user from the mediation processing unit 11.

[0100] In step A14, it is confirmed whether the user was more satisfied with the answer to the user's question.

[0101] If the user is not satisfied with the answer (see No route in step A14), in step A15, the user performs an input operation to indicate their dissatisfaction on the feedback input screen displayed on the terminal device's display, etc. For example, the user may indicate their dissatisfaction by clicking a button or similar displayed on the feedback input screen to indicate that they are not satisfied with the answer. Alternatively, the user may indicate their dissatisfaction by entering a message on the feedback input screen, etc., such as a chat message, stating that they would like a more specialized answer.

[0102] If, after output from the mediation processing unit 11 to the user, no feedback input indicating dissatisfaction is received from the user within a predetermined time, it may be determined that the user is satisfied. Alternatively, if the user performs an input operation indicating satisfaction on the feedback input screen displayed on the terminal device's display, it may be determined that the user is satisfied.

[0103] If the user is satisfied (see the Yes route in step A14), the process ends. On the other hand, if the user is not satisfied (see the No route in step A14), the process proceeds to step A15.

[0104] In step A16, for example, the usage determination unit 15 sends the vectorized question and knowledge data to the second LLM 2b to generate a response to the user question again. As a result, the second LLM 2b generates a higher quality response than the one previously presented to the user. Subsequently, the mediation processing unit 11 outputs the response (answer text) generated by the second LLM 2b to the user and terminates the process.

[0105] Next, the details of the processing performed by the expertise determination unit 14 in the information processing system 1 according to one embodiment will be explained according to the flowchart (steps B1 to B14) shown in Figure 7. This processing represents the details of steps A5 to A7 of the flowchart shown in Figure 6.

[0106] In step B1, the expertise determination unit 14 performs morphological analysis (part-of-speech analysis) of the user question text based on the vectorized question text, and in step B2, extracts nouns.

[0107] In step B3, a loop process is initiated that repeatedly performs the controls up to step B10 for all the extracted nouns.

[0108] In step B4, the expertise determination unit 14 checks whether the extracted noun is already registered in the term frequency table 141. If the noun is not registered in the term frequency table 141 (see No. route in step B4), the process proceeds to step B5. In step B5, the expertise determination unit 14 adds a new row to the term frequency table 141, and in step B6, registers the noun in the newly added row.

[0109] Furthermore, if the noun is already registered in the term frequency table 141 (see the Yes route in step B4), proceed to step B7.

[0110] In step B7, the expertise determination unit 14 increments (+1) the number of times the noun is called in the term frequency table 141.

[0111] In step B8, the expertise determination unit 14 compares the number of times the noun has been called with a first threshold. If the number of calls is equal to or greater than the first threshold (see the Yes route in step B8), in step B9, the expertise determination unit 14 sets the category of the noun to "general".

[0112] On the other hand, if the number of calls is less than the first threshold (see No. route in step B8), in step B10, the expertise determination unit 14 sets the category of the noun to "expert".

[0113] In step B11, the loop end processing corresponding to step B3 is performed. Once processing for all nouns is complete, control proceeds to step B12.

[0114] Steps B3 to B11 described above represent the process by which the expertise determination unit 14 determines the expertise level of the term. Furthermore, the term frequency table 141 is updated as a result of the processing in steps B3 to B11.

[0115] For example, after the completion of step B11, the expertise determination unit 14 may update the first threshold based on the number of times all nouns registered in the term frequency table 141 have been called. The expertise determination unit 14 may also use the updated first threshold to reset (update) the categories for all nouns registered in the term frequency table 141. In step B12, the expertise determination unit 14 checks whether the user question contains at least one term (noun) in the category of "expert". In other words, the expertise determination unit 14 checks whether the number of expert terms included in the user question is at or above the second threshold.

[0116] If the number of technical terms included in the user's question exceeds the second threshold, that is, if the user's question contains at least one term (noun) in the category "technical" (see the Yes route in step B12), then in step B13, the expertise determination unit 14 sets the expertise determination flag to "True" and terminates the process.

[0117] Furthermore, if the number of technical terms included in the user's question is less than the second threshold (see No route in step B12), in step B14, the expertise determination unit 14 sets the expertise determination flag to "False" and terminates the process.

[0118] (C) Effects In one embodiment of the information processing system 1, there are a first LLM2a and a second LLM2b. The first LLM2a is a general-purpose LLM that responds to general questions and has a smaller model size than the second LLM2b. The second LLM2b is a specialized LLM that responds to specialized questions and has a larger model size than the first LLM2a.

[0119] By running the smaller-sized LLM2a on computer 10a using CPU 101, the LLM2a can be used at low cost in a hardware environment with performance appropriate to its model size. For example, CPU 101 consumes less power than GPU 102. Also, CPUs are often less expensive than GPUs as computing resources provided by cloud providers.

[0120] Furthermore, by running the larger LLM2b model on computer 10b using GPU 102, the LLM2b can be used in a hardware environment with performance appropriate to its model size.

[0121] The expertise determination unit 14 determines the expertise level of one or more terms included in the user question based on their frequency of occurrence (number of times they are called). The expertise determination unit 14 also determines the expertise level of the user question based on the results of the term expertise determination. If the expertise determination unit 14 determines that the user question is an expert question, it sets the expertise determination flag to True.

[0122] Then, the usage determination unit 15 causes the first LLM2a to generate an answer to the user question if the value of the expertise determination flag is False, and causes the second LLM2b to generate an answer to the user question if the value of the expertise determination flag is True.

[0123] The usage determination unit 15 directs the second LLM 2b to generate answers to highly specialized user questions (specialized questions), thereby generating highly specialized answers that reflect specialized knowledge and producing answers with appropriate accuracy (expertise) that match the specialization of the user question. Furthermore, the usage determination unit 15 directs the first LLM 2a to generate answers to less specialized user questions (general questions), thereby generating answers at a general knowledge level with low specialization and producing answers with appropriate accuracy (expertise) that match the specialization of the user question. In addition, since the first LLM 2a can be used at a lower cost than the second LLM 2b, answers to user questions can be generated at a lower cost.

[0124] Therefore, it is possible to appropriately select the LLM (Limited Licensing Method) to answer user questions from the perspective of both expertise and cost. In other words, it is possible to efficiently differentiate between processing using the first LLM2a (first LLM) with CPU101 (first processor) and processing using the second LLM2b (second LLM) with GPU102 (second processor).

[0125] The expertise determination unit 14 uses a dynamic threshold, such as the average or median of the number of times multiple nouns registered in the term frequency table 141 are invoked, as the first threshold used to determine the expertise of a term. As a result, the first threshold changes in accordance with the change in the number of invoked values ​​in the term frequency table 141. This allows for accurate determination of the expertise of a term even if the frequency of occurrence of each term increases over time.

[0126] The expertise determination unit 14 registers new or unregistered terms in the term frequency table 141 and assigns a general / specialized category to each term, allowing the term frequency table 141 to be reused as knowledge indicating whether a term is general or specialized. Furthermore, since such knowledge is created automatically, it is economical.

[0127] Furthermore, if a user is not satisfied with the answer to their user question, and provides feedback indicating their dissatisfaction on a feedback input screen or the like, the usage determination unit 15, for example, sends the vectorized question and knowledge data to the second LLM 2b to regenerate the answer to the user question.

[0128] This allows the second LLM2b to generate higher-quality answers than those previously presented to the user. For example, if the previous answer presented to the user was created by the first LLM2a, the second LLM2b can create a new answer, providing the user with a more specialized (higher-quality) response.

[0129] Furthermore, even if the answer previously presented to the user was created by the 2nd LLM2b, the 2nd LLM2b can create a new answer, thereby providing the user with a more specialized (high-quality) response.

[0130] (D) Other Each component and process of this embodiment can be selected or omitted as needed, or combined as appropriate.

[0131] Furthermore, the disclosed technology is not limited to the embodiments described above, and can be implemented in various modified forms without departing from the spirit of this embodiment.

[0132] For example, in the embodiment described above, the information processing system 1 includes computers 10a to 10g, and computers 10a, 10b, 10c, 10d, 10e, 10f, and 10g perform functions as the first LLM2a, second LLM2b, intermediary processing unit 11, data conversion processing unit 12, similarity search execution unit 13, expertise determination unit 14, and usage determination unit 15, respectively, but the system is not limited to this.

[0133] At least one of the computers 10a to 10g may execute two or more functions from among the first LLM2a, second LLM2b, intermediary processing unit 11, data conversion processing unit 12, similarity search execution unit 13, expertise determination unit 14, and usage determination unit 15.

[0134] For example, the functions of the specialization determination unit 14 and the usage determination unit 15 may be performed on a single computer 10.

[0135] In the embodiments described above, the first LLM2a, a general-purpose LLM that responds to general questions, is executed by the CPU101, and the second LLM2b, a specialized LLM that responds to specialized questions, is executed by the GPU102. However, the embodiments are not limited to this. For example, the first LLM2a may be executed by a processor other than the CPU, and the second LLM2b may be executed by a processor other than the GPU. Various modifications can be made to implement the embodiments.

[0136] Furthermore, although the embodiments described above show an example where the second threshold is a fixed value (e.g., 1), the system is not limited to this, and the second threshold may be dynamically changed. For example, the second threshold may be varied according to the number of technical terms among the multiple terms included in the user question. For example, the expertise determination unit 14 may determine that a user question is a technical question if the number of technical terms among the multiple terms included in the user question is above a predetermined percentage. In other words, the second threshold may be a dynamic threshold that is dynamically changed according to the number of technical terms included in the question.

[0137] Furthermore, in the embodiments described above, the expertise determination unit 14 performs expertise determination on nouns included in the user's question sentence, but it is not limited to this. The expertise determination unit 14 may also perform expertise determination on parts of speech other than nouns, such as verbs, and can be implemented in various modified forms.

[0138] Furthermore, the above disclosure makes it possible for those skilled in the art to implement and manufacture this embodiment.

[0139] (E) Note The following additional information is disclosed regarding the embodiments described above. (Note 1) A first processor that executes the first LLM (Large Language Model), A second processor having higher processing performance than the first processor, which runs a second LLM having a larger model size than the first LLM, and To the connected computer, If the frequency of occurrence of a term in the question exceeds a first threshold, the term is determined to be a technical term. If the number of technical terms included in the aforementioned question is equal to or greater than the second threshold, the aforementioned question is determined to be a technical question. If the aforementioned question is a specialized question, the second LLM is made to generate an answer to the aforementioned question. An information processing program characterized by executing a process.

[0140] (Note 2) If the number of technical terms included in the question is less than the second threshold, the question is determined to be a general question. If the aforementioned question is a general question, the first LLM is made to generate an answer to the aforementioned question. The information processing program described in Appendix 1, characterized in that it causes the computer to perform the processing.

[0141] (Note 3) The first threshold is dynamically changed according to the frequency of occurrence of multiple terms included in the question sentence. An information processing program as described in Appendix 1 or 2, characterized by the above.

[0142] (Note 4) The second threshold is dynamically changed according to the number of technical terms included in the question. An information processing program as described in Appendix 1 or 2, characterized by the above.

[0143] (Note 5) A first processor that executes the first LLM (Large Language Model), A second processor having higher processing performance than the first processor, which runs a second LLM having a larger model size than the first LLM, A first determination unit determines that a term is a technical term if the frequency of occurrence of the term in the question is equal to or greater than a first threshold, A second determination unit determines that the question is a technical question if the number of technical terms contained in the question is equal to or greater than a second threshold, If the aforementioned question is a specialized question, the answer generation control unit causes the second LLM to generate an answer to the aforementioned question. An information processing system characterized by having the following features.

[0144] (Note 6) If the number of technical terms included in the question is less than the second threshold, the second determination unit determines the question to be a general question. The response generation control unit causes the first LLM to generate a response to the question if the question is a general question. The information processing system described in Appendix 5, characterized by the features described herein.

[0145] (Note 7) The first threshold is dynamically changed according to the frequency of occurrence of multiple terms included in the question sentence. An information processing system as described in Appendix 5 or 6, characterized by the above.

[0146] (Note 8) The second threshold is dynamically changed according to the number of technical terms included in the question. An information processing system as described in Appendix 5 or 6, characterized by the above.

[0147] (Note 9) A first processor that executes the first LLM (Large Language Model), A second processor having higher processing performance than the first processor, which runs a second LLM having a larger model size than the first LLM, and In an information processing system having, If the frequency of occurrence of a term in the question exceeds a first threshold, the term is determined to be a technical term. If the number of technical terms included in the aforementioned question is equal to or greater than the second threshold, the aforementioned question is determined to be a technical question. If the aforementioned question is a specialized question, the second LLM is made to generate an answer to the aforementioned question. An information processing method characterized by performing a process.

[0148] (Note 10) If the number of technical terms included in the question is less than the second threshold, the question is determined to be a general question. If the aforementioned question is a general question, the first LLM is made to generate an answer to the aforementioned question. The information processing method described in Appendix 9, characterized by performing a process.

[0149] (Note 11) The first threshold is dynamically changed according to the frequency of occurrence of multiple terms included in the question sentence. The information processing method described in Appendix 9 or 10, characterized by the above.

[0150] (Note 12) The second threshold is dynamically changed according to the number of technical terms included in the question. The information processing method described in Appendix 9 or 10, characterized by the above. [Explanation of symbols]

[0151] 1. Information Processing System 2a 1st LLM 2b 2nd LLM 3 Network 10a, 10b, 10c, 10d, 10e, 10f, 10g 10 Computers 101 Processors 102 GPU 103 memory 104 Storage section 105 IF section 106 IO Department 107 Reading Unit 108 Programs 109 Recording media 110 Bus 11. Intermediation Processing Section 12 Data Conversion Processing Unit 13 Similarity Search Execution Unit 14. Specialization Assessment Department 15. Usage determination unit 131 Vector DB 141 Term Frequency Table

Claims

1. A first processor that executes the first LLM (Large Language Model), A second processor having higher processing performance than the first processor, which executes a second LLM having a larger model size than the first LLM, and To the connected computer, If the frequency of occurrence of a term in the question exceeds a first threshold, the term is determined to be a technical term. If the number of technical terms included in the aforementioned question is equal to or greater than the second threshold, the aforementioned question is determined to be a technical question. If the aforementioned question is a specialized question, the second LLM is made to generate an answer to the aforementioned question. An information processing program characterized by executing a process.

2. If the number of technical terms included in the question is less than the second threshold, the question is determined to be a general question. If the aforementioned question is a general question, the first LLM is made to generate an answer to the aforementioned question. The information processing program according to claim 1, characterized in that it causes the computer to perform the processing.

3. The first threshold is dynamically changed according to the frequency of occurrence of multiple terms included in the question. An information processing program according to claim 1 or 2, characterized in that...

4. The information processing program according to claim 1 or 2, characterized in that the second threshold is dynamically changed according to the number of technical terms contained in the question.

5. A first processor that executes the first LLM (Large Language Model), A second processor having higher processing performance than the first processor, which executes a second LLM having a larger model size than the first LLM, A first determination unit determines that a term is a technical term if the frequency of occurrence of the term in the question is equal to or greater than a first threshold, A second determination unit determines that the question is a technical question if the number of technical terms contained in the question is equal to or greater than a second threshold, If the aforementioned question is a specialized question, the answer generation control unit causes the second LLM to generate an answer to the aforementioned question. An information processing system characterized by having the following features.

6. A first processor that executes the first LLM (Large Language Model), A second processor having higher processing performance than the first processor, which executes a second LLM having a larger model size than the first LLM, and In an information processing system having, If the frequency of occurrence of a term in the question exceeds a first threshold, the term is determined to be a technical term. If the number of technical terms included in the aforementioned question is equal to or greater than the second threshold, the aforementioned question is determined to be a technical question. If the aforementioned question is a specialized question, the second LLM is made to generate an answer to the aforementioned question. An information processing method characterized by performing a process.

Images