Data processing device, data processing method, and data processing program

Abstract

A search sentence acquisition unit (101) acquires a search sentence (501). A context acquisition unit (102) acquires a plurality of contexts (502) and an evaluation index (503) for each context by a search using the search sentence. A grouping unit (103) groups the plurality of contexts into a plurality of groups in descending order of evaluation indices. A negative example acquisition unit (104) selects one of the groups from within the range of a prescribed upper limit and a prescribed lower limit in the descending order of evaluation indices, and generates an inquiry sentence (505) to a language model (112) by using the search sentence and a context belonging to the selected group. The negative example acquisition unit (104) inputs the inquiry sentence to the language model (112) and acquires a response sentence (506) to the inquiry sentence from the language model (112). A management unit (105) manages the search sentence and the response sentence in association with each other.

Description

Technical Field

[0001] The present disclosure relates to language model training. A language model is a model used in natural language processing obtained by learning occurrence probabilities of sentences, words, and the like. Background Art

[0002] In recent years, the introduction of sentence generation technology using large language models (LLMs) has progressed. Such sentence generation technology is used in various applications such as supporting human writing work, automatically answering questions, and automatic coding. The scaling up of language models, which has enabled the generation of more natural sentences than before, has promoted the widespread adoption of sentence generation technology.

[0003] On the other hand, although sentences generated by language models appear natural at first glance, they may contain semantic errors in their content. This problem is called hallucination. As an example of hallucination, consider a case where a language model generates the sentence "YY Zoo in XX City breeds ZZ", but in reality YY Zoo is located in WW City.

[0004] Hallucination in language models can be reduced by performing supervised additional training. Performing such additional training requires preparing a large number of pairs of input sentences and correct output sentences. Furthermore, if "high-quality negative examples" can be learned in additional training, hallucination can be effectively suppressed. A negative example is a pair of an input sentence and a sentence that is inappropriate as an output sentence for the input. A "high-quality negative example" refers to a pair among the above negative examples that consists of an input sentence and an output sentence that appears natural and correct at first glance but contains a semantic error. However, it is difficult to prepare a large amount of "high-quality negative examples".

[0005] As a technique for suppressing hallucination, for example, there is a technique disclosed in Non-Patent Document 1. In the technology described in Non-Patent Document 1, the input sentence is used as a query to retrieve text. Furthermore, in the technology described in Non-Patent Document 1, the sentence with the highest search score among the search results (context) and the input sentence are combined and input into the language model as a new input sentence. As a result, the technology described in Non-Patent Document 1 generates a response that also takes contextual information into account. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering, Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, Tat-Seng Chua, https: / / arxiv.org / abs / 2101.00774 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] As mentioned above, hallucination in language models can be suppressed by performing supervised additional learning using high-quality negative examples. The technology described in Non-Patent Document 1 has the problem that while positive examples can be obtained, high-quality negative examples cannot. A positive example is a pair of an input sentence and a natural output sentence that does not contain semantic errors.

[0008] One of the main objectives of this disclosure is to address these challenges. More specifically, the main objective of this disclosure is to obtain high-quality negative examples that can be used for training language models. [Means for solving the problem]

[0009] The data processing device related to this disclosure is A search query retrieval unit that retrieves the search query, A context acquisition unit that acquires multiple contexts and evaluation indicators for each of the multiple contexts by performing a search using the aforementioned search statement, A grouping unit that groups the multiple contexts into multiple groups in order of their superior evaluation metrics, A query statement generation unit selects one of the groups within a predetermined upper and lower limit range in order of excellence of the aforementioned evaluation indicator, and generates a query statement for the language model using one or more contexts belonging to the selected group and the search statement. A response statement acquisition unit inputs the query statement into the language model and obtains a response statement from the language model to the query statement, It has a management unit that manages the aforementioned search query and the aforementioned response query in association. [Effects of the Invention]

[0010] According to this disclosure, high-quality negative examples can be obtained that can be used for training language models. [Brief explanation of the drawing]

[0011] [Figure 1] A diagram showing an example of the functional configuration of a data processing device according to Embodiment 1. [Figure 2] A diagram showing an example of the hardware configuration of the data processing device according to Embodiment 1. [Figure 3] A flowchart showing an example of the operation of the data processing device according to Embodiment 1. [Figure 4] A diagram showing an example of the context and evaluation indicators related to Embodiment 1. [Figure 5] A diagram showing an example of a search query, context, and evaluation indicator according to Embodiment 1. [Figure 6] A diagram showing an example of an inquiry statement according to Embodiment 1. [Figure 7] A diagram showing an example of a response statement according to Embodiment 1. [Figure 8] A figure showing an example of a negative case according to Embodiment 1. MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, embodiments will be described with reference to the drawings. In the following description of embodiments and the drawings, components denoted by the same reference signs indicate the same or corresponding parts.

[0013] Embodiment 1. ***Description of Configuration*** FIG. 1 shows an example of a functional configuration of a data processing apparatus 100 according to the present embodiment. FIG. 2 shows an example of a hardware configuration of the data processing apparatus 100 according to the present embodiment. First, an example of the hardware configuration of the data processing apparatus 100 will be described with reference to FIG. 2.

[0014] The data processing apparatus 100 according to the present embodiment is a computer. The operation procedure of the data processing apparatus 100 corresponds to a data processing method. A program that implements the operation of the data processing apparatus 100 corresponds to a data processing program.

[0015] As hardware, the data processing apparatus 100 includes a processor 901, a main storage device 902 , an auxiliary storage device 903, and a communication device 904. As functional components, the data processing apparatus 100 further includes a query acquisition unit 101, a context acquisition unit 102, a grouping unit 103, a negative example acquisition unit 104, and a management unit 105 shown in FIG. 1. The functional components in FIG. 1 are implemented by, for example, a program. Programs for implementing these functions are stored in the auxiliary storage device 903. These programs are loaded from the auxiliary storage device 903 to the main storage device 902. Then, the processor 901 executes these programs to perform the operations of the functional components shown in FIG. 1. FIG. 2 schematically shows a state where the processor 901 is executing a program for implementing the functions of the functional components shown in FIG. 1.

[0016] Next, an example of the functional configuration of the data processing device 100 will be described with reference to Figure 1.

[0017] The search query acquisition unit 101 acquires, for example, a search query 501 from a user of the data processing device 100. Search query 501 can be a sentence written in natural language, or information corresponding to a sentence written in natural language. Similarly, the context 502, query statement 505, and response statement 506 shown below may also be text written in natural language or information corresponding to text written in natural language. One example of information corresponding to text written in natural language is vector information, which is a vector representation of text written in natural language. The search query acquisition unit 101 outputs the acquired search query 501 to the context acquisition unit 102. The processing performed by the search query acquisition unit 101 corresponds to the search query acquisition process.

[0018] The context acquisition unit 102 acquires the search statement 501 from the search statement acquisition unit 101. Then, the context acquisition unit 102 obtains multiple pairs of contexts 502 and evaluation indicators 503 from the knowledge source 111 by performing a search using the search statement 501. Evaluation metric 503 is an index that serves as the basis for evaluating context 502. Evaluation metric 503 may be, for example, a search score. Alternatively, evaluation metric 503 may be, for example, cosine similarity. In addition, evaluation metric 503 may be an index obtained by, for example, weighting and adding the search score and cosine similarity. The context acquisition unit 102 outputs pairs of multiple contexts 502 and evaluation indicators 503 to the grouping unit 103. Furthermore, the context acquisition unit 102 outputs the search statement 501 and the set of multiple contexts 502 and evaluation index 503 to the negative example acquisition unit 104. The processing performed by the context acquisition unit 102 corresponds to the context acquisition process.

[0019] Knowledge source 111 is, for example, a document database. Alternatively, knowledge source 111 may be a knowledge base. The knowledge source 111 outputs the context 502 corresponding to the search query 501 and the evaluation index 503 of the context 502 to the context acquisition unit 102. Figure 1 shows an example where the knowledge source 111 is located inside the data processing device 100. The knowledge source 111 may also be located outside the data processing device 100. If the knowledge source 111 is located outside the data processing device 100, the context acquisition unit 102 accesses the knowledge source 111 via the network and receives the context 502 and evaluation index 503 from the knowledge source 111.

[0020] The grouping unit 103 obtains pairs of multiple contexts 502 and evaluation indicators 503 from the context acquisition unit 102. Then, the grouping unit 103 sorts multiple contexts in order of their evaluation index 503 performance. Group Street Fighter 502 into multiple groups. The grouping unit 103 outputs the grouping result 504 to the negative example acquisition unit 104. The grouping result 504 shows, for example, the identifiers of the contexts 502 included in each group. As a result of the grouping unit 103 grouping multiple contexts 502, one group may contain multiple contexts 502, or one group may contain one context 502. The processing performed by the grouping unit 103 corresponds to grouping processing.

[0021] The negative example acquisition unit 104 acquires the search statement 501 and a set of multiple contexts 502 and evaluation indicators 503 from the context acquisition unit 102. Furthermore, the negative example acquisition unit 104 acquires the grouping result 504 from the grouping unit 103. The negative example acquisition unit 104 selects a group from among the groups within a predetermined upper and lower limit range in order of performance of the evaluation index 503, excluding the group with the best evaluation index 503. The predetermined upper and lower limit range is the selection range for the group to be targeted for negative example acquisition. The predetermined upper and lower limit range is, for example, the top one-third range. However, the predetermined upper and lower limit range is not limited to the top one-third range. The predetermined upper and lower limit range may be, for example, the top one-quarter range or the top one-fifth range. Furthermore, the predetermined upper and lower limit range may be, for example, the top one-third range or the top one-√5th range, or the top one-N range (where N is a positive real number greater than 1). Also, the predetermined upper and lower limit range can be, for example, set as the top 20% and the lower limit as the top 80%. In this way, the predetermined upper and lower limit range can be arbitrarily set to create suitable negative examples. The predetermined upper and lower limit range can be arbitrarily set by the user, for example. The predetermined upper and lower limits can be set based on the statistics of the evaluation index 503. Specifically, when a user sets the predetermined upper and lower limits, for example, the upper limit can be set to a range less than 0.95 times the maximum value of the evaluation index 503 and greater than 1.1 times the average value of the evaluation index 503, while the lower limit can be set to a range less than 0.9 times the average value of the evaluation index 503 and greater than 1.05 times the minimum value of the evaluation index 503. The maximum, minimum, and average values ​​of the evaluation index 503 can be calculated from multiple evaluation indices 503. The average value may be the median or the mode. In this way, by setting the predetermined upper and lower limits based on the statistics of the evaluation index 503, it is possible to prevent undesirable groups (for example, groups for which positive examples have been obtained, groups with extremely low evaluation indices, groups with unnatural sentences, etc.) from being mistakenly included in the group for which negative examples are to be obtained. Alternatively, the upper and lower bounds may be determined using a pre-trained model obtained through machine learning. Examples of machine learning methods include neural networks, support vector machines, and decision trees. A pre-trained model can be obtained, for example, by machine learning using a large number of pre-given search statements 501 and a set of multiple contexts 502 and evaluation metrics 503. In the negative example acquisition unit 104, the reason for excluding the group with the best evaluation index 503 from the group targeted for negative example acquisition is that this group is likely to have obtained semantically correct output sentences and is therefore unsuitable for generating negative examples. However, if the evaluation index of the group with the best evaluation index 503 is low, the likelihood of obtaining semantically correct output sentences is low, so it is not necessary to exclude the group with the best evaluation index 503. In other words, if the evaluation index of the group with the best evaluation index 503 is low, it may be included in the group targeted for negative example acquisition. A low evaluation index 503 means that the evaluation index 503 is below a predetermined threshold. For example, if the evaluation index 503 is expressed as cosine similarity, the predetermined threshold is 0.5. The predetermined threshold may be appropriately changed, for example, depending on the degree of the nature of the search sentence 501. The degree of the nature of the search sentence 501 is, for example, the degree of complexity or ambiguity of the sentence used for the search. Then, the negative example acquisition unit 104 generates a query statement 505 to the language model 112 using the context 502 belonging to the selected group and the search statement 501. Furthermore, the negative example acquisition unit 104 inputs the query statement 505 into the language model 112 and obtains a response statement 506 to the query statement 505 from the language model 112. The negative example acquisition unit 104 outputs the search statement 501 and the response statement 506 to the management unit 105. The negative example acquisition unit 104 corresponds to the query statement generation unit and the response statement acquisition unit. Furthermore, the processing performed by the negative example acquisition unit 104 corresponds to the query statement generation process and the response statement acquisition process.

[0022] Language model 112 is, for example, a large-scale language model. The language model 112 generates a response statement 506 corresponding to the query statement 505 and returns the response statement 506 to the negative example acquisition unit 104. Figure 1 shows an example where the language model 112 is located inside the data processing unit 100. The language model 112 may also be located outside the data processing unit 100. If the language model 112 is located outside the data processing unit 100, the negative example acquisition unit 104 accesses the language model 112 via the network and sends a query statement 505 to the language model 112. The language model 112 generates a response statement 506 corresponding to the query statement 505 and sends the response statement 506 to the negative example acquisition unit 104 via the network.

[0023] The management unit 105 obtains the search statement 501 and the response statement 506 from the negative example acquisition unit 104. The management unit 105 associates the search query 501 with the response query 506 and manages them as negative examples. More specifically, the management unit 105 associates the search statement 501 with the response statement 506 and stores them as negative examples in the negative example database 113. The processing performed by the management department 105 corresponds to management processing.

[0024] The negative example database 113 stores the search statement 501 and the response statement 506 as negative examples, associating them with each other. The search sentence 501 and response sentence 506 stored in the negative example database 113 are used as negative example training data for supervised additional training of the language model. Supervised additional learning may be performed on language model 112, or on language models other than language model 112. Furthermore, supervised additional learning may be performed, such as in-context learning or fine-tuning. Figure 1 shows an example where the negative example database 113 is located inside the data processing device 100. The negative example database 113 may also be located outside the data processing device 100. If the negative example database 113 is located outside the data processing device 100, the management unit 105 accesses the negative example database 113 via the network and sends a search statement 501 and a response statement 506 to the negative example database 115. Furthermore, the negative example database 113 may be distributed across multiple locations outside the data processing device 100. For example, the search statement 501 and the response statement 506 may be stored separately in multiple locations. The management unit 105 can use a correspondence table showing the relationship between the search statement 501 and the response statement 506 to retrieve them in association from the negative example database 113, which is distributed across multiple locations.

[0025] ***Explanation of operation*** Figure 3 shows an example of the operation of the data processing device 100 according to this embodiment. An example of the operation of the data processing device 100 will be explained with reference to Figure 3.

[0026] First, in step S101, the search statement acquisition unit 101 acquires a search statement 501 from, for example, a user of the data processing device 100. Then, the search query acquisition unit 101 outputs the acquired search query 501 to the context acquisition unit 102.

[0027] Next, in step S102, the context acquisition unit 102 acquires the search statement 501 from the search statement acquisition unit 101. Then, the context acquisition unit 102 obtains multiple pairs of contexts 502 and evaluation indicators 503 from the knowledge source 111 by performing a search using the search statement 501. The context acquisition unit 102 then outputs pairs of multiple contexts 502 and evaluation indicators 503 to the grouping unit 103. Furthermore, the context acquisition unit 102 outputs the search statement 501 and the set of multiple contexts 502 and evaluation index 503 to the negative example acquisition unit 104.

[0028] Next, in step S103, the grouping unit 103 obtains multiple sets of contexts 502 and evaluation indicators 503 from the context acquisition unit 102. Furthermore, the grouping unit 103 groups the multiple contexts 502 into multiple groups in order of superiority of the evaluation indicators 503. For example, the grouping unit 103 groups multiple contexts 502 into approximately 10 to 20 groups. The grouping unit 103 outputs the grouping result 504 to the negative example acquisition unit 104.

[0029] Next, in step S104, the negative example acquisition unit 104 acquires the search statement 501 and a set of multiple contexts 502 and evaluation indicators 503 from the context acquisition unit 102. Furthermore, the negative example acquisition unit 104 acquires the grouping result 504 from the grouping unit 103. Then, the negative example acquisition unit 104 generates the query statement 505. Specifically, the negative example acquisition unit 104 selects one of the groups from the top one-third of the groups in order of performance of the evaluation index 503, excluding the group with the best evaluation index 503. For example, if the grouping unit 103 generates 10 groups, the negative example acquisition unit 104 selects one of the groups that have the second to fourth best evaluation index 503. In this case, the negative example acquisition unit 104 may select only one group from the second to fourth groups, or it may select multiple groups. The negative example acquisition unit 104 then generates a query statement 505 to the language model 112 using the contexts 502 belonging to the selected group and the search statement 501. The negative example acquisition unit 104 may use all of the contexts 502 belonging to the selected group, or it may use only some of the contexts 502.

[0030] Next, in step S105, the negative example acquisition unit 104 inputs the query statement 505 into the language model 112 and obtains a response statement 506 to the query statement 505 from the language model 112. The negative example acquisition unit 104 then outputs the search statement 501 and the response statement 506 to the management unit 105.

[0031] Finally, in step S106, the management unit 105 obtains the search statement 501 and the response statement 506 from the negative example acquisition unit 104. The management unit 105 then associates the search statement 501 with the response statement 506 and manages them as negative examples in the negative example database 113.

[0032] Next, we will explain each step in Figure 3 using a specific example.

[0033] Here, in step S101, the search query acquisition unit 101 acquires a search query 501, for example, "Please tell me which zoos in ABC City have alpacas." The search query acquisition unit 101 outputs the acquired search query 501 to the context acquisition unit 102.

[0034] Next, in step S102, the context acquisition unit 102 acquires a pair of context 502 and evaluation index 503, as illustrated in Figure 4, from the knowledge source 111 by searching using the search statement 501. Here, we will explain assuming that evaluation metric 503 is a search score ranging from "0.0" to "9.9" in increments of "0.1". Contexts 502 with higher evaluation metric 503 values ​​are considered to have a higher evaluation. The context acquisition unit 102 outputs the pair of context 502 and evaluation index 503, as illustrated in Figure 4, to the grouping unit 103. Furthermore, the context acquisition unit 102 outputs the search statement 501 and the pair of context 502 and evaluation index 503, as exemplified in Figure 4, to the negative example acquisition unit 104.

[0035] Next, in step S103, the grouping unit 103 obtains pairs of contexts 502 and evaluation indicators 503, as illustrated in Figure 4, from the context acquisition unit 102. Then, the grouping unit 103 groups the multiple contexts 502 into multiple groups in order of superiority of the evaluation indicator 503. Here, the grouping unit 103 will group the context 502 into 10 groups as follows. 1) Group 1 where the evaluation index 503 is between "0.0" and "0.9" 2) Group 2 where the evaluation index 503 is between "1.0" and "1.9" 3) Group 3 where the evaluation index 503 is between "2.0" and "2.9" 4) Group 4 where the evaluation index 503 is between "3.0" and "3.9" 5) Group 5 where the evaluation index 503 is between "4.0" and "4.9" 6) Group 6 where the evaluation index 503 is between "5.0" and "5.9" 7) Group 7 where the evaluation index 503 is between "6.0" and "6.9" 8) Group 8 where the evaluation index 503 is between "7.0" and "7.9" 9) Group 9 where the evaluation index 503 is between "8.0" and "8.9" 10) Group 10 where the evaluation index 503 is between "9.0" and "9.9" The grouping unit 103 outputs the grouping result 504 to the negative example acquisition unit 104.

[0036] Next, in step S104, the negative example acquisition unit 104 generates the query statement 505. Here, the negative example acquisition unit 104 selects groups 9 to 7 in which the evaluation index 503 is second to fourth best. Then, the negative example acquisition unit 104 uses the contexts 502 belonging to the selected groups 9 to 7 and the search statement 501 to generate a query statement 505 to the language model 112. Here, the negative example acquisition unit 104 generates the query statement 505 shown in Figure 6 using the search statement 501 and context 502 shown in Figure 5.

[0037] Next, in step S105, the negative example acquisition unit 104 inputs the query statement 505 into the language model 112 and obtains a response statement 506 to the query statement 505 from the language model 112. In this case, the negative example acquisition unit 104 acquires the response statement 506 as illustrated in Figure 7.

[0038] Finally, in step S106, the management unit 105 associates the search statement 501 and the response statement 506 and stores them as negative examples in the negative example database 113, as illustrated in Figure 8.

[0039] ***Explanation of the effects of the embodiment*** In this embodiment, context 502 is used from the top one-third of the groups excluding the group with the best evaluation metric (the group ranked 2nd to 4th on a 10-point scale). In other words, in this embodiment, context 502 is used that is related to the search query 501 but is not sufficient as an answer to the search query 501. As a result, in this embodiment, a response sentence 506 can be obtained that appears natural and correct at first glance but contains semantic errors. Therefore, according to this embodiment, high-quality negative examples that can be used for training language models can be obtained. Furthermore, by performing supervised additional learning using the negative examples obtained in this way, hallucination can be effectively suppressed.

[0040] In this embodiment, we have explained how to obtain a negative example using the configuration shown in Figure 1, but it is also possible to obtain a positive example using the configuration shown in Figure 1. When obtaining positive examples, the negative example acquisition unit 104 selects the group with the best evaluation index 503 in the grouping result 504. Then, the negative example acquisition unit 104 generates a query statement 505 to the language model 112 using the context 502 and search statement 501 belonging to the selected group. Furthermore, the negative example acquisition unit 104 inputs the query statement 505 into the language model 112 and obtains a response statement 506 to the query statement 505 from the language model 112. Then, the management unit 105 associates the search statement 501 with the response statement 506 and stores it as a correct example in the correct example database (not shown in Figure 1).

[0041] The procedure described in this embodiment is just one example. Therefore, it is acceptable to perform only a portion of the procedure described in this embodiment. Furthermore, at least some of the procedures described in this embodiment may be combined with procedures not described in this embodiment. Furthermore, the configuration and procedures described in this embodiment may be modified as necessary.

[0042] ***Supplementary explanation of hardware configuration*** Here, we will provide a supplementary explanation of the hardware configuration of the data processing device 100. The processor 901 shown in Figure 2 is an integrated circuit (IC) that performs processing. Processor 901 includes components such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The main memory 902 shown in Figure 2 is RAM (Random Access Memory). The auxiliary storage device 903 shown in Figure 2 includes ROM (Read Only Memory), flash memory, HDD (Hard Disk Drive), etc. The communication device 904 shown in Figure 2 is an electronic circuit that performs data communication processing. The communication device 904 is, for example, a communication chip or a NIC (Network Interface Card).

[0043] Furthermore, the auxiliary storage device 903 also stores the OS (Operating System). Then, at least a portion of the OS is executed by processor 901. The processor 901 executes programs that implement the functions of the functional components shown in Figure 1, while simultaneously running at least a portion of the OS. Processor 901 executes the OS, handling task management, memory management, file management, communication control, and other functions. Furthermore, at least one of the information, data, signal values, and variable values ​​indicating the processing results of the functional components shown in Figure 1 is stored in the main memory 902, auxiliary storage 903, and processor 901. It is stored in at least one of the registers and cache memory. Furthermore, the programs that realize the functions of the functional components shown in Figure 1 may be stored on portable recording media such as magnetic disks, flexible disks, optical disks, compact disks, Blu-ray® disks, and DVDs. Portable recording media containing the programs that realize the functions of the functional components shown in Figure 1 may also be distributed.

[0044] Furthermore, at least one of the "parts" of the functional components shown in Figure 1 may be replaced with "circuit," "process," "procedure," "process," or "circuitry." Furthermore, the data processing device 100 may be implemented by a processing circuit. The processing circuit may be, for example, a logic IC (Integrated Circuit), a GA (Gate Array), or an ASIC (Application Specific Integrated Circuit). These are Circuits (Platform Circuits) and FPGAs (Field-Programmable Gate Arrays). In this case, the functional components shown in Figure 1 are each implemented as part of the processing circuit. In this specification, the higher-level concept encompassing both the processor and the processing circuit is referred to as "processing circuitry." In other words, a processor and a processing circuit are specific examples of "processing circuits," respectively.

[0045] Finally, the various aspects of this disclosure are summarized in the appendix. (Note 1) A search query retrieval unit that retrieves the search query, A context acquisition unit that acquires multiple contexts and evaluation indicators for each of the multiple contexts by performing a search using the aforementioned search statement, A grouping unit that groups the multiple contexts into multiple groups in order of their superior evaluation metrics, A query statement generation unit selects one of the groups within a predetermined upper and lower limit range in order of excellence of the aforementioned evaluation indicator, and generates a query statement for the language model using one or more contexts belonging to the selected group and the search statement. A response statement acquisition unit inputs the query statement into the language model and obtains a response statement from the language model to the query statement, A data processing device having a management unit that manages the aforementioned search statement and the aforementioned response statement in association. (Note 2) The aforementioned management department, The data processing device described in Appendix 1 manages the aforementioned search statement and the aforementioned response statement as training data for machine learning. (Note 3) The aforementioned management department, The data processing device according to Appendix 2, which manages the search statement and the response statement as training data for at least one of in-context learning and fine tuning. (Note 4) The context acquisition unit, The data processing device described in Appendix 1, which obtains at least one of the search score and cosine similarity as the aforementioned evaluation indicator. (Note 5) The computer retrieves the search query, The computer obtains multiple contexts and evaluation metrics for each of the multiple contexts by performing a search using the search statement. The computer then sorts the multiple contexts into multiple groups based on their performance in terms of evaluation metrics. Group them into loops, The computer selects one of the groups within a predetermined upper and lower limit range in order of the quality of the evaluation indicator, and uses one or more contexts belonging to the selected group and the search statement to generate a query statement for the language model. The computer inputs the query statement into the language model and obtains a response statement to the query statement from the language model. A data processing method in which the computer manages the search statement and the response statement in association. (Note 6) The process of obtaining the search query, A context acquisition process that obtains multiple contexts and evaluation metrics for each of the multiple contexts by performing a search using the aforementioned search statement, A grouping process that groups the aforementioned multiple contexts into multiple groups in order of their performance in evaluation metrics, A query generation process that selects one of the groups within a predetermined upper and lower limit range in order of excellence of the aforementioned evaluation indicators, and generates a query statement for the language model using one or more contexts belonging to the selected group and the search statement, A response acquisition process that inputs the query statement into the language model and obtains a response statement from the language model to the query statement, A data processing program that causes a computer to execute a management process that associates the aforementioned search statement with the aforementioned response statement. [Explanation of Symbols]

[0046] 100 Data processing unit, 101 Search statement acquisition unit, 102 Context acquisition unit, 103 Grouping unit, 104 Negative example acquisition unit, 105 Management unit, 111 Knowledge source, 112 Language model, 113 Negative example database, 501 Search statement, 502 Context, 503 Evaluation indicator, 504 Grouping result, 505 Query statement, 506 Response statement, 901 Processor, 902 Main memory, 903 Auxiliary memory, 904 Communication device.

Claims

1. A search query retrieval unit that retrieves the search query, A context acquisition unit that acquires multiple contexts and evaluation indicators for each of the multiple contexts by performing a search using the aforementioned search statement, A grouping unit that groups the multiple contexts into multiple groups in order of their superior evaluation metrics, A query statement generation unit selects one of the groups within a predetermined upper and lower limit range in order of excellence of the aforementioned evaluation indicators, and generates a query statement for the language model using one or more contexts belonging to the selected group and the search statement. A response statement acquisition unit inputs the query statement into the language model and obtains a response statement from the language model to the query statement, A data processing device having a management unit that manages the aforementioned search statement and the aforementioned response statement in association.

2. The aforementioned management department, The data processing device according to claim 1, which manages the search statement and the response statement as training data for machine learning.

3. The aforementioned management department, The data processing device according to claim 2, wherein the search statement and the response statement are managed as training data for at least one of in-context learning and fine tuning.

4. The context acquisition unit, The data processing device according to claim 1, wherein at least one of the search score and cosine similarity is obtained as the evaluation index.

5. The computer retrieves the search query, The computer obtains multiple contexts and evaluation metrics for each of the multiple contexts by performing a search using the search statement. The computer groups the multiple contexts into multiple groups in order of their evaluation metrics, The computer selects one of the groups within a predetermined upper and lower limit range in order of the quality of the evaluation indicator, and uses one or more contexts belonging to the selected group and the search statement to generate a query statement for the language model. The computer inputs the query statement into the language model and obtains a response statement to the query statement from the language model. A data processing method in which the computer manages the search statement and the response statement in association.

6. The process of obtaining the search query, A context acquisition process that obtains multiple contexts and evaluation metrics for each of the multiple contexts by performing a search using the aforementioned search statement, A grouping process that groups the aforementioned multiple contexts into multiple groups in order of their performance in evaluation metrics, A query generation process that selects one of the groups within a predetermined upper and lower limit range in order of excellence of the evaluation indicator, and generates a query statement for the language model using one or more contexts belonging to the selected group and the search statement, A response acquisition process that inputs the query statement into the language model and obtains a response statement from the language model to the query statement, A data processing program that causes a computer to execute a management process that associates the aforementioned search statement with the aforementioned response statement.

Images