Data processing device, data processing method, and data processing program

Abstract

A search sentence acquisition unit (101) acquires a search sentence (501). A context acquisition unit (102) acquires a plurality of contexts (502) and an evaluation index (503) for each context by a search using the search sentence. A grouping unit (103) groups the plurality of contexts into a plurality of groups in descending order of evaluation indices. A negative example acquisition unit (104) selects one of the groups from within the range of a prescribed upper limit and a prescribed lower limit in the descending order of evaluation indices, and generates an inquiry sentence (505) to a language model (112) by using the search sentence and a context belonging to the selected group. The negative example acquisition unit (104) inputs the inquiry sentence to the language model (112) and acquires a response sentence (506) to the inquiry sentence from the language model (112). A management unit (105) manages the search sentence and the response sentence in association with each other.

Description

Data processing device, data processing method and data processing program

[0001] The present disclosure relates to learning a language model, which is a model used in natural language processing obtained by learning the occurrence probability of sentences, words, etc.

[0002] In recent years, the introduction of text generation technology using large-scale language models (LLMs) has progressed. Such text generation technology is being used for a variety of purposes, including assisting people with writing tasks, automatically answering questions, and automatically coding. The increased scale of language models has made it possible to generate more natural-sounding text than before, which has accelerated the spread of text generation technology.

[0003] On the other hand, sentences generated by a language model may appear natural at first glance, but may contain semantic errors. This type of problem is called hallucination. An example of hallucination would be if a language model generated a sentence such as "Zoo YY in City XX keeps ZZ," but Zoo YY is actually located in City WW.

[0004] This hallucination in language models can be reduced by performing supervised incremental learning. To perform such incremental learning, it is necessary to prepare a large number of pairs of input sentences and correct output sentences. Furthermore, if "high-quality negative examples" can be learned in incremental learning, hallucination can be effectively suppressed. A negative example is a pair of an input sentence and an inappropriate sentence as its output sentence. A "high-quality negative example" is a pair of the above-mentioned negative examples, that is, an input sentence and an output sentence that appears natural and correct at first glance but contains a semantic error. However, it is difficult to prepare a large number of "high-quality negative examples."

[0005] One example of a technique for suppressing hallucination is disclosed in Non-Patent Document 1. In the technique of Non-Patent Document 1, sentences are searched (retrieved) using an input sentence as a query. In addition, in the technique of Non-Patent Document 1, a sentence with a high search score among the sentences (contexts) in the search results is combined with the input sentence, and the resulting sentence is input to a language model as a new input sentence. In this way, the technique of Non-Patent Document 1 generates an answer sentence that also takes context information into account.

[0006] Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering, Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, Tat-Seng Chua, https: / / arxiv. org / abs / 2101.00774

[0007] As mentioned above, hallucination in a language model can be suppressed by performing supervised incremental learning using high-quality negative examples. The technology of Non-Patent Document 1 can obtain positive examples, but has the problem of not being able to obtain high-quality negative examples. A positive example is a pair of an input sentence and a natural output sentence that does not contain semantic errors.

[0008] One of the main objectives of the present disclosure is to solve such problems. More specifically, a main objective of the present disclosure is to obtain high-quality negative examples that can be used for training a language model.

[0009] A data processing device according to the present disclosure includes: a search statement acquisition unit that acquires a search statement; a context acquisition unit that acquires a plurality of contexts and evaluation indices for each of the plurality of contexts by searching using the search statement; a grouping unit that groups the plurality of contexts into a plurality of groups in order of best evaluation index; a query statement generation unit that selects one of the groups within a predetermined upper and lower limit range in order of best evaluation index, and generates a query statement for a language model using one or more contexts belonging to the selected group and the search statement; a response statement acquisition unit that inputs the query statement into the language model and acquires a response statement to the query statement from the language model; and a management unit that manages the search statement and the response statement in association with each other.

[0010] According to the present disclosure, it is possible to obtain high-quality negative examples that can be used for training a language model.

[0011] FIG. 1 is a diagram showing an example of the functional configuration of a data processing device according to embodiment 1. FIG. 2 is a diagram showing an example of the hardware configuration of a data processing device according to embodiment 1. FIG. 3 is a flowchart showing an example of the operation of a data processing device according to embodiment 1. FIG. 4 is a diagram showing examples of contexts and evaluation indices according to embodiment 1. FIG. 5 is a diagram showing examples of a search statement, context, and evaluation indices according to embodiment 1. FIG. 6 is a diagram showing an example of a query statement according to embodiment 1. FIG. 7 is a diagram showing an example of a response statement according to embodiment 1. FIG. 8 is a diagram showing an example of a negative example according to embodiment 1.

[0012] Hereinafter, embodiments will be described with reference to the drawings. In the following description of the embodiments and the drawings, the same reference numerals denote the same or corresponding parts.

[0013] Embodiment 1. ***Description of Configuration*** Fig. 1 shows an example of the functional configuration of a data processing device 100 according to this embodiment. Fig. 2 shows an example of the hardware configuration of the data processing device 100 according to this embodiment. First, the example of the hardware configuration of the data processing device 100 will be described with reference to Fig. 2.

[0014] Data processing device 100 according to this embodiment is a computer. The operation procedure of data processing device 100 corresponds to a data processing method. Furthermore, a program that realizes the operation of data processing device 100 corresponds to a data processing program.

[0015] The data processing device 100 includes, as hardware components, a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication device 904. The data processing device 100 also includes, as functional components, a search statement acquisition unit 101, a context acquisition unit 102, a grouping unit 103, a negative example acquisition unit 104, and a management unit 105 shown in FIG. 1 . The functional components of FIG. 1 are realized, for example, by programs. The auxiliary storage device 903 stores programs that realize these functions. These programs are loaded from the auxiliary storage device 903 to the main storage device 902. The processor 901 then executes these programs to perform the operations of the functional components of FIG. 1 . FIG. 2 schematically illustrates a state in which the processor 901 is executing programs that realize the functions of the functional components of FIG. 1 .

[0016] Next, an example of the functional configuration of the data processing device 100 will be described with reference to FIG.

[0017] The search statement acquisition unit 101 acquires, for example, a search statement 501 from a user of the data processing device 100. The search statement 501 may be a sentence written in a natural language or information corresponding to a sentence written in a natural language. Similarly, a context 502, a query statement 505, and a response statement 506 shown below may also be a sentence written in a natural language or information corresponding to a sentence written in a natural language. Information corresponding to a sentence written in a natural language includes, for example, vector information obtained by vectorizing a sentence written in a natural language. The search statement acquisition unit 101 outputs the acquired search statement 501 to the context acquisition unit 102. The processing performed by the search statement acquisition unit 101 corresponds to a search statement acquisition process.

[0018] The context acquisition unit 102 acquires a search statement 501 from the search statement acquisition unit 101. Then, the context acquisition unit 102 acquires pairs of multiple contexts 502 and evaluation indices 503 from the knowledge source 111 by performing a search using the search statement 501. The evaluation index 503 is an index that serves as a criterion for evaluating the context 502. The evaluation index 503 is, for example, a search score. The evaluation index 503 is, for example, a cosine similarity. The evaluation index 503 may be, for example, an index obtained by weighting and adding the search score and the cosine similarity. The context acquisition unit 102 outputs the pairs of multiple contexts 502 and evaluation indices 503 to the grouping unit 103. The context acquisition unit 102 also outputs the search statement 501 and the pairs of multiple contexts 502 and evaluation indices 503 to the negative example acquisition unit 104. The processing performed by the context acquisition unit 102 corresponds to a context acquisition process.

[0019] The knowledge source 111 is, for example, a document database. Alternatively, the knowledge source 111 may be a knowledge base. The knowledge source 111 outputs a context 502 corresponding to a search statement 501 and an evaluation index 503 of the context 502 to the context acquisition unit 102. FIG. 1 shows an example in which the knowledge source 111 is arranged inside the data processing device 100. The knowledge source 111 may be arranged outside the data processing device 100. When the knowledge source 111 is arranged outside the data processing device 100, the context acquisition unit 102 accesses the knowledge source 111 via a network and receives the context 502 and the evaluation index 503 from the knowledge source 111.

[0020] The grouping unit 103 acquires pairs of multiple contexts 502 and evaluation indices 503 from the context acquisition unit 102. Then, the grouping unit 103 groups the multiple contexts 502 into multiple groups in descending order of the evaluation indices 503. The grouping unit 103 outputs grouping results 504 to the negative example acquisition unit 104. The grouping results 504 indicate, for example, the identifiers of the contexts 502 included in each group. As a result of grouping the multiple contexts 502 by the grouping unit 103, multiple contexts 502 may be included in one group, or one context 502 may be included in one group. The processing performed by the grouping unit 103 corresponds to grouping processing.

[0021] The negative example acquisition unit 104 acquires a search statement 501 and a plurality of sets of contexts 502 and evaluation indices 503 from the context acquisition unit 102. The negative example acquisition unit 104 also acquires a grouping result 504 from the grouping unit 103. The negative example acquisition unit 104 selects one of the groups within a predetermined upper and lower limit range in descending order of evaluation index 503, excluding the group with the highest evaluation index 503. The predetermined upper and lower limit range is a selection range of groups from which negative examples are to be acquired. The predetermined upper and lower limit range is, for example, the top one-third range. Note that the predetermined upper and lower limit range is not limited to the top one-third range. The predetermined upper and lower limit range may be, for example, the top one-quarter range or the top one-fifth range. Furthermore, the predetermined upper and lower limit range may be, for example, the top one-Nth range, such as the top 3.3th range or the top √5th range (where N is a positive real number greater than 1). Furthermore, the range of the predetermined upper and lower limits can be, for example, 20% from the top and 80% from the top. In this way, the predetermined upper and lower limit ranges can be arbitrarily set to generate suitable negative examples. The predetermined upper and lower limit ranges can be arbitrarily set by, for example, a user. The predetermined upper and lower limit ranges can be set based on the statistics of the evaluation index 503. Specifically, when the user sets the predetermined upper and lower limit ranges, for example, the upper limit value can be set to a range smaller than 0.95 times the maximum value of the evaluation index 503 and larger than 1.1 times the average value of the evaluation index 503, and the lower limit value can be set to a range smaller than 0.9 times the average value of the evaluation index 503 and larger than 1.05 times the minimum value of the evaluation index 503. The maximum, minimum, and average values ​​of the evaluation index 503 can be calculated from multiple evaluation indexes 503. The average value may be a median or a mode. In this way, by setting a predetermined upper and lower limit range based on the statistics of the evaluation index 503, it is possible to prevent undesirable groups (for example, groups for which positive examples have been obtained, groups with extremely low evaluation indexes, groups containing unnatural sentences, etc.) from being mistakenly included in the groups from which negative examples are to be obtained.The upper and lower limit ranges may also be determined using a trained model obtained by machine learning. Examples of machine learning include neural networks, support vector machines, and decision trees. The trained model can be obtained by machine learning, for example, using a large number of pre-given search sentences 501 and multiple sets of contexts 502 and evaluation indices 503. The reason why the negative example acquisition unit 104 excludes the group with the highest evaluation index 503 from the group from which negative examples are to be acquired is that this group is not suitable for generating negative examples because it is likely to have obtained semantically correct output sentences. However, if the evaluation index of the group with the highest evaluation index 503 is low, it is unlikely that the output sentence is semantically correct, so the group with the highest evaluation index 503 does not need to be excluded. In other words, if the evaluation index of the group with the highest evaluation index 503 is low, it may be included in the group from which negative examples are to be acquired. A low evaluation index 503 refers to a case where the evaluation index 503 is less than a predetermined threshold. For example, if the evaluation index 503 is expressed as cosine similarity, the predetermined threshold is 0.5. The predetermined threshold may be changed as appropriate depending on, for example, the degree of mode of the search statement 501. The degree of mode of the search statement 501 may be, for example, the degree of complexity or ambiguity of the sentence used for the search. Then, the negative example acquisition unit 104 generates a query statement 505 for the language model 112 using the context 502 belonging to the selected group and the search statement 501. Furthermore, the negative example acquisition unit 104 inputs the query statement 505 into the language model 112 and acquires a response statement 506 to the query statement 505 from the language model 112. The negative example acquisition unit 104 outputs the search statement 501 and the response statement 506 to the management unit 105. The negative example acquisition unit 104 corresponds to a query statement generation unit and a response statement acquisition unit. Furthermore, the processing performed by the negative example acquisition unit 104 corresponds to a query statement generation process and a response statement acquisition process.

[0022] The language model 112 is, for example, a large-scale language model. The language model 112 generates a response sentence 506 corresponding to the query sentence 505 and returns the response sentence 506 to the negative example acquisition unit 104. FIG. 1 shows an example in which the language model 112 is arranged inside the data processing device 100. The language model 112 may be arranged outside the data processing device 100. When the language model 112 is arranged outside the data processing device 100, the negative example acquisition unit 104 accesses the language model 112 via a network and transmits the query sentence 505 to the language model 112. The language model 112 generates a response sentence 506 corresponding to the query sentence 505 and transmits the response sentence 506 to the negative example acquisition unit 104 via the network.

[0023] The management unit 105 acquires the search statement 501 and the response statement 506 from the negative example acquisition unit 104. The management unit 105 associates the search statement 501 with the response statement 506 and manages them as negative examples. More specifically, the management unit 105 associates the search statement 501 with the response statement 506 and stores them as negative examples in the negative example database 113. The processing performed by the management unit 105 corresponds to management processing.

[0024] The negative example database 113 stores the search sentence 501 and the response sentence 506 in association with each other as negative examples. The search sentence 501 and the response sentence 506 stored in the negative example database 113 are used as negative example training data for supervised additional learning of the language model. Note that the supervised additional learning may be performed on the language model 112, or on a language model other than the language model 112. Furthermore, the supervised additional learning may be performed by in-context learning or fine tuning. FIG. 1 shows an example in which the negative example database 113 is located inside the data processing device 100. The negative example database 113 may also be located outside the data processing device 100. If the negative example database 113 is located outside the data processing device 100, the management unit 105 accesses the negative example database 113 via a network and transmits the search sentence 501 and the response sentence 506 to the negative example database 115. Furthermore, the negative example database 113 may be located in multiple locations outside the data processing device 100. For example, the search statement 501 and the response statement 506 may be stored separately in multiple locations. The management unit 105 can use a correspondence table showing the correspondence between the search statement 501 and the response statement 506 to read out the search statement 501 and the response statement 506 in association with each other from the negative example database 113 that is distributed and stored in multiple locations.

[0025] ***Explanation of Operation*** Fig. 3 shows an example of operation of the data processing device 100 according to this embodiment. With reference to Fig. 3, an example of operation of the data processing device 100 will be described.

[0026] First, in step S101, the search statement acquisition unit 101 acquires a search statement 501 from, for example, the user of the data processing device 100. Then, the search statement acquisition unit 101 outputs the acquired search statement 501 to the context acquisition unit .

[0027] Next, in step S102, the context acquisition unit 102 acquires a search statement 501 from the search statement acquisition unit 101. Then, the context acquisition unit 102 acquires pairs of multiple contexts 502 and evaluation indices 503 from the knowledge source 111 by performing a search using the search statement 501. Then, the context acquisition unit 102 outputs the pairs of multiple contexts 502 and evaluation indices 503 to the grouping unit 103. In addition, the context acquisition unit 102 outputs the search statement 501 and the pairs of multiple contexts 502 and evaluation indices 503 to the negative example acquisition unit 104.

[0028] Next, in step S103, the grouping unit 103 acquires pairs of multiple contexts 502 and evaluation indices 503 from the context acquisition unit 102. Furthermore, the grouping unit 103 groups the multiple contexts 502 into multiple groups in descending order of the evaluation indices 503. For example, the grouping unit 103 groups the multiple contexts 502 into approximately 10 to 20 groups. The grouping unit 103 outputs grouping results 504 to the negative example acquisition unit 104.

[0029] Next, in step S104, the negative example acquisition unit 104 acquires a search statement 501 and a plurality of sets of contexts 502 and evaluation indices 503 from the context acquisition unit 102. The negative example acquisition unit 104 also acquires a grouping result 504 from the grouping unit 103. The negative example acquisition unit 104 then generates a query statement 505. Specifically, the negative example acquisition unit 104 selects one of the groups in the top one-third range in order of evaluation index 503 quality, excluding the group with the best evaluation index 503. For example, if 10 groups have been generated by the grouping unit 103, the negative example acquisition unit 104 selects one of the groups with the second to fourth best evaluation indices 503. In this case, the negative example acquisition unit 104 may select only one group from the second to fourth best groups, or may select multiple groups. Then, the negative example acquisition unit 104 uses the contexts 502 belonging to the selected group and the search statement 501 to generate a query statement 505 for the language model 112. The negative example acquisition unit 104 may use all of the contexts 502 belonging to the selected group, or may use only some of the contexts 502.

[0030] Next, in step S105, the negative example acquisition unit 104 inputs the query statement 505 to the language model 112 and acquires a response statement 506 to the query statement 505 from the language model 112. Then, the negative example acquisition unit 104 outputs the search statement 501 and the response statement 506 to the management unit 105.

[0031] Finally, in step S106, the management unit 105 acquires the search statement 501 and the response statement 506 from the negative example acquisition unit 104. The management unit 105 then associates the search statement 501 with the response statement 506 and manages them as negative examples in the negative example database 113.

[0032] Next, each step in FIG. 3 will be described using a specific example.

[0033] Here, in step S101, the search statement acquisition unit 101 acquires a search statement 501 such as, "Please tell me about a zoo in ABC city that has alpacas." The search statement acquisition unit 101 outputs the acquired search statement 501 to the context acquisition unit 102.

[0034] Next, in step S102, it is assumed that the context acquisition unit 102 acquires a pair of a context 502 and an evaluation index 503 illustrated in FIG. 4 from the knowledge source 111 by performing a search using the search statement 501. Here, the explanation will be given assuming that the evaluation index 503 is a search score ranging from "0.0" to "9.9" in increments of "0.1". The higher the numerical value of the evaluation index 503, the higher the evaluation of the context 502. The context acquisition unit 102 outputs the pair of the context 502 and the evaluation index 503 illustrated in FIG. 4 to the grouping unit 103. The context acquisition unit 102 also outputs the search statement 501 and the pair of the context 502 and the evaluation index 503 illustrated in FIG. 4 to the negative example acquisition unit 104.

[0035] Next, in step S103, the grouping unit 103 acquires pairs of contexts 502 and evaluation indices 503 illustrated in Fig. 4 from the context acquisition unit 102. Then, the grouping unit 103 groups the multiple contexts 502 into multiple groups in descending order of the evaluation indices 503. Here, it is assumed that the grouping unit 103 groups the contexts 502 into 10 groups as follows: 1) Group 1, where the evaluation index 503 is between "0.0" and "0.9"; 2) Group 2, where the evaluation index 503 is between "1.0" and "1.9"; 3) Group 3, where the evaluation index 503 is between "2.0" and "2.9"; 4) Group 4, where the evaluation index 503 is between "3.0" and "3.9"; 5) Group 5, where the evaluation index 503 is between "4.0" and "4.9"; 6) Group 6, where the evaluation index 503 is between "5.0" and "5.9"; 7) Group 7, where the evaluation index 503 is between "6.0" and "6.9"; 8) Group 8, where the evaluation index 503 is between "7.0" and "7.9"; 9) Group 9, where the evaluation index 503 is between "8.0" and "8.9"; 10) Group 10, where the evaluation index 503 is between "9.0" and "9.9" The grouping unit 103 outputs the grouping result 504 to the negative example acquiring unit 104 .

[0036] Next, in step S104, the negative example acquisition unit 104 generates a query statement 505. Here, it is assumed that the negative example acquisition unit 104 selects groups 9 to 7 having the second to fourth best evaluation indexes 503. Then, the negative example acquisition unit 104 generates a query statement 505 for the language model 112 using a context 502 and a search statement 501 belonging to the selected groups 9 to 7. Here, it is assumed that the negative example acquisition unit 104 generates a query statement 505 shown in FIG. 6 using the search statement 501 and context 502 shown in FIG. 5.

[0037] Next, in step S105, the negative example acquisition unit 104 inputs the query sentence 505 to the language model 112 and acquires a response sentence 506 to the query sentence 505 from the language model 112. Here, it is assumed that the negative example acquisition unit 104 acquires the response sentence 506 illustrated in FIG.

[0038] Finally, in step S106, the management unit 105 associates the search sentence 501 with the response sentence 506 as shown in FIG. 8 and stores them as negative examples in the negative example database 113.

[0039] ***Explanation of Effects of the Embodiment*** In this embodiment, context 502 of a group in the top one-third range excluding the group with the highest evaluation index (the group ranked second to fourth best on a 10-point scale) is used. In other words, in this embodiment, context 502 that is related to search statement 501 but is not sufficient as a response to search statement 501 is used. As a result, this embodiment can obtain response sentence 506 that appears natural and correct at first glance but contains semantic errors. Therefore, according to this embodiment, it is possible to obtain high-quality negative examples that can be used for training a language model. Then, by performing supervised additional learning using the negative examples obtained in this manner, hallucination can be effectively suppressed.

[0040] In this embodiment, the acquisition of negative examples using the configuration shown in Fig. 1 has been described, but positive examples can also be acquired using the configuration shown in Fig. 1. When acquiring a positive example, the negative example acquisition unit 104 selects a group having the best evaluation index 503 from the grouping result 504. Then, the negative example acquisition unit 104 generates a query statement 505 for the language model 112 using a context 502 belonging to the selected group and a search statement 501. The negative example acquisition unit 104 also inputs the query statement 505 to the language model 112 and acquires a response statement 506 to the query statement 505 from the language model 112. Then, the management unit 105 associates the search statement 501 with the response statement 506 and stores them as a positive example in a positive example database (not shown in Fig. 1).

[0041] The procedure described in this embodiment is an example. Therefore, only a part of the procedure described in this embodiment may be implemented. Also, at least a part of the procedure described in this embodiment may be implemented in combination with a procedure not described in this embodiment. Also, the configuration and procedure described in this embodiment may be changed as necessary.

[0042] *** Supplementary Explanation of Hardware Configuration *** Here, a supplementary explanation of the hardware configuration of the data processing device 100 will be provided. The processor 901 shown in FIG. 2 is an IC (Integrated Circuit) that performs processing. The processor 901 is a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or the like. The main storage device 902 shown in FIG. 2 is a RAM (Random Access Memory). The auxiliary storage device 903 shown in FIG. 2 is a ROM (Read Only Memory), a flash memory, an HDD (Hard Disk Drive), or the like. The communication device 904 shown in FIG. 2 is an electronic circuit that executes data communication processing. The communication device 904 is, for example, a communication chip or a NIC (Network Interface Card).

[0043] The auxiliary storage device 903 also stores an OS (Operating System). At least a portion of the OS is executed by the processor 901. The processor 901 executes programs that implement the functions of the functional components shown in FIG. 1 while executing at least a portion of the OS. The processor 901 executes the OS to perform task management, memory management, file management, communication control, and the like. At least one of information, data, signal values, and variable values ​​indicating the results of processing of the functional components shown in FIG. 1 is stored in at least one of the main storage device 902, the auxiliary storage device 903, and registers and cache memory within the processor 901. The programs that implement the functions of the functional components shown in FIG. 1 may be stored on a portable recording medium such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. Portable recording media storing the programs that implement the functions of the functional components shown in FIG. 1 may be distributed.

[0044] Furthermore, the term "unit" of at least one of the functional components shown in FIG. 1 may be read as a "circuit," a "step," a "procedure," a "process," or a "circuitry." Furthermore, the data processing device 100 may be realized by a processing circuit. The processing circuit may be, for example, a logic IC (Integrated Circuit), a GA (Gate Array), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array). In this case, the functional components shown in FIG. 1 are each realized as part of the processing circuit. In this specification, the generic concept of a processor and a processing circuit is referred to as "processing circuitry." In other words, a processor and a processing circuit are each specific examples of "processing circuitry."

[0045] Finally, various aspects of the present disclosure are summarized as appendices. (Appendix 1) A data processing device comprising: a search statement acquisition unit that acquires a search statement; a context acquisition unit that acquires multiple contexts and evaluation indicators for each of the multiple contexts by a search using the search statement; a grouping unit that groups the multiple contexts into multiple groups in order of best evaluation indicator; a query statement generation unit that selects a group from groups within a predetermined upper and lower limit range in order of best evaluation indicator, and generates a query statement for a language model using one or more contexts belonging to the selected group and the search statement; a response statement acquisition unit that inputs the query statement to the language model and acquires a response statement to the query statement from the language model; and a management unit that associates the search statement with the response statement and manages them. (Appendix 2) The data processing device according to Appendices 1, wherein the management unit manages the search statement and the response statement as learning data for machine learning. (Supplementary Note 3) The data processing device according to Supplementary Note 2, wherein the management unit manages the search statement and the response statement as learning data for at least one of in-context learning and fine-tuning. (Supplementary Note 4) The data processing device according to Supplementary Note 1, wherein the context acquisition unit acquires at least one of a search score and cosine similarity as the evaluation index. (Supplementary Note 5) A data processing method in which a computer acquires a search statement, the computer acquires a plurality of contexts and evaluation indexes for each of the plurality of contexts by a search using the search statement, the computer groups the plurality of contexts into a plurality of groups in order of best evaluation index, the computer selects one of the groups within a predetermined upper and lower limit range in order of best evaluation index, and generates a query statement for a language model using the search statement and one or more contexts belonging to the selected group, the computer inputs the query statement into the language model and acquires a response statement to the query statement from the language model, and the computer manages the search statement and the response statement in association with each other.(Supplementary Note 6) A data processing program that causes a computer to execute the following steps: a search statement acquisition process that acquires a search statement; a context acquisition process that acquires multiple contexts and evaluation indicators for each of the multiple contexts by searching using the search statement; a grouping process that groups the multiple contexts into multiple groups in order of best evaluation indicator; a query statement generation process that selects one of the groups within a predetermined upper and lower limit range in order of best evaluation indicator, and generates a query statement for a language model using one or more contexts belonging to the selected group and the search statement; a response statement acquisition process that inputs the query statement into the language model and acquires a response statement to the query statement from the language model; and a management process that associates the search statement with the response statement and manages them.

[0046] 100 Data processing device, 101 Search statement acquisition unit, 102 Context acquisition unit, 103 Grouping unit, 104 Negative example acquisition unit, 105 Management unit, 111 Knowledge source, 112 Language model, 113 Negative example database, 501 Search statement, 502 Context, 503 Evaluation index, 504 Grouping result, 505 Query statement, 506 Response statement, 901 Processor, 902 Main memory device, 903 Auxiliary memory device, 904 Communication device.

Claims

1. A data processing device having: a search statement acquisition unit that acquires a search statement; a context acquisition unit that acquires multiple contexts and evaluation indicators for each of the multiple contexts by searching using the search statement; a grouping unit that groups the multiple contexts into multiple groups in order of best evaluation indicator; a query statement generation unit that selects a group from groups within a predetermined upper and lower limit range in order of best evaluation indicator, and generates a query statement for a language model using one or more contexts belonging to the selected group and the search statement; a response statement acquisition unit that inputs the query statement into the language model and acquires a response statement to the query statement from the language model; and a management unit that associates and manages the search statement and the response statement.

2. The data processing device according to claim 1, wherein the management unit manages the search sentences and the response sentences as learning data for machine learning.

3. The data processing device according to claim 2, wherein the management unit manages the search sentences and the response sentences as learning data for at least one of in-context learning and fine tuning.

4. The data processing device according to claim 1, wherein the context acquisition unit acquires at least one of a search score and a cosine similarity as the evaluation index.

5. A data processing method in which a computer acquires a search statement, the computer acquires multiple contexts and evaluation indicators for each of the multiple contexts by searching using the search statement, the computer groups the multiple contexts into multiple groups in order of best evaluation indicator, the computer selects one of the groups within a predetermined upper and lower limit range in order of best evaluation indicator, and generates a query statement for a language model using one or more contexts belonging to the selected group and the search statement, the computer inputs the query statement into the language model and acquires a response statement to the query statement from the language model, and the computer manages the search statement and the response statement in association with each other.

6. A data processing program that causes a computer to execute the following steps: a search statement acquisition process that acquires a search statement; a context acquisition process that acquires multiple contexts and evaluation indicators for each of the multiple contexts by searching using the search statement; a grouping process that groups the multiple contexts into multiple groups in order of best evaluation indicator; a query statement generation process that selects a group from groups within a predetermined upper and lower limit range in order of best evaluation indicator, and generates a query statement for a language model using one or more contexts belonging to the selected group and the search statement; a response statement acquisition process that inputs the query statement into the language model and acquires a response statement to the query statement from the language model; and a management process that associates and manages the search statement and the response statement.

Images