Language model creation support device, method and program

Abstract

To provide a language model creation support technology that realizes a specific task considering permissible preparation cost and requested target accuracy.SOLUTION: A language model creation support device 10 comprises: a pre-learning unit 15 that performs pre-learning of a large-scale language model 11 on the basis of teacher data 12, in which an example question and an answer previously prepared are combined, to create a verification model 16; a verification unit 17 that verifies accuracy of the verification model 16 on the basis a probability of a right solution derived from this example question; a determination unit 18 that determines a creation policy of a processing model 31 having target level accuracy on the basis of a verification result of the verification model 16; a first creation unit 21 that creates the processing model 31 in which the teacher data 12 is extended to learn on the basis of this creation policy; and a second creation unit 22 that creates a similar determination model 32 which is vector-converted from the teacher data 12, on the basis of this creation policy.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] SUMMARY OF THE DISCLOSURE The present invention relates to a technique for supporting the creation of a language model specialized for a particular task. [Background technology]

[0002] Language models are used in a wide variety of fields, such as estimating the category of newspaper articles, classifying text such as social media posts and product reviews into sentiment categories such as positive, negative, and neutral, and responding to FAQs in chatbots. When using them for specific purposes such as those mentioned above, it is necessary to create a language model specialized for that purpose (task) one by one (for example, a text classification model for ____ or an FAQ model for ____).

[0003] In recent years, large-scale language models (LLMs) that have been trained using large amounts of language data have been developed. Based on pre-trained LLMs, fine-tuning and few-shot learning can be performed using teacher data appropriate for the task to create language models specialized for specific tasks.

[0004] In creating such a language model specialized for a specific task, there is generally a problem that the accuracy is low due to a lack of training data. As a known technique to address this problem, there is a method of expanding language data. Specifically, a method of creating (expanding) language data using an inferable LLM without fine tuning using GPT-3 (Generative Pre-trained Transformer 3) is known.

[0005] There are also known techniques for expanding language data by replacing multiple tokens that make up the language data, and for creating training data by generating token sequences using data expansion operators. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2023-018624 [Patent Document 2] Patent Publication No. 2023-039102 [Patent Document 3] Patent Publication No. 2022-171502 Summary of the Invention [Problem to be solved by the invention]

[0007] In order to improve the accuracy of a language model, there are methods such as extending existing language data to create new language data or adding new language data created manually, as described above. However, there are cases where the difficulty of extending or creating such language data is high, or where adding such new language data does not improve the accuracy of the language model.

[0008] An embodiment of the present invention has been made in consideration of these circumstances, and aims to provide a technology for supporting the creation of a language model that achieves a specific task, taking into account an acceptable preparation cost and a required target accuracy. [Means for solving the problem]

[0009] In a language model creation support device according to an embodiment, the device includes a pre-learning unit that pre-learns a large-scale language model using teacher data that combines previously prepared example questions and correct answers to create a verification model, a verification unit that verifies the accuracy of the verification model based on the probability of the correct answer derived from the example questions, a determination unit that determines a creation policy for a processing model having a target level of accuracy based on a verification result of the verification model, a first creation unit that creates the processing model by expanding and learning the teacher data based on the creation policy, and a second creation unit that creates a similarity determination model by vector-converting the teacher data based on the creation policy. Effect of the Invention

[0010] According to an embodiment of the present invention, a technology for supporting the creation of a language model that realizes a specific task is provided, taking into account an acceptable preparation cost and a required target accuracy. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram of a language model creation assistance device according to a first embodiment of the present invention. [Diagram 2] This dataset shows training data that combines example questions and correct answers for classifying posted text into low and high ratings for a task of rating restaurants. [Diagram 3] A dataset that expands on the training data shown in Figure 2 and shows the expanded combination of example questions and correct answers. [Figure 4] (A) A graph showing the relationship between the number of training data sets used to create the validation model and the accuracy rate. The results indicate that expanding the number of training data sets (to 300) will lead to the creation of a language model with the target level of accuracy (80% accuracy rate). (B) The results indicate that it will be difficult to achieve the target level of accuracy (80% accuracy rate) for the language model even if the number of training data sets is expanded. [Diagram 5] A template showing examples of training data before vector conversion and combinations with correct answers for the task of classifying newspaper articles into categories. [Figure 6] A list showing combinations of newspaper article questions and category answers processed by a similarity judgment model in a task of classifying newspaper articles into categories. [Figure 7] FIG. 1 is a block diagram of a generation system incorporating a model created by a language model creation assistance device according to each embodiment. [Figure 8] FIG. 2 is an explanatory diagram of the operation of a system for generating answer data from question data. [Figure 9] FIG. 11 is a block diagram of a language model creation assistance device according to a second embodiment of the present invention. [Figure 10] 3 is a flowchart showing steps of a language model creation support method and an algorithm of a language model creation support program according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. FIG. 1 is a block diagram of a language model creation support device 10A (10) according to a first embodiment of the present invention. As described above, the language model creation support device 10A (10) includes a pre-learning unit 15 that pre-learns a large-scale language model 11 using teacher data 12 that combines example questions and correct answers (see FIG. 2) prepared in advance to create a verification model 16, a verification unit 17 that verifies the accuracy of the verification model 16 based on the probability of a correct answer derived from the example questions (FIG. 4; correct answer rate), a determination unit 18 that determines a creation policy for a processing model 31 having a target level of accuracy based on the verification result of the verification model 16, a first creation unit 21 that creates a processing model 31 by expanding and learning the teacher data 12 based on the creation policy, and a second creation unit 22 that creates a similarity determination model 32 by vector-transforming the teacher data 12 based on the creation policy.

[0013] The large-scale language model 11 (LLM) can be any model, for example, a pre-trained model such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer)-3, GPT-4, etc. The large-scale language model 11 is stored in a data repository (not shown) together with training data 12 in text format.

[0014] FIG. 2 shows a dataset showing teacher data 12 that combines example questions and correct answers for classifying posted text into high and low ratings in a task of rating restaurants. In this task, information on restaurants through posted text on the Internet is classified into two categories: high and low ratings. Of the multiple datasets that make up teacher data 12 thus prepared in advance, a portion is used by pre-learning unit 15, and the remainder is used by verification unit 17. Here, each of the example questions and questions (question data 14 described below) is text data, and text data with a correct answer is called an example question, and that without a correct answer is called a question.

[0015] The pre-learning unit 15 pre-learns the large-scale language model 11 using the teacher data 12 to create a verification model 16. The large-scale language model 11 to be used will be described using BERT and GPT-3 as examples, but is not limited to these. The large-scale language model 11 to be used may be selected in advance by the user, or may be extracted from multiple models with the highest accuracy or the best balance between cost and accuracy.

[0016] As pre-learning, for example, fine tuning can be performed in the case of BERT, and in the case of GPT-3, short-shot learning can be performed in addition to fine tuning. Fine tuning is exemplified below. If the total number of data sets of pre-prepared teacher data 12 is 60, pre-learning is performed by changing the number of data to be used, such as 10, 20, 30, etc., to create multiple validation models 16.

[0017] The verification unit 17 verifies the accuracy of the multiple verification models 16 created based on the probability of a correct answer (correct answer rate) derived from example questions, preferably not used in pre-learning, in the teacher data 12. The verification unit 17 desirably uses a data set that was not used in creating the target verification model 16, out of the data sets of the teacher data 12. The answers to the example questions in the multiple data sets used for verification are statistically processed to determine whether they are correct or incorrect, and the correct answer rate of the verification model 16 is derived.

[0018] Fig. 4 is a graph showing the relationship between the number of teacher data 12 (X-axis) used to create the verification model 16 and the accuracy rate (Y-axis). Fig. 4(A) shows the determination result of a creation policy for a processing model 31 having a target level of accuracy (80% accuracy rate) by expanding the number of teacher data 12 (to 300). Fig. 4(B) shows the determination result that it is difficult to make the accuracy of the processing model 31 reach the target level (80% accuracy rate) even if the number of teacher data 12 is expanded.

[0019] Since the number of teacher data 12 is limited, in the range exceeding this number, an extrapolation curve is applied to verify the relationship between the number of teacher data 12 and the accuracy rate. Generally, the accuracy rate improves as the number of teacher data 12 increases, but when a certain level is reached, the accuracy rate becomes saturated. In FIG. 4(A), the accuracy can reach the target level (accuracy rate 80%) before the accuracy rate becomes saturated. In this case, the accuracy can be made to reach the target level by increasing the number of teacher data. On the other hand, in FIG. 4(B), the accuracy rate becomes saturated before the accuracy target level (accuracy rate 80%) is reached. In this case, even if the number of teacher data is increased, the accuracy cannot be made to reach the target level.

[0020] Even if the teacher data 12 is the same, the relationship between the number of teacher data 12 and the accuracy rate changes depending on what is used in the large-scale language model 11. Also, even if the large-scale language model 11 is the same, when the task is different and the teacher data 12 used is different, the relationship between the number of teacher data 12 and the accuracy rate changes.

[0021] In addition, the verification unit 17 can improve the verification quality of the relationship between the number of teacher data 12 and the accuracy rate by considering other methods in addition to the method using the extrapolation curve described above. Specifically, the characteristics of the text constituting the examples of the teacher data 12 are analyzed. This analysis is performed by extracting the number of sentence data of the examples constituting the data set and the distribution of sentence data of the examples corresponding to each evaluation (correct answer). In addition, the analysis is performed by evaluating the frequency of words, performing vectorization processing considering the context and semantic relationship of each sentence, and extracting the distribution of similarity for each evaluation label and each sentence.

[0022] By carrying out the text feature analysis of the teacher data 12 in this way, it is possible to predict the effect of the extension process (such as the degree of increase in the accuracy rate) in the extension unit 25 described below. In other words, it is possible to accurately estimate the number of teacher data 12 required to reach a target level of accuracy. Alternatively, it is possible to weight the prediction of the effect of the extension process (such as the degree of increase in the accuracy rate) according to the results of the text feature analysis.

[0023] Furthermore, we consider the case of creating a language model specialized for a different task using the same large-scale language model 11. In this case, if the text feature analysis results of the teacher data 12 are similar, it is possible to verify by analogy application the existing relationship between the number of teacher data 12 and the accuracy rate without going through a pre-learning procedure again.

[0024] The determination unit 18 determines a policy for creating a processing model 31 having a target level of accuracy based on the verification result of the verification model 16. That is, the number of teacher data 12 required to ensure the accuracy of the task to be realized is determined from the relationship between the number of teacher data 12 in the verification model 16 and the accuracy rate. Then, if the number of teacher data 12 can be realistically expanded, the policy proceeds to the first creation unit 21 to create the processing model 31. On the other hand, if the number of teacher data 12 cannot be realistically expanded, the policy proceeds to the second creation unit 22 to create a similarity determination model 32.

[0025] In this way, when the creation of a processing model 31 having a target level of accuracy is abandoned, a similarity determination model 32 is created. In this case, a highly accurate verification model 16 from among those pre-trained may be adopted as the processing model 31, and compared with the processing result by the similarity determination model 32. Alternatively, when processing by the similarity determination model 32 is impossible, the pre-trained verification model 16 may be selected.

[0026] The first creation unit 21 has an extension unit 25 that performs extension processing on example text data in the teacher data 12, and a learning unit 26 that performs learning using the extended teacher data 12 and the large-scale language model 11 to create a processing model 31. In this way, the first creation unit 21 extends the teacher data 12 based on the creation policy of the determination unit 18, and creates the learned processing model 31.

[0027] Figure 3 is a data set showing a combination of example questions and correct answers of teacher data 12a expanded from Figure 2. The evaluations (correct answers) of the expanded example questions of teacher data 12a shown in Figure 3 correspond to the evaluations (correct answers) of the example questions of the original teacher data 12 shown in Figure 2.

[0028] The learning unit 26 expands the teacher data to the number (300) required to reach the target level of accuracy (80% in FIG. 4). The text data expansion processing method includes word replacement, translation / reverse translation, and a method using a pre-trained model, and a method is selected according to the task.

[0029] The learning unit 26 creates a processing model 31 by adding the expanded teacher data 12a to the original teacher data 12. The learning unit 26 performs fine tuning and short-shot learning on the large-scale language model 11 shared with the pre-learning unit 15.

[0030] The second creation unit 22 is composed of a vector conversion unit 27 that converts each word in the teacher data 12 into a numerical value based on rules, and a template creation unit 28 that creates a template of a data set combining correct answers corresponding to the vector-converted example questions. In this way, the second creation unit 22 creates a similarity determination model 32 in which the teacher data 12 is vector-converted based on the creation policy of the determination unit 18.

[0031] Fig. 5 is a template showing combinations of example questions and correct answers of training data 12 before vector conversion in a task of classifying newspaper articles by category. Fig. 6 is a list showing combinations of newspaper article questions and category answers processed by similarity determination model 32 in a task of classifying newspaper articles by category.

[0032] The method of vector-converting the teacher data 12 includes a method that focuses on the frequency of words, a method that takes into account meaning and context, etc. By creating a similarity judgment model 32 that uses the vector-converted teacher data 12 as a template in this way, it is possible to accurately answer new question data (high or low score) based on the similarity in meaning of the words that make up the data.

[0033] 7 is a block diagram of a system 40 incorporating a model (processing model 31 and / or similarity determination model 32) created by the language model creation support device 10 according to each embodiment. As described above, the system 40 includes an input unit 42 for inputting question data 41, a selection unit 45 for selecting the processing model 31 and the similarity determination model 32 stored in the model storage unit 30 that correspond to the input question data 41 (FIG. 6), and a processing unit 46 for processing the question data 41 with the selected processing model 31 or similarity determination model 32 and outputting answer data 47. The processing model 31 and / or the similarity determination model 32 stored in the language model storage unit 30 are stored in a form linked to the task of the question data 41.

[0034] When the processing model 31 is selected by the processing unit 46, answer data 47 having a target level of accuracy is generated in response to the input of question data 41.

[0035] On the other hand, when the similarity determination model 32 is selected by the processing unit 46, the question data 41 is vector-converted. Then, the similarity between the template example of the similarity determination model 32 and the question data 41 is compared between the vector-converted ones. Then, the category (correct answer) corresponding to the most similar template example is generated as answer data 47. Note that as a method for determining the similarity, for example, cosine similarity is used.

[0036] Moreover, if the processing by the similarity determination model 32 is judged to be invalid, the processing unit 46 generates the answer data 47 using the verification model 16 with the highest accuracy that has been adopted as the processing model 31 .

[0037] 8 is an explanatory diagram of the operation of system 40 that generates answer data 47 from question data 41. Suppose question data 41 related to a predetermined task (o× classification task...△△ FAQ task) is input to input unit 42. A language model (o× classification model...△△ FAQ model) corresponding to each task is selected in selection unit 45. Then, each language model is incorporated into processing unit 46, and system 40 becomes an o× document automatic creation system...△△ chatbot system.

[0038] According to this embodiment, answer data 47 with a high accuracy rate can be generated by increasing the amount of teacher data 12 by the extension process and learning the processing model 31. On the other hand, if there is a situation in which it is not possible to increase the amount of teacher data 12, answer data 47 can be generated by similarity determination using vector conversion. Also, if similarity determination using vector conversion is not possible, answer data 47 can be generated using a verification model 16 trained with teacher data 12 that has been prepared in advance. In this case, the accuracy rate verified by the verification unit 17 can be added to the answer data 47.

[0039] Second embodiment Fig. 9 is a block diagram of a language model creation support device 10B (10) according to a second embodiment of the present invention. The language model creation support device 10B according to the second embodiment further includes an acquisition unit 35 and an update unit 33 in addition to the configuration of the language model creation support device 10A according to the first embodiment. In Fig. 9, parts having the same configuration or function as Fig. 1 are indicated by the same reference numerals, and duplicated explanations will be omitted.

[0040] The acquisition unit 35 acquires at least one of task information 36, a target level of accuracy 37, information 38 on the possibility of newly acquiring teacher data 12, and information 39 on the preparation cost required to satisfy the target level 37. These are used to determine a creation policy for the processing model 31.

[0041] The target level of accuracy 37 is the target level of accuracy of the answer data 47 generated by the processing model 31 given the question data 41. In the explanation of the first embodiment, the accuracy rate is set to 80%.

[0042] The new acquisition possibility information 38 is information about the expandability of the teacher data 12. That is, it is information about whether the amount of text-format teacher data 12 will increase in the future due to information on the Internet, questionnaires, manual creation, etc., or whether there are plans to add future data. Note that this possibility information 38 can be obtained in the form of sentences or words, or can be set as a numerical value, such as a five-point scale.

[0043] The preparation cost information 39 sets the time and cost spent on the extension, such as manually creating the teacher data 12 to be extended. For example, a condition such as "time and cost can be spent for two days within 100,000 yen" can be set. The preparation cost information 39 can be set not only by sentences or words, but also by numerical values ​​such as a five-point scale.

[0044] When the conditions of new acquisition possibility information 38 and preparation cost information 39 are not satisfied, even if the graph relationship shown in Fig. 4(A) is obtained, the determination unit 18 determines the creation policy to be taken by the second creation unit 22, not the first creation unit 21. By examining the task information 36, the target level 37, the possibility information 38, and the preparation cost information 39 in this way, an optimal creation support technology for a highly accurate language model is provided while taking into account the burden that will be incurred.

[0045] A specific example of the judgment by the judging unit 18 in the second embodiment is shown below. For example, assume that the target accuracy is 80% and it is predicted that the accuracy will be 80% by performing data expansion. In this case, even if it is not expected that the manually created teacher data 12 will be available in the future, if the cost and time required to expand the data to 300 data are within the cost set as a condition, the creation policy by the first creating unit 21 is determined.

[0046] On the other hand, if the accuracy by data expansion is below the target accuracy with a 50% prediction and further acquisition of teacher data is not expected, the second creation unit 22 determines the creation policy. Also, if the accuracy by data expansion clears the target level with an 80% prediction, but the number of additional teacher data 12 required is 3000 data, and the cost and time required exceed the set cost conditions of one week and 1 million yen, the second creation unit 22 determines the creation policy. Note that the above is just one example, and it is also possible to determine, for example, that the preparation cost information 39 is set to a fixed value and the number of teacher data 12 is determined to make the target level 37 as high as possible while taking into account the possibility information 38.

[0047] The update unit 33 evaluates the probability of a correct answer in the answer data 47, and further expands and re-learns the teacher data 12 as necessary to update the processing model 31. If the accuracy of the processing model 31 after the creation is evaluated in this way and does not reach the target level 37, the update unit 33 is operated with modification of the large-scale language model 11 in mind. Alternatively, in FIG. 4, the regression equation or weighting value may be updated so that the correspondence between the evaluated accuracy (correct answer rate) and the number of teacher data 12 after the expansion fits.

[0048] This not only makes it possible to maintain the accuracy rate of the answer data 47 by the processing model 31 at the target level 37, but also makes it possible to expect further improvement. Furthermore, it is possible to search for an optimal large-scale language model 11 for a specific task. Also, suggestions can be obtained for creation policies (policies for expanding the teacher data 12, policies for selecting the large-scale language model 11, etc.) for improving the accuracy rate of the processing model 31 linked to other tasks.

[0049] According to the language model creation support device 10B of the second embodiment, by acquiring information on task information 36, target level of accuracy 37, possibility information 38 of new acquisition of teacher data 12, and preparation cost information 39, it is possible to automatically determine data scalability based on teacher data 12 at hand and effectiveness of the created processing model 31, and automatically create an optimal processing model 31 that balances the cost to be borne and the obtained performance (accuracy). Furthermore, when an improvement in accuracy is not expected or cannot be obtained even if the number of teacher data 12 is increased, the accuracy of the language model can be improved by adding a process of determining similarity with template data by vector conversion.

[0050] The steps of the language model creation support method and the algorithm of the language model creation support program according to the embodiment will be described with reference to the flowchart of Fig. 10 (see Figs. 9 and 7 as appropriate). First, a plurality of teacher data 12 combining example questions and correct answers (see Fig. 2) are prepared in advance (S11).

[0051] Next, the large-scale language model 11 is pre-trained using some of the multiple teacher data 12 to create a verification model 16 (S12). Then, the accuracy rate of the verification model 16 is calculated using the remaining teacher data 12 that was not used in the pre-learning (S13). Then, the number of teacher data 12 used in the pre-learning is changed and (S12) to (S13) are repeated (S14; Yes, No).

[0052] Next, the change in accuracy rate with respect to changes in the number of teacher data 12 used in pre-learning is analyzed, and the number of teacher data 12 required to create a processing model 31 with a target level of accuracy is verified (S15; see FIG. 4). Then, if the required number of teacher data 12 can be prepared (S16; Yes), the teacher data 12 is expanded (S17), and a processing model 31 is created by learning (S18).

[0053] Next, if the required number of teacher data 12 cannot be prepared (S16; No), the existing teacher data 12 is vector-converted (S19) to create a similarity determination model 32 (S20). Then, the tasks are switched and steps (S11) to (S20) are repeated (S22; Yes, No), and the created processing model 31 and similarity determination model 32 are stored (S21).

[0054] Next, question data 41 for a predetermined task is input to the system 40 (FIG. 7) (S23). Then, a language model (processing model 31 or similarity judgment model 32) linked to the task of this question data 41 is selected from the stored ones (S24). Then, the question data 41 is processed by the selected processing model 31 or similarity judgment model 32, and answer data 47 is output (S25, S26; No, END). Note that if processing by the similarity judgment model 32 is not possible (S26; Yes), the question data 41 is output using the verification model 16 created in pre-learning (S27, END).

[0055] According to at least one of the embodiments of the language model creation support device described above, by having a first creation unit 21 that creates a processing model 31 by expanding and learning the teacher data 12, and a second creation unit 22 that creates a template 32 by vector-converting the teacher data 12 based on this creation policy, it is possible to provide a language model creation support technology that achieves a specific task, taking into account the allowable preparation costs and the required target accuracy.

[0056] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention.

[0057] The language model creation support device described above includes a control device that is a highly integrated processor such as a dedicated chip, an FPGA (Field Programmable Gate Array), a GPU (Graphics Processing Unit), or a CPU (Central Processing Unit), a storage device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), an external storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), a display device such as a display, an input device such as a mouse or a keyboard, and a communication I / F, and can be realized with a hardware configuration using a normal computer. Therefore, the components of the language model creation support device can also be realized by a computer processor and can be operated by a language model creation support program.

[0058] The language model creation assistance program is provided by being pre-installed in a ROM etc. Alternatively, the program may be provided by being stored in a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD) in the form of an installable or executable file.

[0059] The language model creation assistance program according to this embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. The language model creation assistance device may also be configured by combining separate modules that independently fulfill the functions of the components and are mutually connected via a network or dedicated lines. [Explanation of symbols]

[0060] 10 (10A, 10B)...language model creation support device, 11...large-scale language model, 12 (12a)...teacher data, 15...pre-learning unit, 16...verification model, 17...verification unit, 18...judgment unit, 21...first creation unit, 22...second creation unit, 25...extension unit, 26...learning unit, 27...vector conversion unit, 28...template creation unit, 30...language model storage unit, 31...processing model, 32...similarity judgment model, 33...update unit, 35...acquisition unit, 36...task information, 37...target level, 38...feasibility information, 39...preparation cost information, 40...system, 41...question data, 42...input unit, 45...selection unit, 46...processing unit, 47...answer data.

Claims

1. A pre-learning unit that pre-learns a large-scale language model using training data that combines example questions and correct answers prepared in advance, and creates a model for verification; a verification unit that verifies the accuracy of the verification model based on the probability of the correct answer derived from the example problem; a determination unit that determines a policy for creating a processing model having the target level of accuracy based on a verification result of the verification model; a first creation unit that creates the processing model by expanding and learning the teacher data based on the creation policy; a second creation unit that creates a similarity determination model by vector-converting the training data based on the creation policy.

2. 2. The language model creation assistance device according to claim 1, A language model creation support device comprising an acquisition unit that acquires at least one of task information, the target level of accuracy, information on the possibility of newly obtaining the teaching data, and preparation cost information required to meet the target level, which are used to determine the creation policy.

3. 3. The language model creation assistance device according to claim 1, A selection unit that selects one of the stored processing models and the stored similarity determination models that corresponds to the input question data; a processing unit that processes the question data using the selected processing model or similarity determination model and outputs answer data.

4. 4. The language model creation assistance device according to claim 3, A language model creation assistance device comprising an update unit that further expands the teaching data and re-learns and updates the processing model based on the probability of the correct answer in the answer data.

5. 3. The language model creation assistance device according to claim 1, The similarity determination model is created after abandoning the creation of the processing model based on the creation policy; a language model creation assistance device that adopts the verification model with the highest accuracy from among the pre-trained models as the processing model.

6. A step of pre-learning a large-scale language model using training data that combines example questions and correct answers prepared in advance, and creating a model for verification; verifying the accuracy of the validation model based on the probability of the correct answer derived from the example problem; determining a policy for creating a processing model having the target level of accuracy based on a verification result of the verification model; creating the processing model by expanding and learning the training data based on the creation policy; and creating a similarity determination model by vector-converting the training data based on the creation policy.

7. On the computer, A step of pre-learning a large-scale language model using training data that combines example questions and correct answers prepared in advance, and creating a model for verification; verifying the accuracy of the validation model based on the probability of the correct answer derived from the example problem; determining a policy for creating a processing model having the target level of accuracy based on a verification result of the verification model; creating the processing model by expanding and learning the training data based on the creation policy; a step of creating a similarity determination model by vector-converting the training data based on the creation policy.

Images