Adjustment device, adjustment method, and adjustment program

Abstract

An information processing device (10) has a calculation unit (152), a correction unit (154), and an update unit (155). The calculation unit (152) uses a model using BERT to calculate an output for each of a plurality of input vectors. The correction unit (154) corrects a vector to be input into a normalization layer included in the model such that the norm of the vector is constant. The update unit (155) updates the model such that the output is optimized.

Description

[Technical Field]

[0001] The present invention relates to an adjustment device, an adjustment method, and an adjustment program. [Background technology]

[0002] In recent years, natural language processing has been applied in various fields, including chatbots. BERT (Bidirectional Encoder Representations from Transformers) is a well-known machine learning model for natural language processing. BERT can perform tasks such as natural language translation with high accuracy.

[0003] BERT is a huge model with over 100 million parameters. Therefore, training BERT normally requires a huge dataset. However, it is sometimes possible to train BERT with a small dataset through a process called pretraining and fine-tuning.

[0004] In pre-training, parameters are trained using a large dataset, and in fine-tuning, parameters that have been pre-trained are used as initial values ​​and a small dataset specific to the task is used for training.

[0005] For example, in a business setting, if you obtain a pre-trained BERT model, you can use BERT simply by fine-tuning it according to the task. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] ON THE STABILITY OF FINE-TUNING BERT: MISCONCEPTIONS, EXPLANATIONS, AND STRONG BASELINES, [online], [Retrieved November 2, 2022], Internet (https: / / arxiv.org / pdf / 2006.04884.pdf) [Non-patent document 2] On Layer Normalization in the Transformer Architecture, [online], [Retrieved November 2, 2022], Internet (https: / / arxiv.org / pdf / 2002.04745.pdf) [Non-patent document 3] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, [online], [Retrieved November 2, 2022], Internet (https: / / arxiv.org / pdf / 1908.10084.pdf) Summary of the Invention [Problem to be solved by the invention]

[0007] However, the conventional techniques have a problem in that it is not easy to improve the accuracy of the model through fine tuning.

[0008] For example, Non-Patent Document 1 describes that when fine tuning is performed using a pre-trained BERT, the training becomes unstable (for example, accuracy varies significantly depending on the random number seed).

[0009] To improve the accuracy of a model through fine tuning, it is necessary to search for hyperparameters appropriate for the data. Furthermore, because the accuracy of fine tuning is highly dependent on the random number seed, it is necessary to try multiple seeds for each hyperparameter.

[0010] Trying multiple seeds for each hyperparameter is not easy because it requires a large learning cost (e.g., time). On the other hand, without searching for hyperparameters, it is difficult to improve the accuracy of the model. [Means for solving the problem]

[0011] In order to solve the above-mentioned problems and achieve the objectives, the adjustment device is characterized by having a calculation unit that calculates an output for each of a plurality of input vectors using a model using BERT, a correction unit that corrects the vectors input to a normalization layer included in the model so that the norm of the vectors becomes constant, and an update unit that updates the model so that the output is optimized. [Effects of the Invention]

[0012] According to the present invention, the accuracy of the model can be easily improved by fine tuning. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an adjustment device according to the first embodiment. [Figure 2] FIG. 2 is a schematic diagram showing the structure of a BERT. [Figure 3] FIG. 3 is a schematic diagram showing the structure of a Transformer. [Figure 4] FIG. 4 shows the results of a preliminary experiment. [Figure 5] FIG. 5 shows the results of a preliminary experiment. [Figure 6] FIG. 6 shows the results of a preliminary experiment. [Figure 7] FIG. 7 is a diagram illustrating the effect of the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating the effect of the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating the effect of the first embodiment. [Figure 10]FIG. 10 is a flowchart showing the flow of the fine tuning process. [Figure 11] FIG. 11 is a flowchart showing the processing flow by the Transformer. [Figure 12] FIG. 12 is a diagram illustrating an example of application of the first embodiment to a business chat. [Figure 13] FIG. 13 is a diagram illustrating an example of a computer that executes an adjustment program. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an adjustment device, an adjustment method, and an adjustment program according to the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0015] [Configuration of the first embodiment] First, the configuration of an adjustment device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of an adjustment device according to the first embodiment. An information processing device 10 shown in Fig. 1 is an example of an adjustment device. For example, the information processing device 10 is a personal computer, a server device, a smartphone, a tablet terminal, or the like.

[0016] The information processing device 10 can perform fine tuning of BERT. Furthermore, the information processing device 10 can use the fine-tuned BERT to perform tasks related to natural language processing, such as text reading, speech recognition, and translation.

[0017] In this embodiment, it is assumed that the information processing device 10 has already acquired information such as parameters that can construct a pre-trained BERT. However, the information processing device 10 may also perform pre-training of a BERT.

[0018] The information processing device 10 receives input of learning data for fine tuning, and also outputs information (for example, parameters) related to the fine-tuned BERT.

[0019] Furthermore, the information processing device 10 may receive input data for a task using BERT, and output the results of executing the task using the fine-tuned BERT.

[0020] As shown in FIG. 1, the information processing device 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15.

[0021] The communication unit 11 performs data communication with other devices via a network. For example, the communication unit 11 is a network interface card (NIC).

[0022] The input unit 12 receives data input from a user. The input unit 12 is, for example, an input device such as a mouse and a keyboard. Alternatively, the input unit 12 may be an interface that connects the information processing device 10 to the input device.

[0023] The output unit 13 outputs data by displaying it on a screen, etc. The output unit 13 is, for example, an output device such as a display and a speaker, etc. Alternatively, the output unit 13 may be an interface that connects the information processing device 10 to the output device.

[0024] The storage unit 14 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), an optical disk, etc. Note that the storage unit 14 may also be a data-rewritable semiconductor memory such as a RAM (Random Access Memory), a flash memory, or an NVSRAM (Non Volatile Static Random Access Memory). The storage unit 14 stores an OS (Operating System) and various programs executed by the information processing device 10.

[0025] The storage unit 14 stores model information 141 , correct answer information 142 , threshold value information 143 , and norm information 144 .

[0026] The model information 141 is information about a model. For example, the model information 141 is parameters for constructing a pre-trained BERT. For example, the parameters are weights and biases in a neural network included in the BERT.

[0027] The correct answer information 142 is information for performing fine tuning according to the task. The correct answer information 142 is a combination of text in a natural language and correct answer information corresponding to the text.

[0028] The correct answer information 142 may be a combination of text representing a user's utterance and a chatbot's response to the utterance. The correct answer information 142 may also be a combination of text in a first language (e.g., Japanese) and text translated from the first language into a second language (e.g., English).

[0029] The threshold information 143 is a threshold used when executing a task. How the threshold is used in a task will be described later.

[0030] The norm information 144 is a norm recorded during fine tuning. In fine tuning of BERT, a series of vectors is input to the model. At this time, calculation processing is repeatedly performed. For example, the calculation processing is repeated as many times as the number of vectors included in multiple series. The norm information 144 is a norm shared among each of the repeated calculation processing. A specific method for using the norm information 144 will be described later.

[0031] The control unit 15 controls the entire information processing device 10. The control unit 15 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The control unit 15 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory.

[0032] The control unit 15 functions as various processing units by running various programs. For example, the control unit 15 includes an acquisition unit 151, a calculation unit 152, a correction unit 154, and an update unit 155.

[0033] Here, the structure of the BERT in this embodiment and each processing unit of the control unit 15 will be described.

[0034] FIG. 2 is a schematic diagram showing the structure of a BERT. As shown in FIG. 2, Model 2, which is a BERT, has multiple Transformers 21. Model 2 also has a sequence of vectors (E1, E2, ..., E N ) is input. Model 2 also uses a sequence of vectors (T1, T2, ..., T N ) is output. For example, a sequence of vectors (E1,E2,…,E N ) and (T1,T2,…,T N ) represents a string of characters (e.g., a word) that makes up a sentence.

[0035] Model 2 in FIG. 2 has a two-layer structure, and in practice, one Transformer 21 is required for each layer, i.e., two Transformers 21 in total. For example, model information 141 includes parameters for the Transformer 21 in the first layer and parameters for the Transformer 21 in the second layer. Then, the Transformer 21 in the first layer is provided with vectors E1, E2, ..., E N are input sequentially.

[0036] In BERT, N outputs of the Transformer in the previous layer are input to each Transformer 21 from both directions (from left to right and from right to left in FIG. 2). For example, the Transformer 21 in the second layer receives the outputs from the Transformer 21 in the first layer, which are vectors E1, E2, ..., E N The N outputs corresponding to the

[0037] 3 is a schematic diagram showing the structure of a Transformer. As shown in Fig. 3, the Transformer 21 includes an attention layer 21a, an addition layer 21b, a normalization layer 21c, a feed-forward neural network (FFN) 21d, an addition layer 21e, and a normalization layer 21f.

[0038] The attention layer 21a is a neural network that functions as an attention mechanism. The addition layer 21b and addition layer 21e add up multiple input vectors. The FFN 21d is a neural network in which each unit outputs in only one direction (output side).

[0039] The normalization layer 21c and the normalization layer 21f perform layer normalization (Layer Norm). In this embodiment, the norm of a vector input to the normalization layer 21f may be corrected.

[0040] The configurations of the BERT and the Transformer of this embodiment are not limited to those described here. The configuration of the BERT may be the configuration described in Non-Patent Document 1 or a configuration similar to the configuration described in Non-Patent Document 1. The configuration of the BERT may also be the configuration described in Reference Document 1. Reference 1: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (https: / / arxiv.org / pdf / 1810.04805.pdf)

[0041] The configuration of the Transformer may be the configuration described in Non-Patent Document 2 (Post-LN Transformer layer or Pre-LN Transformer layer) or a configuration similar to that described in Non-Patent Document 2.

[0042] 3 performs vector normalization using a method similar to the Layer Norm described in Non-Patent Document 2. However, this embodiment differs from the method described in Non-Patent Document 2 in that the norms of the vectors input to the normalization layer 21c and normalization layer 21f may be corrected.

[0043] Here, as described in Non-Patent Document 1, a preliminary experiment conducted by the inventor to confirm that the fine tuning of the conventional BERT is unstable will be described.

[0044] In the experiment, a model called Pooling BERT (see, for example, Non-Patent Document 3) was used. The training data was Recognizing Textual Entailment (RTE). Figures 4, 5, and 6 show the results of the preliminary experiment.

[0045] The horizontal axis in Figure 4 is the number of iterations (number of steps) of the calculation process. The vertical axis in Figure 4 is the accuracy of the model after learning. The difference between the solid and dashed lines is the difference in the random number seed given during learning. The dashed line corresponds to the random number seed in cases where learning was successful (accuracy improved). The solid line corresponds to the random number seed in cases where learning failed (accuracy did not improve).

[0046] The horizontal axis in Figure 5 represents the number of iterations (steps) of the calculation process. The vertical axis in Figure 4 represents the gradient of the loss in the normalization layer during training. The gradient of the loss is calculated, for example, during the backpropagation procedure.

[0047] As shown in Figures 4 and 5, in cases where learning fails, the gradient of the loss in the normalization layer disappears.

[0048] The vertical axis in Figure 6 represents the number of iterations (number of steps) of the calculation process. The vertical axis in Figure 6 also represents the magnitude of the norm of the vector input to the normalization layer during learning. The dashed line corresponds to the random number seed for cases where learning was successful (accuracy improved). The solid line corresponds to the random number seed for cases where learning failed (accuracy did not improve).

[0049] As shown in Figure 6, in the case where learning fails, the norm increases rapidly at the step where the gradient vanishes.

[0050] From the results of the preliminary experiment, it can be said that the theory of equation (1) holds true.

[0051]

number

[0052] d is the dimension of the vector x input to the normalization layer. ||·|| is the norm of the vector. O is the order. Equation (1) shows that the derivative (gradient) of the normalization layer becomes smaller as the norm of the input vector becomes larger.

[0053] Furthermore, as proven in Lemma 3 of Non-Patent Document 2, the derivative of the loss with respect to the input vector is always proportional to the x derivative of the normalization layer due to the chain rule of differentiation, so the gradient disappears when the norm of x is large.

[0054] 1, the acquiring unit 151 acquires data necessary for fine tuning from the supervision information 142. For example, the acquiring unit 151 acquires a sequence of input vectors corresponding to text in a natural language according to a task, and a sequence of output vectors corresponding to supervision information corresponding to the text.

[0055] The calculation unit 152 sequentially inputs the sequence of input vectors acquired by the acquisition unit 151 to a pre-trained BERT constructed based on the model information 141. Then, the calculation unit 152 executes calculations using the BERT including the Transformer 21. The calculation unit 152 uses a model using the BERT to calculate an output for each of the multiple input vectors.

[0056] The correction unit 154 corrects the vectors so that the norms of the vectors input to the normalization layers included in the model are constant. The correction unit 154 performs the correction process at the timing when the vectors are input to the normalization layers 21c and 21f of the Transformer 21 during the calculation process by the calculation unit 152. The correction process by the correction unit 154 is called context normalization.

[0057] Furthermore, the correction unit 154 corrects the vector so that the norm of the vector input to a normalization layer that performs layer normalization and is included in the Transformer that constitutes the BERT becomes constant.

[0058] Here, it is assumed that the norm information 144 is initialized and the recorded norm is erased when the learning process starts. Therefore, when a vector is input to any normalization layer for the first time in the learning process, no norm is recorded in the norm information 144.

[0059] In the correction process, first, the correction unit 154 checks whether a norm is recorded in the norm information 144. If a norm is not recorded in the norm information 144, the correction unit 154 records the norm of the vector input to the normalization layer in the norm information 144. In this case, the correction unit 154 does not correct the norm.

[0060] On the other hand, if a norm is recorded in the norm information 144, the norm of the vector input to the normalization layer is corrected to the norm recorded in the norm information 144. In this case, the vector whose norm has been corrected is input to the normalization layer.

[0061] In this way, the correction unit 154 corrects the norm of a vector input to the normalization layer in a certain step so that the norm is equal to the norm of the vector input to the normalization layer in the previous step. For example, the correction process by the correction unit 154 is described in Pytorch as follows: if self.scale is not None: current_scale = context_layer.mean(0).norm().detach() context_layer = (context_layer / current_scale)* self.scale if self.training: self.scale = context_layer.mean(0).norm().detach()

[0062] In this way, by referring to the norm information 144, the correction unit 154 can correct the first vector so that the norm of the first vector input to the normalization layer included in the model is equal to the norm of the second vector last input to the normalization layer.

[0063] The update unit 155 updates the parameters of the model, i.e., the model information 141, so that the sequence of vectors output from the model approaches the sequence of output vectors acquired by the acquisition unit 151. In other words, the update unit 155 updates the model so that the output is optimized.

[0064] For example, the update unit 155 uses the backpropagation algorithm to update the parameters of the attention layer 21a and the FNN 21d of the Transformer 21. At that time, the update unit 155 calculates the derivatives of the losses of the normalization layer 21c and the normalization layer 21f.

[0065] The effects of the first embodiment will be described below. Figures 7, 8 and 9 are diagrams showing the effects of the first embodiment.

[0066] The vertical axis in Fig. 7 represents the number of iterations (number of steps) of the calculation process. The vertical axis in Fig. 7 also represents the magnitude of the norm of the vector input to the normalization layer during learning. The dashed line corresponds to the case where context normalization is not used (prior art). The solid line corresponds to the case where context normalization is used (first embodiment).

[0067] The horizontal axis in Fig. 8 represents the number of iterations (number of steps) of the calculation process. The vertical axis in Fig. 8 represents the gradient of the loss in the normalization layer during learning. The dashed line corresponds to the case where context normalization is not used (prior art). The solid line corresponds to the case where context normalization is used (first embodiment).

[0068] The horizontal axis in FIG. 9 represents the number of iterations (number of steps) of the calculation process. The vertical axis in FIG. 9 represents the accuracy of the model after learning. The difference between the solid line and the dashed line is the difference in the random number seed given during learning. The dashed line corresponds to the case where context normalization is not used (prior art). The solid line corresponds to the case where context normalization is used (first embodiment).

[0069] As can be seen from FIG. 7, in the first embodiment, the magnitude of the norm of the vector input to the normalization layer is constant and stable. Furthermore, as can be seen from FIG. 8, in the first embodiment, the disappearance of the gradient is prevented. Furthermore, as can be seen from FIG. 9, in the first embodiment, accuracy is improved. Note that in the example of FIG. 9, the learning result at the iteration number at which accuracy is highest due to early stopping may be adopted, and therefore the first embodiment can obtain a trained model with higher accuracy than conventional techniques.

[0070] [Processing of the first embodiment] 10 is a flowchart showing the flow of the fine-tuning process. The number of vector sequences in the fine-tuning is set to N (where N is an integer equal to or greater than 1).

[0071] 10, first, the information processing device 10 assigns 1 to i to initialize the norm information 144 (step S11). Note that the norm information 144 is initialized to a state in which no norm has been recorded. Then, the information processing device 10 inputs the i-th vector out of N vectors to a model (BERT) (step S12).

[0072] Here, the information processing device 10 executes calculations by each Transformer (step S13). Details of step S13 will be described later with reference to FIG.

[0073] Subsequently, the information processing device 10 updates the model based on the calculation result, that is, the vector output from the model (step S14).

[0074] If i=N (step S15, Yes), the information processing device 10 ends the process. On the other hand, if i=N is not true (step S15, No), the information processing device 10 increments i by 1 (step S16) and returns to step S12.

[0075] 11 is a flowchart showing the processing flow by the Transformer. The processing shown in FIG. 11 corresponds to step S13 in FIG.

[0076] 11, the information processing device 10 receives an input of a vector (step S131).The information processing device 10 determines whether or not there is a next layer (step S132).

[0077] For example, the processing order is assumed to be attention layer 21a, summation layer 21b, normalization layer 21c, FFN 21d, summation layer 21e, and normalization layer 21f. If there is a layer next to the layer for which processing has been completed, the information processing device 10 determines that there is a next layer.

[0078] For example, after the processing of the addition layer 21b is completed, the information processing device 10 determines that there is a next layer, the normalization layer 21c. Also, for example, after the processing of the normalization layer 21f is completed, the information processing device 10 determines that there is no next layer.

[0079] If there is no next layer (step S132, No), the information processing device 10 outputs the processed vector (step S139). For example, the information processing device 10 outputs the output of the normalization layer f as the processed vector.

[0080] If there is a next layer (Yes at step S132), the information processing device 10 determines whether the next layer is a normalization layer (the normalization layer 21c or the normalization layer 21f) (step S133).

[0081] If the next layer is not a normalization layer (step S133, No), the information processing device 10 executes processing on the next layer (step S138). If the next layer is a normalization layer (step S133, Yes), the information processing device 10 determines whether the norm has been recorded in the norm information 144 (step S134).

[0082] If the norm has not been recorded (step S134, No), the information processing device 10 records the norm of the vector input to the normalization layer (step S135). After the norm information 144 is initialized in step S11 of Fig. 10, the norm information 144 is in a state in which the norm has not been recorded. Then, in step S135, the norm information 144 transitions to a state in which the norm has been recorded.

[0083] If the norm has been recorded (step S134, Yes), the information processing device 10 corrects the norm of the vector input to the normalization layer based on the recorded norm (step S136). The information processing device 10 makes the norm of the vector input to the normalization layer equal to the recorded norm.

[0084] Then, the information processing device 10 performs layer normalization in the normalization layer. The norm of the vector to be subjected to layer normalization is constant.

[0085] [Example] An example in which the first embodiment is applied to a business chat will be described with reference to Fig. 12. Fig. 12 is a diagram for explaining an example in which the first embodiment is applied to a business chat.

[0086] A business chat application (for example, Slack) provided in the information processing device 10 acquires a first character string input by a user and a second character string indicating a skill. Next, the application inputs the acquired first character string (input sentence) and second character string (skill list) into Sentence Bert (an example of BERT), which has been trained for the entire language, and converts them into vectors.

[0087] The application then measures the distance between the vector representing the meaning of the character string and the vector representing the meaning of the skill by measuring the distance between the converted vectors using cosine distance. The application then selects a skill based on the measured distance. At this time, if the distance measured by measurement unit 123 is greater than the threshold indicated by threshold information 143, selection unit 124 selects general conversation.

[0088] A skill is a series of processes including the execution of a specific program, etc. On the other hand, a general conversation is a response to a user by outputting a voice, etc.

[0089] The information addition phase and operation phase executed by the information processing device 10 with respect to the application will be described below.

[0090] In the information addition phase, first, the information processing device 10 performs learning using Sentence Bert for the entire language. The information processing device 10 may acquire pre-trained Sentence Bert and store it as model information 141.

[0091] Next, the information processing device 10 performs fine tuning by the method of the first embodiment using a small number of input sentences and correct answer skills. In addition, the information processing device 10 determines a threshold value that maximizes accuracy, including for general conversations, and stores the threshold value information 143.

[0092] The operation phase will be described. First, the information processing device 10 acquires text input to an application. For example, if the application is Slack, the information processing device 10 acquires the text using Slack's official API called Bolt API.

[0093] Next, the information processing device 10 passes the text to the fine-tuned Sentence Bert and converts it into a vector. Similarly, the information processing device 10 passes the skills to the fine-tuned Sentence Bert and converts them into a vector. The application selects and executes a skill based on the cosine distance between the vectors.

[0094] For example, the information processing device 10 acquires the text "Let's work!" input into the application, and by selecting the skills to start work, it becomes possible to clock in to the company's internal system.

[0095] [Effects of the first embodiment] As described above, the information processing device 10 includes a calculation unit 152, a correction unit 154, and an update unit 155. The calculation unit 152 calculates an output for each of a plurality of input vectors using a model that uses BERT. The correction unit 154 corrects the vectors so that the norms of the vectors input to the normalization layer included in the model are constant. The update unit 155 updates the model so that the output is optimized.

[0096] Furthermore, the correction unit 154 corrects the vector so that the norm of the vector input to a normalization layer that performs layer normalization and is included in the Transformer that constitutes the BERT becomes constant.

[0097] Furthermore, the correction unit 154 corrects the first vector so that the norm of the first vector input to the normalization layer included in the model is equal to the norm of the second vector last input to the normalization layer.

[0098] This keeps the norm of the vector input to the normalization layer constant, preventing the gradient of the loss in the normalization layer from disappearing. Therefore, according to the first embodiment, the accuracy of the model can be easily improved by fine tuning. For example, according to the first embodiment, the accuracy of the model can be improved even if the search for hyperparameters is omitted.

[0099] [System configuration, etc.] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0100] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0101] [program] In one embodiment, the information processing device 10 can be implemented by installing an adjustment program that executes the above-described adjustment process as package software or online software on a desired computer. For example, by executing the above-described adjustment program on an information processing device, the information processing device can function as the information processing device 10. The information processing device referred to here includes desktop and notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).

[0102] The information processing device 10 can also be implemented as an adjustment server device that provides a service related to the adjustment process to a client terminal device used by a user. For example, the adjustment server device is implemented as a server device that provides an adjustment service that receives a small amount of training data according to a task and outputs information about a fine-tuned model. In this case, the adjustment server device may be implemented as a web server or as a cloud that provides a service related to the adjustment process by outsourcing.

[0103] 13 is a diagram showing an example of a computer that executes an adjustment program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0104] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0105] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the information processing device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the information processing device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0106] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0107] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0108] 10. Information processing equipment 11 Communications Department 12 Input section 13 Output section 14 Storage section 15 Control Unit 141 Model Information 142 Correct Answer Information 143 Threshold Information 144 Norm Information 151 Acquisition Department 152 Calculation section 154 Correction Unit 155 Update Department

Claims

1. a calculation unit that calculates an output for each of a plurality of input vectors using a model that uses BERT; a correction unit that corrects a vector input to a normalization layer included in the model so that the norm of the vector becomes constant; an update unit that updates the model so as to optimize the output; An adjustment device comprising:

2. 2. The adjustment device according to claim 1, wherein the correction unit corrects the vector so that a norm of the vector input to a normalization layer that performs layer normalization included in a Transformer that constitutes a BERT becomes constant.

3. 2. The adjustment device according to claim 1, wherein the correction unit corrects a first vector input to a normalization layer included in the model so that a norm of the first vector is equal to a norm of a second vector last input to the normalization layer.

4. A regulation method performed by a regulation device, comprising: a calculation step of calculating an output for each of a plurality of input vectors using a BERT-based model; a correction step of correcting a vector input to a normalization layer included in the model so that the norm of the vector becomes constant; an updating step for updating the model so that the output is optimized; An adjustment method comprising:

5. On the computer, Calculating an output for each of a plurality of input vectors using a BERT-based model; correcting the vectors input to a normalization layer included in the model so that the norms of the vectors are constant; Update the model so that the output is optimized. An adjustment program that causes a process to be executed.

Images