Information processing device, program, information processing method

Abstract

To provide an information processing device capable of improving the performance of language models. [Solution] The information processing device 30 comprises a language model 340, a first arithmetic unit 352a, a second arithmetic unit 352b, a third arithmetic unit 352c, and a parameter adjustment unit 352d. The first arithmetic unit 352a calculates a first loss value based on the output result of the output layer OL of the language model 340. The second arithmetic unit 352b calculates a second loss value based on the output result of the intermediate layer ML. The third arithmetic unit 352c calculates a final loss value from the first loss value and the second loss value. The parameter adjustment unit 352d adjusts the parameters of the language model 340 based on the final loss value.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a program, and an information processing method. [Background technology]

[0002] Conventionally, there is a device described in Patent Document 1 below. This device pre-trains a language model using first data and fine-tunes the language model using second data. As for methods for training the language model, for example, there is the method described in Non-Patent Document 1 below. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-26202 [Non-patent literature]

[0004] [Non-Patent Document 1] Rafael Rafailov, et al. "Direct preference optimization: Your language model is secretly a reward model", In Thirty-seventh Conference on Neural Information Processing Systems, arXiv:2305.18290, 2023. [Overview of the project] [Problems that the invention aims to solve]

[0005] Regarding the language models used in the apparatus described in Patent Document 1, and the language models trained using the method described in Non-Patent Document 1, there is room for improvement in their performance.

[0006] The present invention has been made in view of such circumstances, and an object thereof is to provide an information processing apparatus, a program, and an information processing method capable of improving the performance of a language model.

Means for Solving the Problems

[0007] The information processing apparatus for solving the above problems includes a language model, a first calculation unit, a second calculation unit, a third calculation unit, and a parameter adjustment unit. The language model has an input layer, an intermediate layer, and an output layer. When the first calculation unit inputs predetermined data to the input layer, it calculates a first loss value representing the deviation of each of the probability that a desirable output result is generated in the output layer and the probability that an undesirable output result is generated in the output layer from the ideal value. When the second calculation unit inputs predetermined data to the input layer, it calculates a second loss value representing the deviation of each of the probability that a desirable output result is generated in the intermediate layer and the probability that an undesirable output result is generated in the intermediate layer from the ideal value. The third calculation unit calculates a final loss value from the first loss value and the second loss value. The parameter adjustment unit adjusts the parameters of the language model based on the final loss value.

[0008] Another information processing apparatus for solving the above problems includes a storage medium and a processor. The storage medium stores a language model having an input layer, an intermediate layer, and an output layer, and a program readable by a computer. The processor executes the program. By executing the program, when the processor inputs predetermined data to the input layer, it calculates a first loss value representing the deviation of each of the probability that a desirable output result is generated in the output layer and the probability that an undesirable output result is generated in the output layer from the ideal value, calculates a second loss value representing the deviation of each of the probability that a desirable output result is generated in the intermediate layer and the probability that an undesirable output result is generated in the intermediate layer from the ideal value when the processor inputs predetermined data to the input layer, calculates a final loss value from the first loss value and the second loss value, and adjusts the parameters of the language model based on the final loss value.

[0009] A program for solving the above problems causes a computer to calculate a first loss value representing the deviation of each of the probability that a desired output result is generated in the output layer and the probability that an undesired output result is generated in the output layer from an ideal value when predetermined data is input to the input layer of a language model having an input layer, an intermediate layer, and an output layer, calculate a second loss value representing the deviation of each of the probability that a desired output result is generated in the intermediate layer and the probability that an undesired output result is generated in the intermediate layer from an ideal value when predetermined data is input to the input layer, calculate a final loss value from the first loss value and the second loss value, and adjust the parameters of the language model based on the final loss value.

[0010] In an information processing method for solving the above problems, when a computer inputs predetermined data to the input layer of a language model having an input layer, an intermediate layer, and an output layer, it calculates a first loss value representing the deviation of each of the probability that a desired output result is generated in the output layer and the probability that an undesired output result is generated in the output layer from an ideal value, calculates a second loss value representing the deviation of each of the probability that a desired output result is generated in the intermediate layer and the probability that an undesired output result is generated in the intermediate layer from an ideal value when the predetermined data is input to the input layer, calculates a final loss value from the first loss value and the second loss value, and adjusts the parameters of the language model based on the final loss value.

[0011] According to these, since not only the first loss value based on the output result of the output layer but also the second loss value based on the output result of the intermediate layer are taken into account and the parameters of the language model are adjusted, the performance of the language model can be improved.

Effects of the Invention

[0012] According to the information processing apparatus, program, and information processing method of the present invention, it is possible to improve the performance of a language model.

Brief Description of the Drawings

[0013] [Figure 1]A block diagram showing the schematic configuration of the information processing system of the embodiment. [Figure 2] A block diagram showing the schematic configuration of the information processing device of the embodiment. [Figure 3] A block diagram showing an example of the schematic configuration of the language model of the embodiment. [Figure 4] Figures (A) to (C) schematically illustrate an example of a language model optimization method for the embodiment. [Figure 5] A block diagram showing an example of the schematic configuration of the language model of the embodiment. [Figure 6] A block diagram showing an example of the schematic configuration of the language model of the embodiment. [Figure 7] A diagram showing an example of experimental results by the inventor. [Figure 8] A diagram showing an example of experimental results by the inventor. [Figure 9] A diagram showing an example of experimental results by the inventor. [Figure 10] A block diagram showing the schematic configuration of the preference optimization unit of the embodiment. [Figure 11] A block diagram showing the hardware configuration of the computer in this embodiment. [Figure 12] (A) and (B) are diagrams showing examples of experimental results by the inventor. [Figure 13] A diagram showing an example of a screen displayed on a terminal device of a second modified embodiment. [Figure 14] A diagram showing an example of a screen displayed on a terminal device of a second modified embodiment. [Modes for carrying out the invention]

[0014] Hereinafter, an embodiment of an information processing device, a program, and an information processing method will be described with reference to the drawings. To facilitate understanding of the explanation, the same reference numerals are used for identical components in each drawing whenever possible, and redundant explanations are omitted.

[0015] <Embodiment> First, the general configuration of the information processing system 10 of this embodiment will be described.

[0016] (Overview of the Information Processing System) The information processing system 10 shown in Figure 1 is a system for performing natural language processing, such as document creation, on predetermined input text entered by a user. The information processing system 10 includes a terminal device 20 and an information processing device 30. The terminal device 20 and the information processing device 30 are connected to each other via a network line N such as the Internet, enabling them to communicate with each other.

[0017] Terminal device 20 is a device used by the user. Terminal device 20 can be a personal computer, smartphone, tablet, etc. By operating terminal device 20, the user can input, for example, a predetermined input sentence. The input sentence entered by the user is transmitted from terminal device 20 to information processing device 30 via the network line N. In addition, terminal device 20 displays a response sentence or the like created by the information processing device 30 in response to the input sentence.

[0018] The information processing device 30 generates documents and performs other operations based on user input text using large language models (LLMs). The information processing device 30 also trains the language models. As shown in Figure 2, the information processing device 30 comprises an input unit 31, an output unit 32, a communication unit 33, a storage unit 34, and a control unit 35.

[0019] The input unit 31 receives input operations from administrators and other personnel who manage the information processing device 30. Administrators and other personnel can operate the information processing device 30 and input various information by operating the input unit 31. The output unit 32 displays various screens to the administrator and outputs audio. The communication unit 33 communicates with the terminal device 20 via the network line N. For example, the communication unit 33 obtains input text entered by the user from the terminal device 20 via the network line N. The communication unit 33 also transmits response text created in the information processing device 30 to the terminal device 20 via the network line N, thereby displaying the response text on the terminal device 20.

[0020] The memory unit 34 stores various types of information held by the information processing device 30. For example, the memory unit 34 stores various programs executed by the control unit 35. The memory unit 34 also stores the language model 340, the pre-training dataset 341, the fine-tuning dataset 342, and the preference dataset 343.

[0021] The language model 340 is a machine learning model that performs natural language processing, such as document creation, on a given input sentence. Figure 3 is a block diagram showing an example of the configuration of the language model 340.

[0022] As shown in Figure 3, the language model 340 comprises an input layer IL, n hidden layers ML, and an output layer OL. Here, n is an integer greater than or equal to 1. For example, n is set to 32.

[0023] The input layer IL receives the user's input sentence. The input layer IL includes, for example, an embedding layer EL. The embedding layer EL divides the input sentence into units called tokens using, for example, pre-trained word embeddings, and converts multiple tokens into vectors. Hereafter, the vectors converted by the embedding layer EL will be referred to as embedding vectors. The input layer IL may further include a positional encoding layer that assigns positional encoding to the embedding vectors, which are the output of the embedding layer EL.

[0024] The n-layer hidden layer ML is a neural network model for performing sequence transformation tasks, which convert user input sentences into output sentences of another sequence by using embedding vectors generated by the input layer IL as input data. The hidden layer ML includes, for example, a transformer layer TL. The transformer layer TL includes, for example, at least one of an encoder layer and a decoder layer.

[0025] The encoder layer, for example, when a predetermined input vector is input, generates an output vector by referencing all input vectors, not just the input vector corresponding to machine learning. In the encoder layer, for example, if the input vector sequence is a sequence of embedding vectors, the output vector sequence will also be a sequence of embedding vectors. The encoder layer includes, for example, a self-attention mechanism and a feed-forward layer as sublayers. The self-attention mechanism takes a sequence of embedding vectors as input, calculates features that take into account the relevance between them, and calculates a new sequence of embedding vectors to which the information of the calculated features has been added. The feed-forward layer further processes the output of the self-attention mechanism. The self-attention mechanism and the feed-forward layer of the encoder layer may each have a residual connection and a layer normalization layer. The encoder layer has a set of weights unique to each layer.

[0026] The decoder layer generates output sentences of another sequence based on features extracted by, for example, the encoder layer. The decoder layer includes, for example, a self-attention mechanism and a forward propagation layer as sublayers, similar to the encoder layer, as well as an encoder-decoder self-attention mechanism. The decoder layer's self-attention mechanism is a masked self-attention mechanism, trained during its operation to exclude future words from self-attention calculations. The input to the masked self-attention mechanism is, for example, an embedding vector, which is the output of the decoder layer, with a position code assigned by a position coding layer (not shown). The encoder-decoder self-attention mechanism is a self-attention mechanism that uses the output of the decoder layer in combination with the output of the decoder layer's self-attention mechanism. Each of the decoder layer's self-attention mechanism, forward propagation layer, and encoder-decoder self-attention mechanism may have residual connections and normalization layers. The decoder layer, like the encoder layer, has a set of weights unique to each layer.

[0027] The transformer layer TL may consist of, for example, one or more encoder layers only, or one or more decoder layers only. Alternatively, the transformer layer TL may consist of a combination of one or more encoder layers and one or more decoder layers.

[0028] As shown in Figure 3, the n-layer hidden layers ML are arranged in series between the input layer IL and the output layer OL. Hereafter, the n-layer hidden layers ML closest to the input layer IL will be referred to as "hidden layer ML(1)", and those arranged sequentially toward the output layer OL will be referred to as hidden layer ML(2), hidden layer ML(3), ..., hidden layer ML(n).

[0029] Hidden layer ML(1) takes the embedding vector generated by the input layer IL as input and converts it into an embedding vector of a different system. Hidden layer ML(2) takes the embedding vector generated by hidden layer ML(1) as input and converts it into an embedding vector of a different system. In this way, each hidden layer ML(1) to ML(n) sequentially converts the embedding vector generated by the preceding hidden layer into an embedding vector of a different system. The embedding vector generated by the final hidden layer ML(n) is input to the linear layer LL.

[0030] The output layer OL includes a linear layer LL and a softmax layer SL. The linear layer LL converts the embedding vector generated by the intermediate layer ML(n) into a vector of a predetermined size. The softmax layer SL receives the embedding vector generated by the linear layer LL and converts it into a vector having the same dimension and a sum of probabilities of 1. The language model 340 generates a response sentence based on the output vector of the output layer OL. This response sentence is transmitted to the terminal device 20 via the communication unit 33 and displayed on the terminal device 20.

[0031] The language model 340 shown in Figure 3 is constructed by training its initial model and stored in the memory unit 34. As shown in Figure 2, the memory unit 34 stores the pre-training dataset 341, the fine-tuning dataset 342, and the preference dataset 343 as data used for training the language model 340.

[0032] Furthermore, various calculation processes during the training of the language model 340 are performed by the control unit 35. The control unit 35 has a functional configuration realized by executing a program stored in the memory unit 34, and includes a display control unit 350, a pre-training unit 351, a fine-tuning unit 352, and a preference optimization unit 353.

[0033] The display control unit 350 displays various screens on the terminal device 20. For example, the display control unit 350 displays a response sentence generated by the language model 340 on the terminal device 20.

[0034] The pre-training unit 351, the fine-tuning unit 352, and the preference optimization unit 353 train the language model 340.

[0035] Next, we will specifically describe the configuration for training the language model 340.

[0036] The language model 340 is trained in three steps, for example, as shown in Figure 4: pre-training, fine-tuning, and preference optimization. Each process is performed by the pre-training unit 351, the fine-tuning unit 352, and the preference optimization unit 353 of the control unit 35, respectively.

[0037] The pre-training unit 351 improves the fit of the language model 340 by training the initial model of the language model 340, which has not been trained, with the pre-training dataset 341. The pre-training dataset 341 includes, for example, a large amount of raw text data. The pre-training process updates each layer and parameter of the language model 340. The parameters of the language model 340 include, for example, the weights of each of the n intermediate layer MLs. Hereinafter, the language model 340 that has undergone the pre-training process by the pre-training unit 351 will be referred to as the "pre-trained language model 340".

[0038] The fine-tuning unit 352 optimizes the language model 340 to apply the requirements of the target task by further training the pre-trained language model 340 with the fine-tuning dataset 342. The fine-tuning data included in the fine-tuning dataset 342 is a labeled dataset, as shown in Figure 4, which includes, for example, a predetermined prompt and an example sentence corresponding to that prompt. Thus, the training of the language model 340 performed by the fine-tuning unit 352 in this embodiment is so-called supervised fine-tuning. Through the fine-tuning process, each layer and parameter of the language model 340 is further updated. Hereinafter, the language model 340 after the fine-tuning process performed by the fine-tuning unit 352 will be referred to as the "fine-tuned language model 340".

[0039] The preference optimization unit 353 further optimizes the language model 340 by fine-tuning its parameters by further training the fine-tuned language model 340 with the preference dataset 343. Each preference data included in the preference dataset 343, as shown in Figure 4, includes, for example, a predetermined prompt and positive and negative examples corresponding to that prompt. A positive example is, for example, an example sentence of a desirable output result for a predetermined prompt. A negative example is, for example, an example sentence of an undesirable output result for a predetermined prompt. The preference optimization unit 353 optimizes the language model 340 using a method called Direct Preference Optimization (DPO), which directly fine-tunes the parameters of the language model 340 from the preference dataset 343 without using a reward model. In the information processing device 30 of this embodiment, an intermediate DPO, which is an improved version of the normal DPO, is used.

[0040] (Summary of Intermediate DPO) Next, we will explain the overview of the intermediate DPO. As mentioned above, the intermediate DPO is an improved version of the standard DPO, so we will first explain the overview of the standard DPO. Note that the standard DPO is as described in Non-Patent Document 1, so only an overview will be provided below.

[0041] A typical DPO is a method for adjusting the parameters of a finely tuned language model so that, given preference data as shown in Figure 4, the probability of the language model generating a desirable response sentence is higher than the probability of the language model generating an undesirable response sentence. In a typical DPO, the generation probability of the language model is calculated using the output of the final layer of the finely tuned language model. For example, when using language model 340 as shown in Figure 3, the output of the output layer OL is used as the output of the final layer.

[0042] In a typical DPO, the loss value λ is calculated using the following formula f1. DPO The following calculation is performed: Loss value λ DPOrepresents the deviation of each of the probability of generating a desirable output result in the output layer OL and the probability of generating an undesirable output result in the output layer OL from the ideal value.

[0043] [Number]

[0044] In the formula f1, x is a prompt included in the preference data input to the target language model. y l is a positive example included in the preference data. y w is a negative example included in the preference data. πθ represents the target language model, and πθ(y|x) represents the conditional probability that the response sentence y is generated when the input sentence x is given to the target language model. π ref represents the reference model, and π ref (y|x) represents the conditional probability that the response sentence y is generated when the input sentence x is given to the reference model. In the formula f1, π ref (y|x) is used as the ideal value. The reference model is, for example, a fine-tuned language model. σ represents the sigmoid function. β represents a hyperparameter.

[0045] In the normal DPO, the language model is optimized by finely adjusting the parameters of the language model so that the loss value λ DPO calculated by the above formula f1 becomes small. Hereinafter, the loss value λ DPO calculated based on the output of the output layer OL, which is the final layer of the language model 340, is referred to as "the first loss value λ DPO ".

[0046] In the information processing system 10 of the present embodiment, in order to further improve the performance of the normal DPO as described above, not only the first loss value λ DPO is calculated based on the output of the final layer of the language model, but also a second loss value is further calculated based on the output of the intermediate layer of the language model. For example, when using the language model 340 as shown in FIG. 3, the output of the intermediate layer is the output of each of the intermediate layers ML of the n layers.

[0047] Specifically, in the intermediate DPO, one or more intermediate layer MLs are selected from the n intermediate layer MLs to calculate the second loss value. Hereafter, the intermediate layers selected to calculate the second loss value will be referred to as "selected intermediate layer ML(K)". K indicates the index of the selected intermediate layer. For example, if intermediate layers ML(5) and TL(8) are selected as intermediate layers to calculate the second loss value, K=5,8.

[0048] In the intermediate DPO, in order to calculate the second loss value from the output of the selected hidden layer ML(K), the language model 340 further includes an output layer OL(K) for processing the output of the selected hidden layer ML(K), as shown in Figure 5. The output layer OL(K) includes a linear layer LL(K) and a softmax layer SL(K). For example, when K=5,8, the language model 340 is provided with an output layer OL(5) corresponding to the selected hidden layer ML(5) and an output layer OL(8) corresponding to the selected hidden layer ML(8), as shown in Figure 6.

[0049] Note that the parameters used in the linear layer LL(K) of the selected intermediate layer ML(K) are shared. For example, if K=5,8, the parameters of the linear layer LL(5) and the linear layer LL(8) are shared.

[0050] In the intermediate DPO, the second loss value λ is calculated from the output of the output layer OL(K) based on the above equation f1. DPO_K The following calculation is performed. For example, as shown in Figure 6, when K=5,8, the second loss value λ of the selected hidden layer ML(5) is calculated based on the output of the output layer OL(5). DPO_5 The above equation f1 is used to calculate the second loss value λ of the selected hidden layer ML(8) based on the output of the output layer OL(8). DPO_8 This is calculated using the above formula f1. In the intermediate DPO, the second loss value λ of the selected intermediate layer ML(K) calculated in this way is obtained. DPO_K Based on the following equation f2, the final second loss value λ DPO_int Perform the calculation.

[0051]

number

[0052] In equation f2, |K| represents the number of occurrences of a number in K. For example, if K=5,8, then |K| is 2. Equation f2 represents multiple second loss values ​​λ DPO_K This is equivalent to the formula for calculating the average value.

[0053] In the intermediate DPO, the final second loss value λ is obtained by equation f2. DPO_int In order to use this as an auxiliary loss function, the first loss value λ is as shown in the following equation f3. DPO The final loss value λ is calculated by performing a weighted sum with respect to these factors.

[0054]

number

[0055] In equation f3, γ represents a hyperparameter.

[0056] In the intermediate DPO, the language model is optimized by fine-tuning the parameters of the language model 340 so that the final loss value λ, calculated by equation f3, becomes small.

[0057] In this embodiment, the language model 340 performs decoding by calculating the posterior probability from the output of the softmax layer SL, which is the output layer, during inference.

[0058] On the other hand, experiments conducted by the inventors have confirmed that performing preference optimization on a language model using the intermediate DPO described above improves the performance of the language model compared to a language model that has undergone preference optimization using a normal DPO, and also compared to a finely tuned language model.

[0059] Furthermore, experiments conducted by the inventors have confirmed that when using an intermediate DPO, the performance of the language model differs depending on which of the n intermediate layer MLs is selected as the selected intermediate layer ML(K).

[0060] Next, we will describe the outline of the experiments conducted by the inventor.

[0061] The inventor experimentally measured the performance difference of language models depending on the position of the selected hidden layer. The positions used for the selected hidden layer were K=11, 16, and 22, i.e., the positions of hidden layers ML(11), ML(16), and ML(22). Figure 7 shows the results of this experiment. Figure 7 shows the win rate of the language model that underwent preference optimization processing using the intermediate DPO compared to the fine-tuned language model. The inventor's experiment used a language model 340 with n set to 32, in other words, a language model 340 with 32 hidden layers ML. As shown in Figure 7, the inventor's experiment confirmed that the win rate when the selected hidden layer was set to ML(22) was higher than when the selected hidden layer was set to ML(11) or ML(16). Therefore, the experimental results shown in Figure 7 indicate that among the n-layer hidden layers ML, the second loss value λ is higher in the hidden layer closest to the output layer. DPO_int It can be seen that the performance of the language model improves by performing this calculation.

[0062] Furthermore, the inventor experimentally measured the performance difference of the language model depending on the number of selective hidden layers (ML(K)). Figure 8 shows the results of that experiment. As shown in Figure 8, it was confirmed that the win rate was higher when there were two selective hidden layers (ML(K)) than when there was one selective hidden layer (ML(K)). Therefore, the experimental results shown in Figure 8 show that the performance of the language model can be improved by selecting multiple selective hidden layers (ML(K)).

[0063] Furthermore, the inventors experimentally measured the performance differences of the language model due to differences in the selection method when selecting multiple selective hidden layers ML(K). Figure 9 shows the results of that experiment. As shown in Figure 9, the win rates when ML(11) and ML(22) are selected as the selective hidden layers ML(K), and the win rates when ML(30) and ML(31) are selected as the selective hidden layers ML(K). The former win rate indicates the win rate when two distant layers out of n hidden layers ML are selected as the selective hidden layers ML(K). The latter win rate indicates the win rate when two consecutive layers are selected starting from the layer closest to the output layer OL. As shown in Figure 9, it has been confirmed that the win rate when ML(11) and ML(22) are selected as the selective hidden layers ML(K) is higher than the win rate when ML(30) and ML(31) are selected as the selective hidden layers ML(K). Therefore, the experimental results shown in Figure 9 demonstrate that selecting multiple distant layers from the n-layer hidden layer ML as selected hidden layer ML(K) improves the performance of the language model.

[0064] (Configuration of the preference optimization unit) Next, the configuration of the preference optimization unit of this embodiment, which utilizes the overview of the intermediate DPO described above, will be explained.

[0065] As shown in Figure 2, the preference optimization unit 353 has a functional configuration realized by the control unit 35 executing a program stored in the memory unit 34, and includes a first calculation unit 352a, a second calculation unit 352b, a third calculation unit 352c, and a parameter adjustment unit 352d.

[0066] Figure 10 is a block diagram showing the flow of calculation processes for the first calculation unit 352a, the second calculation unit 352b, the third calculation unit 352c, and the parameter adjustment unit 352d.

[0067] As shown in Figure 10, the preference optimization unit 353 inputs one of the preference data from the preference dataset 343 into the input layer IL of the language model 340. As a result, the language model 340 outputs predetermined response data corresponding to the preference data from its output layer OL. In addition, predetermined response data is also output from the output layer OL(K) corresponding to the selected intermediate layer ML(K).

[0068] The first arithmetic unit 352a uses the above equation f1 to calculate the first loss value λ from the output result of the output layer OL of the language model 340. DPO Perform the calculation.

[0069] The second arithmetic unit 352b uses the above equation f1 to calculate the second loss value λ from the output result of the output layer OL(K) of the language model 340. DPO_K The calculation is performed. If the selected intermediate layer ML(K) is one layer, the second calculation unit 352b calculates the second loss value λ. DPO_K This is used as the final second loss value λ DPO_int It is used as follows. If there are two or more selected hidden layers ML(K), the second arithmetic unit 352b calculates multiple loss values ​​λ corresponding to each of the multiple selected hidden layers ML(K). DPO_K By using the above equation f2, the final second loss value λ DPO_int Perform the calculation.

[0070] The third calculation unit 352c calculates the first loss value λ calculated by the first calculation unit 352a. DPO And the second loss value λ calculated by the second calculation unit 352b DPO_int Therefore, the final loss value λ is calculated using the above formula f3.

[0071] The parameter adjustment unit 352d fine-tunes the parameters of the language model 340 so that the final loss value λ calculated by the third calculation unit 352c becomes smaller.

[0072] (Hardware configuration of information processing equipment) Next, with reference to Figure 11, an example of a hardware configuration when the terminal device 20 and the information processing device 30 are implemented by a computer 100 will be described.

[0073] As shown in Figure 11, the computer 100 includes, for example, a processor 101, memory 102, storage device 103, input I / F unit 104, data I / F unit 105, communication I / F unit 106, and display device 107.

[0074] Computer 100 may be, for example, a cloud computer, a server computer, a personal computer (e.g., a desktop, laptop, tablet, etc.), a media computer platform (e.g., a cable or satellite set-top box, a digital video recorder, etc.), a handheld computer device (e.g., a PDA, an email client, etc.), or any other type of computer or communication platform.

[0075] The processor 101 is a control unit that controls various processes in the computer 100 by executing programs stored in the memory 102.

[0076] Memory 102 is a storage medium such as RAM (Random Access Memory). Memory 102 temporarily stores the code of the program executed by the processor 101 and the data required when the program is executed.

[0077] The storage device 103 is a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. The storage device 103 stores the operating system and various programs necessary to implement the above configurations.

[0078] The input interface unit 104 is a device for receiving input from the user. The input interface unit 104 can be, for example, a keyboard, mouse, touch panel, various sensors, or a wearable device. The input interface unit 104 may be connected to the computer 100 via an interface such as USB (Universal Serial Bus).

[0079] The data I / F unit 105 is a device for inputting data from outside the computer 100. The data I / F unit 105 is, for example, a drive device for reading data stored on various storage media. The data I / F unit 105 may be located outside the computer 100. If the data I / F unit 105 is located outside the computer 100, it is connected to the computer 100 via an interface such as USB.

[0080] The communication interface unit 106 is a device for performing data communication with external devices of the computer 100 via a network such as the Internet, either by wire or wireless connection. The communication interface unit 106 may be located outside the computer 100. If the communication interface unit 106 is located outside the computer 100, it is connected to the computer 100 via an interface such as USB.

[0081] The display device 107 is a device for displaying various types of information. The display device 107 may be, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, or a display for a wearable device. The display device 107 may be located outside the computer 100. If the display device 107 is located outside the computer 100, it is connected to the computer 100, for example, via a display cable. Furthermore, if a touch panel is used as the input I / F unit 104, the display device 107 may be integrated with the input I / F unit 104.

[0082] (Operation and effects of information processing devices) As described above, the information processing device 30 of this embodiment comprises a language model 340, a first arithmetic unit 352a, a second arithmetic unit 352b, a third arithmetic unit 352c, and a parameter adjustment unit 352d. The language model 340 has an input layer IL, an intermediate layer ML, and an output layer OL. The first arithmetic unit 352a calculates a first loss value λ based on the output result of the output layer OL when preference data is input to the input layer IL.DPO The second calculation unit 352b calculates the second loss value λ based on the output result of the selected intermediate layer ML(K) when the preferred data is input to the input layer IL. DPO_int The third calculation unit 352c calculates the first loss value λ. DPO or second loss value λ DPO_int The final loss value λ is calculated from this. The parameter adjustment unit 352d adjusts the parameters of the language model 340 based on the final loss value λ.

[0083] According to this configuration, the first loss value λ is based on the output result of the output layer OL. DPO In addition, the second loss value λ is based on the output result of the intermediate layer ML. DPO_int Because the parameters of the language model are adjusted taking this into account, for example, the first loss value λ is not the same as in a normal DPO. DPO Compared to tuning the language model parameters based solely on this, the performance of language model 340 can be improved.

[0084] The second arithmetic unit 352b calculates the second loss value λ in multiple selected intermediate layers ML(K) of the language model 340. DPO_K The third calculation unit 352c calculates the first loss value λ. DPO And, multiple second loss values ​​λ calculated in each of the multiple selected hidden layer ML(K) DPO_K Based on this, the final loss value λ is calculated. Specifically, the third calculation unit 352c calculates the first loss value λ DPO And, multiple second loss values ​​λ calculated in each of the multiple selected hidden layer ML(K) DPO_K The second loss value λ is the average value of the two values. DPO_int The final loss value λ is obtained by calculating a weighted sum with respect to the factors.

[0085] This configuration allows for a more accurate calculation of the final loss value λ, making it possible to build a higher-performing language model 340.

[0086] (First variation) Next, a first modified example of the information processing system 10 of the embodiment will be described.

[0087] The inventors analyzed the relationship between the position of the selected intermediate layer ML(K) and the tendency of the response sentences of the language model 340 in an experiment using the language model 340 shown in Figure 5. In this experiment as well, the language model 340 having 32 intermediate layers ML was used.

[0088] Figure 12(A) shows a pie chart of the results of an analysis of the response sentences of language model 340 with K=22, i.e., language model 340 in which intermediate layer ML(22) is selected as the preferred intermediate layer. As shown in Figure 12(A), when K=22, language model 340 showed a tendency to excel at technical or expert answers, as well as analysis and reasoning.

[0089] Figure 12(B) shows a pie chart of the results of analyzing the response sentences of language model 340 with K=11, i.e., language model 340 in which intermediate layer ML(11) is selected as the preferred intermediate layer. As shown in Figure 12(B), when K=11, language model 340 tends to be somewhat weak in analysis and reasoning, but tends to be strong in technical or expert answers, as well as advice and problem solving.

[0090] In this way, the characteristics of the trained language model 340 can be fine-tuned depending on which of the n hidden layers ML is selected as the selected hidden layer ML(K). Specifically, if K is set to 11, that is, if the hidden layer ML located closer to the input layer IL than the output layer OL is selected as the selected hidden layer ML(K), a language model 340 that excels at giving advice and solving problems can be obtained. On the other hand, if K is set to 22, that is, if the hidden layer ML located closer to the output layer OL than the input layer IL is selected as the selected hidden layer ML(K), a language model 340 that excels at analysis and inference can be obtained.

[0091] (Second variation) Next, a second modified example of the information processing system 10 of the embodiment will be described.

[0092] In this modified example, the display control unit 350 displays a screen 70 on the terminal device 20, as shown in Figure 13. The screen 70 shown in Figure 13 is provided with a plurality of input areas 71a, 71b, 71c, 71d, ... that can be used to specify the intermediate layer ML to be set as the selected intermediate layer ML(K). For example, if the user operates the terminal device 20 and inputs the number "5" into input area 71a, intermediate layer ML(5) is selected as the selected intermediate layer ML(K). Also, if the user operates the terminal device 20 and inputs the numbers "5" and "8" into the two input areas 71a and 71b respectively, intermediate layer ML(5) and ML(8) are selected as the selected intermediate layer ML(K).

[0093] As shown by the dashed line in Figure 2, the control unit 35 further includes a selection information acquisition unit 354. The selection information acquisition unit 354 acquires information on the numerical values ​​X entered in the input areas 71a, 71b, 71c, 71d, ... on the screen 70 shown in Figure 13. Based on the numerical values ​​X acquired by the selection information acquisition unit 354, the preference optimization unit 353 sets the intermediate layer ML(X) corresponding to the numerical values ​​X as the selected intermediate layer ML(K) and executes the preference optimization process.

[0094] This configuration allows users to arbitrarily set the selected intermediate layer ML(K), thereby improving usability.

[0095] As explained in the first modified example above, it is known that the characteristics of the language model 340 change depending on which of the n intermediate layer MLs is selected as the selected intermediate layer ML(K). Therefore, instead of directly specifying the layer number of the intermediate layer ML as shown in Figure 13, the user may be allowed to select the characteristics of the language model 340, for example, as shown in Figure 14. The screen 80 shown in Figure 14 is provided with a first selection button 81 and a second selection button 82.

[0096] The first selection button 81 is operated when the user wishes to optimize the language model 340 to be better at giving advice and solving problems. When the user operates the first selection button 81, the intermediate layer ML(11) is selected as the selected intermediate layer ML(K).

[0097] The second selection button 82 is operated when the user wishes to optimize the language model 340 to be better suited to analysis and inference. When the user operates the second selection button 82, the hidden layer ML(22) is selected as the selected hidden layer ML(K).

[0098] With this configuration, the characteristics of the language model 340 can be tuned to the user's preference, further improving usability.

[0099] <Other Embodiments> This disclosure is not limited to the specific examples given above.

[0100] For example, the information processing device 30 obtains the second loss value λ from all of the n-layer hidden layer ML DPO_1 , λ DPO_2 ,···,λ DPO_n After performing the calculations for each of these, the final second loss value λ is calculated based on the above formula f2. DPO_int You may perform the calculation.

[0101] The configuration of language model 340 can be modified as needed. For example, the number of intermediate ML layers in language model 340 can be changed arbitrarily.

[0102] In the above embodiment, we have illustrated a case where the information processing device 30 performs all of the pre-training, fine-tuning, and preference optimization shown in Figure 4. However, instead, the information processing device 30 may, for example, acquire a fine-tuned language model 340 from an external device and perform only the preference optimization of the language model 340.

[0103] Even the above-mentioned examples, with appropriate design modifications by those skilled in the art, are included within the scope of this disclosure, as long as they possess the features of this disclosure. The elements, their arrangement, conditions, shapes, etc., of each of the above-mentioned examples are not limited to those exemplified and can be modified as appropriate. The elements of each of the above-mentioned examples can be combined in different ways as appropriate, as long as no technical inconsistencies arise. [Explanation of symbols]

[0104] IL: Input Layer, ML: Intermediate Layer, OL: Output Layer, 30: Information Processing Unit, 100: Computer, 101: Processor, 103: Memory Device (Storage Medium), 340: Language Model, 353a: First Calculation Unit, 353b: Second Calculation Unit, 353c: Third Calculation Unit, 353d: Parameter Adjustment Unit, 354: Selected Information Acquisition Unit.

Claims

1. A language model having an input layer, a hidden layer, and an output layer, A first calculation unit calculates a first loss value that represents the deviation between the ideal value and the probability that a desirable output result is generated in the output layer, and the probability that an undesirable output result is generated in the output layer, when predetermined data is input to the input layer. A second calculation unit calculates a second loss value that represents the deviation between the ideal value and the probability that a desirable output result is generated in the intermediate layer, and the probability that an undesirable output result is generated in the intermediate layer, when the predetermined data is input to the input layer. A third calculation unit calculates a final loss value from the first loss value and the second loss value, The system includes a parameter adjustment unit that adjusts the parameters of the language model based on the final loss value. Information processing device.

2. The second arithmetic unit calculates the second loss value in each of the intermediate layers of the language model, The third calculation unit calculates the final loss value based on the first loss value and the multiple second loss values ​​calculated in each of the multiple intermediate layers. The information processing apparatus according to claim 1.

3. The third calculation unit calculates the final loss value based on the first loss value and the average value of the multiple second loss values ​​calculated in each of the multiple intermediate layers. The information processing apparatus according to claim 2.

4. The third calculation unit calculates the final loss value as a weighted sum of the first loss value and the average value of the multiple second loss values. The information processing apparatus according to claim 3.

5. The second calculation unit uses, as an intermediate layer for calculating the second loss value, an intermediate layer among the multiple intermediate layers of the language model that is located closer to the output layer than the input layer. The information processing apparatus according to claim 1.

6. The second calculation unit uses, as an intermediate layer for calculating the second loss value, an intermediate layer among the multiple intermediate layers of the language model that is located closer to the input layer than the output layer. The information processing apparatus according to claim 1.

7. The system further includes a selection information acquisition unit that acquires an intermediate layer selected by the user from among a plurality of intermediate layers of the language model, The second calculation unit uses the intermediate layer acquired by the selection information acquisition unit as the intermediate layer for calculating the second loss value. The information processing apparatus according to claim 1.

8. The predetermined data includes a predetermined prompt, a positive example representing a desirable output, and a negative example representing an undesirable output. The information processing apparatus according to claim 1.

9. A language model having an input layer, an intermediate layer, and an output layer, a program readable by a computer, and a storage medium for storing these. A processor that executes the aforementioned program, The processor executes the program, When predetermined data is input to the input layer, a first loss value is calculated that represents the deviation between the ideal value and the probability that a desirable output result is generated in the output layer, and the probability that an undesirable output result is generated in the output layer. When the predetermined data is input to the input layer, a second loss value is calculated that represents the deviation between the ideal value and the probability that a desirable output result is generated in the intermediate layer, and the probability that an undesirable output result is generated in the intermediate layer. The final loss value is calculated from the first loss value and the second loss value. The parameters of the language model are adjusted based on the final loss value. Information processing device.

10. On the computer, When predetermined data is input to the input layer of a language model having an input layer, an intermediate layer, and an output layer, a first loss value is calculated that represents the deviation between the probability of a desirable output result being generated in the output layer and the probability of an undesirable output result being generated in the output layer, and the ideal value. When the predetermined data is input to the input layer, a second loss value is calculated that represents the deviation between the probability of a desirable output result being generated in the intermediate layer and the probability of an undesirable output result being generated in the intermediate layer, and the ideal value. The final loss value is calculated from the first loss value and the second loss value. The parameters of the language model are adjusted based on the final loss value. program.

11. Computers When predetermined data is input to the input layer of a language model having an input layer, a hidden layer, and an output layer, a first loss value is calculated that represents the deviation between the probability of a desirable output result being generated in the output layer and the probability of an undesirable output result being generated in the output layer, and the ideal value. When the predetermined data is input to the input layer, a second loss value is calculated that represents the deviation between the ideal value and the probability that a desirable output result is generated in the intermediate layer, and the probability that an undesirable output result is generated in the intermediate layer. The final loss value is calculated from the first loss value and the second loss value. The parameters of the language model are adjusted based on the final loss value. Information processing methods.

Images