Learning device, utilization device, learning method and program

Abstract

To provide a learning device, a utilization device, a learning method and a program capable of determining a reply by an LLM.SOLUTION: According to a learning device of the present disclosure, a reply input unit accepts an input of a reply by a Large Language Model (LLM) to a prompt. A data perturbing unit generates, using a reward function for determining the reply, perturbation data obtained by adding perturbation to the reply. An output unit outputs the perturbation data. An annotation data input unit accepts an input of data that is the data to which perturbation is added, and annotation is further added. A training process unit trains the reward function based on the data to which the annotation is added.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning device, a utilization device, a learning method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technology related to learning of a large language model (LLM). Patent Document 1 discloses a technology for generating extended data by transforming or extending existing data based on a language model, and transferring knowledge recognized by the language model to another language model through the extended data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-018624 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 has a problem in that it is not possible to evaluate the answers of the LLM.

[0005] In consideration of such issues, one of the objectives of the present disclosure is to provide a learning device, a usage device, a learning method, and a program that can evaluate LLM answers. [Means for solving the problem]

[0006] The learning device of the present disclosure includes: an answer input section for accepting input of the LLM's answer to the prompt; a data perturbation unit that generates perturbation data by adding a perturbation to the answer using a reward function for determining the answer; an output unit that outputs the perturbation data; an annotation data input unit that receives input of data in which annotations are added to the perturbation data; and a training processor that trains the reward function based on the annotated data.

[0007] The utilization device of the present disclosure is a prompt input unit that accepts a prompt input; an answer input unit that accepts input of the LLM's answer to the prompt; a reward assigning unit that assigns a reward value to the answer of the LLM using a learned reward function; A learning processing unit that learns the LLM so as to maximize the reward value using a loss function, The learned reward function is The learning device learns.

[0008] Other utilization devices of the present disclosure include: a prompt input unit that accepts a prompt input; an answer input unit that accepts input of the LLM's answer to the prompt; a reward assigning unit that assigns a reward value to the answer of the LLM using a learned reward function; A selection unit that selects the answers of the LLM based on the assigned reward value; an output unit that outputs the selected answer, The learned reward function is The learning device learns.

[0009] The computer Accepts LLM responses to prompts, generating perturbation data in which a perturbation is added to the answer using a reward function for judging the answer; outputting the perturbation data; accepting input of data annotated with the perturbation data; training the reward function based on the annotated data How to learn.

[0010] Accepts LLM responses to prompts, generating perturbation data in which a perturbation is added to the answer using a reward function for judging the answer; outputting the perturbation data; accepting input of data annotated with the perturbation data; and causing a computer to perform a process of training the reward function based on the annotated data. program. [Effects of the Invention]

[0011] The present disclosure provides a learning device, a usage device, a learning method, and a program that can evaluate LLM answers. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of the configuration of a learning device 100 according to the present embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a learning device 200 according to the present embodiment. [Figure 3] 10 is a flowchart showing an example of the operation of the learning device 200 according to the present embodiment. [Figure 4] 10 is a flowchart showing a detailed example of the operation of step S102 of the learning device 200 according to the present embodiment. [Figure 5] FIG. 2 is a block diagram showing an example of the configuration of a user device 300 according to the present embodiment. [Figure 6] 10 is a flowchart showing an example of the operation of the device using device 300 according to the present embodiment. [Figure 7] FIG. 2 is a block diagram showing an example of the configuration of a user device 400 according to the present embodiment. [Figure 8] 10 is a flowchart showing an example of the operation of the device using device 400 according to the present embodiment. [Figure 9]FIG. 5 is a block diagram showing an example of the hardware configuration of a computer 500 according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and for clarity of explanation, duplicate explanations will be omitted as necessary.

[0014] (First embodiment) First, the configuration of a learning device 100 according to the first embodiment will be described with reference to FIG.

[0015] 1 is a block diagram showing an example of the configuration of a learning device 100 according to the first embodiment. As shown in FIG. 1, the learning device 100 includes an answer input unit 101, a data perturbation unit 102, an output unit 103, an annotation data input unit 104, and a training processing unit 105.

[0016] The answer input unit 101 accepts input of an LLM (large language model) answer to a prompt. The data perturbation unit 102 generates perturbed data by perturbing the answer using a reward function to determine the answer. The output unit 103 outputs the perturbed data to the user. The annotation data input unit 104 accepts input of data to which the user has added annotations to the perturbed data. The training processing unit 105 trains the reward function based on the annotated data.

[0017] In the learning device 100 according to the first embodiment, the answer to the LLM (or perturbation data obtained by adding perturbations to the answer to the LLM) serves as a data set for training the reward function. The learning device 100 can determine the answer to the LLM (or perturbation data) based on the reward function.

[0018] (Second embodiment) Next, the configuration of a learning device 200 according to the second embodiment will be described with reference to FIG.

[0019] First, the problem associated with this embodiment will be described. Reinforcement Learning from Human Feedback (RLHF) is used to train large language models (LLMs) such as ChatGPT. By using RLHF, LLMs are fine-tuned by human feedback, allowing the LLM's behavior (e.g., its responses to prompts) to reflect user intent.

[0020] The learning process of an LLM using RLHF includes (1) pre-learning the LLM policy, (2) training the reward function, and (3) learning the LLM policy through reinforcement learning to maximize the trained reward function. The policy is a strategy that determines what response to return when a prompt is input. The reward function is a function that provides feedback to the LLM instead of a human in reinforcement learning.

[0021] In the (2) step of training the reward function, the LLM's responses to prompts are evaluated by humans, and the reward function is trained based on the evaluated data. This requires a large number of manual evaluations of the LLM's responses, which poses a significant burden on both personnel and time.

[0022] The following describes a learning device 200 that can reduce the human and time burden when learning a reward function used in reinforcement learning of LLM.

[0023] Fig. 2 is a block diagram showing an example of the configuration of a learning device 200 according to this embodiment. As shown in Fig. 2, the learning device 200 includes an AE generation unit 21, an answer example storage unit 22, a reward function storage unit 23, an AE generation function storage unit 24, an annotation addition unit 25, and a reward function training unit 26.

[0024] The AE generation unit 21 includes a response input unit 211 , a data perturbation unit 212 , a data selection unit 213 , and an output unit 214 . The answer input unit 211 accepts input of an LLM answer A to a prompt stored in the answer example storage unit 22. The answer example storage unit 22 may store a plurality of types of LLM answers other than the LLM answer A.

[0025] The data perturbation unit 212 generates adversarial examples (AE), which are data obtained by applying perturbations to the answer A, based on the AE creation function 241 stored in the AE generation function storage unit 24. The adversarial samples correspond to the perturbation data according to the first embodiment. A perturbation refers to a small change such as noise. For example, the data perturbation unit 212 applies perturbation to the answer A by changing part of the text of the answer A to another text. The AE creation function 241 is a function that determines what kind of perturbation to apply to the answer A. The AE generation function storage unit 24 may store multiple types of AE creation functions 241.

[0026] The data perturbation unit 212 assigns a reward value to each of the LLM answer A and the generated adversarial samples using the reward function 231 stored in the reward function storage unit 23. Then, the data perturbation unit 212 assigns a reward value to each of the LLM answer A and the generated adversarial samples based on the reward value. n (x1, , x n(n is a natural number). The reward function 231 is a function that rewards the LLM's answer A or an adversarial sample. How the reward is given by the reward function 231 is designed in advance according to a user's request. For example, the reward function 231 is designed to reward the LLM's answer so that "the LLM gives an answer that is in line with public morals." In this case, the reward function 231 has a pre-trained decision boundary for determining whether an answer is ethical or unethical. The reward function 231 gives a higher reward value to an answer that is more ethical. On the other hand, the reward function 231 gives a lower reward value to an answer that is more unethical. On the other hand, the reward function 231 is designed to reward the LLM's answer so that "the LLM gives an answer that is as concise and correct as possible." In this case, the reward function 231 has a pre-trained decision boundary for determining whether an answer is correct or incorrect. The reward function 231 gives a higher reward value to an answer that is more correct. On the other hand, the reward function 231 assigns a lower reward value to an answer that is more incorrect.

[0027] Specifically, the data perturbation unit 212 generates adversarial samples with different reward values ​​assigned by the reward function 231 as a sequence of adversarial samples x n That is, the data perturbation unit 212 classifies the data, among the generated adversarial samples, whose reward value is changed from the reward value assigned to the answer A, into a sequence of adversarial samples x n In addition, the data perturbation unit 212 classifies the adversarial samples into a sequence x n The reward value given to the adversarial sample already classified as x is used to calculate the data that changes from the reward value given to the adversarial sample already classified as x. n Classify into:

[0028] More specifically, the data perturbation unit 212 generates an adversarial sample sequence x as follows: nThe data perturbation unit 212 assigns a reward value to the answer A using the reward function 231. The data perturbation unit 212 generates a first adversarial sample by adding perturbation to the answer A. The data perturbation unit 212 assigns a reward value to the first adversarial sample using the reward function 231. If the reward value for the first adversarial sample changes from the reward value for the answer A by more than a predetermined threshold, the data perturbation unit 212 assigns the first adversarial sample together with the answer A to the sequence of adversarial samples x n In this case, the adversarial sample sequence x n In the example, the answer A is treated as an adversarial sample x1, and the first adversarial sample is treated as an adversarial sample x2. Then, the data perturbation unit 212 generates a second adversarial sample by perturbing the first adversarial sample. The data perturbation unit 212 assigns a reward value to the second adversarial sample using a reward function 231. If the reward value for the second adversarial sample changes from the reward value for the first adversarial sample by more than a predetermined threshold, the data perturbation unit 212 assigns the second adversarial sample to the sequence of adversarial samples x n In this case, the adversarial sample sequence x n The second adversarial sample is treated as the adversarial sample x3. The data perturbation unit 212 perturbs the sequence of adversarial samples x n Adversarial samples x4, , x n is generated in the same way.

[0029] The data perturbation unit 212 generates the adversarial sample sequence x as follows: n The data perturbation unit 212 assigns a reward value to the answer A using the reward function 231. The data perturbation unit 212 generates a first adversarial sample by adding a first perturbation to the answer A. The data perturbation unit 212 assigns a reward value to the first adversarial sample using the reward function 231. If the reward value for the first adversarial sample changes from the reward value for the answer A by more than a first threshold, the data perturbation unit 212 assigns the first adversarial sample together with the answer A to the sequence of adversarial samples x n In this case, the adversarial sample sequence x nIn the example, answer A is treated as an adversarial sample x1, and the first adversarial sample is treated as an adversarial sample x2. Then, the data perturbation unit 212 generates a second adversarial sample by adding a second perturbation different from the first perturbation to the answer. The data perturbation unit 212 assigns a reward value to the second adversarial sample using the reward function 231. If the reward value for the second adversarial sample has changed from the reward value for answer A by more than a second threshold different from the first threshold, the data perturbation unit 212 assigns the second adversarial sample to the sequence of adversarial samples x n In this case, the adversarial sample sequence x n The second adversarial sample is treated as the adversarial sample x3. The data perturbation unit 212 perturbs the sequence of adversarial samples x n Adversarial samples x4, , x n is generated in the same way.

[0030] For example, the reward function 231 is designed to reward the LLM's answer so that the LLM's answer conforms to public morals. In this case, the adversarial sample sequence x n When ordered according to reward value, x1 (x1 = answer A), , x n (x n = unethical answer B). On the other hand, suppose that the reward function 231 rewards the LLM's answer so that the LLM gives the correct answer as succinctly as possible. In that case, the generated adversarial sample sequence x n When ordered according to reward value, x1(x n = Original answer A), , x n (x n =Incorrect answer B).

[0031] The data selection unit 213 selects a sequence of adversarial samples x n From the adversarial sample y around the decision boundary of the reward function 231, m (y1, , y m Specifically, the data selection unit 213 selects a sequence of adversarial samples x n, adversarial samples y , which are assigned reward values ​​that are within a predetermined value from a predetermined reward value as a decision boundary of the reward function 231. m Select .

[0032] For example, the reward function 231 is designed to reward the LLM's answer so that the LLM's answer conforms to public morals. In this case, the adversarial sample sequence x n is x1 (x1 = answer A), , x n (x n = unethical answer B). Here, the data selection unit 213 selects answers that are judged as either ethical or unethical from the adversarial samples y around the decision boundary of the reward function 231. m On the other hand, the reward function 231 is assumed to reward the LLM's answer so that "the LLM gives the correct answer as succinctly as possible." In this case, the adversarial sample sequence x generated by the data perturbation unit 212 is n is x1 (x1 = answer A), , x n (x n = incorrect answer B). Here, the data selection unit 213 selects answers that are judged as correct or incorrect from the adversarial samples y around the decision boundary of the reward function 231. m Select as.

[0033] The output unit 214 outputs the adversarial samples y around the decision boundary of the reward function 231 selected by the data selection unit 213. m is output to the user.

[0034] The annotation adding unit 25 adds the hostile sample y m For example, the user can annotate the adversarial samples y by using equality and inequality signs, such as adversarial sample y1 > adversarial sample y2 or adversarial sample y3 = adversarial sample y4. m The annotation adding unit 25 evaluates the hostile sample y based on the user's evaluation. m Add annotation to the

[0035] The reward function training unit 26 includes an annotation data input unit 261 and a training processing unit 262. The annotation data input unit 261 receives input of data to which annotations have been added by the annotation adding unit 25 . The training processing unit 262 trains the reward function 231 based on the annotated data. The trained reward function 231 is used for reinforcement learning of the LLM.

[0036] Next, the operation of the learning device 200 according to the second embodiment will be described with reference to FIGS.

[0037] FIG. 3 is a flowchart showing an example of the operation of the learning device 200 according to this embodiment. As shown in FIG. 3, first, in step S101, the answer input unit 211 of the AE generation unit 21 of the learning device 200 accepts input of an LLM answer A to a prompt stored in the answer example storage unit 22.

[0038] Next, in step S102, the data perturbation unit 212 generates a sequence x of adversarial samples, which is data obtained by adding perturbations to the answer A, using the AE generation function 241 and the reward function 231. n (x1, , x n The process of step S102 by the learning device 200 will be specifically described with reference to FIG.

[0039] FIG. 4 is a flowchart showing a detailed example of the operation of step S102 of the learning device 200 according to this embodiment.

[0040] As shown in FIG. 4, in step S1021, the data perturbation unit 212 calculates a reward value for the answer A using the reward function 231. Next, in step S1022, the data perturbation unit 212 generates a first adversarial sample by perturbing the answer A using the AE creation function 241. Next, in step S1023, the data perturbation unit 212 calculates a reward value for the first adversarial sample based on the reward function 231.

[0041] Next, in step S1024, the data perturbation unit 212 determines whether the reward value for the first adversarial sample has changed by a predetermined threshold or more from the reward value for answer A. If it is determined that there has been a change (YES in step S1024), the process proceeds to step S1025. On the other hand, if it is determined that there has not been a change (NO in step S1024), the process returns to step S1022.

[0042] In step S1025, the data perturbation unit 212 converts the first adversarial sample into an adversarial sample sequence x n In this case, the sequence of adversarial samples x n In, answer A is treated as adversarial example x1, and the first adversarial example is treated as adversarial example x2.

[0043] Next, in step S1026, the data perturbation unit 212 generates the adversarial sample sequence x n It is determined whether or not to end the generation of the adversarial sample sequence x. If it is determined that the generation is to be ended (YES in step S1026), the process ends. On the other hand, if it is determined that the generation is not to be ended (NO in step S1026), the process returns to step S1022. Not ending the generation means that other adversarial samples are added to the adversarial sample sequence x n In this case, after step S1026, in step S1022 and the subsequent processing, "answer A" is treated as the "first adversarial sample" and "first adversarial sample" is treated as the "second adversarial sample." Also, an AE creation function 241 having an algorithm different from the algorithm used in the above processing may be used.

[0044] Returning to FIG. 3, next, in step S103, a sequence of adversarial samples x n From the reward function 231, the adversarial sample y around the decision boundary m (y1, , y mSpecifically, the data selection unit 213 selects a sequence x of adversarial samples. n , adversarial samples y , which are assigned reward values ​​that are within a predetermined value from a predetermined reward value as a decision boundary of the reward function 231. m Select .

[0045] Next, in step S104, the output unit 214 outputs the adversarial samples y around the decision boundary of the reward function 231. m By doing so, the user can m Annotation can be performed on the

[0046] Next, in step S105, the annotation adding unit 25 adds the hostile sample y m Add annotation to the Next, in step S106, annotation data input unit 261 of reward function training unit 26 accepts input of data to which annotations have been added. Next, in step S107, the training processing unit 262 trains the reward function 231 based on the annotated data.

[0047] The learning device 200 according to the second embodiment calculates a sequence of adversarial data x , which is data obtained by adding perturbations to the answer A of the LLM, based on a reward value assigned to the reward function 231. n By adding a perturbation to the answer A, the learning device 200 can efficiently increase the data set for learning the reward function 231. The learning device 200 also calculates the sequence x of adversarial samples. n , and outputs to the user adversarial samples that lie near the decision boundary of the reward function 231. Adversarial samples that lie near the decision boundary of the reward function 231 are data that are difficult for the reward function 231 to judge. In other words, the learning device 200 selectively outputs to the user data for which a human should provide feedback. By doing so, the learning device 200 can reduce the human burden and time burden when learning the reward function 231.

[0048] (Third embodiment) Next, the configuration of the device 300 used according to the third embodiment will be described with reference to FIG. The utilization device 300 according to this embodiment is a device that utilizes the reward function 231 learned in the second embodiment described above for learning the LLM.

[0049] 5 is a block diagram showing an example of the configuration of the utilization device 300 according to this embodiment. As shown in FIG. 5, the utilization device 300 includes an LLM learning unit 31, an LLM storage unit 32, a reward function storage unit 33, and a loss function storage unit 34.

[0050] The LLM storage unit 32 stores the LLM 321. The policy of the LLM 321 is pre-trained. A policy is a strategy for determining what response should be returned when a certain prompt is input. The reward function storage unit 33 stores a trained reward function 331. The trained reward function 331 is a function that provides a reward for the answer of the LLM 321, and is trained by the method according to the second embodiment described above. For example, the reward function 331 is designed to provide a reward for the answer of the LLM 321 so that "the LLM provides an answer that is in accordance with public morals." In this case, the reward function 331 provides a higher reward value for an answer that is more ethical. On the other hand, the reward function 331 provides a lower reward value for an answer that is more unethical.

[0051] The loss function storage unit 34 stores a loss function 341. The loss function 341 is, for example, a PPO (Proximal Policy Optimization) algorithm, and is a function that optimizes the policy of the LLM 321 according to the reward assigned to the answer by the trained reward function 331.

[0052] The LLM learning unit 31 includes a prompt input unit 311 , an answer input unit 312 , a reward granting unit 313 , and a learning processing unit 314 .

[0053] The prompt input unit 311 accepts the input of a prompt. The answer input unit 312 obtains the answer from the LLM 321 to the prompt. The reward granting unit 313 grants a reward for the answer of the LLM 321 using the learned reward function 331.

[0054] The learning processing unit 314 learns the LLM 321 so as to maximize the reward, using the loss function 341. Specifically, the learning processing unit 314 uses the loss function 341 to update the policy of the LLM 321 to a policy that is expected to obtain a higher reward.

[0055] Next, the operation of the device 300 according to the third embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the operation of the device 300 according to this embodiment.

[0056] First, in step S201, the prompt input unit 311 accepts the input of a prompt. In step S202, the answer input unit 312 acquires the answer of the LLM 321 to the prompt.

[0057] In step S203, the reward assigning unit 313 assigns a reward value to the response of the LLM 321 using the learned reward function 331. In step S204, the learning processing unit 314 learns the LLM 321 using the loss function 341 so as to maximize the reward value.

[0058] Note that the utilization device 300 may repeat the processes of steps S201 to S204 when further learning the LLM.

[0059] As described above, the utilization device 300 according to the third embodiment utilizes the reward function 231 learned in the second embodiment for training the LLM 421. By doing so, the utilization device 300 can reduce the human and time burdens required for the entire training process of the LLM 421.

[0060] (Fourth embodiment) Next, the configuration of the device 400 used in the fourth embodiment will be described with reference to FIG. The utilization device 400 according to this embodiment is a device that can utilize the reward function 231 learned in the second embodiment to select an LLM response to a prompt.

[0061] The utilization device 400 includes an answer selection unit 41, an LLM storage unit 42, and a reward function storage unit 43.

[0062] The LLM storage unit 42 stores the LLM 421. The policy of the LLM 321 is pre-trained. The reward function storage unit 43 stores a trained reward function 431. The trained reward function 431 is a function that provides a reward for the answer of the LLM 421, and is trained by the method according to the second embodiment described above. For example, the reward function 431 is designed to provide a reward for the answer of the LLM 421 so that "the LLM provides an answer that is in accordance with public morals." In this case, the reward function 431 provides a higher reward value to an answer that is more ethical. On the other hand, the reward function 431 provides a lower reward value to an answer that is more unethical.

[0063] The answer selection unit 41 includes a prompt input unit 411 , an answer input unit 412 , a reward granting unit 413 , a selection unit 414 , and an output unit 415 .

[0064] The prompt input unit 411 receives a prompt input from the user via an interface. The answer input unit 412 acquires an answer to the prompt from the LLM 421. Note that the answer input unit 412 may acquire multiple answers from the LLM 421.

[0065] The reward assigning unit 413 assigns a reward value to the answer of the LLM 421 using the learned reward function 431. Note that the reward assigning unit 413 may assign a reward value to each of a plurality of answers of the LLM 421.

[0066] Based on the assigned reward value, the selection unit 414 selects answers from the LLM 421. For example, the selection unit 414 selects answers whose reward value is equal to or greater than a predetermined threshold. The output unit 415 outputs the selected answer. For example, the output unit 415 outputs the selected answer to the user via an interface.

[0067] Next, the operation of the device 400 according to the fourth embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the operation of the device 400 according to this embodiment.

[0068] As shown in FIG. 8, first, in step S301, the prompt input unit 411 accepts a prompt input from the user. Next, in step S302, the answer input unit 412 acquires the answer of the LLM 421 to the prompt. Next, in step S303, the reward granting unit 413 grants a reward value to the response of the LLM 321 using the learned reward function 431.

[0069] Next, in step S304, the selection unit 414 determines whether the reward value assigned to the answer is equal to or greater than a predetermined threshold. If the reward value is equal to or greater than the predetermined threshold (YES in step S304), the process proceeds to step S305. On the other hand, if the reward value is not equal to or greater than the predetermined threshold (YES in step S304), the process ends. In step S305, the output unit 415 outputs the response of the LLM 421 to the user.

[0070] In step S302, the answer input unit 412 may acquire multiple answers from the LLM 421 in response to the prompt. In this case, in step S303, the reward granting unit 413 grants rewards to the multiple answers from the LLM 421 using the learned reward function 431. In step S304, the selection unit 414 determines whether the reward values ​​assigned to the multiple answers are equal to or greater than a predetermined threshold. Then, in step S305, the output unit 415 outputs the selected answers from the LLM 421 to the user.

[0071] As described above, the utilization device 400 according to the fourth embodiment can utilize the learned reward function 431 to select (i.e., determine) the LLM's response to the prompt.

[0072] <Example of hardware configuration> Each functional component of learning device 100, learning device 200, utilization device 300, and utilization device 400 according to the above-described embodiments may be realized by hardware (e.g., a hardwired electronic circuit) that realizes each functional component, or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it). Below, we further explain the case where each functional component of the above devices is realized by a combination of hardware and software.

[0073] Figure 9 is a block diagram showing an example of the hardware configuration of computer 500 according to an embodiment. Learning device 100, learning device 200, utilization device 300, and utilization device 400 can all be realized by computer 500 having the hardware configuration shown in Figure 9. Computer 500 may be a portable computer such as a smartphone or tablet terminal, or a stationary computer such as a PC (Personal Computer). Computer 500 may be a dedicated computer or a general-purpose computer. For example, by installing a predetermined application on the computer 500, the computer 500 can be given desired functions.

[0074] The computer 500 has a bus 501, a processor 502, a memory 503, a storage device 504, an input / output interface (I / F) 505, and a network interface (I / F) 506. The bus 501 is a data transmission path for the processor 502, the memory 503, the storage device 504, the input / output interface 505, and the network interface 506 to transmit and receive data to and from each other. However, the method for connecting the processor 502 and the like to each other is not limited to bus connection.

[0075] The processor 502 is one of various processors such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 503 is a main storage device realized using a random access memory (RAM) or the like. The storage device 504 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read-only memory (ROM) or the like.

[0076] The input / output interface 505 is an interface for connecting the computer 500 to an input / output device. For example, the input / output interface 505 is connected to an input device such as a keyboard and an output device such as a display device.

[0077] The network interface 506 is an interface for connecting the computer 500 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0078] Programs for realizing desired functions are stored in storage device 504. For example, programs for realizing the functions of learning device 100, learning device 200, utilization device 300, and utilization device 400 are stored in storage device 504 of computer 500. Processor 502 reads these programs into memory 503 and executes them to realize each function.

[0079] These programs include instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The programs may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The programs may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0080] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention.

[0081] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) an answer input unit that accepts input of an LLM (large language model) answer to a prompt; a data perturbation unit that generates perturbation data by adding a perturbation to the answer using a reward function for determining the answer; an output unit that outputs the perturbation data; an annotation data input unit that receives input of data in which annotations are added to the perturbation data; a training processor that trains the reward function based on the annotated data. Learning device. (Appendix 2) The data perturbation unit assigning a reward value to each of the answer and perturbation data obtained by adding a perturbation to the answer using the reward function, and generating a sequence of the perturbation data based on the reward values; The learning device a data selection unit that selects perturbation data around a decision boundary of the reward function from the series of perturbation data, The output unit outputting the selected perturbation data 2. The learning device of claim 1. (Appendix 3) The data selection unit From the series of perturbation data, perturbation data to which a reward value is assigned that is within a predetermined value from a predetermined reward value as a decision boundary of the reward function is selected. 3. The learning device according to claim 2. (Appendix 4) The data perturbation unit Classifying perturbation data having different reward values ​​assigned by the reward function into a series of the perturbation data. 4. The learning device according to claim 2 or 3. (Appendix 5) The data perturbation unit assigning a reward value to the answer using the reward function; generating first perturbation data by adding a perturbation to the answer; assigning a reward value to the first perturbed data using the reward function; if a reward value for the first perturbation data has changed from a reward value for the answer by a predetermined threshold or more, adding the first perturbation data together with the answer to the sequence of perturbation data; generating second perturbed data by adding a perturbation to the first perturbed data; assigning a reward value to the second perturbed data using the reward function; If the reward value for the second perturbation data has changed from the reward value for the first perturbation data by a predetermined threshold or more, the second perturbation data is added to the sequence of perturbation data. 5. The learning device of claim 4. (Appendix 6) The data perturbation unit assigning a reward value to the answer using the reward function; generating first perturbation data by adding a first perturbation to the answer; assigning a reward value to the first perturbed data using the reward function; adding the first perturbation data together with the response to the sequence of perturbation data when the reward value for the first perturbation data has changed from the reward value for the response by more than a first threshold; generating second perturbation data by adding a second perturbation different from the first perturbation to the answer; assigning a reward value to the second perturbed data using the reward function; adding the second perturbation data to the perturbation data when the reward value for the second perturbation data changes from the reward value for the answer by more than a second threshold different from the first threshold; 5. The learning device of claim 4. (Appendix 7) The perturbation data are adversarial samples 7. A learning device according to any one of claims 1 to 6. (Appendix 8) a prompt input unit that accepts a prompt input; an answer input unit that accepts input of the LLM's answer to the prompt; a reward assigning unit that assigns a reward value to the answer of the LLM using a learned reward function; A learning processing unit that learns the LLM so as to maximize the reward value using a loss function, The learned reward function is The learning device according to any one of Supplementary Notes 1 to 7 is used for learning. Equipment used. (Appendix 9) a prompt input unit that accepts a prompt input; an answer input unit that accepts input of the LLM's answer to the prompt; a reward assigning unit that assigns a reward value to the answer of the LLM using a learned reward function; A selection unit that selects the answers of the LLM based on the assigned reward value; an output unit that outputs the selected answer, The learned reward function is The learning device according to any one of Supplementary Notes 1 to 7 is used for learning. Equipment used. (Appendix 10) The computer Accepts input of LLM (large language model) answers to prompts, generating perturbation data in which a perturbation is added to the answer using a reward function for judging the answer; outputting the perturbation data; accepting input of data annotated with the perturbation data; training the reward function based on the annotated data How to learn. (Appendix 11) Accepts input of LLM (large language model) answers to prompts, generating perturbation data in which a perturbation is added to the answer using a reward function for judging the answer; outputting the perturbation data; accepting input of data annotated with the perturbation data; and causing a computer to perform a process of training the reward function based on the annotated data. program. [Explanation of symbols]

[0082] 21 AE generation section 22 Answer example storage section 23 Reward function memory 24 AE generation function memory section 25 Annotation section 26 Reward Function Training Department 31 LLM Learning Department 32 LLM storage section 33 Reward function memory 34 Loss function memory 41 Answer Selection Department 42 LLM storage section 43 Reward function memory 100 Learning Device 101 Answer input section 102 Data Perturbation Unit 103 Output section 104 Annotation data input section 105 Training Processing Unit 200 Learning Device 211 Answer input section 212 Data Perturbation Unit 213 Data Selection Unit 214 Output section 231 Reward Function 241 AE creation functions 261 Annotation Data Input Section 262 Training Processing Unit 300 Equipment used 311 Prompt Input Section 312 Answer input section 313 Reward Division 314 Learning processing unit 331 Reward Function 341 Loss Function 400 Equipment used 411 Prompt Input Section 412 Answer input section 413 Reward Division 414 Sorting Department 415 Output Section 431 Reward Function 500 computers 501 Bus 502 processor 503 memory 504 Storage Devices 505 Input / Output Interface (I / F) 506 Network Interface (I / F)

Claims

1. an answer input unit that accepts input of an LLM (large language model) answer to a prompt; a data perturbation unit that generates perturbation data by adding a perturbation to the answer using a reward function for determining the answer; an output unit that outputs the perturbation data; an annotation data input unit that receives input of data in which annotations are added to the perturbation data; a training processor that trains the reward function based on the annotated data. Learning device.

2. The data perturbation unit assigning a reward value to each of the answer and perturbation data obtained by adding a perturbation to the answer using the reward function, and generating a sequence of the perturbation data based on the reward values; The learning device a data selection unit that selects perturbation data around a decision boundary of the reward function from the series of perturbation data, The output unit outputting the selected perturbation data The learning device according to claim 1 .

3. The data selection unit From the series of perturbation data, perturbation data to which a reward value is assigned that is within a predetermined value from a predetermined reward value as a decision boundary of the reward function is selected. The learning device according to claim 2 .

4. The data perturbation unit Classifying perturbation data having different reward values ​​assigned by the reward function into a series of the perturbation data. The learning device according to claim 2 .

5. The data perturbation unit assigning a reward value to the answer using the reward function; generating first perturbation data by adding a perturbation to the answer; assigning a reward value to the first perturbed data using the reward function; adding the first perturbation data together with the response to the sequence of perturbation data when a reward value for the first perturbation data has changed from a reward value for the response by a predetermined threshold or more; generating second perturbed data by adding a perturbation to the first perturbed data; assigning a reward value to the second perturbed data using the reward function; If a reward value for the second perturbation data has changed from a reward value for the first perturbation data by a predetermined threshold or more, the second perturbation data is added to the sequence of perturbation data. The learning device according to claim 4 .

6. The data perturbation unit assigning a reward value to the answer using the reward function; generating first perturbation data by adding a first perturbation to the answer; assigning a reward value to the first perturbed data using the reward function; adding the first perturbation data together with the response to the sequence of perturbation data when the reward value for the first perturbation data has changed from the reward value for the response by more than a first threshold; generating second perturbation data by adding a second perturbation different from the first perturbation to the answer; assigning a reward value to the second perturbed data using the reward function; adding the second perturbation data to the perturbation data when the reward value for the second perturbation data changes from the reward value for the answer by more than a second threshold different from the first threshold; The learning device according to claim 4 .

7. a prompt input unit that accepts a prompt input; an answer input unit that accepts input of the LLM's answer to the prompt; a reward assigning unit that assigns a reward value to the answer of the LLM using a learned reward function; a learning processing unit that learns the LLM so as to maximize the reward value using a loss function; The learned reward function is A learning device according to any one of claims 1 to 6 is used for learning. Equipment used.

8. a prompt input unit that accepts a prompt input; an answer input unit that accepts input of the LLM's answer to the prompt; a reward assigning unit that assigns a reward value to the answer of the LLM using a learned reward function; a selection unit that selects the LLM's answer based on the assigned reward value; an output unit that outputs the selected answer, The learned reward function is A learning device according to any one of claims 1 to 6 is used for learning. Equipment used.

9. The computer Accepts input of large language model (LLM) responses to prompts; generating perturbation data in which a perturbation is added to the answer using a reward function for judging the answer; outputting the perturbation data; accepting input of data annotated with the perturbation data; training the reward function based on the annotated data How to learn.

10. Accepts input of large language model (LLM) responses to prompts; generating perturbation data in which a perturbation is added to the answer using a reward function for judging the answer; outputting the perturbation data; accepting input of data annotated with the perturbation data; and causing a computer to perform a process of training the reward function based on the annotated data. program.

Images