Information processing apparatus, program, and information processing method

Abstract

To provide an information processing apparatus for improving the reliability of a content generation system using generative AI.SOLUTION: According to the present invention, an information processing apparatus for verifying a content generation system using a generative AI comprises control means and storage means, wherein the control means comprises an output information acquisition unit, an output information verification unit, a feedback information acquisition unit, a history information recording unit, and a reward model learning unit, the output information acquisition unit acquires AI output information from the generative AI in response to input from an input person who inputs a prompt to the content generation system, the output information verification unit verifies the AI output information, the feedback information acquisition unit acquires feedback information from the input person in response to system output information from the content generation system, the history information recording unit records the AI output information and the feedback information as history information in the storage means, and the reward model learning unit trains a reward model based on the history information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a program, and an information processing method. [Background technology]

[0002] In recent years, applications and services that generate text using large-scale language models (LLMs) have been developed. For example, Patent Document 1 discloses a device that generates question sentences (prompts) to be input into LLMs in order to improve the accuracy of answers from LLMs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7313757 Summary of the Invention [Problem to be solved by the invention]

[0004] However, even with the improved prompts, there was still a risk that LLMs and sentence generation systems using LLMs would generate inappropriate answers, making it impossible to provide or use the sentence generation systems safely. Furthermore, the problem of outputting such undesired results was not limited to LLMs and corresponding sentence generation systems, but also occurred in other generative AIs such as image and video generation AIs and corresponding content generation systems.

[0005] The present invention has been made in view of the above circumstances, and provides an information processing device for improving the reliability of a content generation system using a generation AI. [Means for solving the problem]

[0006] According to the present invention, the following inventions are provided. [1] An information processing device for verifying a content generation system using a generation AI, comprising a control means and a storage means, wherein the control means comprises an output information acquisition unit, an output information verification unit, a feedback information acquisition unit, a history information recording unit, and a reward model learning unit, wherein the output information acquisition unit acquires AI output information from the generation AI in response to input from an input person who inputs a prompt to the content generation system, the output information verification unit verifies the AI ​​output information, the feedback information acquisition unit acquires feedback information from the input person in response to system output information from the content generation system, the history information recording unit records the AI ​​output information and the feedback information in the storage means as history information, and the reward model learning unit learns a reward model based on the history information. [2] An information processing device according to [1], wherein the control means further includes an input information acquisition unit, which acquires at least one of user input information from the user to the content generation system and system input information from the content generation system to the generation AI, the history information recording unit records at least one of the user input information and the system input information as the history information in the storage means in addition to the AI ​​output information and the feedback information, and the reward model learning unit learns the reward model based on the history information. [3] An information processing device according to [2], wherein the input information acquisition unit and the feedback information acquisition unit provide an API to the content generation system in advance, and use the API to acquire at least one of the user input information and the system input information, and the feedback information, from the content generation system, respectively. [4] An information processing device according to any one of [1] to [3], wherein the output information acquisition unit provides an API to the content generation system in advance and acquires the AI ​​output information from the content generation system using the API. [5] An information processing device according to any one of [1] to [4], wherein the control means further includes an output information presentation unit, and the output information presentation unit presents the verification result to the person inputting information together with or instead of the system output information based on the verification result of the output information verification unit. [6] An information processing device according to any one of [1] to [5], wherein the reward model learning unit pre-learns the reward model based on the AI ​​output information and an evaluation of the AI ​​output information in a learning phase, the output information verification unit verifies the AI ​​output information using the reward model in a provision phase, and the reward model learning unit updates the reward model based on the history information in the provision phase. [7] [6] An information processing device according to the invention, wherein the control means further includes an input information acquisition unit, which acquires, during the learning phase, at least one of input user input information from the input user to the content generation system and system input information from the content generation system to the generation AI, and wherein, during the learning phase, the reward model learning unit trains the reward model based on at least one of the input user input information and the system input information for a learning dataset input by the input user, the AI ​​output information corresponding thereto, and an evaluation of the AI ​​output information. [8] An information processing device according to [6] or [7], wherein the output information verification unit further verifies at least one of harmfulness, accuracy, and emotion of the AI ​​output information in addition to verification using the reward model during the provision phase. [9] An information processing device according to [2] or [3], wherein the control means further includes an input information verification unit, and the input information verification unit verifies at least one of the harmfulness and emotions of the input information of the user.

[10] An information processing device according to any one of [1] to [9], wherein the reward model learning unit learns the reward model for each individual user entering data or for each of multiple users belonging to the same group.

[11] An information processing device according to any one of [1] to

[10] , wherein the generation AI is a large-scale language model, and the content generation system is a sentence generation system.

[12] An information processing method for verifying a content generation system using a generation AI, comprising: an output information acquisition process; an output information verification process; a feedback information acquisition process; a history information recording process; and a reward model learning process; wherein the output information acquisition process acquires AI output information from the generation AI in response to input from an input user who inputs a prompt to the content generation system; the output information verification process verifies the AI ​​output information; the feedback information acquisition process acquires feedback information from the input user in response to system output information from the content generation system; the history information recording process records the AI ​​output information and the feedback information in a storage means as history information; and the reward model learning process learns a reward model based on the history information.

[13] A program for verifying a content generation system using a generation AI, the program causing a computer to execute an output information acquisition process, an output information verification process, a feedback information acquisition process, a history information recording process, and a reward model learning process, wherein the output information acquisition process acquires AI output information from the generation AI in response to input from an input user who inputs a prompt to the content generation system, the output information verification process verifies the AI ​​output information, the feedback information acquisition process acquires feedback information from the input user in response to system output information from the content generation system, the history information recording process records the AI ​​output information and the feedback information in a storage means as history information, and the reward model learning process learns a reward model based on the history information. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the reliability of a content generation system that uses a generation AI. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing the overall configuration of a system including an information processing device 100 according to a first embodiment of the present invention. [Figure 2] FIG. 2A is a block diagram showing the hardware configuration of information processing device 100 in FIG. 1, and FIG. 2B is a block diagram showing the hardware configuration of user terminal 400. As shown in FIG. [Figure 3] 2B is a block diagram showing the functional configuration of the control means 1 of the information processing device 100 of FIG. 2A. FIG. [Figure 4] 1 is a sequence diagram showing the processing performed among the user terminal 400, the content generation system 200, the generation AI 300, and the information processing device 100 when the user U uses the content generation system 200. FIG. [Figure 5] 1. This is an example D1 of a comparison screen output by the content generation system 200 and displayed on the display means 405 of the user terminal 400 of FIG. [Figure 6] 1. This is an example screen D2 output by the content generation system 200 and displayed on the display means 405 of the user terminal 400 of FIG. [Figure 7] 1. This is an example D3 of a comparison screen output by the content generation system 200 and displayed on the display means 405 of the user terminal 400 of FIG. [Figure 8] 1. This is an example screen D4 output by the information processing device 100 and displayed on the display means 405 of the user terminal 400 in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following describes embodiments of the present invention. The various features shown in the following embodiments can be combined with each other. Furthermore, each feature can be an independent invention.

[0010] 1. First embodiment 1.1 Overall Configuration of a System Including an Information Processing Device 100 1 is a diagram showing the overall configuration of a system including an information processing device 100 according to one embodiment of the present invention, a content generation system 200 to which the information processing device 100 is applied, a generation AI 300, and a user terminal 400 used by a user U who uses the content generation system 200. Here, the generation AI 300 to which the information processing device 100 according to this embodiment is applied is an LLM (large-scale language model), and the content generation system 200 is a text generation system that generates text as content using the generation AI 300.

[0011] More specifically, LLM is an AI that learns from large amounts of text data and generates text that sounds like it would be written by a human. For example, it can complete sentences, answer questions, write essays, and return text in response to instructions or questions entered as prompts.

[0012] More specifically, the content generation system 200 receives user input information (inputter input information) from a user U as an inputter via a user terminal 400, processes the inputter input information appropriately, and then transmits it to the generation AI 300 as system input information. The content generation system 200 acquires AI output information from the generation AI and provides it to the user U as system output information. Examples of content generation systems 200 to which the information processing device 100 of this embodiment is applied include those created or customized for organizations in the same industry, such as call centers, sales departments, and technical departments. For customization, organization-specific data, such as response manuals and business manuals, is imported and used when processing user input information to generate system input information and when processing AI output information to generate system output information. However, the content generation system 200 may also be customized for each individual.

[0013] The information processing device 100 of this embodiment is a device used to verify whether such a content generation system 200 outputs an inappropriate answer sentence, that is, whether the generation AI 300 operating behind the content generation system 200 outputs an inappropriate answer sentence. The information processing device 100 of this embodiment will be described in detail below. In this embodiment, the information processing device 100, the content generation system 200, the generation AI 300, and the user terminal 400 are connected via a network N.

[0014] 1.2 Hardware Configuration of Information Processing Device 100 2A is a block diagram showing the hardware configuration of the information processing device 100. As shown in FIG. 2A, the information processing device 100 specifically includes a control unit 1, a storage unit 2, and a communication unit 3.

[0015] The control means 1 is composed of one or more processors such as a CPU (Central Processing Unit), and controls the overall operation of the information processing device 100 by executing a predetermined program stored in the storage means 2. The control means 1 may also include other processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor). At least a part of the control means 1 may be an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0016] A part of the storage means 2 is configured, for example, by RAM (Random Access Memory) or DRAM (Dynamic Random Access Memory), and is used as a work area when the control means 1 executes processes based on various programs. Also, a part of the storage means 2 is, for example, a non-volatile memory such as ROM (Read Only Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive), and stores various data and programs used in the processing of the control means 1. At least a part of the storage means 2 may be configured from an external cloud or distributed storage.

[0017] The programs stored in the storage means 2 include, for example, an OS (Operating System) for realizing the basic functions of the information processing device 100, drivers for controlling various hardware, programs for realizing various functions, etc., and include the computer program related to this embodiment.

[0018] Furthermore, as shown in Fig. 2A, the storage means 2 of this embodiment also stores history information HS and a reward model RM. The history information HS is information including system input information, AI output information, and feedback information that is feedback from the user U on the system output information. The reward model RM is a model for evaluating the output (answer) of the generation AI 300, and in this embodiment, is trained based on the history information HS. Details of the history information HS and the reward model RM will be described later.

[0019] The communication means 3 is, for example, a NIC (Network Interface Controller) and has a function of connecting to the network N. Note that the communication means 3 may have, instead of or together with the NIC, a function of connecting to a wireless LAN (Local Area Network), a function of connecting to a wireless WAN (Wide Area Network), a function of enabling short-range wireless communication such as Bluetooth (registered trademark), infrared communication, etc. The information processing device 100 is connected to the content generation system 200 and the user terminal 400 via the network N and can transmit and receive various data.

[0020] The control means 1, storage means 2, and communication means 3 are electrically connected to one another via a bus 4. Therefore, the control means 1 can access the storage means 2 and communicate with the content generation system 200, user terminal 400, etc. via the communication means 3.

[0021] 2, the information processing device 100 may be realized by a plurality of devices using so-called cloud or distributed computing technology, etc. Also, the various data (information) stored in the storage means 2 may not be configured in a single database as shown in the figure, but may be configured in a distributed database.

[0022] 1.3 Hardware Configuration of User Terminal 400 2B is a block diagram showing the hardware configuration of the user terminal 400. The user terminal 400 is a terminal used by a user U who uses the content generation system 200, and is, for example, an information processing terminal such as a personal computer (PC), a smartphone, a tablet terminal, or an in-vehicle terminal. As shown in FIG. 2B, the user terminal 400 specifically includes a control means 401, a storage means 402, a communication means 403, an input means 404, and a display means 405. These components are electrically connected to each other via a bus 406.

[0023] The general configurations of the control means 401, storage means 402, and communication means 403 are the same as those of the information processing device 100 described above, and therefore a description thereof will be omitted.

[0024] The input means 404 is a device that accepts input from the user U, and is composed of various input means such as a mouse, keyboard, touch panel, microphone, etc. The display means 405 is a liquid crystal display, touch panel display, etc., and displays images, etc. to the user U.

[0025] 1.4 Functional configuration of the information processing device 100 (control means 1) As shown in Fig. 3, the control means 1 of the information processing device 100 includes an input information acquisition unit 10, an output information acquisition unit 11, a feedback information acquisition unit 12, a history information recording unit 13, a reward model learning unit 14, an output information presentation unit 15, an input information verification unit 16, and an output information verification unit 17. With these functional components, the information processing device 100 of this embodiment verifies whether the content generation system 200 (generation AI 300) outputs an inappropriate answer sentence. Each functional component will be described below.

[0026] In this embodiment, the verification function of the information processing device 100 is provided by exchanging information with the content generation system 200 via an API. In addition, in FIG. 1 and the following description, "user input information," "system input information," "AI output information," "system output information," and "feedback information" may refer to the natural language itself input by or presented to the user U, depending on the context, or may refer to information that has been digitized (encoded) for transmission / reception or recording in the storage means 2. Furthermore, it is preferable that this information be encrypted.

[0027] The input information acquisition unit 10 acquires system input information from the content generation system 200 to the generation AI 300 using an API (Application Programming Interface) as an input information acquisition process (see step S4 in FIG. 4). Specifically, the input information acquisition unit 10 provides the API to the content generation system 200 in advance, and when the content generation system 200 transmits system input information to the generation AI 300, the input information acquisition unit 10 causes the content generation system 200 to also transmit the system input information to the information processing device 100 at the same time or around the same time.

[0028] As the output information acquisition process (see step S10 in FIG. 4), the output information acquisition unit 11 uses an API to acquire AI output information from the generation AI 300 to the content generation system 200. Specifically, the output information acquisition unit 11 provides the API to the content generation system 200 in advance, and when the content generation system 200 acquires the AI ​​output information from the generation AI 300, the output information acquisition unit 11 causes the content generation system 200 to transmit the AI ​​output information to the information processing device 100 at the same time or at a later timing.

[0029] The feedback information acquisition unit 12 acquires feedback information from the user U regarding the system output information using an API as a feedback information acquisition process (see step S18 in FIG. 4). Specifically, the feedback information acquisition unit 12 provides the API to the content generation system 200 in advance, and when the content generation system 200 acquires feedback information from the user terminal 400, the feedback information acquisition unit 12 causes the content generation system 200 to transmit the feedback information to the information processing device 100 simultaneously or at a later timing. The feedback information is the system output information presented to the user U by the content generation system 200 via the display means 405 of the user terminal 400, in other words, information on the user U's evaluation of the response to the prompt input by the user U. In this embodiment, the feedback information is specifically information on a two-level evaluation of "good / bad" (a three-level evaluation if no response is included), for example. However, the feedback information may be a multi-level evaluation (for example, a numerical evaluation such as 0 to 100 points) or a written evaluation.

[0030] As part of the history information recording process (see steps S5, S11, and S19 in FIG. 4), the history information recording unit 13 records the system input information acquired by the input information acquiring unit 10, the AI ​​output information acquired by the output information acquiring unit 11, and the feedback information acquired by the feedback information acquiring unit 12 as history information HS in the storage means 2. More specifically, the history information recording unit 13 associates the system input information, AI output information, and feedback information acquired in the series of steps in which the user U inputs a prompt, obtains a response, and provides feedback on the response, and records them as history information HS.

[0031] As a reward model learning process (see step S20 in FIG. 4), reward model learning unit 14 learns reward model RM based on history information HS recorded by history information recording unit 13. In this embodiment, learning of reward model RM includes pre-learning in the learning phase and updating in the provision phase. Here, pre-learning in the learning phase is mainly learning of reward model RM before user U uses content generation system 200, and updating in the provision phase is updating of reward model RM while user U actually uses content generation system 200.

[0032] Specifically, the pre-learning in the learning phase is performed by having an inputter other than the user U, such as a developer of the content generation system 200, an operator who operates the content generation system 200, a beta user of the content generation system 200, or even a provider of the information processing device 100 according to this embodiment, input a large number of prompts (at least several hundred prompts) as a learning dataset, and obtaining AI output information and evaluations (feedback information) of the system output information from the inputter for each prompt (i.e., system input information). Here, the number of inputters receiving feedback may be one or more. However, one reward model RM is created and trained for each inputter or for multiple inputters belonging to an organization (group) that uses the content generation system 200 for the same purpose. It is also conceivable to train the reward model RM by having another generation AI perform evaluations instead of a human.

[0033] On the other hand, updates in the provision phase are specifically performed each time history information HS is added by user U using content generation system 200, that is, each time a series of events occurs in which user U inputs a prompt, receives a response, and provides feedback on that response, or each time a predetermined amount of history information HS is accumulated (for example, every 100 times feedback is obtained).

[0034] Based on the verification result of the output information verification unit 17, the output information presentation unit 15 presents the verification result by the output information verification unit 17 to the user U via the display means 405 of the user terminal 400 together with or instead of the system output information as output information presentation processing (see warning message W in FIG. 6). Here, "instead of the system output information" means that if the verification result of the output information verification unit 17 is unfavorable and it is determined that the system output information should not be presented to the user U, the system output information is not displayed on the display means 405. Note that a specific method for presenting the verification result will be described later. Furthermore, the output information presentation unit 15 presents statistical information on inputs and outputs of the content generation system 200 based on the history information HS (see FIG. 8).

[0035] The input information verification unit 16 verifies the user input information acquired by the input information acquisition unit 10 as an input information verification process (see step S6 in FIG. 4). Specifically, as shown in FIG. 3, the input information verification unit 16 of this embodiment includes an input harmfulness filter 16a and an input emotion filter 16b, and is configured to verify the harmfulness and emotion of the user input information. Here, "harmfulness of user input information" refers to a state in which the user input contains inappropriate terms or expressions such as discrimination or prejudice, and "emotion of user input information" refers to a positive or negative emotional expression inferred from the user input. Note that the input information verification unit 16 may include only either the input harmfulness filter 16a or the input emotion filter 16b. Furthermore, the input information verification unit 16 may also include additional filters other than these.

[0036] The output information verification unit 17 verifies the AI ​​output information acquired by the output information acquisition unit 11 as an output information verification process (see step S13 in FIG. 4). Specifically, as shown in FIG. 3, the output information verification unit 17 of this embodiment includes an output harmfulness filter 17a, an output emotion filter 17b, a fact check filter 17c, and a reward model filter 17d. The output harmfulness filter 17a verifies the harmfulness of the AI ​​output information, and the output emotion filter 17b verifies the emotion of the AI ​​output information. Here, "harmfulness of the AI ​​output information" refers to a state in which the AI ​​output contains inappropriate terms or expressions such as discrimination or prejudice, and "emotion of the AI ​​output information" refers to positive or negative emotional expressions inferred from the AI ​​output. Furthermore, the fact check filter 17c verifies whether the content of the AI ​​output information is in line with the facts (accuracy) and outputs the degree of agreement with the facts. In addition, the reward model filter 17d verifies the AI ​​output information using a reward model RM. Specifically, the reward model filter 17d evaluates (e.g., a numerical evaluation from 0 to 1) the system output information (which is essentially the same as the AI ​​output information output by the generation AI 300) output by the content generation system 200 in response to the system input information. However, the evaluation of the system output information by the reward model filter 17d can also be classified as, for example, A, B, C, etc. Verification of the AI ​​output information by the reward model filter 17d will be described later. Note that the output information verification unit 17 does not necessarily have to include at least one of the output emotion filter 17b, the fact check filter 17c, and the reward model filter 17d. Furthermore, the output information verification unit 17 may also include additional filters other than these.

[0037] The above-described functional configurations and processes may be realized by software (including so-called applications) appropriately installed in the information processing device 100, or may be realized by hardware. When realized by software, the control means 1 can realize various functions by executing programs that constitute the software. Furthermore, the functions may be realized by multiple pieces of software rather than a single piece of software.

[0038] When the information processing device 100 is implemented by executing a program, the program may be stored in a storage unit 2 built into the information processing device 100, or may be stored in a computer-readable non-transitory recording medium. Alternatively, the information processing device 100 may be implemented by reading out a program stored in an external storage device and using so-called cloud computing. Alternatively, when the information processing device 100 is implemented by hardware, the information processing device 100 may be implemented by various circuits such as an ASIC, a SOC, an FPGA, or a DRP. Furthermore, at least a part of the above-described functional configuration may be processed by software or hardware on a user terminal 400 that accepts input.

[0039] Furthermore, the above-described functional configuration may be realized by a plurality of computers, and in that case, each of the above-described functional configurations may be distributed and arranged among the plurality of computers.

[0040] 1.5 Information Processing Method by Information Processing Device 100 Next, an information processing method in which the information processing device 100 of this embodiment verifies the output of the content generation system 200 (output of the generation AI 300) will be described using the sequence diagram of Fig. 4 and screen examples D1 to D4 of Fig. 5 to Fig. 8. Note that the sequence diagram of Fig. 4 includes not only the processing of the information processing device 100, but also the processing of the user terminal 400, the content generation system 200, and the generation AI 300. Also, as described above, verification by the information processing device 100 of this embodiment is realized by sending and receiving necessary information to and from the content generation system 200 via the API.

[0041] Specifically, in the information processing method of this embodiment, first, in step S1, the user terminal 400 accepts an input, i.e., a prompt, from the user U to the content generation system 200 via the input means 404. Next, in step S2, the user terminal 400 transmits the accepted user input information to the content generation system 200. Next, in step S3, the content generation system 200 processes the user input information as necessary. Here, one possible example of processing the user input information by the content generation system 200 is to add supplemental information to the prompt entered by the user U using a Retrieval Augmented Generation (RAG) mechanism.

[0042] Next, in step S4, the content generation system 200 transmits the processed user input information to the information processing device 100 as system input information, and the input information acquisition unit 10 of the control means 1 of the information processing device 100 acquires the system input information.

[0043] Next, in step S5, the history information recording unit 13 of the control means 1 records the acquired history information HS by the input information acquisition unit 10 in the storage means 2. In step S6, the input information verification unit 16 of the control means 1 verifies the system input information, and in step S7, transmits the verification result of the system input information to the content generation system 200. Note that how the verification result by the information processing device 100 is utilized can be freely set on the content generation system 200 side. For example, if the input contains harmful words or sentences, it is conceivable to warn the user U or block transmission of the input from the user U to the generation AI 300.

[0044] Next, in step S8, the content generation system 200 sends the system input information to the generation AI 300 unless the verification result of the system input information indicates that the system input information should not be sent to the generation AI 300. In step S9, the generation AI 300 receives the system input information as a prompt and sends the AI ​​output information to the content generation system 200 as a response.

[0045] Next, in step S10, the content generation system 200 transmits the AI ​​output information acquired from the generated AI 300 to the information processing device 100, and the output information acquisition unit 11 of the control means 1 of the information processing device 100 acquires the AI ​​output information.

[0046] Next, in step S11, the history information recording unit 13 of the control means 1 records the AI ​​output information acquired by the output information acquisition unit 11 as history information HS in the storage means 2. In addition, the output information verification unit 17 of the control means 1 verifies the AI ​​output information in step S12, and transmits the verification result to the content generation system 200 in step S13.

[0047] Next, in step S14, the content generation system 200 processes the AI ​​output information based on the verification result of the AI ​​output information acquired from the information processing device 100, and generates system output information to be transmitted to the user terminal 400. In step S15, the generated system output information is transmitted to the user terminal 400 and displayed on the display means 405.

[0048] FIG. 5 shows a comparative screen example D1 in which a user U asks, "What is the smallest prefecture in Japan?" in the prompt input field A1 of the content generation system 200, and the answer, "The smallest prefecture is Hokkaido," is displayed in the response field A2. Here, the smallest prefecture is not actually Hokkaido, so this answer can be considered inappropriate. When the generation AI 300 outputs such an answer as the AI ​​output information, the output information verification unit 17 of the information processing device 100, for example, determines that the AI ​​output information is an output that may be hallucination (i.e., plausible false information), and the output information presentation unit 15 transmits the verification result to the content generation system 200. As a result, the content generation system 200 can process the AI ​​output information and generate system output information that includes a warning message W to warn that the output may be hallucination, as shown in the screen example D2 of FIG. 6.

[0049] 7 is a comparison screen example D3 showing an example in which the user U points out an error in the prompt input field A1 in response to an incorrect answer from the generation AI 300 as shown in FIG. 5, resulting in the generation AI 300 providing a harmful answer (insult) to the user U as AI output information. When the generation AI 300 outputs such an answer as AI output information, the output information verification unit 17 of the information processing device 100 determines, for example, that the AI ​​output information is an output that should not be presented to the user U, and transmits the verification result to the content generation system 200. This enables the content generation system 200 to avoid presenting a harmful answer to the user U. In this case, the content generation system 200 can generate, instead of the AI ​​output information, system output information indicating, for example, that the output content cannot be displayed because the output was problematic.

[0050] Next, in step S16, the user terminal 400 receives feedback on the system output information (for example, a two-level evaluation of good / bad) from the user U via the input means 404. Then, in step S17, the user terminal 400 transmits the received feedback information to the content generation system 200. In step S18, the content generation system 200 transmits the feedback information to the information processing device 100, and the feedback information acquisition unit 12 of the control means 1 acquires the feedback information.

[0051] Next, in step S19, the history information recording unit 13 of the control means 1 records the feedback information acquired by the feedback information acquisition unit 12 as history information HS in the storage means 2. The history information recording unit 13 is configured to record the system input information in step S5, the AI ​​output information in step S11, and the feedback information in step S19 in association with each other.

[0052] Finally, in step S20, the reward model learning unit 14 of the control means 1 updates the reward model RM based on the recorded history information HS.

[0053] In the information processing method of this embodiment, through the series of steps S1 to S20 described above, the reward model RM is updated each time the user U provides feedback using the content generation system 200, or each time a predetermined amount of history information HS is accumulated.

[0054] 8, the output information presenting unit 15 of the information processing device 100 can also present the latest statistical information on inputs and outputs of the content generation system 200 to the user U on a web page, by email, or the like, based on the recorded history information HS. A screen example D4 shown in FIG. 8 shows statistical data on the harmfulness of the AI ​​output information (harmfulness of text).

[0055] 1.6 Effects (1) In the information processing device 100 according to this embodiment, the output information acquisition unit 11 acquires AI output information from the generated AI 300, and the output information verification unit 17 verifies the AI ​​output information based on the reward model RM. The feedback information acquisition unit 12 acquires feedback information regarding the system output information from the user U via the content generation system 200, the history information recording unit 13 records the feedback information together with the AI ​​output information as history information HS, and the reward model learning unit 14 learns and updates the reward model RM based on the history information HS. Because the information processing device 100 according to this embodiment has this configuration, the output information verification unit 17 can verify the content generation system 200 (system output information) based on the reward model RM, which is updated daily based on the feedback information. This makes it possible to improve the reliability of the content generation system 200 in a constantly changing society.

[0056] (2) In the information processing device 100 according to this embodiment, the input information acquisition unit 10 acquires system input information and records it together with AI output information and feedback information as history information HS, and the reward model learning unit 14 learns and updates the reward model RM based on the history information HS. In this way, by also using the system input information in learning and updating the reward model RM, it is possible to improve the accuracy of learning and updating the reward model RM.

[0057] (3) In the information processing device 100 according to this embodiment, the input information acquisition unit 10, the output information acquisition unit 11, and the feedback information acquisition unit 12 are configured to acquire corresponding information from the content generation system 200 via APIs. This allows the output information verification unit 17 to verify the content generation system 200 and to train and update the reward model RM, even when information cannot be acquired directly from the generation AI 300 or the user terminal 400. It is also preferable to design the API so that when the content generation system 200 provides each piece of information to the information processing device 100, the information is encrypted (vectorized). This allows the developer and operator of the content generation system 200, and each user U, to use the information processing device 100 with peace of mind.

[0058] (4) In the information processing device 100 according to this embodiment, the output information presenting unit 15 displays the verification result by the output information verifying unit 17 on the display means 405 of the user terminal 400 used by the user U, together with or instead of the system output information from the content generation system 200. This allows the user U of the content generation system 200 to determine whether the response from the content generation system 200 (or the response from the generation AI) is trustworthy.

[0059] (5) In the information processing device 100 according to this embodiment, the reward model learning unit 14 pre-learns the reward model RM using a training dataset in the learning phase, and then in the subsequent provision phase, the output information verification unit 17 verifies the AI ​​output information using the trained reward model RM. This makes it possible to output verification results customized for the user U from the beginning of providing the content generation system 200 incorporating the information processing device 100 to the user U.

[0060] (6) In the information processing device 100 according to this embodiment, the control means 1 includes the output information verification unit 17 that includes the output harmfulness filter 17a, the output emotion filter 17b, and the fact check filter 17c in addition to the reward model filter 17d. This makes it possible to perform comprehensive verification, not limited to verification using the reward model filter 17d that is tuned based on feedback from the user U.

[0061] (7) Furthermore, the control means 1 of this embodiment also includes an input information verification unit 16 equipped with an input harmfulness filter 16a and an input emotion filter 16b, which makes it possible to prevent the reward model RM from inappropriately learning due to inappropriate user input information, AI output information (system output information) based on the inappropriate user input information, and feedback information thereto. In addition, it is possible to protect the content generation system 200 from inappropriate input.

[0062] (8) In this embodiment, the reward model learning unit 14 creates, pre-trains, and updates a reward model RM for each user U or for each group of users U (inputters) who use the content generation system 200 for the same purpose. As a result, the reward model RM is generated and updated to suit the purpose of the single user or group, and the verification results are fine-tuned, making it possible to provide each user U with more accurate verification results than when all users U use the same reward model (general-purpose reward model).

[0063] (9) In this embodiment, the content generation system 200 to which the information processing device 100 is applied is a text generation system, and the generation AI 300 is an LLM (large-scale language model). Inappropriate responses (answers) from LLMs and text generation systems using them have become particularly problematic in recent years because they are easily identified as inappropriate. When an inappropriate response is reported, developers and operators of the content generation system 200 using external LLM services may be forced to suspend system provision until the LLM service responds, or may be forced to deal with the system-side problem. In this regard, the information processing device 100 according to this embodiment verifies the output of the content generation system 200 and the generation AI 300 from a third-party perspective, neither the content generation system 200 nor the generation AI 300, and updates the reward model RM daily. This allows developers and operators of the content generation system 200 to provide the system with peace of mind, even if they do not themselves respond to inappropriate responses.

[0064] 3. Variations The present invention can also be implemented in the following aspects.

[0065] In the above embodiment, the information processing device 100 was configured to verify a text generation system using a large-scale language model (LLM) as the content generation system 200 using the generation AI 300. However, the information processing device 100 of the present invention may also be configured to verify an image generation system using an AI to generate images, a video generation system using an AI to generate videos, or even a music generation system using an AI to generate music.

[0066] In the above embodiment, information processing device 100 was configured such that history information recording unit 13 records, as history information HS, not only the AI ​​output information acquired by output information acquisition unit 11 but also the user input information acquired by input information acquisition unit 10, together with feedback information acquired by feedback information acquisition unit 12, and reward model learning unit 14 learns and updates reward model RM based on the history information HS. However, history information recording unit 13 may learn and update reward model RM based on the AI ​​output information and feedback information, i.e., not based on user input information.

[0067] The input information acquisition unit 10 in this embodiment is configured to acquire, as AI input information, the processed prompt sent by the content generation system 200 to the generation AI 300. However, instead of this, the input information acquisition unit 10 may acquire, as AI input information, the raw prompt sent by the user terminal 400 to the content generation system 200 (the prompt itself entered by the user U). In this case, the reward model RM is learned and updated based on this information in addition to the AI ​​output information and feedback information.

[0068] In the above embodiment, the input information acquisition unit 10, the output information acquisition unit 11, and the feedback information acquisition unit 12 are each configured to acquire corresponding information from the content generation system 200 via an API. However, the information acquisition method of the input information acquisition unit 10, the output information acquisition unit 11, and the feedback information acquisition unit 12 according to the present invention is not limited to this. For example, at least one of the input information acquisition unit 10 and the feedback information acquisition unit 12 may acquire corresponding information directly from the user terminal 400 via the network N. For example, acquisition may be via a web browser or using an application resident on the terminal, such as antivirus software. Alternatively, the output information acquisition unit 11 may acquire a response directly from the generation AI 300. Furthermore, the information processing device 100 may directly receive input from the content generation system 200 and send the input to the generation AI 300 as a prompt, and the information processing device 100 may directly receive a response from the generation AI 300 and send the response to the content generation system 200. In this case, the information processing device 100 is disposed between the content generation system 200 and the generation AI 300 and functions as a firewall for the content generation system 200 or the generation AI 300.

[0069] In the above embodiment, the output information acquisition unit 11 of the information processing device 100 acquires the AI ​​output information, records the AI ​​output information together with the user input information and feedback information as history information HS, and uses it for training the reward model RM. However, it is also possible for the output information acquisition unit 11 to acquire system output information instead of the AI ​​output information, record it as history information HS, and use it for training the reward model RM. However, if the system output information already includes the verification result sent by the output information presentation unit 15 of the information processing device 100, a loop will occur if the output information verification unit 17 verifies it again. Therefore, it is necessary to extract only the output of the generated AI 300 from the system output information and perform verification, etc.

[0070] In the above embodiment, the output information presenting unit 15 is configured to transmit the verification result by the output information verifying unit 17 to the content generation system 200, and process the AI ​​output information on the content generation system 200 to display the warning message W (see FIG. 6). However, the verification result by the output information verifying unit 17 can also be displayed separately from the content generation system 200, for example, via a web browser, a pop-up, or a push notification. It is also preferable to display the verification result on the content generation system 200 and further enable confirmation together with separate statistical data on a web browser or the like.

[0071] In the above embodiment, reward model learning unit 14 pre-learns reward model RM after applying information processing device 100 to content generation system 200 and before content generation system 200 is provided to user U. However, information processing device 100 according to the present invention can also provide reward model RM to user U without pre-learning it. In this case, reward model RM is learned based only on feedback from user U.

[0072] In the above embodiment, the information processing device 100 acquires system input information by the input information acquisition unit 10 and AI output information by the output information acquisition unit 11 at different times (step S4 / step S10). However, it is also possible to acquire system input information and AI output information simultaneously. In this case, subsequent recording and verification of the system input information and AI output information as history information HS can also be performed simultaneously.

[0073] In the above embodiment, it was assumed that feedback would be received from the user U regarding the answer (AI output information) by the generated AI 300 among the system output information, but separately from this, feedback may also be received regarding the verification results by the output information verification unit 17 presented by the output information presentation unit 15.

[0074] In the above embodiment, the information processing device 100 is connected to the content generation system 200 via a network N and verifies the content generation system 200 (generation AI 300) as a so-called web service. However, the information processing device 100 may be configured on-premise on the same local network or the same private cloud as the content generation system 200.

[0075] In the above embodiment, the content generation system 200 and the generation AI 300 are each connected to the user terminal 400 via a network N to provide various functions to the user U. However, the content generation system 200 and the generation AI 300 may each be provided as a system running on an integrated circuit mounted on the user terminal 400, specifically, a local device such as a PC, mobile terminal, automobile, or medical device, known as an edge AI. In this case, at least some of the functional elements of the information processing device 100 can be located on-premise on the same user terminal 400 (local device). Furthermore, even if the generation AI 300 is on the network N and only the content generation system 200 is on the user terminal 400, the information processing device 100 can also be located on-premise on the same device as the content generation system 200. [Explanation of symbols]

[0076] 1: Control means 2: Storage means 3: Means of communication 4: Bus 10: Input information acquisition section 11: Output information acquisition section 12: Feedback information acquisition unit 13: History information recording section 14: Reward model learning section 15: Output information presentation section 16: Input information verification section 16a: Input Toxicity Filter 16b: Input emotion filter 17: Output information verification section 17a: Output Toxicity Filter 17b: Output emotion filter 17c: Fact-checking filters 17d: Reward model filter 100: Information processing device 200: Content Generation System 300: Generation AI 400: User terminal 401: Control means 402: Storage means 403: Communication methods 404: Input method 405:Display means 406: Bus A1: Prompt input field A2: Response column HS: History information N: Network RM: Remuneration Model U: User W: Warning text

Claims

1. An information processing device for verifying a content generation system using generation AI, A control means and a storage means are provided, the control means includes an output information acquisition unit, an output information verification unit, a feedback information acquisition unit, a history information recording unit, and a reward model learning unit; The output information acquisition unit acquires AI output information from the generation AI in response to an input from an input person who inputs a prompt to the content generation system, The output information verification unit verifies the AI ​​output information, the feedback information acquisition unit acquires feedback information from the person inputting information in response to system output information from the content generation system; the history information recording unit records the AI ​​output information and the feedback information as history information in the storage means, The reward model learning unit learns a reward model based on the history information.

2. 2. The information processing device according to claim 1, The control means further includes an input information acquisition unit, The input information acquisition unit acquires at least one of input person input information from the input person to the content generation system and system input information from the content generation system to the generation AI, the history information recording unit records at least one of the user input information and the system input information as the history information in addition to the AI ​​output information and the feedback information in the storage means; The reward model learning unit learns the reward model based on the history information.

3. 3. The information processing device according to claim 2, The input information acquisition unit and the feedback information acquisition unit provide an API to the content generation system in advance, and use the API to acquire at least one of the user input information and the system input information, and the feedback information, from the content generation system, respectively.

4. 2. The information processing device according to claim 1, The output information acquisition unit provides an API to the content generation system in advance and acquires the AI ​​output information from the content generation system using the API.

5. 2. The information processing device according to claim 1, The control means further includes an output information presentation unit, The information processing device wherein the output information presenting unit presents the verification result to the person inputting information together with or instead of the system output information based on the verification result of the output information verifying unit.

6. 2. The information processing device according to claim 1, The reward model learning unit, in a learning phase, pre-learns the reward model based on the AI ​​output information and an evaluation of the AI ​​output information; The output information verification unit verifies the AI ​​output information using the reward model in the provision phase, The information processing device wherein the reward model learning unit updates the reward model based on the history information in the provision phase.

7. 7. The information processing device according to claim 6, The control means further includes an input information acquisition unit, The input information acquisition unit acquires, in the learning phase, at least one of input person input information from the input person to the content generation system and system input information from the content generation system to the generation AI; The reward model learning unit, during the learning phase, learns the reward model based on at least one of the user input information and the system input information for the learning dataset input by the user, the AI ​​output information corresponding thereto, and an evaluation of the AI ​​output information.

8. 7. The information processing device according to claim 6, The information processing device, wherein the output information verification unit further verifies at least one of harmfulness, accuracy, and emotion of the AI ​​output information in addition to verification using the reward model in the provision phase.

9. 3. The information processing device according to claim 2, The control means further includes an input information verification unit, The input information verification unit verifies at least one of harmfulness and emotion of the information input by the user.

10. 2. The information processing device according to claim 1, The information processing device, wherein the reward model learning unit learns the reward model for each individual user who inputs data or for each of multiple users who belong to the same group.

11. The information processing device according to any one of claims 1 to 10, An information processing device, wherein the generative AI is a large-scale language model and the content generation system is a sentence generation system.

12. An information processing method for verifying a content generation system using generation AI, comprising: An output information acquisition process, an output information verification process, a feedback information acquisition process, a history information recording process, and a reward model learning process are performed; In the output information acquisition process, AI output information is acquired from the generation AI in response to an input from an input person who inputs a prompt to the content generation system, In the output information verification process, the AI ​​output information is verified, the feedback information acquisition process acquires feedback information from the input person in response to system output information from the content generation system; In the history information recording process, the AI ​​output information and the feedback information are recorded as history information in a storage means; In the reward model learning process, a reward model is learned based on the history information.

13. A program for verifying a content generation system using generative AI, causing a computer to execute an output information acquisition process, an output information verification process, a feedback information acquisition process, a history information recording process, and a reward model learning process; In the output information acquisition process, AI output information is acquired from the generation AI in response to an input from an input person who inputs a prompt to the content generation system, In the output information verification process, the AI ​​output information is verified, the feedback information acquisition process acquires feedback information from the input person in response to system output information from the content generation system; In the history information recording process, the AI ​​output information and the feedback information are recorded as history information in a storage means; In the reward model learning process, a reward model is learned based on the history information.

Images