Guardrail system and program

Abstract

To enable safer text generation using generative models. [Solution] The guardrail system (1) includes an acquisition unit (11) that sequentially acquires text to be input to a generative model that receives input from a user interface (2), or text output by the generative model; a determination unit (12) that determines whether the acquired text satisfies predetermined action start conditions each time a predetermined determination condition is met; and an action unit (14) that performs predetermined action when the determination unit determines that the text satisfies the action start conditions.

Description

Technical Field

[0001] The present disclosure relates to a guardrail system and a program.

Background Art

[0002] Non-Patent Document 1 describes the situation of efforts in a guardrail system for ensuring the safety of large language models (LLMs) and AI.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Means for Solving the Problems

[0004] In order to solve the above problems, a guardrail system according to one aspect of the present disclosure includes an acquisition unit that sequentially acquires text for inputting to a generation model that has received an input at a user interface, or text output by the generation model, a determination unit that determines whether the acquired text satisfies a predetermined countermeasure start condition every time a predetermined determination condition is satisfied, and a countermeasure unit that performs a predetermined countermeasure when the determination unit determines that the text satisfies the countermeasure start condition.

[0005] Furthermore, a program relating to another aspect of this disclosure causes the processor to execute an acquisition process that sequentially acquires text to be input to a generative model that receives input at a user interface, or text output by the generative model; a determination process that determines whether the acquired text satisfies predetermined action start conditions each time a predetermined determination condition is met; and an action process that performs predetermined action when the determination process determines that the text satisfies the action start conditions. [Brief explanation of the drawing]

[0006] [Figure 1] This block diagram shows an example of a schematic configuration of an information generation system equipped with a guardrail system according to the embodiment of this disclosure. [Figure 2] This figure shows an example of a screen displayed by the user interface of the information generation system according to the embodiment. [Figure 3] This is a block diagram showing an example of the functional configuration of a guardrail system according to an embodiment. [Figure 4] This table shows an example of the relationship between the categories used by the guardrail system according to the embodiment when classifying text and the actions taken. [Figure 5] This table shows another example of the relationship between the categories used by the guardrail system according to the embodiment when classifying text and the actions taken. [Figure 6] This is a sequence diagram showing an example of the operation of the guardrail system according to the embodiment. [Figure 7] This figure shows an example of a screen displayed by the user interface according to the embodiment. [Figure 8] This figure shows another example of a screen displayed by the user interface according to the embodiment. [Figure 9] This is a sequence diagram showing another example of the operation of the guardrail system according to the embodiment. [Modes for carrying out the invention]

[0007] <Background, challenges, solutions, and principles of solutions> Prior to describing each embodiment of this disclosure, the background of this disclosure, the problems of the prior art, the means for solving the problems, and the principle of solving them by the means for solving them will be explained.

[0008] 〔background〕 In recent years, guardrail systems have become capable of detecting various types of hazardous materials.

[0009] [Challenge 1] However, conventional guardrail systems could only offer monotonous solutions (such as warning signs) regardless of the type of hazardous material present.

[0010] [Solution 1] To solve the above problems, a guardrail system according to one aspect of Disclosure 1 includes: an acquisition unit that acquires text to be input to a generation model that receives input from a user interface, or text generated by the generation model; a determination unit that determines whether the acquired text satisfies predetermined action initiation conditions; a classification unit that, if the determination unit determines that the text satisfies the action initiation conditions, classifies the text into one of several categories according to predetermined classification criteria; and an action unit that performs action according to the category classified by the classification unit.

[0011] [Solution Principle 1] This guardrail system classifies text that meets the conditions for initiating action (for example, containing harmful or specialized content) into various categories and takes appropriate action for each category. Therefore, this guardrail system allows for actions such as issuing warnings or requesting re-entry for content with relatively low harmfulness, and taking safer actions such as stopping operation or regenerating content with relatively high harmfulness.

[0012] [Challenge 2] In addition, the conventional guardrail monitored whether the text contained harmful content after the user's input or the output of the generation model was completed. For this reason, there were problems such as it taking time until the text was displayed, or harmful content being temporarily displayed.

[0013] [Solution 2] In order to solve the above problems, a guardrail system according to one aspect of Disclosure 2 includes an acquisition unit that sequentially acquires text for inputting to a generation model that has received an input through a user interface, or text generated by the generation model, and a determination unit that determines whether the acquired text satisfies a predetermined countermeasure start condition every time a predetermined determination condition is satisfied, and a countermeasure unit that performs a predetermined countermeasure when the determination unit determines that the text satisfies the countermeasure start condition.

[0014] [Solution Principle 2] This guardrail system repeatedly determines whether the text sequentially acquired by the acquisition unit satisfies a predetermined countermeasure start condition every time a predetermined determination condition is satisfied. That is, this guardrail system monitors the acquired text in real time. For this reason, according to this guardrail system, when the determination unit determines that the countermeasure start condition is satisfied (for example, the text contains harmful content) during the acquisition of the text, a countermeasure (for example, display of a warning, temporary stop of text input / output, etc.) can be performed at the time of determination.

[0015] [Embodiment] Next, an embodiment of the present disclosure will be described in detail with reference to the drawings.

[0016] [Configuration of Information Generation System 100] FIG. 1 is a block diagram showing an example of the schematic configuration of an information generation system 100 including a guardrail system according to the present embodiment. As shown in FIG. 1, the information generation system 100 includes, in addition to the guardrail system 1, at least one user interface 2 (hereinafter, UI2) and a generation model 3. The guardrail system 1, UI2, and generation model 3 are connected to each other via a communication network.

[0017] 〔UI2〕 UI2 receives text input by the user. Also, UI2 presents the output of the generation model 3 to the user. The UI2 according to the present embodiment is composed of, for example, a PC, a tablet terminal, a mobile phone, etc. UI2 includes a communication unit 21, a reception unit 22, and a display unit 23.

[0018] (Communication unit 21) The communication unit 21 communicates with the guardrail system 1. The communication unit 21 according to the present embodiment is composed of a communication module.

[0019] (Reception unit 22) The reception unit 22 is for the user to input text. The reception unit 22 according to the present embodiment is composed of a keyboard, a touch panel laminated on the display unit 23, etc. Note that the reception unit 22 may be configured to sequentially convert the user's speech into text.

[0020] (Display unit 23) The display unit 23 displays various texts acquired from the guardrail system 1. In this embodiment, the display unit 23 displays an input / output screen S, for example, as shown in Figure 2. The display unit 23 displays the text received as input to the generation model in the input field Sa of the input / output screen S, and displays the text generated by the generation model 3 in the answer field Sb. The display unit 23 also displays various information (for example, warnings, etc.) in the other display field Sc of the input / output screen S. The input / output screen S may be displayed in a browser or it may be a dedicated application installed in the UI2. The input / output screen S may also be a chat-style screen that alternately displays the received input text and the text generated by the generation model 3. Hereinafter, the received input text will be referred to as "input text," and the text generated by the generation model 3 will be referred to as "generated text." When there is no distinction between input text and generated text, it will simply be referred to as "text."

[0021] [Generative Model 3] Generative model 3 is constructed to generate generated text with content corresponding to the input text when input text is received. Generative model 3 in this embodiment is a Large Language Model (LLM). Generative model 3 may be constructed to generate generated text after the input text has been entered, or it may be constructed to sequentially generate text with content corresponding to the entered portion from a stage in the input text being entered.

[0022] [Guardrail System 1] Figure 3 is a block diagram showing an example of the functional configuration of the guardrail system 1 according to this embodiment. The guardrail system 1 intervenes between the UI 2 and the generation model 3 and monitors the input text and the generated text. As shown in Figure 3, the guardrail system 1 includes an acquisition unit 11, a determination unit 12, a classification unit 13, and a handling unit 14. The guardrail system 1 according to this embodiment further includes a pre-processing unit 15, an input unit 16, an output unit 17, and a recording unit 18.

[0023] (Acquisition part 11) The acquisition unit 11 acquires text. In this embodiment, the acquisition unit 11 acquires text sequentially. That is, each time the UI2 receives a character input, or each time the generation model 3 outputs a character, the acquisition unit 11 acquires that character.

[0024] (Preprocessing step 15) The preprocessing unit 15 performs normalization on the text. Normalization includes processes such as unifying text containing a mix of full-width and half-width characters to either all full-width or all half-width characters, removing line breaks and unnecessary spaces, and normalizing special characters and symbols.

[0025] (Judgment part 12) The determination unit 12 determines whether the acquired text satisfies predetermined action initiation conditions. These conditions include whether the text contains harmful content. In this embodiment, the determination unit 12 uses a language model or a machine learning model to determine whether the acquired text satisfies predetermined action initiation conditions. When using a language model, the determination unit 12 utilizes natural language processing technology to analyze phrases contained in the text and the context of the sentences it constitutes. When using a machine learning model (e.g., a classifier), the determination unit 12 detects harmful keywords through vocabulary-based filtering. The action initiation conditions may also include whether the text is a question or consultation on a specialized field, or an answer to such a question or consultation.

[0026] In this embodiment, the determination unit 12 determines whether the acquired text satisfies a predetermined action start condition each time a predetermined determination condition is met. The determination condition is that a predetermined time (for example, within the range of 0.1 to 1.0 seconds) has elapsed since the previous determination of whether the action start condition is met. As a result, the determination is repeated multiple times while the user is inputting text or while the generation model 3 is generating text, and the user's input or the output of the generation model 3 can be monitored in real time. The determination condition may also be that a predetermined number of characters (for example, within the range of 10 to 100 characters) of text has been input. Furthermore, in this embodiment, the determination unit 12 determines whether the action start condition is met between the time the predetermined determination condition is met and the time a predetermined waiting period has elapsed.

[0027] The decision condition may also be the acceptance of text input in UI2, or the completion of text generation by the generation model 3. In other words, the decision unit 12 may be configured to determine whether the action start condition is met after the acquisition unit 11 has acquired all the text. Furthermore, if an interaction takes place between the user and the generation model 3, the decision unit 12 may be configured to determine whether the action start condition is met for the input text corresponding to the user's last utterance (input) in that interaction. Whether or not an interaction has taken place can be determined, for example, by referring to the internal log (details described later) stored in the recording unit 18. For example, if an interaction takes place in which the user utters "Hello," the generation model 3 replies "Hello," and the user utters "How are you?", the decision unit 12 will determine whether the action start condition is met for the user's utterance "How are you?".

[0028] (Input section 16) The input unit 16 inputs the input text to the generation model 3. As described above, the determination unit 12 determines whether the action start condition is met between the time a predetermined determination condition is met and the time a waiting period has elapsed. For this reason, the input unit 16 in this embodiment inputs the input text, after a predetermined waiting period has elapsed since the predetermined determination condition was met, into the generation model, after it has been determined whether the action start condition is met. The input unit 16 may also be configured to input the output text acquired by the acquisition unit 11 into the generation model 3. Furthermore, if the input text is configured to be sent directly from the UI2 to the generation model 3, the guardrail system 1 does not need to have an input unit 16.

[0029] (Output section 17) The output unit 17 outputs the generated text to the UI2. The output unit 17 in this embodiment is composed of a communication module. That is, the output unit 17 in this embodiment transmits the generated text data to the UI2. After a predetermined waiting time has elapsed since the acquisition unit 11 acquired the text, the output unit 17 in this embodiment outputs the output text to the UI2, after determining whether or not the action start condition is met. Note that if the generated text is configured to be transmitted directly from the generation model 3 to the UI2, the guardrail system 1 does not need to have an output unit 17.

[0030] (Classification section 13) The classification unit 13 classifies the text into one of several categories according to predetermined classification criteria when the determination unit 12 determines that the text meets the conditions for initiating action. The categories include at least one of the following: sexual content, copyright infringement, violent content, hate speech, false information, promotion of illegal activities, and information leakage.

[0031] In this embodiment, the classification unit 13 classifies text into the category of sexual expression if the text contains pornographic content, sexual allusions, or explicit language.

[0032] Furthermore, if the text contains at least a portion of another person's copyrighted work for which permission has not been obtained, the classification unit 13 classifies the text into the category of copyright infringement.

[0033] Furthermore, if the text contains violent depictions, threatening language, or content that encourages extreme behavior, the classification unit 13 classifies the text into the category of violent expression.

[0034] Furthermore, if the text contains discriminatory remarks based on race, religion, gender, sexual orientation, etc., the classification unit 13 classifies the text as hate speech.

[0035] Furthermore, if the text contains false information or content that intentionally distorts the facts, the classification unit 13 classifies the text into the category of false information.

[0036] Furthermore, if the text contains content that encourages the use of illegal drugs or the performance of illegal acts, the classification unit 13 classifies the text into the category of promoting illegal acts.

[0037] Furthermore, the classification unit 13 classifies text into the category of information leakage if it contains personal or confidential information. Personal information includes the names, addresses, telephone numbers, email addresses, social security numbers, medical information, and financial information of third parties. Confidential information includes corporate and internal documents, strategic plans, unpublished patent information, national secrets, and defense information.

[0038] If the conditions for initiating action include a question or consultation concerning a specialized field, or an answer to such a question or consultation, the categories may include at least one of the following: medical consultation, legal consultation, mental health consultation, and financial consultation. In this case, the judgment unit 12 classifies the text as follows:

[0039] Classification unit 13 classifies text into the medical consultation category if it contains questions about disease symptoms or treatments (for example, "I've been having really bad headaches lately, are there any good medicines?").

[0040] Furthermore, if the text contains questions about specific legal issues or legal rights and obligations (for example, "I want a divorce, what procedures should I follow?"), the classification section 13 classifies the text into the legal consultation category.

[0041] Furthermore, if the text contains questions related to mental health issues or self-assessment (for example, "I've been feeling very depressed lately, what can I do to cheer myself up?"), the classification unit 13 classifies the text into the category of mental health consultation.

[0042] Furthermore, if the text contains questions related to investment, loans, financial management, etc., the classification unit 13 classifies the text into the category of financial consultation.

[0043] The classification unit 13 may also assign a severity level to each of the texts classified into categories. The severity level is an indicator of the level of harmfulness. The severity level may be expressed as a level such as low, medium, or high, or as a numerical value. Furthermore, if the response unit 14, which will be described later, is configured to take predetermined action regardless of the content of the text, the guardrail system 1 does not need to have a classification unit 13.

[0044] (Handling Section 14) The handling unit 14 performs a predetermined action when the determination unit 12 determines that the text satisfies the action start conditions. In this embodiment, the handling unit 14 performs at least one of the following predetermined actions: stopping operation, issuing a warning, requesting re-input, correcting, and regenerating.

[0045] If the action is to stop operation, the action unit 14 stops acquiring the rest of the text from UI2 or the generation model 3. While the acquisition of the rest of the text is stopped, the input unit 16 inputs the rest of the input text to the generation model 3 (display of the rest of the input text by UI2) is stopped. Also, while the acquisition of the rest of the generated text is stopped, the output unit 17 outputs the rest of the generated text to UI2 (display of the rest of the generated text by UI2) is also stopped.

[0046] Furthermore, if the action to be taken is a warning, the action unit 14 will cause a warning message to be displayed on the UI2 in at least one of the input text and the generated text. The warning message may be a statement indicating that the content contains highly harmful material, a statement indicating that the content has been modified to remove highly harmful material, or a statement indicating that the content cannot be displayed because highly harmful material was input or generated. The action unit 14 may also output the warning as audio to the UI2.

[0047] Furthermore, if the action required is a request for re-entry, the action unit 14 displays a message on the UI2 requesting that the user change the content and re-enter the text. The action unit 14 may also output the message as audio to the UI2.

[0048] Furthermore, if the action is a correction, the action unit 14 automatically corrects the acquired text. The correction may be the deletion or replacement of harmful parts, or it may be a correction of the entire acquired text.

[0049] Furthermore, if the action is regeneration, the action unit 14 first removes the harmful parts from the acquired input text. Then, the action unit 14 causes the generation model 3 to regenerate the generated text by having the input unit 16 input the input text from which the harmful parts have been removed. The action unit 14 may also change the prompt to be input to the generation model 3 to include an instruction to consider harmfulness. Then, the generation model 3 may be caused to regenerate the generated text by having the input unit 16 input the changed prompt.

[0050] If the conditions for initiating action include a question or consultation concerning a specialized field, or an answer to such a question or consultation, the action unit 14 will take at least one of the following actions as a prescribed action: a warning, a recommendation for consultation, the provision of general information, and the provision of contact information.

[0051] If the recommended course of action is to seek advice, the response unit 14 displays a message on the UI2 encouraging the user to consult with a specialist (such as a doctor, lawyer, or psychological counselor). The response unit 14 may also output the message as audio to the UI2.

[0052] Furthermore, if the response involves providing general information, the response unit 14 displays text containing very general information in the relevant field on the UI2. This prevents the user from being presented with potentially misleading information. The response unit 14 may also output the text as audio on the UI2.

[0053] Furthermore, if the solution involves providing contact information, the solution unit 14 displays information about experts on the UI2. This allows the user to directly consult with experts closely related to the content of the input text based on the outputted information. The user can then obtain appropriate advice from the expert they contact.

[0054] As described above, in the guardrail system 1 according to this embodiment, the classification unit 13 classifies the text into one of several categories. Therefore, the handling unit 14 according to this embodiment takes action according to the category classified by the classification unit 13. Figures 4 and 5 are tables showing an example of the relationship between the categories used by the guardrail system 1 when classifying text and the actions taken.

[0055] As shown in Figure 4, if the input text is classified as containing sexual content, the handling unit 14 will, for example, issue a warning and request re-input. The handling unit 14 will also make corrections as necessary. On the other hand, if the generated text is classified as containing sexual content, the handling unit 14 will issue a warning and make corrections. The handling unit 14 will also regenerate the text as necessary.

[0056] Furthermore, if the input text is classified as a copyright infringement category, the response unit 14 will, for example, issue a warning, request correction, and request re-input. On the other hand, if the generated text is classified as a copyright infringement category, the response unit 14 will issue a correction, regeneration, and warning.

[0057] Furthermore, if the input text is classified as containing violent content, the handling unit 14 will, for example, issue a warning or request re-input. On the other hand, if the generated text is classified as containing violent content, the handling unit 14 will stop operation and issue a warning.

[0058] Furthermore, if the input text is classified as hate speech, the handling unit 14 will, for example, issue a warning and request re-entry. On the other hand, if the generated text is classified as hate speech, the handling unit 14 will make corrections. The handling unit 14 will also regenerate the text as necessary.

[0059] Furthermore, if the input text is classified as false information, the handling unit 14 will, for example, issue a warning and request re-input. On the other hand, if the generated text is classified as false information, the handling unit 14 will issue a warning and regenerate the text.

[0060] Furthermore, if the input text is classified as a category that encourages illegal activity, the response unit 14 will, for example, issue a warning, make corrections, or request re-entry. On the other hand, if the generated text is classified as a category that encourages illegal activity, the response unit 14 will issue a warning, make corrections, or request re-entry.

[0061] Furthermore, if the input text is classified as an information leak category, the response unit 14 will, for example, issue a warning, make corrections, and request re-input. On the other hand, if the generated text is classified as a recommendation of illegal activity, the response unit 14 will issue a warning, make corrections, and regenerate the text.

[0062] As shown in Figure 5, if the input text is classified as a medical consultation or a mental health consultation, the response unit 14 issues a warning and recommends consultation. On the other hand, if the generated text is classified as a medical consultation or a mental health consultation, the response unit 14 issues a warning and provides contact information.

[0063] Furthermore, if the generated text is classified as a legal consultation, the response unit 14 provides general information and recommends consultation.

[0064] Furthermore, if the generated text is classified as a financial consultation, the response unit 14 provides general information. On the other hand, if the generated text is classified as a legal consultation, the response unit 14 provides general information.

[0065] Furthermore, if a re-input request is made or if an action is taken that takes longer than a predetermined amount of time, the system may also be temporarily shut down. This prevents highly harmful content from being displayed on the UI before the user has finished re-inputting or while the action unit 14 is taking action. In addition, if the classification unit 13 is configured to set a severity level for the text, the action unit 14 may be configured to decide on an action based on the category and severity level. In this case, the action unit 14 may also be configured to determine whether the severity level is above a predetermined level, and if it is determined to be above the predetermined level, to temporarily shut down the operation of the entire information generation system 100. In addition, if the guardrail system 1 does not have a classification unit 13, the action unit 14 may be configured to perform a single action (for example, only a warning).

[0066] (Record Section 18) The recording unit 18 records the text acquired by the acquisition unit 11, the text that the judgment unit 12 determines to satisfy the conditions for initiating action, the classification results of the classification unit 13, and the details of the action taken by the action unit 14 as an internal log. The recorded internal log serves as data to be used to improve each part of the guardrail system 1.

[0067] (Example of implementation using software) The function of guardrail system 1 is a program for causing at least one computer to function as the system, and can be realized by a program for causing each control block of the system to function as a computer.

[0068] In this case, the system includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, the functions described in each of the embodiments are realized.

[0069] The above program may be recorded on one or more computer-readable recording media, not temporary ones. These recording media may or may not be provided by the system. In the latter case, the program may be supplied to the system via any wired or wireless transmission medium.

[0070] [Example of operation of information generation system 100 1] Figure 6 is a sequence diagram showing an example of the operation of the guardrail system 1. The parts 11 to 17 of the guardrail system 1 operate in a sequence such as that shown in Figure 6.

[0071] First, UI2 receives input text from the user (A11). Then, UI2 sends the received input text to guardrail system 1.

[0072] Next, the acquisition unit 11 acquires the input text (B11).

[0073] Next, the preprocessing unit 15 performs normalization processing on the input text acquired by the acquisition unit 11 (B12).

[0074] If monitoring is performed in real time, the next step is for the decision unit 12 to determine whether the decision conditions are met (B13). If the decision unit 12 determines that the decision conditions are not met (B13: NO), the process returns to obtaining the input text (B11). If monitoring is not performed in real time, step B13 is skipped.

[0075] If the determination unit 12 determines that the determination conditions have been met (B13: YES), the determination unit 12 determines whether the input text meets the predetermined conditions for initiating action (B14).

[0076] If the determination unit 12 determines that the input text does not meet the processing start conditions (B14: NO), the output unit 17 outputs the input text to UI2 (B15). This causes UI2 to display the input text (A12). Additionally, the input unit 16 inputs the input text to the generation model 3 (B16). This causes the generation model 3 to generate the generated text (C11).

[0077] Next, the generative model 3, which has the input text as input, generates the corresponding generated text (C11).

[0078] Next, the acquisition unit 11 acquires the generated text (B19).

[0079] Next, the output unit 17 outputs the acquired generated text to UI2 (B20). As a result, UI2 displays the generated text (A13).

[0080] In B14, if the determination unit 12 determines that the input text meets the processing start conditions (B14: YES), the classification unit 13 classifies the input text into one of several categories according to predetermined classification criteria (B17).

[0081] Next, the handling unit 14 takes action according to the category classified by the classification unit 13 (at least one of the following: operation stop, warning, re-input request, correction, and regeneration) (B18).

[0082] In B18, if a warning or re-entry request is issued, as shown in Figure 7, UI2 displays at least one of the following on the input / output screen S: a warning message and a message requesting the user to re-enter the text (Figure 7 shows an example of a warning message) (A12). If the user re-enters the text, the system returns to accepting text input (A11). On the other hand, if correction or regeneration is performed in B18, generation model 3 generates or regenerates the generated text based on the corrected input text (C11). Also, if the system stops working in B18, as shown in Figure 8, UI2 displays at least one of the following on the input / output screen S: a warning message and a message requesting the user to re-enter the text (Figure 8 shows an example of a warning message), along with the input text that was partially entered (highly harmful content was accepted but not displayed) (A12).

[0083] After the generation model 3 generates the generated text, the acquisition unit 11 acquires the generated text (B19).

[0084] Next, the output unit 17 outputs the generated text to UI2. As a result, as shown in Figure 7, UI2 displays the generated text (Figure 7 illustrates the case of generated text based on the modified input text or regenerated generated text) (A13).

[0085] [Example of operation of information generation system 100 2] Figure 9 is a sequence diagram showing another example of the operation of the information generation system 100. The guardrail system 1, UI 2, and generation model 3 that constitute the information generation system 100 can also operate as shown in Figure 9, for example.

[0086] First, UI2 receives input text from the user (A21). Then, UI2 sends the received input text to guardrail system 1.

[0087] Next, the acquisition unit 11 acquires the input text (B21).

[0088] Next, the output unit 17 outputs the acquired input text to UI2 (B22). As a result, UI2 displays the input text (A22).

[0089] Next, the preprocessing unit 15 performs normalization processing on the input text acquired by the acquisition unit 11 (B23).

[0090] Next, the input unit 16 inputs the input text to the generation model 3 (B24). As a result, the generation model 3 generates the generated text (C21).

[0091] Next, the acquisition unit 11 acquires the generated text (B25).

[0092] If monitoring is performed in real time, the next step is for the decision unit 12 to determine whether the decision conditions have been met (B26). If the decision unit 12 determines that the decision conditions have not been met (B26: NO), the process returns to obtaining the generated text (B25). If monitoring is not performed in real time, step B26 is skipped.

[0093] If the determination unit 12 determines that the determination conditions have been met (B26: YES), the determination unit 12 determines whether the generated text meets the predetermined conditions for initiating action (B27).

[0094] If the determination unit 12 determines that the generated text does not meet the conditions for initiating action (B27: NO), the output unit 17 outputs the generated text to UI2 (B28). As a result, UI2 displays the generated text (A23).

[0095] In B27, if the determination unit 12 determines that the generated text meets the conditions for initiating action (B27: YES), the classification unit 13 classifies the generated text into one of several categories according to predetermined classification criteria (B29).

[0096] Next, the handling unit 14 takes action according to the category classified by the classification unit 13 (at least one of operation stop, warning, re-input request, correction, and regeneration) (B30).

[0097] In B30, if a warning or re-entry request is issued, UI2 displays at least one of the following on the input / output screen S: a warning message and a message requesting the user to re-enter the text (A23). If the user re-enters the text, the system returns to accepting text input (A21). On the other hand, if correction or regeneration is performed in B30, generation model 3 generates or regenerates the generated text based on the corrected input text (C21). Also, if the system stops working in B30, UI2 displays a warning message and the generated text in the middle of being displayed (highly harmful content has been generated but is not yet displayed) on the input / output screen S (A12).

[0098] Furthermore, the sequence diagram shown in Figure 6 can be shown as an operation that monitors both input text and generated text by replacing each operation enclosed by the dashed line X with each operation enclosed by the dashed line Y in Figure 9.

[0099] [Effects of Information Generation System 100 (Guardrail System) 1] The guardrail system 1 according to this embodiment classifies harmful or specialized content contained in text into various categories and takes appropriate action according to each category. Therefore, the guardrail system 1 allows for actions such as warnings and requests for re-entry for content with relatively low harmfulness, and safer actions such as stopping operation and regeneration for content with relatively high harmfulness. As a result, text generation using the generative model can be performed more safely.

[0100] [Effects of Information Generation System 100 (Guardrail System) 2] Furthermore, in the guardrail system 1 according to this embodiment, the judgment unit 12 repeatedly determines whether predetermined action start conditions are met for the text sequentially acquired by the acquisition unit 11, each time the predetermined judgment conditions are met. In other words, the guardrail system 1 monitors the acquired text in real time. Therefore, with the guardrail system 1, if the judgment unit 12 determines that the action start conditions are met during text acquisition (for example, that the text contains harmful content), action can be taken at that time (for example, displaying a warning, temporarily suspending text input / output, etc.). As a result, text generation using the generation model can be performed more safely.

[0101] <Variation> This disclosure is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Furthermore, embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of this disclosure.

[0102] For example, the guardrail system 1 described above monitored both the text obtained from UI2 (entered by the user) and the generated text. However, the guardrail system 1 may monitor only the text obtained from UI2, or it may monitor only the generated text.

[0103] Furthermore, in the above embodiment, it was explained that the functions of the guardrail system 1 can be realized by a program that causes a computer to function as each control block of the system. On the other hand, some or all of the functions of each of the above control blocks can also be realized by logic circuits. For example, an integrated circuit in which logic circuits that function as each of the above control blocks are formed is also included in the scope of the present invention. In addition, it is also possible to realize the functions of each of the above control blocks by, for example, a quantum computer.

[0104] <Summary> A guardrail system according to aspect 1 of the present invention comprises: an acquisition unit that sequentially acquires text to be input to a generation model that receives input from a user interface, or text output by the generation model; a determination unit that determines whether the acquired text satisfies predetermined action start conditions each time a predetermined determination condition is met; and an action unit that performs predetermined action when the determination unit determines that the text satisfies the action start conditions.

[0105] In the guardrail system according to aspect 2 of the present invention, the judgment condition in aspect 1 described above may be that a predetermined amount of time has elapsed since the previous judgment on whether or not the condition for initiating countermeasures is met.

[0106] The guardrail system according to aspect 3 of the present invention may be configured such that, in aspect 1 or 2 described above, the condition for initiating action includes the text being harmful, and the action unit performs at least one of the following as the predetermined action: stopping operation, issuing a warning, requesting re-input, correcting, and regenerating.

[0107] A guardrail system according to aspect 4 of the present invention may be configured such that, in any of aspects 1 to 3 above, the response initiation condition includes the text being a question or consultation concerning a specialized field, or an answer to such a question or consultation, and the response unit displays a warning on the user interface.

[0108] A guardrail system according to aspect 5 of the present invention may further include an input unit for inputting text to the generation model into the generation model, wherein the input unit inputs the text into the generation model after a predetermined waiting time has elapsed since the predetermined judgment condition was met, and the judgment unit determines whether or not the action start condition is met before the waiting time has elapsed.

[0109] A guardrail system according to aspect 6 of the present invention may further include an output unit that outputs text output by the generation model to the user interface, wherein the output unit outputs the text to the user interface after a predetermined waiting time has elapsed since the predetermined judgment condition was met, and the judgment unit determines whether or not the action start condition is met during the waiting time.

[0110] A guardrail system according to aspect 7 of the present invention may further include an input unit for inputting text to the generation model into the generation model, in the above aspect 3, wherein the processing unit removes harmful parts from the acquired text and causes the input unit to input the text from which the harmful parts have been removed into the generation model, thereby causing the generation model to regenerate the text.

[0111] A guardrail system according to aspect 8 of the present invention may further include an input unit for inputting text to the generation model into the generation model, in the above aspect 3, wherein the handling unit modifies the prompt to be input to the generation model to include an instruction to consider the harmfulness, and causes the generation model to regenerate the text by having the input unit input the modified prompt.

[0112] The program according to aspect 9 of the present invention is configured to cause the processor to execute an acquisition process that sequentially acquires text to be input to a generation model that receives input from a user interface, or text output by the generation model; a determination process that determines whether the acquired text satisfies a predetermined action start condition each time a predetermined determination condition is met; and an action process that performs a predetermined action if the determination process determines that the text satisfies the action start condition. [Explanation of symbols]

[0113] 100 Information Generation Systems 1. Guardrail System 11 Acquisition Department 12 Judgment Department 13 Classification section 14. Response Department 15 Pre-processing section 16 Input section 17 Output section 18 Records Section 2. User Interface 21 Communications Department 22 Reception Department 23 Display section S input / output screen Sa input field Sb answer column Sc Other display field 3 Generative Models

Claims

1. The acquisition unit sequentially retrieves the text output by the generative model, Each time the acquisition unit acquires a predetermined number of characters of text, a determination unit determines whether the acquired text satisfies predetermined processing start conditions. The aforementioned text is used to determine if the determination unit determines that the conditions for initiating the action are met, and the action unit performs a predetermined action. If the determination unit determines that the text does not satisfy the conditions for initiating action, the output unit outputs the text to the user interface. Equipped with, Guardrail system.

2. The aforementioned handling unit, as the handling, A warning message is output to the user interface. Alternatively, the generation model may modify the text or regenerate the text. The guardrail system according to claim 1.

3. An acquisition unit that sequentially acquires text to be input to a generative model that receives input from a user interface, or text output by the generative model, Each time a predetermined judgment condition is met, a determination unit determines whether the acquired text satisfies a predetermined action initiation condition, The aforementioned text is used to determine if the determination unit determines that the conditions for initiating the action are met, and the action unit performs a predetermined action. The system includes an input unit for inputting text to the generation model, The input unit inputs the text to the generation model after a predetermined waiting time has elapsed since the predetermined judgment condition was met. The determination unit determines whether the conditions for initiating action are met before the waiting time has elapsed. Guardrail system.

4. An acquisition unit that sequentially acquires text to be input to a generative model that receives input from a user interface, or text output by the generative model, Each time a predetermined judgment condition is met, a determination unit determines whether the acquired text satisfies a predetermined action initiation condition, The aforementioned text is used to determine if the determination unit determines that the conditions for initiating the action are met, and the action unit performs a predetermined action. The system includes an output unit that outputs the text output by the generation model to the user interface, The output unit outputs the text to the user interface after a predetermined waiting time has elapsed since the predetermined judgment condition was met. The determination unit determines whether the conditions for initiating action are met before the waiting time has elapsed. Guardrail system.

Images