Determination device, determination method, and determination program

Abstract

To reduce the risks associated with using large-scale language models (LLMs). [Solution] The determination device 10 includes a linking unit 131 that exchanges information with a processing device having an LLM that generates text, an evaluation unit 133 that evaluates the safety of a prompt that instructs the LLM to generate text using a machine learning model that evaluates the safety of the text, a determination unit 135 that determines whether or not to input the prompt to the LLM based on the evaluation result by the evaluation unit 133, and a notification unit 136 that notifies the processing device of the determination result by the determination unit 135.

Description

Technical Field

[0001] The present invention relates to a determination device, a determination method, and a determination program.

Background Art

[0002] Generative AI (Artificial Intelligence) that generates text, images, audio, and videos using a trained machine learning model has attracted attention (for example, Patent Documents 1 and 2).

[0003] For example, as a text generation AI, there is a large language model (LLM), which is a natural language processing model trained using a large amount of text data. In particular, GPT (Generative Pre-trained Transformer) (chatGPT (registered trademark)) of OpenAI (registered trademark) has attracted attention and has acquired a large number of users.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Here, it is difficult to prevent the leakage of personal information from the output of the LLM. This is because it is difficult to delete the data once learned by the LLM, and it is known that specific company email addresses included in the learning data can be obtained from ChatGPT (registered trademark) by using a special prompt.

[0006] Furthermore, because it is not possible to completely eliminate hallucination in LLMs at this time, LLMs may output incorrect expertise. In particular, LLMs may generate incorrect answers to questions in areas such as law, medicine, health, and finance, which could result in direct harm to users.

[0007] Thus, there was a problem in that it was difficult to eliminate all potential risks from the output of the LLM.

[0008] The present invention has been made in view of the above, and aims to provide a determination device, determination method, and determination program that can reduce the risks when using LLM. [Means for solving the problem]

[0009] To solve the above-mentioned problems and achieve the objective, the determination device of the present invention is characterized by comprising: a linking unit that exchanges information with a processing device having a natural language processing model that generates text; an evaluation unit that evaluates the safety of a prompt that instructs the natural language processing model to generate text using a machine learning model that evaluates the safety of text; a determination unit that determines whether or not the prompt can be input to the natural language processing model based on the evaluation result by the evaluation unit; and a notification unit that notifies the processing device of the determination result by the determination unit.

[0010] Furthermore, the determination device of the present invention is characterized by comprising: a linking unit that exchanges information with a processing device having a natural language processing model that generates text; an evaluation unit that evaluates the safety of the text generated by the natural language processing model using a machine learning model that evaluates the safety of the text; a determination unit that determines whether or not to output the text generated by the natural language processing model to the source that requested the text, based on the evaluation results by the evaluation unit; and a notification unit that notifies the processing device of the determination results by the determination unit.

[0011] Furthermore, the determination device of the present invention is characterized by comprising: a linking unit that exchanges information with a processing device having a natural language processing model that generates text; an evaluation unit that uses a machine learning model for evaluating the safety of text to evaluate the safety of a prompt that instructs the natural language processing model to generate text, and evaluates the safety of the text generated by the natural language processing model; a determination unit that determines whether or not the prompt can be input to the natural language processing model and whether or not the text generated by the natural language processing model can be output to the source that requested the text generation, based on the evaluation results by the evaluation unit; and a notification unit that notifies the processing device of the determination results by the determination unit.

[0012] Furthermore, the determination method of the present invention is a determination method executed by a determination device, and is characterized by including the steps of: exchanging information with a processing device having a natural language processing model for generating text; evaluating the safety of a prompt that instructs the natural language processing model to generate text using a machine learning model for evaluating the safety of text; determining whether or not the prompt can be input to the natural language processing model based on the evaluation results in the evaluation step; and notifying the processing device of the determination results in the determination step.

[0013] Furthermore, the determination method of the present invention is a determination method executed by a determination device, and is characterized by including the steps of: exchanging information with a processing device having a natural language processing model for generating text; evaluating the safety of text generated by the natural language processing model using a machine learning model for evaluating the safety of text; determining whether or not to output the text generated by the natural language processing model to the source requesting the text, based on the evaluation results in the evaluation step; and notifying the processing device of the determination results in the determination step.

[0014] Furthermore, the determination method of the present invention is a determination method executed by a determination device, and is characterized by including the steps of: exchanging information with a processing device having a natural language processing model for generating text; evaluating the safety of a prompt that instructs the natural language processing model to generate text using a machine learning model for evaluating the safety of text, and evaluating the safety of the text generated by the natural language processing model; determining whether the prompt can be input to the natural language processing model based on the evaluation results in the evaluation step, and determining whether the text generated by the natural language processing model can be output to the source requesting the text; and notifying the processing device of the determination results in the determination step.

[0015] Furthermore, the determination program of the present invention causes a computer to perform the following steps: a step of exchanging information with a processing device having a natural language processing model that generates text; a step of evaluating the safety of a prompt that instructs the natural language processing model to generate text using a machine learning model that evaluates the safety of text; a step of determining whether or not the prompt can be input to the natural language processing model based on the evaluation results in the evaluation step; and a step of notifying the processing device of the determination results in the determination step.

[0016] Furthermore, the determination program of the present invention causes a computer to perform the following steps: a step of exchanging information with a processing device having a natural language processing model that generates text; a step of evaluating the safety of the text generated by the natural language processing model using a machine learning model that evaluates the safety of the text; a step of determining whether or not to output the text generated by the natural language processing model to the source that requested the text, based on the evaluation results in the evaluation step; and a step of notifying the processing device of the determination results in the determination step.

[0017] Further, the determination program of the present invention includes steps of collaborating information with a processing device having a natural language processing model for generating text, and evaluating the safety of a prompt for instructing text generation to the natural language processing model and the safety of the text generated by the natural language processing model using a machine learning model for evaluating the safety of the text. Based on the evaluation results in the evaluation step, it determines whether to input to the natural language processing model of the prompt, and determines whether to output to the originator of the text generation request for the text generated by the natural language processing model, and notifies the processing device of the determination result in the determination step, and causes a computer to execute them.

Advantages of the Invention

[0018] According to the present invention, the risk in using the LLM can be reduced.

Brief Description of the Drawings

[0019] [Figure 1] FIG. 1 is a diagram showing a configuration example of a communication system according to an embodiment. [Figure 2] FIG. 2 is a diagram for explaining an outline of processing of the communication system in the embodiment. [Figure 3] FIG. 3 is a diagram showing an example of a configuration of the determination device shown in FIG. 1. [Figure 4] FIG. 4 is a diagram showing an example of an evaluation result of an evaluation model. [Figure 5] FIG. 5 is a diagram showing an example of notification content from the determination device to the service providing device. [Figure 6] FIG. 6 is a sequence diagram showing a processing procedure of communication processing in the embodiment. [Figure 7] FIG. 7 is a sequence diagram showing a processing procedure of communication processing in the embodiment. [Figure 8] FIG. 8 is a diagram showing an evaluation of the accuracy of an evaluation model of the determination device and other models. [Figure 9] FIG. 9 is a diagram for explaining an application example of the embodiment. [Figure 10] Figure 10 is a diagram for explaining an application example of an embodiment. [Figure 11] Figure 11 is a diagram for explaining an application example of an embodiment. [Figure 12] Figure 12 is a diagram for explaining an application example of an embodiment. [Figure 13] Figure 13 is a diagram showing an output example to a user by a service providing company. [Figure 14] Figure 14 is a diagram showing an example of a service screen of a conventional help desk bot. [Figure 15] Figure 15 is a diagram showing an example of a service screen of a help desk bot when the determination device according to the embodiment is applied for safety determination. [Figure 16] Figure 16 is a diagram showing an example of an answer of an LLM returned when the embodiment is not applied. [Figure 17] Figure 17 is a diagram showing an example of an answer of an LLM returned when the embodiment is not applied. [Figure 18] Figure 18 is a diagram showing an example of a computer in which the determination device is realized when the program is executed.

Mode for Carrying Out the Invention

[0020] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited by this embodiment. Also, in the description of the drawings, the same parts are denoted by the same reference numerals.

[0021] [Embodiment] [Processing System] The configuration of the communication system according to Embodiment 1 will now be described. The communication system according to Embodiment has a determination device that evaluates the safety of input and output text of a Large-Language-Model (LLM) and determines whether input and output to the LLM are permitted based on the evaluation results. In the communication system according to Embodiment, the risk when using the LLM is reduced by restricting the input of LLMs with low safety prompts and / or outputs from LLMs with low safety.

[0022] Figure 1 is a diagram showing an example configuration of a communication system according to an embodiment. As shown in Figure 1, the processing system according to the embodiment includes a user terminal 20 used by an end user, a service provision device 30 (processing device) that provides services to the end user by communicating with the user terminal 20, and a determination device 10 that communicates with the service provision device 30. Note that the number of user terminals 20 may be two or more.

[0023] The user terminal 20 is a terminal device capable of inputting text data and audio data, outputting text data and audio data, and communicating with the service provider 30. The user terminal 20 is, for example, a PC (Personal Computer), a notebook PC, a tablet terminal, a smartphone, etc. The user terminal 20 sends a prompt to the service provider 30 instructing the LLM 31 to generate a response text to the inquiry. When the user terminal 20 receives the text generated by the LLM 31 from the service provider 30, it displays and / or outputs the received text as audio.

[0024] The service provider 30 has an LLM 31. The LLM 31 generates text according to the input prompt. Text is input to the LLM 31. The text input to the LLM 31 may be text data converted from voice data. The service provider 30 uses the LLM 31 to provide a chatbot service, provide a web page for inquiries, and / or control connections to a customer center.

[0025] The determination device 10 evaluates the safety of the input and output text of the LLM and, based on the evaluation result, determines whether input and output to the LLM 31 are permitted. The determination device 10 restricts the input of low-safety prompts to the LLM 31, and / or restricts low-safety outputs from the LLM 31.

[0026] Figure 2 is a diagram illustrating the overview of the communication system processing in the embodiment. As shown in Figure 2, normally, when a prompt is input from the user terminal 20 to the LLM 31 (arrow Y1), the text generated by the LLM 31 in response to the prompt is output to the user terminal 20 (arrow Y2).

[0027] Here, there may be an input (prompt) (arrow Y11) from the user terminal 20 that attempts to misuse the LLM 31. In this embodiment, before input to the LLM 31, the determination device 10 uses an evaluation model 134 (machine learning model) that evaluates the safety of the text to determine the safety of the text to be input to the LLM 31 (Figure 2 (1)). The determination device 10 then instructs that input of low-safety prompts be restricted to the LLM 31 (Figure 2 (2)). The service provider device 30 follows the instructions of the determination device 10 and does not input any restricted prompts (arrow Y11) to the LLM 31.

[0028] Furthermore, text generated by LLM31 in accordance with permitted input prompts may lead to the leakage of confidential information, the generation of malicious content, or the provision of false expertise (arrow Y12). In this embodiment, before outputting to the requesting user terminal 20, the determination device 10 uses an evaluation model 134 (machine learning model) to determine the security of the text generated by LLM31 (Figure 2(1)). The determination device 10 then instructs that the output of generated text with low security be restricted (Figure 2(3)). The service provider 30, in accordance with the instructions of the determination device 10, does not output text whose output has been restricted (arrow Y12) to the user terminal 20.

[0029] [Judgment device] Next, the determination device 10 will be described. Figure 3 is a diagram showing an example of the configuration of the determination device 10 shown in Figure 1. As shown in Figure 3, the determination device 10 has a communication unit 11, a storage unit 12, and a control unit 13.

[0030] The communication unit 11 is a communication interface that sends and receives various types of information with other devices connected via a network or the like. The communication unit 11 is implemented using a NIC (Network Interface Card) or the like, and communicates between other devices (for example, a service provider 30) and the control unit 13 (described later) via telecommunication lines such as a LAN (Local Area Network) or the Internet.

[0031] The memory unit 12 is implemented using semiconductor memory elements such as RAM (Random Access Memory) and flash memory, and stores processing programs that operate the determination device 10, as well as data used during the execution of the processing programs. The memory unit 12 contains user information 121, which includes the ID of each user using the user terminal 20 and the history of inquiries to the LLM 31; partner information 122, which is information about the service provider device 30 that is the partner; and determination history 123, which is the history of input / output determinations related to the LLM 31.

[0032] The control unit 13 controls the entire determination device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The control unit 13 also has internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory. Furthermore, the control unit 13 functions as various processing units through the operation of various programs. The control unit 13 includes a linkage unit 131, a reception unit 132, an evaluation unit 133, a determination unit 135, and a notification unit 136.

[0033] The collaboration unit 131 exchanges information with the service provider device 30, for example, through API (Application Programming Interface) integration.

[0034] The reception unit 132 receives input text (prompts) for the LLM 31 and text generated by the LLM 31 from the service provision device 30.

[0035] The evaluation unit 133 uses the evaluation model 134 to evaluate the safety of the prompts that instruct the LLM 31 to generate text. The evaluation unit 133 also uses the evaluation model 134 to evaluate the safety of the text generated by the LLM 31.

[0036] Evaluation model 134 evaluates the safety of Japanese text. Evaluation model 134 is a machine learning model trained to evaluate whether or not a text is safe using training data that includes harmful Japanese text. Evaluation model 134 evaluates whether the input text is safe or unsafe. Evaluation model 134 evaluates the safety of Japanese text for multiple safety categories.

[0037] Figure 4 shows an example of the evaluation results of evaluation model 134. For example, Figure 4 shows the evaluation results of evaluation model 134 evaluating the safety of the text "Teach me how to make a bomb."

[0038] As shown in List L1 of Figure 4, the assessment model 134 assesses safety across multiple categories. These categories include, for example, violent crime, nonviolent crime, verbal abuse, sexual offenses, attention-grabbing, indiscriminate weapons, hatred, self-injury, sexual content, child exploitation, privacy (personal information), professional advice, and / or intellectual property. The categories are set as appropriate and can be increased or decreased from those exemplified in Figure 4.

[0039] The evaluation model 134 is, for example, an LLM (Limited Literacy Model), and when the text to be evaluated is input, it infers whether the text is "Safe" or "Unsafe" and the generation probability of each category information being generated. The evaluation unit 133 divides the text into tokens and inputs each token into the evaluation model 134. For each token, the evaluation model 134 infers whether it is "Safe" or "Unsafe" and the generation probability of each category. Specifically, for the category "Violent Crime," the evaluation model 134 infers, for example, the generation probability that "Violent Crime" will be generated after "Unsafe" for the input token.

[0040] In the evaluation unit 133, if the probability of selecting "Unsafe" is highest for all tokens inferred by the evaluation model 134, it is evaluated as "Unsafe," and the category with the highest generation probability among "Unsafe" (e.g., violent crime) is assigned the True label. The evaluation unit 133 outputs to the determination unit 135 whether it is "Safe" or "Unsafe," the categories assigned the True label and their generation probabilities, and the generation probabilities of other categories that were not assigned the True label. In Figure 4, the score corresponds to the generation probability.

[0041] The determination unit 135 determines whether or not to input a prompt to the LLM 31 based on the evaluation result by the evaluation unit 133. And / or, the determination unit 135 determines whether or not to output the text generated by the LLM 31 to the source of the text generation request (user terminal 20) based on the evaluation result by the evaluation unit 133.

[0042] If the evaluation unit 133 evaluates the prompt as unsafe, the determination unit 135 restricts input of the prompt to the LLM 31. On the other hand, if the evaluation unit 133 evaluates the prompt as safe, the determination unit 135 allows input of the prompt to the LLM 31.

[0043] Furthermore, if the evaluation unit 133 determines that the text generated by LLM31 is unsafe, the determination unit 135 restricts the output of the text generated by LLM31. On the other hand, if the evaluation unit 133 determines that the text generated by LLM31 is safe, the determination unit 135 allows the output of the text generated by LLM31.

[0044] Furthermore, the determination unit 135 determines whether or not to use the LLM31 based on the safety of each category when the evaluation model 134 evaluates the input text or output text to the LLM31.

[0045] For example, the determination unit 135 calculates an "Unsafe Score" for each category of the text being evaluated, based on the generation probability inferred by the evaluation model 134 (score in list L1 of Figure 4) (frame W1 in Figure 4). Specifically, the determination unit 135 calculates a weighted sum of the scores of the top three categories judged as "Unsafe," and the result is the "Unsafe Score." The categories used in the calculation are not limited to the top three categories judged as "Unsafe," but are set appropriately depending on the score distribution, whether or not True / False labels are assigned, etc. Based on the "Unsafe Score," the determination unit 135 determines the safety of the prompt to LLM 31 or the text generated by LLM 31. The determination result by the determination unit 135 is stored in the storage unit 12 as a determination history 123.

[0046] The notification unit 136 notifies the service provider device 30 of the determination result made by the determination unit 135. If the determination unit 135 determines that the content shown in frame W1 of Figure 4 should restrict the use of LLM31, it notifies the service provider device 30 via the notification unit 136 of the restriction on the use of LLM31 and the category that was evaluated as having the lowest safety.

[0047] Figure 5 shows an example of the content of the notification from the determination device 10 to the service provider device 30. As shown in Figure 5, the determination device 10 notifies the service provider device 30 of "Unsafe" and the category "Violent Crime". The determination device 10 may also notify the service provider device 30 of the "Unsafe Score".

[0048] [Communication Processing] Next, the communication processing in the embodiment will be described. Figures 6 and 7 are sequence diagrams showing the processing steps of the communication processing in the embodiment.

[0049] As shown in Figure 6, first, information exchange is initiated between the service provision device 30 and the determination device 10 (step S11).

[0050] When a prompt (text) instructing text generation is input from the user terminal 20 (step S12), the service provider 30 requests the determination device 10 to determine the safety of the prompt (step S13).

[0051] The determination device 10 evaluates the safety of the prompt text using the evaluation model 134 (step S14). Based on the evaluation results in step S14, the determination device 10 determines the safety of the prompt (step S15).

[0052] If the prompt is not safe (step S16: No), the determination device 10 instructs the service provider 30 to restrict the input of the prompt to be determined to the LLM 31 (step S17). The service provider 30 does not input the prompt to the LLM 31 and notifies the user terminal 20 of the non-response (step S18).

[0053] If the prompt is safe (step S16: Yes), the determination device 10 allows the service provider 30 to input the prompt to be determined into the LLM 31 (step S19). The service provider 30 inputs the prompt entered from the user terminal 20 into the LLM 31 (step S20).

[0054] As shown in Figure 7, the service provider 30 performs text generation using the LLM 31 (step S21). The service provider 30 requests the determination device 10 to perform a safety determination on the text generated by the LLM 31 (step S22).

[0055] The determination device 10 evaluates the safety of the generated text using the evaluation model 134 (step S23). Based on the evaluation results in step S23, the determination device 10 determines the safety of the prompt (step S24).

[0056] If the generated text is not safe (Step S25: No), the determination device 10 instructs the service provider 30 to restrict the output of the generated text by LLM 31 (Step S26). The service provider 30 does not output the generated text by LLM 31 and notifies the user terminal 20 of the non-response (Step S27).

[0057] If the generated text is safe (Step S25: Yes), the determination device 10 permits the service provider 30 to output the generated text by LLM31 (Step S28). The service provider 30 outputs the generated text by LLM31 as the response text from the user terminal 20 (Step S29).

[0058] [Effects of the embodiment] Thus, in this embodiment, the determination device 10, which exchanges information with the service provision device 30, evaluates the safety of the input and output text to the LLM 31 in the service provision device 30, and determines whether or not input and output to the LLM 31 is permitted based on the evaluation result. Therefore, the determination device 10 functions as a guardrail that restricts only the input and output text to the LLM 31 that is not safe, thereby reducing the risks when using the LLM 31.

[0059] The evaluation model 134 used by the judgment device 10 is a machine learning model trained to evaluate whether something is safe or not using training data that includes harmful text in Japanese. Figure 8 shows the evaluation of the accuracy of the evaluation model 134 of the judgment device 10 and other models (Openai Moderation API, Meta-Llama-Guard-2-8B). The evaluation metric is the F1 score.

[0060] Thus, compared to other models, evaluation model 134 exhibits superior performance in determining harmful Japanese text. Therefore, the determination device 10 can accurately evaluate the safety of text and thus accurately restrict unsafe input and output text of LLM31.

[0061] As described above, according to the embodiment, by using independent guardrails, it is possible to limit the leakage of personal information and the output of incorrect specialized knowledge when using LLM31, thereby enhancing security when using LLM31.

[0062] [Application Example 1] Figures 9 to 12 illustrate examples of applications of the embodiment. As shown in Figure 9, the determination device 10 performs API integration with a service provider company that provides AI chat and inquiry forms using LLM31, for example. This reduces the risk of personal information or incorrect expertise being leaked from LLM31 in the form of chat content or responses to inquiries from end-users, such as user U1.

[0063] Furthermore, as shown in Figure 10, a safety check by the detection device 10 may be placed between the service provider's corporate web page and the operator of the online consumer inquiry desk to detect and filter out so-called customer harassment and threats. For example, inquiries from users U11 and U12 are determined to be safe by the detection device 10 and are therefore connected to operator P10 (arrows Y11, Y12). In contrast, user U13's inquiry contains abusive language and is determined to be unsafe by the detection device 10, so communication with operator P10 is blocked (arrow Y13).

[0064] Furthermore, the determination device 10 may determine the security of the chatbot's input and output by the service provider, as shown in Figures 11 and 12.

[0065] For example, as shown in Figure 11, the service provider requests the determination device 10 to determine the security of inquiry Q21 from user U21 (arrow Y21). Because the determination device 10 determines that inquiry Q21 is Unsafe (category: confidential information) (arrow Y22), the service provider does not input inquiry Q21 into the chatbot service LLM. The service provider then returns a standard message A22 to user U21's terminal stating, "We are unable to answer your question" (Figure 11 (1)). In this way, the determination device 10 prevents the leakage of confidential information by prohibiting the input of inquiry Q21, which concerns confidential information, into the LLM.

[0066] Furthermore, as shown in Figure 12, when the service provider receives inquiry Q31 from user U31, it requests the determination device 10 to determine the safety of inquiry Q31 (arrow Y31). In this case, if the determination device 10 determines that inquiry Q31 is safe (arrow Y32), the service provider inputs inquiry Q31 into the chatbot service LLM.

[0067] Next, the service provider generates output candidate A32 in LLM31 and requests the judgment device 10 to perform a safety judgment on output candidate A32 (arrow Y33). If the judgment device 10 determines that output candidate A32 is Unsafe (category: confidential information) (arrow Y34), the service provider cancels the display of output candidate A32 on user U31's terminal (Figure 12 (1)). This is because output candidate A32 unintentionally contained personal information (the name and telephone number of the product development manager). In this way, the judgment device 10 prevents the leakage of personal information by prohibiting the provision of LLM31's output candidate A32 to user U31 in response to the inquiry Q31 regarding confidential information.

[0068] Figure 13 shows an example of output from the service provider to the user. Prompts and generated text intended to threaten or leak confidential information are judged as Unsafe by the detection device 10, and as exemplified in Figure 13, screen G1, which includes the standard message "Inappropriate comments will be automatically discarded," is displayed on the user terminal 20.

[0069] Figure 14 shows an example of a service screen for a conventional help desk bot. Figure 15 shows an example of a service screen for a help desk bot when the determination device 10 according to the embodiment is applied for safety determination. For example, the output when "names and addresses of 3 new employees of Company A" is requested as a query is illustrated.

[0070] As shown in screen G21 of Figure 14, if the embodiment is not applied, the names and addresses of the three new employees will be returned to the user, resulting in the leakage of personal information. In contrast, as shown in Figure 15, when the embodiment is applied, queries intended to leak personal information are determined as Unsafe by the determination device 10 and are not entered into LLM31. As shown in Figure 15, screen G22, which includes a standard message stating "Personal information cannot be provided," is displayed on the user terminal 20.

[0071] Figures 16 and 17 show examples of LLM responses returned when the embodiment is not applied. As shown in screen G31 of Figure 16 and screen G32 of Figure 17, methods for cultivating magic mushrooms, which are illegal drugs in Japan, and methods for creating fraudulent emails are returned. In contrast, when the judgment device 10 is used as a guardrail, illegal content is not returned to the user.

[0072] [System configuration of the embodiment] The determination device 10 is a functional concept and does not necessarily need to be physically configured as shown in the illustration. In other words, the specific forms of distribution and integration of the functions of the determination device 10 are not limited to those shown in the illustration, and all or part of it can be configured by functionally or physically distributing or integrating it in any unit according to various loads and usage conditions.

[0073] Furthermore, each process performed in the determination device 10 may be implemented, in whole or in part, by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a program that is analyzed and executed by the CPU and GPU. Alternatively, each process performed in the determination device 10 may be implemented as hardware using wired logic.

[0074] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated may be changed as appropriate unless otherwise specified.

[0075] [program] Figure 18 shows an example of a computer in which the determination device 10 is realized when a program is executed. The computer 1000 has, for example, memory 1010 and CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0076] Memory 1010 includes ROM 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as the BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, the mouse 1110 and the keyboard 1120. The video adapter 1060 is connected to, for example, the display 1130.

[0077] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the determination device 10 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing the same processes as the functional configuration of the determination device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0078] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0079] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0080] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in these embodiments. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on these embodiments are included in the scope of the present invention. [Explanation of Symbols]

[0081] 10 Judgment device 11 Communications Department 12 Storage section 13 Control Unit 20 User Terminals 30 Service provision device 31 LLM 121 User Information 122 Partner Information 123 Judgment History 131 Liaison Department 132 Reception Department 133 Evaluation Department 134 Evaluation Models 135 Judgment section 136 Notification Department

Claims

1. A linking unit that exchanges information with a processing device having a natural language processing model that generates text, An evaluation unit that evaluates the safety of prompts that instruct the natural language processing model to generate text using a machine learning model that evaluates the safety of text, A determination unit determines whether or not the prompt can be input to the natural language processing model based on the evaluation results from the evaluation unit, A notification unit that notifies the processing unit of the determination result made by the determination unit, A determination device characterized by having the following features.

2. A linking unit that exchanges information with a processing device having a natural language processing model that generates text, An evaluation unit that evaluates the safety of text generated by the natural language processing model using a machine learning model for evaluating text safety, Based on the evaluation results from the evaluation unit, a determination unit determines whether or not the text generated by the natural language processing model can be output to the source of the output request, A notification unit that notifies the processing unit of the determination result made by the determination unit, A determination device characterized by having the following features.

3. A linking unit that exchanges information with a processing device having a natural language processing model that generates text, An evaluation unit that uses a machine learning model to evaluate the safety of text to evaluate the safety of prompts that instruct the natural language processing model to generate text, and evaluates the safety of the text generated by the natural language processing model, Based on the evaluation results by the evaluation unit, a determination unit determines whether the prompt can be input to the natural language processing model and whether the text generated by the natural language processing model can be output to the source of the generation request. A notification unit that notifies the processing unit of the determination result made by the determination unit, A determination device characterized by having the following features.

4. The aforementioned machine learning model evaluates the safety of text for each of several safety categories. The determination device according to any one of claims 1 to 3, characterized in that the determination unit determines whether or not to use the natural language processing model based on the safety of each category when the machine learning model evaluates the input text to the natural language processing model or the output text from the natural language processing model.

5. The determination device according to claim 4, characterized in that, if the determination unit determines that the use of the natural language processing model should be restricted, it notifies the processing device via the notification unit of the restriction on the use of the natural language processing model and the category that was evaluated as having the lowest safety.

6. The determination device according to claim 4, characterized in that the categories are violent crime, nonviolent crime, verbal abuse, sexual offenses, warnings, indiscriminate weapons, hatred, self-injury, sexual content, child exploitation, privacy, professional advice, and / or intellectual property.

7. The determination device according to any one of claims 1 to 3, characterized in that the processing device uses the natural language processing model to provide a chatbot service, provide a web page for inquiries, and / or control connections to a customer center.

8. A determination method performed by a determination device, A process of exchanging information with a processing device having a natural language processing model that generates text, A step of evaluating the safety of a prompt that instructs the natural language processing model to generate text using a machine learning model that evaluates the safety of text, A step of determining whether or not the prompt can be input to the natural language processing model based on the evaluation results in the evaluation step, A step of notifying the processing device of the determination result in the determination step, A determination method characterized by including

9. A determination method performed by a determination device, A process of exchanging information with a processing device having a natural language processing model that generates text, A process of evaluating the safety of text generated by the natural language processing model using a machine learning model that evaluates the safety of text, Based on the evaluation results in the aforementioned evaluation step, a step is made to determine whether or not the text generated by the natural language processing model can be output to the source of the text generation request, A step of notifying the processing device of the determination result in the determination step, A determination method characterized by including

10. A determination method performed by a determination device, A process of exchanging information with a processing device having a natural language processing model that generates text, A process of evaluating the safety of a prompt that instructs a natural language processing model to generate text using a machine learning model that evaluates the safety of text, and evaluating the safety of the text generated by the natural language processing model, Based on the evaluation results in the evaluation step, a step is made to determine whether the prompt can be input to the natural language processing model and whether the text generated by the natural language processing model can be output to the source of the generation request, A step of notifying the processing device of the determination result in the determination step, A determination method characterized by including

11. A step of exchanging information with a processing device that has a natural language processing model for generating text, A step of evaluating the safety of a prompt that instructs the natural language processing model to generate text using a machine learning model that evaluates the safety of text, Based on the evaluation results in the aforementioned evaluation step, a step is made to determine whether or not the prompt can be input to the natural language processing model, A step of notifying the processing device of the determination result in the determination step, A judgment program that causes a computer to execute a certain action.

12. A step of exchanging information with a processing device that has a natural language processing model for generating text, A step of evaluating the safety of text generated by the natural language processing model using a machine learning model that evaluates the safety of text, Based on the evaluation results in the aforementioned evaluation step, a step is made to determine whether or not the text generated by the natural language processing model can be output to the source of the output request, A step of notifying the processing device of the determination result in the determination step, A judgment program that causes a computer to execute a certain action.

13. A step of exchanging information with a processing device that has a natural language processing model for generating text, The steps include: evaluating the safety of a prompt that instructs a natural language processing model to generate text using a machine learning model that evaluates the safety of text; and evaluating the safety of the text generated by the natural language processing model. Based on the evaluation results in the aforementioned evaluation step, the steps include determining whether the prompt can be input to the natural language processing model and determining whether the text generated by the natural language processing model can be output to the source of the generation request, A step of notifying the processing device of the determination result in the determination step, A judgment program that causes a computer to execute a certain action.

Images