Generation program, generation method, and information processing device

Abstract

An information processing device (100) acquires first question information that is generated by inputting a prompt, which includes a message pertaining to an existing question, into a generative artificial intelligence (AI), and that pertains to the message. The information processing device (100): executes filtering processing for determining whether or not the acquired first question information is related to existing second question information; and outputs the result of the executed filtering processing.

Description

GENERATION PROGRAM, GENERATION METHOD, AND INFORMATION PROCESSING APPARATUS

[0001] The present invention relates to a generation program and the like.

[0002] FAQs (Frequently Asked Questions) are created based on the knowledge of the FAQ creator, so if the FAQ creator lacks knowledge in a particular category, problems may arise, such as questions in that particular category not being included in the FAQ.

[0003] To address the above problem, there is a conventional technology that uses LLMs (Large Language Models) to generate questions that are difficult for FAQ creators to anticipate.

[0004] Fig. 10 is a diagram for explaining the conventional technology. As shown in Fig. 10, the conventional technology uses an Operator 10, an FAQ DB 11, and an LLM 12. For example, a User 5 is a FAQ creator. The User 5 requests the Operator 10 to generate a new question (step S10).

[0005] The Operator 10 requests an existing question from the FAQ DB 11 (step S11). The FAQ DB 11 responds with the existing question to the Operator 10 (step S12). The Operator 10 combines the existing question with a prompt sentence (step S13).

[0006] The Operator 10 sends the prompt to the LLM 12 (step S14). The LLM 12 responds with a question generated based on the prompt (step S15). The Operator 10 notifies the user 5 of the result (step S16).

[0007] US Patent No. 11676044 JP 2023-76413 A International Publication No. 2022 / 180990 JP 2023-124315 A

[0008] However, the above-mentioned conventional techniques have a problem in that new questions cannot be acquired efficiently.

[0009] For example, the questions sent in step S15 of Figure 10 include a mixture of "questions similar to existing questions" and "new questions that are not similar to existing questions," making it impossible to efficiently acquire new questions.

[0010] In one aspect, the present invention aims to provide a generation program, a generation method, and an information processing device that can efficiently acquire new questions.

[0011] In the first proposal, a computer is caused to execute the following process: the computer acquires first question information related to a message, which is generated by inputting a prompt including a message related to an existing system into a generation AI (artificial intelligence), the computer executes a filtering process for determining whether the acquired first question information is related to existing second question information, and outputs the result of the executed filtering process.

[0012] New questions can be acquired efficiently.

[0013] FIG. 1 is a diagram showing an example of a system according to this embodiment. FIG. 2 is a diagram (1) for explaining the processing of an information processing device. FIG. 3 is a diagram (2) for explaining the processing of the information processing device. FIG. 4 is a functional block diagram showing the configuration of an information processing device according to this embodiment. FIG. 5 is a diagram showing an example of the data structure of an FAQDB. FIG. 6 is a flowchart showing the processing procedure of an information processing device according to this embodiment. FIG. 7 is a flowchart (1) showing the processing procedure of a filtering process. FIG. 8 is a flowchart (2) showing the processing procedure of a filtering process. FIG. 9 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device according to this embodiment. FIG. 10 is a diagram for explaining the prior art.

[0014] Hereinafter, embodiments of the generating program, generating method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0015] First, an example of a system according to this embodiment will be described. Fig. 1 is a diagram showing an example of a system according to this embodiment. As shown in Fig. 1, this system includes an LLM server 30 and an information processing device 100. The LLM server 30 and the information processing device 100 are connected to each other via a network 20.

[0016] The LLM server 30 is a server equipped with LLMs (Large Language Models). For example, when the LLM server 30 receives prompt sentence information from the information processing device 100, it inputs the received prompt sentence into the LLM to obtain an output result of the LLM. The LLM server 30 transmits the output result of the LLM to the information processing device 100. The LLM is a large-scale language model, such as a trained machine learning model. The LLM server 30 performs machine learning using training data, which is a group of texts in various formats including question sentences, to update the parameters of the machine learning model and construct a trained machine learning model. The LLM server 30 also stores the constructed trained machine learning model. The LLM is, for example, generative AI (artificial intelligence).

[0017] The information processing device 100 transmits prompt information to the LLM server 30, instructing the user to create a new question based on an existing question, and thereby obtains the output result of the LLM from the LLM server 30. For example, the output result includes information on multiple questions created in the LLM. The information processing device 100 performs a filtering process on the multiple questions included in the output result to determine whether they are related to existing questions. This allows new questions to be obtained efficiently.

[0018] For example, the information processing device 100 includes an FAQ DB 141 , an Operator 151 , a filtering unit 152 , and a category detection unit 153 .

[0019] The information processing device 100 will be described in more detail with reference to Figures 2 and 3. Figures 2 and 3 are diagrams for explaining the processing of the information processing device.

[0020] 2 will be described. The Operator 151 of the information processing device 100 receives a request to create a new question from the user 6 (step S20). The Operator 151 requests existing questions from the FAQ DB 141 (step S21). The FAQ DB 141 responds with the existing questions to the Operator 151 (step S22).

[0021] The Operator 151 combines an existing question with a prompt (step S23). For example, the prompt combined in step S23 includes the content of the existing question and instructions to create multiple (e.g., 10) new questions based on the existing questions. The Operator 151 then sends the prompt to the LLM server 30 (step S24). The LLM server 30 then inputs the prompt into the LLM and responds with a question generated by the LLM (step S25).

[0022] The Operator 151 outputs the questions generated by the LLM to the filtering unit 152 (step S26). The filtering unit 152 of the information processing device 100 performs a filtering process on the questions generated by the LLM, and selects new questions (a plurality of new questions) (step S27).

[0023] Next, we will move on to the explanation of Fig. 3. The filtering unit 152 outputs the new questions to the category detection unit 153 (step S28). The category detection unit 153 sends the contents of the multiple new questions and a prompt statement requesting a category list for the multiple new questions to the LLM server 30 (step S29).

[0024] The LLM server 30 inputs the prompt sentence to the LLM, and the LLM returns a category list generated by the LLM (step S30). The category detection unit 153 outputs the category list to the Operator 151 (step S31).

[0025] The Operator 151 combines the existing question, the category list, and the prompt sentence (step S32). For example, the prompt sentence combined in step S32 includes the content of the existing question and content instructing the creation of multiple (e.g., 10) new questions in the "{category list}" category. Note that the Operator 151 may further combine the content of a new question selected by the filtering unit 152 with the content of the existing question.

[0026] The Operator 151 sends the prompt to the LLM server 30 (step S33). The LLM server 30 inputs the prompt to the LLM, and the LLM responds with a question generated by the LLM (step S34). The Operator 151 notifies the user 6 of the result (the question generated by the LLM) (step S35).

[0027] An example of the processing executed by the information processing device 100 has been described above with reference to Figures 2 and 3. For example, the question generated in step S34 of Figure 3 is a "question that is not similar to existing questions." This allows the FAQ to be efficiently expanded to include questions that anticipate a wide range of user concerns.

[0028] Here, an example of the content of the questions and category list described with reference to Figures 2 and 3 will be described. For example, the existing questions set in step S23 of Figure 2 are the following questions 1a, 1b, and 1c.

[0029] Existing Question 1a: How do I create an account on the website? Existing Question 1b: How do I delete an account on the website? Existing Question 1c: What are the website's opening hours?

[0030] The responses to the questions generated by the LLM described in step S25 of FIG. 2 include the following questions 2a and 2b.

[0031] Generated Question 2a: How does the website ensure the security of my personal information? Generated Question 2b: How do I delete my account on the website?

[0032] The new question selected in step 27 of FIG. 2 is question 2a out of the above questions 2a and 2b.

[0033] The category list returned in step 30 of FIG. 3 is "Security."

[0034] The responses to the questions generated by the LLM described in step S34 of FIG. 3 include the following questions 3a and 3b.

[0035] Generated Question 3a: How do I report a security issue on a website? Generated Question 3a: How do I secure my website account?

[0036] The above describes an example of the content of the questions and category list described with reference to Figures 2 and 3. That is, the information processing device 100 can use the LLM to generate new questions 3a and 3b from existing questions 1a, 1b, and 1c, and notify the user 6 of the new questions.

[0037] In the examples shown in Figures 1 to 3, the information processing device 100 sends prompt statement information to the LLM server 30 and obtains the LLM output result, but this is not limited to this, and the information processing device 100 may also have the function of LLM.

[0038] Next, a configuration example of the information processing device 100 that executes the above-mentioned processing will be described. Fig. 4 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 4, the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0039] The communication unit 110 executes data communication with the LLM server 30 and other external devices via a network. The communication unit 110 is realized by a NIC (Network Interface Card) or the like.

[0040] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc. For example, the user 6 operates the input unit 120 to make a request to generate a new question, etc.

[0041] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, etc. For example, the display unit 130 displays a new question generated by the LLM.

[0042] The storage unit 140 includes an FAQ DB 141. The storage unit 140 is a memory or the like.

[0043] The FAQDB 141 stores information about existing questions. FIG. 5 is a diagram showing an example of the data structure of the FAQDB. As shown in FIG. 5, the FAQDB 141 associates item numbers with existing questions. The item numbers are numbers assigned to each existing question. The existing questions are text information about questions that have already been used in the FAQ.

[0044] For example, the existing question for item number "1" is "I want to know how to create a website account." The existing question for item number "2" is "I want to know how to delete a website account." The existing question for item number "3" is "I want to know the website's business hours."

[0045] The control unit 150 includes an operator 151, a filtering unit 152, and a category detection unit 153. The control unit 150 is, for example, a CPU (Central Processing Unit).

[0046] 2, when the Operator 151 receives a request for a new question from the user 6, the Operator 151 acquires an existing question from the FAQ DB 141 and transmits a prompt statement that combines the existing question and the prompt statement to the LLM server 30. When the Operator 151 acquires a question generated by the LLM from the LLM server 30, the Operator 151 outputs the acquired question to the filtering unit 152. Note that the Operator 151 may also output an existing question to the filtering unit 152.

[0047] 3, when the Operator 151 receives a category list from the category detection unit 153, it sends a prompt statement that combines an existing question, the category list, and a prompt statement to the LLM server 30. When the Operator 151 obtains a question generated by the LLM from the LLM server 30, it causes the display unit 130 to display the results of the obtained question.

[0048] Other explanations regarding the Operator 151 are the same as those regarding the Operator 151 described with reference to FIGS.

[0049] The filtering unit 152 performs filtering processing on the questions generated by the LLM to select new questions, as described in Fig. 2. The filtering unit 152 outputs the selected new questions to the category detection unit 153, as described in Fig. 3.

[0050] Here, an example of the processing of the filtering unit 152 will be described. For convenience of explanation, it is assumed that the questions generated by the LLM include questions A1, A2, and A3, and that the existing questions include questions B1, B2, and B3. Questions A1 to A3 and questions B1 to B3 are text information of the questions.

[0051] The filtering unit 152 calculates the vectors of questions A1, A2, and A3. For example, the filtering unit 152 vectorizes the questions (text) using bag of words, distributed representation, etc. The vector of question A1 is V LLM1 Let the vector for question A2 be V LLM2 Let the vector for question A3 be V LLM3 Let's say.

[0052] The filtering unit 152 calculates the vectors of questions B1, B2, and B3. For example, the vector of question B1 is V list1 Let the vector for question B2 be V list2 Let the vector for question B3 be V list3 Let's say.

[0053] The filtering unit 152 filters the vector of the question A1 as V LLM1 and Vlist1 Distance, V LLM1 and V list2 Distance, V LLM1 and V list3 If the distances of all the vectors are equal to or greater than the threshold, the filtering unit 152 selects the question A1 as a new question.

[0054] The filtering unit 152 filters the vector of the question A2 as V LLM2 and V list1 Distance, V LLM2 and V list2 Distance, V LLM2 and V list3 If the distances of all the vectors are equal to or greater than the threshold, the filtering unit 152 selects question A2 as a new question.

[0055] The filtering unit 152 filters the vector of the question A3 as V LLM3 and V list1 Distance, V LLM3 and V list2 Distance, V LLM3 and V list3 If the distances of all the vectors are equal to or greater than the threshold, the filtering unit 152 selects question A3 as a new question.

[0056] Other aspects of the filtering unit 152 are the same as those of the filtering unit 152 described with reference to FIG.

[0057] The category detection unit 153 detects the category of a new question using the LLM, as described in FIG. 3 . For example, the category detection unit 153 acquires new questions from the filtering unit 152. The category detection unit 153 receives a category list from the LLM server 30 by transmitting the contents of multiple new questions and a prompt statement requesting a category list for the multiple new questions to the LLM server 30. The category detection unit 153 outputs the category list to the Operator 151.

[0058] Other aspects of the category detection unit 153 are the same as those of the category detection unit 153 described with reference to FIG.

[0059] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 6 is a flowchart showing the processing procedure of the information processing device according to this embodiment. As shown in Fig. 6, when the Operator 151 of the information processing device 100 receives a request to generate a new question from the user 6, the Operator 151 acquires an existing question from the FAQ DB 141 (step S101).

[0060] The Operator 151 combines an existing question with a prompt sentence and sends the prompt sentence to the LLM server 30 (step S102). The Operator 151 receives a response to the question generated by the LLM (step S103).

[0061] The filtering unit 152 of the information processing device 100 executes a filtering process (step S104). The category detection unit 153 of the information processing device 100 transmits a prompt statement requesting a category list for the new question to the LLM server 30 (step S105). The category detection unit 153 receives the category list (step S106).

[0062] The Operator 151 combines the existing question, category list, and prompt sentence, and sends the prompt sentence to the LLM server 30 (step S107). The Operator 151 receives the response to the question generated by the LLM (step S108). The Operator 151 outputs the result to the display unit 130 for display (step S109).

[0063] Next, an example of the processing procedure of the filtering process described in step S104 of Fig. 6 will be described. Fig. 7 is a flowchart (1) showing the processing procedure of the filtering process. As shown in Fig. 7, the filtering unit 152 of the information processing device 100 acquires a question generated by the LLM and an existing question from the Operator 151 (step S201).

[0064] The filtering unit 152 calculates a vector of the question generated by the LLM (step S202).The filtering unit 152 calculates a vector of the existing question (step S203).

[0065] The filtering unit 152 selects a new question based on the distance between the question vector generated by the LLM and the vector of an existing question (step S204), and outputs the selected new question to the category detection unit 153 (step S204).

[0066] The filtering process performed by the filtering unit 152 is not limited to the process shown in Fig. 7. For example, the filtering unit 152 may display the selected results to the user and further receive an instruction as to whether the selected question is a new question. Fig. 8 is a flowchart (2) showing the processing procedure of the filtering process.

[0067] 8, the filtering unit 152 of the information processing device 100 acquires a question generated by the LLM and an existing question from the Operator 151 (step S301). The filtering unit 152 calculates a vector of the question generated by the LLM (step S302). The filtering unit 152 calculates a vector of the existing question (step S303).

[0068] The filtering unit 152 selects questions whose distance is equal to or greater than a threshold based on the distance between the question vector generated by the LLM and the vector of an existing question (step S304).The filtering unit 152 displays the selected questions and the existing questions on the display unit in a manner that allows them to be compared (step S305).

[0069] The filtering unit 152 receives a selection of a new question from the user 6 (step S306), and outputs the selected new question to the category detection unit 153 (step S307).

[0070] 8, the filtering unit 152 displays the questions selected using the vectors and existing questions so that they can be compared, but this is not limiting. For example, the filtering unit 152 may display the questions before selection, which are generated by the LLM, and existing questions so that they can be compared, and allow the user 6 to select a new question.

[0071] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 transmits prompt information, which instructs the creation of new questions based on existing questions, to the LLM server 30, and thereby obtains LLM output results from the LLM server 30. For example, the output results include information on multiple questions created in the LLM. The information processing device 100 performs a filtering process on the multiple questions included in the output results to determine whether they are related to existing questions. This makes it possible to efficiently obtain new questions.

[0072] The information processing device 100 selects new questions based on the vectors of existing questions and the vectors of questions generated by the LLM, thereby enabling appropriate selection of new questions.

[0073] The information processing device 100 displays existing questions and questions generated by the LLM on the display unit 130, and accepts the selection of a new question from the questions generated by the LLM, thereby enabling the user to appropriately select a new question from their perspective.

[0074] The information processing device 100 acquires a category list for the selected new question using the LLM. The information processing device 100 also acquires multiple pieces of question information generated by inputting a prompt to the LLM requesting the generation of multiple pieces of new question information belonging to a category. This allows new questions to be acquired efficiently. The information processing device 100 also generates a prompt for the category to which the acquired new question belongs, and can acquire the new question generated by the LLM based on the generated prompt for the category. This allows new questions to be appropriately selected.

[0075] FIG. 9 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing apparatus of this embodiment. As shown in FIG. 9, the computer 200 has a CPU 201 that executes various arithmetic processes, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that exchanges data with external devices via a wired or wireless network, and an interface device 205. A microphone, speaker, etc. may be connected to the interface device 205. The computer 200 also has a RAM 206 that temporarily stores various information, and a hard disk drive 207. The devices 201 to 207 are connected to a bus 208.

[0076] The hard disk drive 207 stores an operator program 207 a, a filtering program 207 b, and a category detection program 207 c. The CPU 201 reads out each of the programs 207 a to 207 c and loads them into the RAM 206.

[0077] The Operator program 207a functions as the Operator process 206a, the Filtering program 207b functions as the Filtering process 206b, and the Category detection program 207c functions as the Category detection process 206c.

[0078] The processing of the Operator process 206a corresponds to the processing of the Operator 151. The processing of the filtering process 206b corresponds to the processing of the filtering unit 152. The processing of the category detection process 206c corresponds to the processing of the category detection unit 153.

[0079] It should be noted that each of the programs 207a to 207c does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207c.

[0080] 30 LLM server 100 Information processing device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 FAQ DB 150 Control unit 151 Operator 152 Filtering unit 153 Category detection unit

Claims

1. A generation program that causes a computer to execute the following processes: acquire first question information related to a message generated by inputting a prompt containing a message related to an existing question into a generation AI (Artificial Intelligence); execute a filtering process to determine whether the acquired first question information is related to existing second question information; and output the results of the executed filtering process.

2. The generation program described in claim 1, characterized in that the filtering process determines whether the first question information is related to the second question information based on the vector of the first question information and the vector of the second question information.

3. The generation program described in claim 1, characterized in that the filtering process displays the first question information and the second question information on a screen and accepts a selection as to whether the first question information is related to the second question information.

4. The generation program described in claim 1, characterized in that if the result of the filtering process is that the first question information is not related to the second question information, the generation program further causes the computer to execute a process of obtaining a category to which the first question information belongs, which is generated by inputting a prompt including the first question information into the generation AI.

5. The generation program described in claim 4, further comprising causing the computer to execute a process of acquiring multiple pieces of question information that are generated by inputting a prompt to the generation AI requesting the AI ​​to generate multiple pieces of new question information that belong to the category based on the first question information.

6. The generation program according to claim 4, wherein the generation AI is a large-scale language model, generates a prompt related to the category to which the acquired first question information belongs, and acquires the question information generated by the large-scale language model based on the generated prompt related to the category.

7. A generation method characterized by a computer executing the following processes: acquiring first question information related to the message, which is generated by inputting a prompt including a message related to an existing question into a generation AI (Artificial Intelligence); executing a filtering process to determine whether the acquired first question information is related to existing second question information; and outputting the results of the executed filtering process.

8. The generation method described in claim 7, characterized in that the filtering process determines whether the first question information is related to the second question information based on the vector of the first question information and the vector of the second question information.

9. The generation method described in claim 7, characterized in that the filtering process displays the first question information and the second question information on a screen and accepts a selection as to whether the first question information is related to the second question information.

10. The generation method described in claim 7, characterized in that if the result of the filtering process is that the first question information is not related to the second question information, the computer is further made to execute a process of obtaining the category to which the first question information belongs, which is generated by inputting a prompt including the first question information into the generation AI.

11. The generation method described in claim 10, characterized in that the computer further executes a process of acquiring multiple pieces of question information that are generated by inputting a prompt to the generation AI requesting that the AI ​​generate multiple pieces of new question information belonging to the category based on the first question information.

12. An information processing device having a control unit that executes the following processes: acquire first question information related to a message generated by inputting a prompt including a message related to an existing question into a generation AI (Artificial Intelligence); execute a filtering process to determine whether the acquired first question information is related to existing second question information; and output the results of the executed filtering process.

13. The information processing device described in claim 12, characterized in that the filtering process determines whether the first question information is related to the second question information based on the vector of the first question information and the vector of the second question information.

14. An information processing device as described in claim 12, characterized in that the filtering process displays the first question information and the second question information on a screen and accepts a selection as to whether the first question information is related to the second question information.

15. The information processing device described in claim 12, characterized in that if the result of the filtering process is that the first question information is not related to the second question information, the computer is further made to execute a process of obtaining a category to which the first question information belongs, which is generated by inputting a prompt including the first question information into the generation AI.

16. The information processing device described in claim 15, characterized in that the control unit further executes a process of acquiring multiple pieces of question information generated by inputting a prompt to the generation AI requesting the generation AI to generate multiple pieces of new question information belonging to the category based on the first question information.

Images