Processing apparatus, processing method, and program

Abstract

To improve the quality of answers output from LLM (Large-scale Language Model).SOLUTION: A processing apparatus of the present disclosure accepts input of questions and an analysis viewpoint. Then, the processing apparatus repeatedly executes the process of inputting questions and prompts indicating an analysis viewpoint into the LLM and generating answers, using the input questions and new questions generated based on the answers.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a processing device, a processing method, and a program. [Background technology]

[0002] A technology that uses large-scale language models (LLMs) to perform natural language processing such as question-and-answer sessions is widely used. However, this technology faces the challenge of improving the quality of answers output from LLMs. Related technology is disclosed in Patent Document 1.

[0003] The technology disclosed in Patent Document 1 generates prompts that include useful sentences as reference information for the input question within a set character limit. By generating prompts in this way, this technology improves the quality of answers output from LLM. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7313757 Summary of the Invention [Problem to be solved by the invention]

[0005] One example of a goal of the present disclosure is to improve the quality of answers output from LLMs. [Means for solving the problem]

[0006] According to the present disclosure, an input receiving means for receiving input of a question and an analysis point of view; a search means for repeatedly executing a process of inputting a question and a prompt indicating an analytical viewpoint into a large-scale language model to generate an answer, using the input question and a new question generated based on the answer; A processing device is provided having:

[0007] Further, according to the present disclosure, One or more computers Accepts input of questions and analytical viewpoints, A processing method is provided in which a process of inputting a question and a prompt indicating an analytical viewpoint into a large-scale language model to generate an answer is repeatedly performed using the input question and a new question generated based on the answer.

[0008] Further, according to the present disclosure, Computer, an input receiving means for receiving input of a question and an analysis point of view; a search means for repeatedly executing a process of inputting a question and a prompt indicating an analytical viewpoint into a large-scale language model to generate an answer, using the input question and a new question generated based on the answer; A program is provided to function as a [Effects of the Invention]

[0009] According to one aspect of the present disclosure, the quality of answers output from the LLM can be improved. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of a functional block diagram of a processing device according to the present disclosure. [Figure 2] 10 is a flowchart illustrating an example of a processing flow of a processing device according to the present disclosure. [Figure 3] FIG. 1 is a diagram for explaining an overview of a processing apparatus according to the present disclosure. [Figure 4] FIG. 1 is a diagram illustrating an example of a hardware configuration of a processing device according to the present disclosure. [Figure 5] FIG. 1 illustrates an example of a functional block diagram of a processing device according to the present disclosure. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional block diagram of a search unit according to the present disclosure. [Figure 7]10 is a flowchart illustrating another example of the processing flow of the processing device according to the present disclosure. [Figure 8] FIG. 2 is a diagram for explaining a processing flow of a processing device according to the present disclosure. [Figure 9] FIG. 10 is a diagram for explaining an example of processing of a processing device according to the present disclosure. [Figure 10] FIG. 10 is a diagram for explaining another example of processing by the processing device according to the present disclosure. [Figure 11] FIG. 10 is a diagram for explaining another example of processing by the processing device according to the present disclosure. [Figure 12] FIG. 10 is a diagram for explaining another example of processing by the processing device according to the present disclosure. [Figure 13] FIG. 10 is a diagram for explaining another example of processing by the processing device according to the present disclosure. [Figure 14] FIG. 10 is a diagram for explaining another example of processing by the processing device according to the present disclosure. [Figure 15] FIG. 10 is a diagram for explaining another example of processing by the processing device according to the present disclosure. [Figure 16] FIG. 10 is a diagram illustrating an example of information output by a processing device according to the present disclosure. [Figure 17] FIG. 10 is a diagram for explaining another example of processing by the processing device according to the present disclosure. [Figure 18] FIG. 10 is a diagram for explaining another example of processing by the processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are designated by similar reference numerals, and descriptions thereof will be omitted as appropriate.

[0012] <<First embodiment>> Fig. 1 is a functional block diagram showing an overview of the processing device 10. Fig. 2 is a flowchart showing an example of the flow of processing executed by the processing device 10.

[0013] 1, the processing device 10 includes an input receiving unit 11 and a search unit 12. These functional units execute the processing of the flowchart in FIG.

[0014] In S10, the input receiving unit 11 receives input of a question and an analysis point of view. In S11, the search unit 12 repeatedly executes the process of inputting a question and a prompt indicating an analytical viewpoint into the LLM and generating an answer, using the input question and a new question generated based on the answer.

[0015] That is, as shown in Fig. 3, when the processing device 10 receives input of a question and an analytical viewpoint, it inputs a prompt into the LLM and repeats the process of generating an answer. The processing device 10 repeats such a process, making it possible to simulate the behavior of a human being who repeatedly thinks and delves deeper. As a result, it is possible to generate higher quality answers.

[0016] Furthermore, the processing device 10 automatically generates prompts and inputs them into the LLM during repeated processing. In other words, the user only needs to input the question and analysis perspective once at the beginning, and does not need to input prompts each time the processing is repeated. In this way, the processing device 10 can simulate the human behavior of repeatedly thinking and digging deeper without requiring the user's effort. This allows for the generation of higher quality answers.

[0017] The processing device 10 can also generate questions based on answers and generate prompts using the new questions. This type of processing device 10 can mimic human behavior, where a person obtains a thought result and then delves deeper based on that result. This allows for the generation of higher quality answers.

[0018] Furthermore, the processing device 10 can input not only a question but also a prompt indicating an analytical perspective into the LLM to obtain an answer. By adding an analytical perspective, a higher quality answer can be generated.

[0019] <<Second embodiment>> <Summary> The processing apparatus 10 of the second embodiment is a specific embodiment of the configuration of the processing apparatus 10 of the first embodiment, which will be described in detail below.

[0020] <Hardware configuration> First, an example of the hardware configuration of the processing device 10 will be described. Each functional unit of the processing device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are many variations in the realization method and device. The software includes programs that are pre-loaded in the device before shipping, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0021] FIG. 4 is a block diagram illustrating an example of the hardware configuration of a processing device 10. As shown in FIG. 4, the processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device 10 does not necessarily have to have the peripheral circuit 4A. Note that the processing device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices may have the above hardware configuration.

[0022] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a central processing unit (CPU) or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, and touch panel. Examples of output devices include a display, projector, speaker, printer, and mailer. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0023] <Functional configuration> Next, the functional configuration of the processing device 10 will be described in detail. Fig. 5 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 has an input receiving unit 11, a search unit 12, and a result output unit 13. As shown in Fig. 6, the search unit 12 has an information retrieval unit 121, an answer generation unit 122, an answer verification unit 123, and a question generation unit 124.

[0024] These functional units execute the processes shown in the flowchart of Fig. 7. An overview of each process and the overall picture of the process will be described with reference to Fig. 8.

[0025] Input Receiving Process S20: The input receiving unit 11 receives input of a question and an analysis viewpoint. For example, the input receiving unit 11 receives input of a question Q and an analysis viewpoint P, as shown in FIG.

[0026] Related information search process S21: The information search unit 121 searches for related information related to the question. For example, as shown in FIG. 8, the information search unit 121 searches for related information R related to the question Q from the information stored in the database. 1,1 ~R 4,3 Search for.

[0027] Answer generation process S22: The answer generation unit 122 generates a prompt using the question, the analysis viewpoint, and the related information, and inputs the generated prompt to the LLM to generate a primary answer. For example, the information search unit 121 generates a primary answer using the question Q, the analysis viewpoint P, and the related information R shown in FIG. 1,1 ~R 4,3 The generated prompts are input to a large-scale language model to generate primary answers A1 to A4.

[0028] Answer verification process S23: The answer verification unit 123 determines whether the relevance between the primary answer and the question satisfies the pass condition, and outputs the primary answer that satisfies the pass condition as an answer. For example, the answer verification unit 123 determines whether the relevance between the question Q and each of the primary answers A1 to A4 shown in FIG. 8 satisfies the pass condition. Then, the answer verification unit 123 outputs the primary answers A1 to A3 that satisfy the pass condition as answers for the next process. Note that, as shown in FIG. 8, the answer verification unit 123 may discard the primary answer A4 that does not satisfy the pass condition rather than output it for the next process. Also, as shown in FIG. 8, the answer verification unit 123 may correct the primary answer A2 that does not satisfy the pass condition, and output the corrected primary answer A2 that satisfies the pass condition as an answer for the next process.

[0029] Loop end determination process S24: The answer verification unit 123 determines whether to end the loop. If the loop is not to be ended, the processing device 10 proceeds to question generation process S25. If the loop is to be ended, the processing device 10 proceeds to result output process S26.

[0030] Question generation process S25: The question generation unit 124 generates hypotheses based on the answers output in the answer verification process S23, and generates new questions based on the generated hypotheses. For example, as shown in FIG. 8, the question generation unit 124 generates hypotheses H1 to H3 based on the answers A1 to A3 output in the answer verification process S23, respectively. Then, the question generation unit 124 generates new questions based on the generated hypotheses H1 to H3, respectively. Thereafter, the processing device 10 returns to the related information search process S21 and repeats the same process. In the second and subsequent processes, the new questions generated in the question generation process S25 are used instead of the question Q input in the input reception process S20.

[0031] Result output process S26: The result is output.

[0032] Next, each process will be described in detail.

[0033] "Input reception process S20" The input receiving unit 11 receives input of a question and an analysis viewpoint. The question is made up of, for example, a sentence. The analysis viewpoint is made up of, for example, a word or a sentence.

[0034] There are no particular restrictions on the format of the input of the question and analysis viewpoint. For example, a free description format may be adopted. In this case, the input receiving unit 11 can use techniques such as morphological analysis to decipher the input content and identify the question sentence and analysis viewpoint. An example of such input is shown below. The data entered by the user is enclosed in " ". (Example 1) "What are the challenges in data utilization? The analysis perspective is issues, causes, and solutions." (Example 2) "Please answer the questions about data utilization issues from the perspective of analyzing the issues, causes, and solutions."

[0035] Alternatively, the input receiving unit 11 may provide a separate field for receiving question input and a field for receiving analysis viewpoint input on a UI (user interface) screen for receiving input. The input receiving unit 11 may then accept content entered in the question field as a question, and content entered in the analysis viewpoint field as an analysis viewpoint. An example of such input is shown below. The data entered by the user is enclosed in " ". (Example 3) Question: "What are the challenges in data utilization?" Analysis perspective: "Challenges, causes, solutions"

[0036] The input receiving unit 11 can receive the input via any input device such as a keyboard, a touch panel, a mouse, a microphone, a physical button, etc. The processing device 10 may also be a server. In this case, the input receiving unit 11 can receive questions and analysis perspectives sent from a client terminal.

[0037] "Related information search process S21" The information search unit 121 searches for relevant information related to the question.

[0038] 7 and 8, the processing device 10 repeats a loop. In the first loop, the information search unit 121 can search for related information related to the question input in the input reception process S20. In the second and subsequent loops, the information search unit 121 can search for related information related to a new question generated by the question generation unit 124.

[0039] The related information may be, but is not limited to, document data, image data (drawings, photographs, etc.), data in other formats (tables, etc.), etc. The source (search destination) of the related information may be one or multiple. The related information may be information that is widely available on the web. The related information may also be information that is locally stored. The locally stored information may be data prepared for processing by the processing device 10, or may be information that is widely available locally.

[0040] The information search unit 121 can acquire related information using a search model that searches for related information based on some degree of relevance to the question.

[0041] For example, the information search unit 121 may use a web search engine to search for related information. In this example, the information search unit 121 inputs a question to the web search engine and performs a search. The information search unit 121 then acquires information on web pages included in the search results as related information related to the input question. If the search results have a large number of hits, the information search unit 121 may acquire information from a predetermined number of the top search results as related information.

[0042] Alternatively, the information search unit 121 may input a query to a search engine that searches locally stored information to perform a search. Then, the information search unit 121 may acquire local information included in the search results as related information related to the input query. If the search results have a large number of hits, the information search unit 121 may acquire information from a predetermined number of top search results as related information.

[0043] It should be noted that the method illustrated here is merely an example, and the information search unit 121 may search for related information relating to the question using other methods.

[0044] However, the related information retrieved by the above-described method may actually contain data (noise data) that has a low degree of relevance to the question (when judged by a person). Therefore, the information retrieval unit 121 may perform a process to remove the noise data from the retrieved related information.

[0045] For example, the information search unit 121 may calculate the degree of relevance between the question and the searched related information, and remove related information whose degree of relevance is equal to or less than a threshold as noise data. The information search unit 121 can calculate the degree of relevance between the question and the searched related information using a relationship determination model prepared in advance.

[0046] An example of the relationship determination model is to receive an input of a question and searched related information, and then output the similarity between them as the relevance. In this case, the information search unit 121 removes, as noise data, related information whose similarity (relevance) with the question is equal to or less than a threshold.

[0047] In recent years, technology for calculating similarity between sentences has become widely used. In one example, sentences are converted into vectors, and the similarity between the vectors (such as cosine similarity) is calculated as the similarity between the sentences. One example of a relationship judgment model can use this technology to calculate the similarity between a question and retrieved related information.

[0048] If the related information is image data such as a drawing or a photograph, the information search unit 121 may generate document data from the image data using a learning model that has been trained to generate document data representing an image from the image data.The information search unit 121 may then calculate the similarity between the generated document data and the question as the similarity (relevance) between the image data and the question.If the related information is data in another format such as a table, the similarity (relevance) between the question and the searched related information can be calculated using the same method as for image data.

[0049] Another example of a relevance judgment model is LLM. LLM is a natural language processing model trained using large amounts of text data. For example, when you input a query into LLM, it outputs an answer to that query. The sentence (query) input into LLM is called a "prompt."

[0050] The information search unit 121 may generate a prompt asking whether the question and the searched related information are related, and input the prompt into the LLM to obtain an answer as to whether the question and the searched related information are related.The information search unit 121 may then remove related information for which the answer is "not related" as noise data.There are various possible examples of prompts for this process, but one example is as follows. "Are the following questions and related information relevant? If they are relevant, output 1; if they are not relevant, output 0." [Question] What are the challenges in utilizing data? [Related information] Recent...

[0051] The information search unit 121 may generate a prompt using, for example, a template prepared in advance. For example, the template may be a sentence with blank spaces for inserting a question and for inserting related information. The answer generation unit 122 may then generate a prompt by inserting the question and related information into predetermined positions in the template. Alternatively, the information search unit 121 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0052] "Answer generation process S22" The answer generation unit 122 generates a prompt (hereinafter sometimes referred to as a "prompt for generating a primary answer") using the question, the analytical perspective, and related information, and inputs the generated prompt for generating a primary answer into the LLM to generate a primary answer.

[0053] An example is shown in Figure 9. The example prompt in the figure is an example of a prompt for generating a primary answer. The example answer in the figure is an example of a primary answer output from the LLM. As shown in the figure, in response to the input of one prompt for generating a primary answer, the LLM may generate and output multiple primary answers (three primary answers in the example shown). Note that in response to the input of one prompt for generating a primary answer, the LLM may generate and output one primary answer.

[0054] The answer generation unit 122 can generate a prompt for generating a primary answer by using, for example, a template prepared in advance. For example, the template may be a sentence in which positions for inserting a question, a position for inserting an analytical viewpoint, and a position for inserting related information are left blank. The answer generation unit 122 may then generate a prompt for generating a primary answer by inserting a question, a analytical viewpoint, and related information into predetermined positions in the template. Alternatively, the answer generation unit 122 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0055] The answer generation unit 122 can provide the related information in the context of the LLM, and then refer to the related information to generate a primary answer generation prompt that asks for an answer to the question from a specified perspective. The primary answer obtained from such a primary answer generation prompt is generated by referring to the related information and is an "answer to the question" related to the specified perspective. The technology that uses related information in this way is known as Retrieval Augmented Generation (RAG).

[0056] 7 and 8, the processing device 10 repeats a loop. In the first loop, the answer generation unit 122 can inquire about an answer to the question input in the input reception process S20. The related information to be included in the primary answer generation prompt in the first loop is related information related to the question input in the input reception process S20.

[0057] Then, in the second and subsequent loops, the answer generation unit 122 can inquire about an answer to the new question generated by the question generation unit 124. The related information to be included in the primary answer generation prompt in the second and subsequent loops is related information that is related to the new question generated by the question generation unit 124.

[0058] The analysis viewpoint to be included in the prompt for generating a primary answer is determined based on the analysis viewpoint input in the input receiving process S20, for example.

[0059] It is preferable to include different analytical perspectives in the prompts for generating a primary answer for each of the multiple loops. That is, it is preferable to use different analytical perspectives for each iteration of the process, which is repeatedly executed as shown in Figures 7 and 8. In this way, the more the loop is repeated, the more in-depth the thinking from various perspectives can be achieved.

[0060] The prompt for generating the primary answer may include one analytical perspective or a combination of multiple analytical perspectives.

[0061] When only one analytical perspective is to be included in the prompt for generating a primary answer, the answer generation unit 122 can select one from each of the multiple analytical perspectives input in the input reception process S20, for example, and include it in the prompt for generating a primary answer for each of the multiple loops.

[0062] This allows you to narrow your thinking in each loop to one analytical perspective, which increases the likelihood that the LLM will provide you with a more accurate answer that delves deeper into that one analytical perspective.

[0063] On the other hand, if multiple analytical perspectives are allowed to be included in the prompt for generating a primary answer, the answer generation unit 122 generates all combinations consisting of at least one component (analytical perspective) using the multiple analytical perspectives input in the input reception process S20, for example. Then, the answer generation unit 122 can select one set from each of the multiple combinations generated and include it in the prompt for generating a primary answer for each of the multiple loops.

[0064] In this case, "Perspective A" and "Perspective A + Perspective B" are considered to be different combinations of perspectives. Both include the same perspective, "Perspective A," but when combined with other perspectives, the LLM's answer may differ. By using a combination of various perspectives to obtain answers from the LLM, you can delve deeper into thinking from various perspectives.

[0065] The answer generation unit 122 may include an analytical perspective other than the analytical perspective input in the input receiving process S20 in the prompt for generating a primary answer. For example, candidates for analytical perspectives may be registered in advance in the processing device 10. Then, the answer generation unit 122 may determine an analytical perspective to be included in the prompt for generating a primary answer from among the candidates for analytical perspectives registered in advance, using the same method as described above.

[0066] Alternatively, the answer generation unit 122 may acquire a preferred analytical perspective for the question by generating a prompt asking for a preferred analytical perspective for the question and inputting the generated prompt into the LLM. Then, the answer generation unit 122 may determine an analytical perspective to be included in the primary answer generation prompt from the analytical perspectives acquired in this manner, using the same method as described above.

[0067] There are various possible examples of prompts for this process, but one example is as follows: "Please list five preferred analytical perspectives for the following questions. [Question] What are the challenges in utilizing data?

[0068] The answer generation unit 122 may generate a prompt using, for example, a template prepared in advance. For example, the template may be a sentence with a blank space where a question is to be inserted. The answer generation unit 122 may then generate a prompt by inserting a question into a predetermined position in the template. Alternatively, the answer generation unit 122 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0069] The analysis perspective may include business framework perspectives such as "people," "things," and "money." This will enable us to expect answers that focus on important business perspectives.

[0070] In this way, the analytical perspectives to be included in the prompts for generating a primary answer are not particularly limited, and various analytical perspectives can be included. However, in this case, there is a possibility that the LLM does not have sufficient knowledge about the analytical perspectives. As a result, the accuracy of the LLM's answers will be reduced. Therefore, related information related to various analytical perspectives may be registered in advance in the processing device 10. Then, the answer generation unit 122 may further include related information about the analytical perspectives in the prompts for generating a primary answer, as shown in FIG. 10.

[0071] Alternatively, the information search unit 121 may acquire related information related to the analytical viewpoint using a method similar to that for acquiring related information related to the question. Then, the answer generation unit 122 may include the related information of the analytical viewpoint acquired using such a method in the prompt for generating a primary answer.

[0072] By including relevant information about the analytical perspective in the prompts used to generate the first-stage answers, it becomes easier to obtain highly accurate answers from the LLM.

[0073] If the amount of related information is too large to fit within the context length that can be handled by the LLM, the answer generation unit 122 may select the related information in descending order of relevance and include it in the prompt for generating a primary answer. Alternatively, the answer generation unit 122 may generate a summary of the related information using the LLM and include the summary of the related information in the prompt for generating a primary answer instead of the related information.

[0074] The answer generation unit 122 may also classify the related information by topic and generate multiple prompts for generating a primary answer, each including the related information for each topic. The answer generation unit 122 may then input each of the multiple prompts for generating a primary answer generated in this manner into the LLM and generate answers to questions in the LLM for each topic. In this case, the prompts for generating a primary answer corresponding to each topic include the related information for that topic, but do not include the related information for other topics. Classification of the related information can be achieved by various methods. One example is, but is not limited to, a method of vectorizing each piece of related information and clustering it based on the similarity between the vectors.

[0075] "Answer verification process S23" The answer verification unit 123 determines whether the degree of association between the primary answer and the question (the question included in the primary answer generation prompt for obtaining that primary answer) satisfies the pass condition. Then, the answer verification unit 123 outputs the primary answer that satisfies the pass condition as the answer to the question. The answer verification unit 123 may output one primary answer as the answer to the question, or may output multiple primary answers as answers to the question. This processing allows subsequent processing (such as question generation, loop processing based on a new question, and output of results) to be performed based on the primary answer that has a high degree of association with the question.

[0076] The answer verification unit 123 can determine whether the degree of relevance between the primary answer and the question satisfies the pass condition based on one of the following verification processes 1 to 4.

[0077] ○Verification process 1 The answer verification unit 123 can calculate the degree of relevance between the primary answer and the question using a dual-encoder as shown in Fig. 11. If the calculated degree of relevance is equal to or greater than a threshold, the answer verification unit 123 can determine that the relationship between the primary answer and the question satisfies the pass condition. On the other hand, if the calculated degree of relevance is less than the threshold, the answer verification unit 123 can determine that the relationship between the primary answer and the question does not satisfy the pass condition.

[0078] The Dual-encoder includes a Query encoder and a Document encoder. When the query encoder receives a question (sentence), it converts the question into a vector and outputs that vector. When the document encoder receives a primary answer (text), it converts the primary answer into a vector and outputs the vector.

[0079] Then, the answer verification unit 123 calculates the similarity (cosine similarity, etc.) between the two vectors output from the two encoders as the degree of association between the primary answer and the question.

[0080] ○Verification process 2 The answer verification unit 123 can calculate the degree of relevance between the primary answer and the question using a cross encoder as shown in Fig. 12. If the calculated degree of relevance is equal to or greater than a threshold, the answer verification unit 123 can determine that the relationship between the primary answer and the question satisfies the pass condition. On the other hand, if the calculated degree of relevance is less than the threshold, the answer verification unit 123 can determine that the relationship between the primary answer and the question does not satisfy the pass condition.

[0081] The Cross encoder receives a question (sentence) and a primary answer (sentence) as input and outputs the relevance.

[0082] ○Verification process 3 The answer verification unit 123 can calculate the degree of relevance between the primary answer and the question using the LLM as shown in FIG.

[0083] The answer verification unit 123 generates a prompt asking whether the primary answer and the question are related, and inputs the prompt into the LLM to obtain an answer as to whether the primary answer and the question are related. If the answer indicates that the answer is "related," the answer verification unit 123 can determine that the relationship between the primary answer and the question satisfies the pass condition. On the other hand, if the answer indicates that the answer is "not related," the answer verification unit 123 can determine that the relationship between the primary answer and the question does not satisfy the pass condition.

[0084] Various examples of the prompt for this process are possible, and one example is as shown in FIG. 13. The answer verification unit 123 may generate the prompt using, for example, a template prepared in advance. For example, the template may be a sentence in which the positions where a question is to be inserted and the positions where a primary answer is to be inserted are left blank. The answer verification unit 123 may then generate the prompt by inserting the question and the primary answer into predetermined positions in the template. The template may also have blank spaces at the positions where an analytical perspective is to be inserted (the position of "Problem" in FIG. 13). Alternatively, the answer verification unit 123 may generate the prompt based on a generative model instead of generating the prompt based on a template. The generative model here can be realized using any widely known technology.

[0085] ○Verification process 4 The answer verification unit 123 can determine whether the relevance between the primary answer and the question satisfies the pass condition by a method that combines at least two of the verification processes 1 to 3.

[0086] In this example, if the verification results of each of the multiple verification processes satisfy a predetermined condition, the answer verification unit 123 determines that the degree of association between the primary answer and the question satisfies the pass condition.

[0087] The predetermined condition is one of the following: - In all of the multiple verification processes, the relationship between the first answer and the question meets the passing conditions. - In at least one of the multiple verification processes, the relationship between the primary answer and the question meets the passing condition. - The relationship between the primary answer and the question satisfies the pass criteria in a specified percentage of multiple verification processes - The relationship between the primary answer and the question satisfies the pass condition for a specified number of verification processes or more. The evaluation value calculated (for example, added together) based on the weighted scores of each verification process that determines that the relationship between the primary answer and the question meets the pass criteria is greater than or equal to the threshold.

[0088] If the relevance between the primary answer and the question does not satisfy the pass condition, the answer verification unit 123 may discard the primary answer. This process can prevent the inconvenience of subsequent processes (such as question generation, loop processing based on a new question, and output of results) being performed based on a primary answer with a low relevance to the question.

[0089] Alternatively, if the relevance between the primary answer and the question does not satisfy the pass condition, the answer verification unit 123 may modify the primary answer. Then, it may determine whether the relevance between the primary answer and the question after modification satisfies the pass condition. The answer verification unit 123 may then output the modified primary answer that satisfies the pass condition as the answer to the question. Through this processing, subsequent processing (such as question generation, loop processing based on the new question, and output of results) will be performed based on the primary answer that has been modified so that its relevance with the question satisfies the pass condition.

[0090] If the degree of relevance between the primary answer and the question is lower than a predetermined level, the answer verification unit 123 may discard the primary answer. If the degree of relevance between the primary answer and the question is higher than a predetermined level but does not satisfy the pass condition, the answer verification unit 123 may modify the primary answer.

[0091] The predetermined level here may be defined by the degree of association calculated in the verification process 1 and the verification process 2.

[0092] In one example, if the relevance is less than a first threshold, the response verification unit 123 discards the primary response. If the relevance is equal to or greater than the first threshold but less than a second threshold, the response verification unit 123 modifies the primary response. If the relevance is equal to or greater than the second threshold, the response verification unit 123 determines that the primary response satisfies the pass condition.

[0093] Alternatively, the predetermined level may be defined based on the verification results of each of a plurality of verification processes.

[0094] In one example, a weighted score is predetermined for each of the multiple verification processes. Then, the response verification unit 123 calculates (for example, adds up) the evaluation value of the primary response based on the weighted scores of each verification process that has been determined to satisfy the pass condition.

[0095] If the evaluation value is less than the first threshold, the response verification unit 123 discards the primary response. If the evaluation value is equal to or greater than the first threshold but less than the second threshold, the response verification unit 123 modifies the primary response. If the evaluation value is equal to or greater than the second threshold, the response verification unit 123 determines that the primary response satisfies the pass condition.

[0096] Next, a process for correcting the primary response will be described. The response verification unit 123 can perform either of the following correction processes 1 and 2.

[0097] Correction process 1 The answer verification unit 123 generates a prompt requesting correction of the primary answer and inputs it into the LLM to obtain the corrected primary answer. There are various examples of prompts for this process, but one example is as follows. "Please revise the following primary answer to the following question so that it is more relevant to the question. [Question] What are the challenges in utilizing data? [First Answer] In order to utilize the data...

[0098] The answer verification unit 123 may generate a prompt using, for example, a template prepared in advance. For example, the template may be a sentence in which the positions where a question is to be inserted and the positions where a primary answer is to be inserted are left blank. The answer verification unit 123 may then generate a prompt by inserting the question and the primary answer into predetermined positions in the template. Alternatively, the answer verification unit 123 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0099] Correction process 2 When correcting a primary answer using LLM, if the prompt content is inappropriate, the desired correction may not be made, and the content may end up being almost the same as the content before the correction. To prevent this problem, the answer verification unit 123 can generate a prompt as follows.

[0100] First, the response verification unit 123 calculates the difference between the primary response and the reference information using LLM, as shown in Fig. 14. There are various possible examples of prompts for this process, but one example is as shown above the prompt example in Fig. 14.

[0101] The answer verification unit 123 may generate a prompt using, for example, a template prepared in advance. For example, the template may be a sentence in which the positions where the primary answer and the positions where the reference sentence are to be inserted are left blank. The answer verification unit 123 may then generate a prompt by inserting the primary answer and the reference sentence into predetermined positions in the template. Alternatively, the answer verification unit 123 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0102] The reference sentence is information obtained in the previous processing. For example, the answer verification unit 123 can use the answer obtained in the previous loop and related information used to generate the answer as the reference sentence.

[0103] After the difference is obtained, the answer verification unit 123 prompts the user to correct the primary answer by focusing on the difference. There are various possible examples of the prompt for this process, but one example is shown below the prompt example in Figure 14.

[0104] The answer verification unit 123 may generate a prompt using, for example, a template prepared in advance. For example, the template may be a sentence in which the positions where a question is to be inserted, the positions where a primary answer is to be inserted, and the positions where a difference is to be inserted are left blank. The answer verification unit 123 may then generate a prompt by inserting the question, the primary answer, and the difference into predetermined positions in the template. Alternatively, the answer verification unit 123 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0105] "Loop end determination process S24" The answer verification unit 123 determines whether to end the loop. If the loop is not to be ended, the processing device 10 proceeds to a question generation process S25. If the loop is to be ended, the processing device 10 proceeds to a result output process S26.

[0106] A condition for terminating the loop is determined in advance. Then, the response verification unit 123 determines whether the termination condition for terminating the loop is met. If the termination condition is met, the response verification unit 123 decides to terminate the loop. On the other hand, if the termination condition is not met, the response verification unit 123 decides not to terminate the loop.

[0107] The loop termination condition can be various. For example, the loop termination condition may be one of the following: - The loop has been repeated more than the specified number of times. All or all combinations of analytical perspectives used have been used at least once The desired search results are no longer obtained in the related information search process S21 (the number of search results is below a threshold value)

[0108] The analysis perspective to be used may be the analysis perspective input in the input reception process S20. Alternatively, the analysis perspective to be used may be a candidate analysis perspective that has been registered in advance. Alternatively, the analysis perspective to be used may be an analysis perspective acquired using LLM. Alternatively, the analysis perspective to be used may be a combination of at least two of these.

[0109] "Question generation process S25" The question generation unit 124 generates a hypothesis based on the answer output in the answer verification process S23. Then, the question generation unit 124 generates a new question based on the generated hypothesis. By incorporating the viewpoint of the hypothesis in this way, the question becomes more specific, and it is expected that a more interesting answer will be more likely to be obtained in the next loop trial.

[0110] When one answer is output in the answer verification process S23, the question generation unit 124 generates one or more hypotheses corresponding to the one answer and generates one or more questions. Thereafter, the processing device 10 repeats the above-described loop using the one question or each of the multiple questions.

[0111] On the other hand, if multiple answers are output in the answer verification process S23, the question generation unit 124 generates one or multiple hypotheses corresponding to each of the multiple answers, and generates one or multiple questions. Thereafter, the processing device 10 repeats the above-mentioned loop using the one question or each of the multiple questions.

[0112] Next, the process of generating hypotheses will be described.

[0113] The question generator 124 generates hypotheses by inputting prompts to the LLM to generate hypotheses based on the answers output in the answer verification process S23 and obtaining the results.

[0114] The question generation unit 124 may generate a prompt using, for example, a template prepared in advance. For example, the template may be a sentence with blank spaces where the answer (or a summary thereof) output in the answer verification process S23 is to be inserted. The question generation unit 124 may then generate a prompt by inserting the answer (or a summary thereof) output in the answer verification process S23 into a predetermined position in the template. Alternatively, the question generation unit 124 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0115] The hypothesis generated here may be a hypothesis of the next analytical perspective of the answer output in the answer verification process S23. That is, in the question generation process S25 for the nth loop (n is an integer equal to or greater than 1), the question generation unit 124 may generate a hypothesis of the analytical perspective of the (n+1)th loop of the answer output in the answer verification process S23 for the nth loop. For example, assume that the analytical perspective of the nth loop is "issue," and the answer (issue) is output in the answer verification process S23 for the nth loop. In this case, in the question generation process S25 for the nth loop, the question generation unit 124 can generate a hypothesis of the cause (analysis perspective of the (n+1)th loop) of the answer (issue) output in the answer verification process S23 for the nth loop.

[0116] The next perspective is a perspective (or a combination of perspectives) that has not been used in any of the previous loops.

[0117] An example of hypothesis generation will now be described with reference to FIG. 15. In the example of FIG. 15, the process is structured in two steps: an outline of the hypothesis is generated, and then details of the hypothesis are generated. This two-step structure allows for the generation of highly accurate hypotheses. However, it is also possible to generate a prompt that directly asks, "What is the next analytical perspective hypothesis for the answer output in the answer verification process S23?" and generate details of the hypothesis in one step. This method also allows for the generation of hypotheses with a certain degree of accuracy.

[0118] 15, the question generator 124 first generates a prompt for generating an outline of a hypothesis for the next analytical perspective of the answer output in the answer verification process S23, as shown above the prompt example in the figure. Then, the question generator 124 inputs the prompt into the LLM to obtain the outline of the hypothesis.

[0119] Note that the question generation unit 124 may generate a prompt by using a summary of the answer instead of the answer output in the answer verification process S23, as in the example of Fig. 15. The question generation unit 124 can generate the summary by using the LLM.

[0120] The question generation unit 124 may generate a prompt using, for example, a template prepared in advance. For example, the template may be a sentence in which the position where the answer (or its summary) output in the answer verification process S23 is to be inserted and the position where the next analytical perspective is to be inserted are left blank. The question generation unit 124 may then generate a prompt by inserting the answer (or its summary) output in the answer verification process S23 and the next analytical perspective into predetermined positions in the template. Alternatively, the question generation unit 124 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0121] After obtaining the outline of the hypothesis, the question generator 124 generates a prompt that digs deeper into the outline of the hypothesis and generates details of the hypothesis, as shown below the example prompt in the figure.The question generator 124 then inputs the prompt into the LLM to obtain the details of the hypothesis.

[0122] The prompt for generating the details of a hypothesis asks for details of the hypothesis from the next analytical perspective after the hypothesis summary. The "next analytical perspective" here can be the same analytical perspective as the "next analytical perspective" used when generating the hypothesis summary. In other words, the question generation unit 124 can generate the hypothesis summary and the details of the hypothesis using the same analytical perspective as the next analytical perspective.

[0123] In the example of Fig. 15, "lack of skills" is obtained as the summary of the hypothesis of the cause (next analysis perspective) of the answer output in the answer verification process S23. Then, a prompt is generated asking about the hypothesis of the cause (next analysis perspective) of "lack of skills" (summary of the hypothesis).

[0124] The question generator 124 may generate a prompt using, for example, a template prepared in advance. For example, the template may be a sentence with blank spaces at the positions where the outline of the hypothesis and the next analytical perspective are to be inserted. The question generator 124 may then generate a prompt by inserting the outline of the hypothesis and the next analytical perspective at predetermined positions in the template. The template may also have blank spaces at the positions where questions are to be inserted. Alternatively, the question generator 124 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0125] Next, the process of generating a question based on a hypothesis will be described.

[0126] The question generation unit 124 generates a question asking about the next analytical perspective based on the generated hypothesis. The "next analytical perspective" here can be the same analytical perspective as the "next analytical perspective" used when generating the hypothesis. In other words, the question generation unit 124 uses the same analytical perspective as the next analytical perspective to generate an outline of the hypothesis, generate details of the hypothesis, and generate a new question.

[0127] For example, suppose the analysis perspective of the n-th loop is "problem," and the answer (problem) is output in the answer verification process S23 of the n-th loop. In this case, the question generation unit 124 generates a hypothesis (analysis perspective of the (n+1)-th loop) of the cause of the answer (problem) output in the answer verification process S23 of the n-th loop in the question generation process S25 of the n-th loop. Then, the question generation unit 124 can generate a question asking about the cause of the hypothesis (analysis perspective of the (n+1)-th loop) in the question generation process S25 of the n-th loop.

[0128] The question generator 124 may generate a question using, for example, a template prepared in advance. For example, the template may be a sentence with blank spaces where a hypothesis and a next analytical perspective are to be inserted. The question generator 124 may then generate a question by inserting the hypothesis and the next analytical perspective into predetermined positions in the template. Alternatively, the question generator 124 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0129] "Result output process S26" The result output unit 13 outputs to the user an answer screen showing the answer output by the answer verification unit 123 in the answer verification process S23. The result output unit 13 may display the answer screen via an output device such as a display or a projection device. Alternatively, if the processing device 10 is a server, the result output unit 13 may transmit the answer screen to a client terminal.

[0130] As a result of performing multiple loops, the answer verification unit 123 may output multiple answers in the answer verification process S23. For example, by repeating the loop, the processing device 10 can obtain a result in which answers from various analytical perspectives are linked together in the order in which the loop is executed. The result output unit 13 may output to the user an answer screen that displays a list of the multiple answers. Alternatively, the result output unit 13 may output to the user an answer screen that displays an answer selected from the multiple answers according to a predetermined rule.

[0131] Furthermore, the result output unit 13 may further output related information used when obtaining each answer. For example, the result output unit 13 may link each answer to the related information used when obtaining each answer on the answer screen and display it. Additionally, the result output unit 13 may specify one answer on the answer screen and output the related information used when obtaining the specified answer on a different screen (for example, on a different window) in response to a user input to display related information.

[0132] Furthermore, the result output unit 13 may further output a question for each answer. For example, the result output unit 13 may display each answer in association with a question for each answer on the answer screen. Alternatively, the result output unit 13 may designate one answer on the answer screen and, in response to a user input to display a question, output the question for the designated answer on the answer screen or on another screen (for example, on another window).

[0133] Furthermore, the result output unit 13 may further output the analytical viewpoint used in obtaining each answer. For example, the result output unit 13 may display each answer in association with the analytical viewpoint used in obtaining each answer on the answer screen. Additionally, the result output unit 13 may specify one answer on the answer screen and, in response to a user input to display the analytical viewpoint, output the analytical viewpoint used in obtaining the specified answer on the answer screen or on a separate screen (for example, on a separate window).

[0134] Furthermore, the result output unit 13 may further output hypotheses (including summaries and details) used to obtain each answer. For example, the result output unit 13 may link each answer to the hypothesis used to obtain that answer on the answer screen. Additionally, the result output unit 13 may specify one answer on the answer screen and, in response to a user input to display a hypothesis, output the hypothesis used to obtain the specified answer on the answer screen or on a separate screen (for example, on a separate window).

[0135] The amount of information output by the result output unit 13 can be enormous. Therefore, the result output unit 13 may output the information using various display methods such as a tree structure or a graph. Specific examples of this display will be described in the following embodiments.

[0136] "Action and effect" According to the processing apparatus 10 of the second embodiment, the same effects as those of the processing apparatus 10 of the first embodiment are achieved.

[0137] Furthermore, the processing device 10 can automatically generate prompts for obtaining desired answers based on input questions and analytical viewpoints using a template-based or generative model-based method, thereby reducing the time and effort required for users to create templates.

[0138] Furthermore, the processing device 10 can automatically collect related information necessary to obtain a desired answer by cross-sectionally searching multiple information sources based on the generated question. Then, the processing device 10 can increase the accuracy and specificity of the answer by generating an answer to the question using the collected related information.

[0139] Furthermore, the processing device 10 can generate multiple questions from the obtained answers and separately use the multiple questions to collect the related information and generate the answers. Such a processing device 10 allows for divergent thinking and allows for the generation of answers that have been considered from various perspectives.

[0140] The processing device 10 generates questions and searches for related information across a wide range of topics in a chain reaction, allowing it to simulate the human process of digging deeper and automate analysis from a desired perspective. Furthermore, by diverging thoughts as described above and using various related information, it is possible to discover unexpected relationships and new policy ideas that humans would not have noticed.

[0141] Furthermore, the processing device 10 of the second embodiment can generate answers to questions by combining LLM and RAG. Using RAG can improve the quality of answers output from LLM. Combining LLM with RAG makes it possible to extract various insights from collected related information, which is expected to greatly promote data utilization in various scenarios, such as information exploration, hypothesis verification, and idea generation support.

[0142] In recent years, attempts have been made to create an organization-wide data infrastructure and eliminate data silos. However, even if an organization-wide data infrastructure is created and data silos are eliminated, the problem remains that the number of users does not increase. The following are some examples of causes of this problem:

[0143] ·Prompt creation issues To effectively extract information from LLM, prompts must be written well. This means that users must have the skills to write prompts well. Since few users have these skills, the use of the data infrastructure across the organization is hindered.

[0144] -Problems with utilizing related information To effectively extract information from LLM, users must be able to effectively use related information. This means that users must have the skills to effectively search for and use related information. This includes not only information search skills, but also the skills to identify what related information should be used to obtain a desired answer. The lack of users with sufficient skills hinders the use of data infrastructure across an organization.

[0145] Deeper thinking In some cases, the desired answer can be obtained by repeatedly thinking from various perspectives. In this case, users need to have the skills to repeatedly think from various perspectives while using the organization-wide data infrastructure. However, because there are few users who have sufficient skills, the use of the organization-wide data infrastructure is not progressing well.

[0146] The processing device 10 of the second embodiment is configured to solve such problems. That is, the processing device 10 automatically creates prompts, utilizes related information, and delves deeper into thoughts. The user's first task is to input a question and an analytical perspective. The processing device 10 then automatically creates prompts, utilizes related information, and delves deeper into thoughts, and outputs the results.

[0147] By using such a processing device 10, it is expected that the above-mentioned problem, that is, the problem of slow progress in utilizing the data infrastructure of the entire organization, will be solved.

[0148] <<Third embodiment>> The processing device 10 of the third embodiment can output the result of thinking about a question in a characteristic tree structure in the result output process S26, as will be described in detail below.

[0149] As described in the second embodiment, the processing device 10 repeats a loop to obtain a result in which answers from various analytical perspectives are linked together in the order in which the loop is executed. Since multiple questions may be generated in one loop as described above, the linked results are not connected in a single linear line but branch into multiple lines along the way. For example, if m questions are generated in a certain loop, subsequent lines will branch into m lines starting from that loop.

[0150] The result output unit 13 can display multiple answers obtained through multiple loops in a tree structure showing such a chain or branching situation. That is, the result output unit 13 can generate and output a result screen that displays the relationship between multiple answers obtained by repeatedly executing a loop (processing) in a tree structure.

[0151] 16 shows an example of a result screen output by the result output unit 13. The result screen shown in the figure has a graph panel, a summary panel, and a details panel. Note that the result screen does not necessarily have to have at least one of the summary panel and the details panel.

[0152] The graph panel displays a tree structure that shows the relationships between multiple answers obtained by repeatedly executing a loop (process).

[0153] Node Q is a node corresponding to the question input in the input reception process S20. The result output unit 13 can display the question corresponding to node Q. However, if the question is relatively long, the amount of information displayed on the result screen at one time becomes large, making it difficult to view. Therefore, as shown in FIG. 16, the result output unit 13 can display a label that succinctly indicates the question ("Data Utilization" in the figure). Note that the result output unit 13 may not normally display the question or label, but may display the question or label in response to a user input that specifies node Q by hovering the mouse over or other operation. Furthermore, the result output unit 13 may display the question or label in the display area of ​​node Q. That is, the result output unit 13 may display the question or label instead of the letter "Q" in the figure.

[0154] Multiple nodes A (nodes marked with the letter A) are nodes corresponding to the answers output in the answer verification process S23. As described above, one loop may output one answer or multiple answers. Nodes for multiple answers generated in the same loop are displayed in the same hierarchical layer. In the example of FIG. 16, the result output unit 13 displays three nodes A1 to A3 connected to node Q. The three answers corresponding to nodes A1 to A3, respectively, are answers generated in one loop using the question of node Q. In the example of FIG. 16, the three nodes A1 to A3 corresponding to the three answers, respectively, are connected in parallel to node Q and displayed in the same hierarchical layer.

[0155] Furthermore, the result output unit 13 can display an answer corresponding to node A. However, if the answer is a relatively long sentence, the amount of information displayed on the result screen at one time becomes large, making it difficult to view. Therefore, as shown in FIG. 16, the result output unit 13 may display a label that succinctly indicates the answer (such as "Securing Human Resources" in the figure). Note that the result output unit 13 may not normally display the answer or label, but may display the answer or label in response to a user input that specifies node A by hovering the mouse over or other operation. Furthermore, the result output unit 13 may display the answer or label in the display area of ​​node A. That is, the result output unit 13 may display the answer or label instead of characters such as "A1" in the figure.

[0156] In the example of FIG. 16, the result output unit 13 displays the analytical perspectives (such as "issue," "cause," and "solution" in the diagram) used to obtain answers for each layer. Note that the result output unit 13 may not normally display the analytical perspectives, but may display them in response to user input such as specifying node A by hovering the mouse over or other operations. The result output unit 13 may also display the analytical perspectives in the display area of ​​node A. That is, the result output unit 13 may display the analytical perspectives in place of characters such as "A1" in the diagram.

[0157] In addition, in the example of FIG. 16, in response to a user input specifying one node A by hovering the mouse over or other operation, the result output unit 13 displays information related to the answer of that node A in the details panel.

[0158] The information related to the answer includes at least one of the question, the answer, the analytical perspective, the relevance of the question and the answer, a label that succinctly describes the answer, a summary of the answer, related information used to obtain the answer, a summary of the related information, and a hypothesis (including an overview and details).

[0159] Note that the amount of related information tends to be relatively large. Therefore, the result output unit 13 may display only the headings of the related information in a list in the details panel, as shown in Fig. 16. Then, the result output unit 13 may display the related information in response to a user input specifying the heading of one piece of related information by hovering the mouse over or by other operation.

[0160] 16, in response to a user input specifying one node A, the result output unit 13 displays a node R corresponding to at least one piece of related information used to obtain the answer for that node A. The result output unit 13 may also display related information or a summary of the related information in response to a user input specifying one node R by hovering the mouse over or by other operation.

[0161] 16, the result output unit 13 displays, in the summary panel, a summary of the answers of a node group that is a part of a tree structure and is made up of multiple nodes that hang down from the same node. The user may be able to specify the node group.

[0162] "Summary 1" shown in Figure 16 is a summary of the answers from the node group consisting of multiple nodes A hanging from node A1. "Summary 2" shown in Figure 16 is a summary of the answers from the node group consisting of multiple nodes A hanging from node A2. "Summary 3" shown in Figure 16 is a summary of the answers from the node group consisting of multiple nodes A hanging from node A3.

[0163] Although not explicitly shown in the example of Fig. 16, the result output unit 13 may change the display mode (color, size, etc.) of the node based on a predetermined score. The score may be, for example, a score proportional to the relevance of the answer to the question or the number of edges connected to each node. The search unit 12 or the result output unit 13 can calculate a score for each node (each answer) in accordance with predetermined score calculation rules.

[0164] 16, the result output unit 13 may change the display mode (color, line type, line thickness, etc.) of edges according to a predetermined rule. For example, the result output unit 13 may display edges in the same layer in the same display mode, and may display edges in different modes for each layer.

[0165] 16, the result output unit 13 may group multiple nodes and distinguishably display the groups. For example, the result output unit 13 may display a frame or the like that encompasses nodes that belong to the same group. Alternatively, the result output unit 13 may display nodes that belong to the same group in the same display mode (color, size, etc.) and display the nodes in different modes for each group.

[0166] There are various grouping methods, but for example, similar questions or answers may be grouped together based on the similarity of the questions or answers. For example, the similarity of questions or answers can be calculated using the technique for calculating the similarity between sentences described above. The search unit 12 or the result output unit 13 can perform the grouping process.

[0167] Next, the process of generating a "label" and a "summary" will be described.

[0168] The result output unit 13 can generate at least one of a label and a summary using the LLM, and display the generated at least one of the label and the summary on the result screen described above.

[0169] The result output unit 13 may generate a prompt using, for example, a template prepared in advance. For example, the template may be a sentence with blank spaces where a label or a target for creating a summary (such as an answer, question, or related information) is to be inserted. The result output unit 13 may then generate a prompt by inserting a target for creating a label or summary (such as an answer, question, or related information) into a predetermined position in the template. Alternatively, the result output unit 13 may generate a prompt based on a generative model instead of generating a prompt based on a template. The generative model here can be realized using any widely known technology.

[0170] An example of label generation will now be described with reference to FIG. 17. Note that the example of FIG. 17 has a two-step configuration in which multiple keywords are identified (for example, using technologies such as LLM or MultipartieRank), and then labels are generated based on the identified multiple keywords. This two-step configuration allows for highly accurate labels to be generated. However, it is also possible to generate labels in one step by generating a prompt that directly asks for "a label that succinctly expresses the target sentence." Furthermore, keywords identified using technologies such as LLM or MultipartieRank may be used as labels. Even in this way, labels can be generated with a certain degree of accuracy.

[0171] In the example of Fig. 17, the result output unit 13 first generates a prompt requesting multiple keywords that succinctly represent the target (answer) for which a label is to be created, as shown in the upper part of the prompt example in Fig. 17. Then, the result output unit 13 inputs the prompt into the LLM to acquire multiple keywords.

[0172] After obtaining the multiple keywords, the result output unit 13 generates a prompt requesting a title that comprehensively represents the multiple obtained keywords, as shown at the bottom of the prompt example in Figure 17. The result output unit 13 then inputs the prompt into the LLM to obtain the title. This title becomes the label.

[0173] As shown in the prompt example in Figure 17, by adding a character limit to the answer output from the LLM, a label of the desired length can be obtained.

[0174] Next, an example of summary generation will be described with reference to Fig. 18. In the example of Fig. 18, the result output unit 13 generates a prompt that prompts the user to answer a predetermined question within a predetermined number of characters based on the target (answer) for which a summary is to be created. In this way, by imposing a character limit on the answer output from the LLM, a summary of the desired length can be obtained.

[0175] Other configurations of the processing apparatus 10 of the third embodiment are similar to those of the processing apparatus 10 of the first and second embodiments.

[0176] The processing device 10 of the third embodiment achieves the same effects as the processing device 10 of the first and second embodiments. Furthermore, the processing device 10 can display the results obtained by the characteristic processing described in the second embodiment in a characteristic tree structure.

[0177] As explained in the second embodiment, the processing device 10 generates questions and searches for related information across a wide range of topics in a chain reaction, simulating the process of deep human exploration and automating analysis from a desired perspective. Furthermore, by diverging thoughts and using various related information, it is possible to discover unexpected relationships and new policy ideas that humans would not have noticed.

[0178] However, when processing is performed to simulate such complex thinking, the results obtained may also have complex relationships. By displaying the results of such complex relationships in a tree structure, users can systematically grasp the results. As a result, users can easily understand the results.

[0179] <<Modifications>> <Variation 1> The search unit 12 does not need to have the information search unit 121. In this case, the related information search process S21 is not executed. The processing device 10 then executes the above-described process without using related information. Even if the above-described process is repeated without using related information, it is possible to obtain an answer to the question with a certain degree of accuracy.

[0180] <Variation 2> The search unit 12 does not need to have the answer verification unit 123. In this case, the answer verification process S23 is not executed. Then, the processing device 10 performs subsequent processes (such as generating questions, loop processing based on new questions, and outputting results) based on all primary answers output in the answer generation process S22. In this modified example, it is impossible to avoid the inconvenience of subsequent processes being performed based on primary answers that have a low degree of relevance to the question, but it is possible to obtain answers to questions with a certain degree of accuracy.

[0181] <Variation 3> In the third embodiment, the processing device 10 outputs a result screen with a tree structure. That is, the processing device 10 generates an answer to a question through the process described in the second embodiment, and then outputs a result screen with a tree structure.

[0182] In this modification, an information providing device that is physically and / or logically separate from the processing device 10 that generated the answer to the question using the process described in the second embodiment acquires the result data of the answer to the question from the processing device 10. The information providing device then outputs a tree-structured result screen using the process described in the third embodiment. An example of the hardware configuration of the information providing device is the same as that of the processing device 10, and is shown in FIG. 4, for example. In this modification as well, the same effects as those of the above embodiment are achieved.

[0183] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0184] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of the content.

[0185] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. 1. An input receiving means for receiving input of a question and an analysis point of view; a search means for repeatedly executing a process of inputting a question and a prompt indicating an analytical viewpoint into a large-scale language model to generate an answer, using the input question and a new question generated based on the answer; A processing device having: 2. The searching means 1. The processing device according to claim 1, further comprising a question generation means for generating a hypothesis based on the answer and generating the new question based on the generated hypothesis. 3. The question generation means 3. A processing device according to claim 2, which generates hypotheses by inputting prompts into a large-scale language model to obtain results, thereby generating hypotheses regarding a predetermined analytical perspective based on the answers. 4. The question generation means 3. The processing device of claim 2, wherein the generated hypothesis is included in the prompt indicating the new question. 5. The searching means an answer generation means for generating a prompt indicating a question and an analytical viewpoint, and inputting the generated prompt into a large-scale language model to generate a primary answer; an answer verification means for determining whether the relevance between the primary answer and the question satisfies a passing condition, and outputting the primary answer that satisfies the passing condition as an answer to the question; 5. The processing device according to any one of 1 to 4, comprising: 6. The answer verification means 6. The processing device according to 5, wherein the primary answer that does not satisfy the pass condition is corrected, and the corrected primary answer that satisfies the pass condition is output as the answer to the question. 7. The answer verification means A processing device as described in 6, which corrects the primary answer by calculating the difference between the primary answer and reference information, and inputting a prompt to correct the primary answer based on the difference into a large-scale language model to obtain a result. 8. The answer verification means 8. The processing device according to any one of 5 to 7, wherein the primary response that does not satisfy the pass condition is discarded. 9. The searching means an information retrieval means for retrieving relevant information related to the query; an answer generation means for generating a prompt using the question, the analysis viewpoint, and the related information, and inputting the generated prompt into a large-scale language model to generate an answer; 9. A processing device according to any one of 1 to 8, comprising: 10. The searching means 10. The processing device according to any one of 1 to 9, wherein the processing is repeatedly executed with a different analysis viewpoint used for each iteration of the processing. 11. A processing device described in any one of 1 to 10, further comprising a result output means for generating and outputting a result screen that displays the relationships between the multiple answers obtained by repeatedly executing the process in a tree structure. 12. The result output means 12. The processing device according to claim 11, wherein information related to the answers is displayed in the tree structure by being linked to a node corresponding to each of the multiple answers. 13. The result output means 13. The processing device according to 12, which generates at least one of a label and a summary using a large-scale language model and displays at least one of the generated label and summary. 14. The result output means A processing device described in any one of 11 to 13, which generates and displays a summary of the answers of a group of nodes in the tree structure, the group of nodes being composed of multiple nodes hanging from the same node, using a large-scale language model. 15. One or more computers Accepts input of questions and analytical viewpoints, A processing method in which a process of inputting a question and a prompt indicating an analytical viewpoint into a large-scale language model to generate an answer is repeatedly performed using the input question and a new question generated based on the answer. 16. Computer, an input receiving means for receiving input of a question and an analysis point of view; a search means for repeatedly executing a process of inputting a question and a prompt indicating an analytical viewpoint into a large-scale language model to generate an answer, using the input question and a new question generated based on the answer; A program that functions as a

[0186] Some or all of Supplements 2 to 14 that are dependent on the processing device of Supplement 1 described above may also be dependent on the processing method of Supplement 15 and the program of Supplement 16 in the same dependent relationship as Supplement 1 and Supplements 2 to 14. Furthermore, within the scope of each of the above-mentioned embodiments, some or all of the configurations described as Supplements can be realized in various hardware, software, various recording means for recording software, or systems. [Explanation of symbols]

[0187] 10 Processing equipment 11 Input reception section 12 Exploration Department 121 Information Search Department 122 Answer generation part 123 Answer Verification Department 124 Question generation part 1A processor 2A Memory 3A input / output I / F 4A peripheral circuit 5A Bus

Claims

1. an input receiving means for receiving input of a question and an analysis point of view; a search means for repeatedly executing a process of inputting a question and a prompt indicating an analytical viewpoint into a large-scale language model to generate an answer, using the input question and a new question generated based on the answer; A processing device having:

2. The searching means 2. The processing device according to claim 1, further comprising a question generating means for generating a hypothesis based on the answer and generating the new question based on the generated hypothesis.

3. The question generation means The processing device of claim 2 , wherein the hypothesis is generated by inputting prompts into a large-scale language model to obtain results, the prompts generating hypotheses about a predetermined analysis perspective based on the answers.

4. The question generation means The processing device of claim 2 , wherein the prompt indicating the new question includes the generated hypothesis.

5. The searching means an answer generation means for generating a prompt indicating a question and an analysis point, and inputting the generated prompt into a large-scale language model to generate a primary answer; an answer verification means for determining whether the relevance of the primary answer to the question satisfies a passing condition, and outputting the primary answer that satisfies the passing condition as an answer to the question; The processing device of claim 1 , comprising:

6. The answer verification means The processing device according to claim 5 , wherein the primary answer that does not satisfy the pass condition is corrected, and the corrected primary answer that satisfies the pass condition is output as the answer to the question.

7. The answer verification means The processing device according to claim 6, wherein the primary answer is corrected by calculating a difference between the primary answer and reference information, and inputting a prompt to correct the primary answer based on the difference into a large-scale language model to obtain a result.

8. The answer verification means The processing device according to claim 5 , wherein the primary response that does not satisfy the pass condition is discarded.

9. The searching means an information retrieval means for retrieving relevant information related to the query; an answer generation means for generating a prompt using the question, the analysis viewpoint, and the related information, and inputting the generated prompt into a large-scale language model to generate an answer; The processing device of claim 1 , comprising:

10. The searching means The processing device according to claim 1 , wherein the processing is repeatedly executed with a different analysis viewpoint used for each iteration of the processing.

11. 2. The processing device according to claim 1, further comprising a result output means for generating and outputting a result screen that displays, in a tree structure, the relationships between the plurality of answers obtained by repeatedly executing the process.

12. The result output means The processing device according to claim 11 , wherein information relating to the answers is displayed in the tree structure by being linked to a node corresponding to each of the plurality of answers.

13. The result output means The processing device according to claim 12, wherein at least one of a label and a summary is generated using a large-scale language model, and the generated at least one of the label and the summary is displayed.

14. The result output means The processing device according to claim 11, wherein a summary of the answers of a group of nodes in the tree structure, the group of nodes being composed of multiple nodes hanging from the same node, is generated and displayed using a large-scale language model.

15. One or more computers Accepts input of questions and analytical viewpoints, A processing method in which a process of inputting a question and a prompt indicating an analytical viewpoint into a large-scale language model to generate an answer is repeatedly performed using the input question and a new question generated based on the answer.

16. Computer, an input receiving means for receiving input of a question and an analysis point of view; a search means for repeatedly executing a process of inputting a question and a prompt indicating an analytical viewpoint into a large-scale language model to generate an answer, using the input question and a new question generated based on the answer; A program that functions as a

Images