Generating AI information consistency verification device, generating AI information consistency verification method, and computer program

Abstract

This invention provides a generator AI information consistency verification device that can verify the consistency between chunk information corresponding to input information and generated response information, and can also verify the conditions under which hallucination risks occur based on statistical evidence. In turn, it enhances the reliability of the generating AI, enables evidence-based explanations, and provides a device that can continuously avoid hallucination. [Solution] The generated AI information consistency verification device 1 includes an input control unit 14 that acquires chunk information based on input information and causes the generated AI to generate answer information based on the acquired chunk information and input information; a calculation unit 15 that calculates a basis score for the chunk information for each logical unit included in the generated answer information; a verification unit 16 that verifies the consistency of the answer information with the chunk information based on the calculated basis score; and an output control unit 18 that modifies and outputs the answer information based on the consistency verification result of the answer information.

Description

Technical Field

[0001] The present invention relates to a technique for avoiding hallucinations in so-called generative AI (Artificial Intelligence).

Background Art

[0002] In recent years, generative AI utilizing large language models (LLMs) has been rapidly spreading in a wide range of fields such as summarization and search assistance. However, generative AI has a serious problem of hallucinations, which are plausible lies. As a countermeasure, for example, a prompt for reexamination is repeatedly input to the dialog-based AI until it is determined that the answer does not contain hallucinations, and when predetermined conditions are satisfied, the answer generated by the dialog-based AI is output, and a system for suppressing errors by regeneration control has been proposed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional system, each time it is determined whether the generated answer is correct or incorrect, and if it is inappropriate, only on-the-spot countermeasures such as regeneration or fine-tuning of the prompt are taken. With such symptomatic approaches, it is difficult to identify the root cause of why hallucinations occur. Also, there is a lack of a specific quality control perspective on when the risk of hallucinations increases under what input conditions and settings.

[0005] Given this background, this type of technology requires a more sophisticated mechanism than simply determining whether something is true or false. Specifically, there is a need for technology that can objectively verify whether the generated response is truly based on the source information, and then statistically analyze the results, thereby visualizing the trend of hallucination.

[0006] The present invention has been made in view of the above points, and aims to provide a generated AI information consistency verification device, a generated AI information consistency verification method, and a computer program that can verify the consistency between chunk information corresponding to input information and generated response information, can verify from a statistical basis the conditions under which the risk of hallucination occurs, and thereby enhance the reliability of generated AI, enable evidence-based explanations, and continuously avoid hallucination. [Means for solving the problem]

[0007] One aspect of the present invention is, A generation AI information consistency verification device that performs information consistency verification when generating response information in response to user input information and outputting it, The aforementioned input information As information units that are divided into meaningful fragments A unit for obtaining chunk information, An input control unit that causes the generating AI to generate response information based on the acquired chunk information and the input information, The generated response information Extract a logical unit of semantic units from the chunk information, and for each extracted logical unit, apply the following to the chunk information. A calculation unit that calculates the basis score, Calculated Root Based on the score, verify the consistency of the aforementioned response information with the aforementioned chunk information. The consistency verification results are then generated to indicate the logical units whose justification score is below a predetermined threshold as non-justification statements. Verification Department, Based on the results of the consistency verification of the aforementioned response information, an output control unit determines whether to correct or output the response information and then outputs it. We provide a generated AI information consistency verification device equipped with the necessary components.

[0008] In one aspect of the present invention, chunk information obtained from an external source based on input information is input to the generating AI, a similarity or justification score is calculated for each logical unit of the generated response information, the consistency between the response information and the chunk information is verified based on the calculated similarity or justification score, and the response information is modified or output is determined based on the consistency verification result before being output. According to this aspect of the present invention, the consistency between chunk information corresponding to input information and generated response information can be verified using similarity and evidence scores. The conditions under which hallucination risk occurs can also be verified from statistical evidence based on similarity and evidence scores, thereby increasing the reliability of the generating AI, enabling evidence-based explanations, and sustainably avoiding hallucination.

[0009] While one aspect of the present invention is the category of apparatus, methods and computer programs can also achieve similar functions and effects specific to their respective categories. [Effects of the Invention]

[0010] According to the present invention, it is possible to verify the consistency between input information and generated response information, and to verify, from a statistical standpoint, the conditions under which hallucination risk occurs. This, in turn, enhances the reliability of the generating AI, enables evidence-based explanations, and allows for the sustained avoidance of hallucination. [Brief explanation of the drawing]

[0011] [Figure 1] This diagram shows an overview of the AI ​​generation information consistency verification device. [Figure 2] This is a functional block diagram of the AI ​​generation information consistency verification device. [Figure 3] This diagram illustrates an example of the information flow via the AI ​​information consistency verification device that generates the information. [Figure 4] This diagram illustrates another example of information flow via a generated AI information consistency verification device. [Figure 5]This is a flowchart showing the input / output control process by the generated AI information consistency verification device. [Figure 6] This is a flowchart showing the statistical evaluation process performed by the generated AI information consistency verification device. [Modes for carrying out the invention]

[0012] Hereinafter, an embodiment for carrying out the present invention (hereinafter referred to as "this embodiment") will be described in detail with reference to the attached drawings. In the drawings, the same or similar elements are denoted by the same numbers or reference numerals throughout the description of this embodiment.

[0013] [Overview of the Generated AI Information Integrity Verification Device] As shown in Figure 1, the Generative AI Information Consistency Verification Device 1 of this embodiment is connected to the Generative AI Server 2, which has the functionality of Generative AI with a Large Language Model (LLM) as its core technology, as an orchestrator that controls input and output information. The Generative AI Server 2 has the functionality of Generative AI, which probabilistically selects the next word that should follow from a vast amount of pre-learned knowledge based on information (input information) such as questions input by the user, and generates information (answer information) that shows a plausible and natural answer. The Generative AI Information Consistency Verification Device 1 has the purpose and role of avoiding so-called Generative AI hallucination, so as not to output answer information that contains plausible lies while utilizing Generative AI. The Generative AI Information Consistency Verification Device 1 and the Generative AI Server 2 are connected to an external server 3 so as to be able to communicate with it in order to refer to external information related to the input information, etc., as needed.

[0014] The generated AI information integration verification device 1 is communicably connected to the generated AI server 2 and the external server 3 via a wide area network (WAN) or a local area network (LAN), and is realized by a computer capable of input operations and display outputs of various information. The generated AI server 2 is realized by a computer suitable for high-speed inference calculations and the exchange of huge amounts of knowledge data. Each computer includes, as main hardware, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a read-only memory (ROM), a random access memory (RAM), an internal storage device, an external storage device, an input / output interface, a network card, etc., and, as peripheral devices as required, a keyboard, a mouse, a microphone, a display (including a touch panel), a printer, a speaker, etc.

[0015] Note that the generated AI information integration verification device 1 may be one that complementarily performs the functions of the generated AI and may be integrally incorporated as an internal module of the generated AI server 2. Further, the generated AI information integration verification device 1 may be one in which the input / output part related to the user's input operation and display output to the user is separated, and one or more input / output terminals are connected.

[0016] [Functional configurations such as the generated AI information integration verification device] As shown in FIG. 2, the generated AI information integration verification device 1 includes, as components that function when a computer executes a predetermined program, a communication unit 10, an input unit 11, an output unit 12, an acquisition unit 13, an input control unit 14, a calculation unit 15, a verification unit 16, a statistical evaluation unit 17, an output control unit 18, and a storage unit 19.

[0017] The communication unit 10 is implemented, for example, by a network card, and transmits and receives various information with the generation AI server 2. Specifically, the communication unit 10 sends a prompt to the generation AI server 2 that contains the input information entered by the user as concrete content. Subsequently, it receives response information that has been generated by the generation AI based on the prompt (input information) and sent from the generation AI server 2.

[0018] The input unit 11 is implemented, for example, by a keyboard or mouse, and accepts input information such as text from the user.

[0019] The output unit 12 is implemented, for example, by a display or printer, and outputs various types of information, including response information, to the user.

[0020] The acquisition unit 13, for example, is implemented by a CPU and searches for chunk information related to the input information on an external server 3, and acquires the necessary chunk information from the external server 3 via the communication unit 10. Chunk information is a unit of information obtained by dividing knowledge information into small meaningful fragments in order to make it easier to perform processing such as referencing on a vast amount of knowledge information, and is used as a reference source when the generating AI generates answer information. Chunk information includes, for example, the text itself extracted from the main text, as well as supplementary information such as "which file and which page it is," "when it was updated," and "the connection to the preceding and succeeding chunk information," and metadata such as vector data in which the text has been converted into meaningful numerical values ​​so that it can be easily calculated by a search engine.

[0021] The input control unit 14, implemented by, for example, a CPU and an input interface, generates a prompt for the generating AI to generate response information based on the chunk information acquired by the acquisition unit 13 and the input information from the input unit 11, and transmits this prompt to the generating AI server 2 via the communication unit 10. The input control unit 14 also adjusts the input information based on the results of the response information consistency verification by the verification unit 16 (described later), regenerates the prompt, and transmits it to the generating AI server 2 via the communication unit 10. Alternatively, instead of adjusting the input information, the input control unit 14 may regenerate the prompt after adjusting the generation characteristics of the generating AI itself by prompt engineering, which involves devising instructions for the generating AI, or by adjusting parameters to change the behavior of the generating AI.

[0022] The calculation unit 15, implemented by, for example, a CPU, extracts logical units that constitute each sentence or semantic unit from the response information generated by the AI, and calculates a justification score for each logical unit decomposed by the extraction, relative to the source chunk information. The justification score expresses the presence or absence of justification for each logical unit as a numerical percentage, indicating whether the response information was correctly generated based on the chunk information. For example, a high justification score close to 100% means that almost all logical units of the response information are clearly stated in the chunk information. On the other hand, a low justification score close to 0% means that the response information contains many logical units that do not match the chunk information, or contains many logical units whose content has been distorted and misinterpreted. The justification score for each logical unit is aggregated and recorded and stored in the storage unit 19 for use as statistical information in the statistical evaluation unit 17 described later. Alternatively, the calculation unit 15 may calculate a similarity score, which expresses the degree to which each logical unit is similar to the content of the chunk information as a numerical percentage, instead of a justification score. The following explanation will describe examples of applying evidence scores throughout, but the same process can be applied by substituting similarity scores, so the explanation of examples using similarity scores will be omitted.

[0023] The verification unit 16 verifies the consistency of the response information with the chunk information based on the basis score for each logical unit calculated by the calculation unit 15, which is implemented, for example, by the CPU. For example, response information consisting of logical units with mostly high basis scores is full grounding information that perfectly matches the source chunk information without contradiction, and a consistency verification result to that effect is created. On the other hand, response information where the basis score for the logical units as a whole is between high and low scores is partial grounding information in which some logical units are based on chunk information, but the remaining logical units contain general knowledge information or logical reasoning that is not present in the chunk information, and a consistency verification result to that effect is created. On the other hand, response information consisting of logical units with mostly low basis scores is hallucination information in which the content contradicts the description of the chunk information, or where content that is completely unrelated is generated, and a consistency verification result to that effect is created.

[0024] As described above, the verification unit 16 distinguishes between full grounding response information, partial grounding response information, and hallucination response information as a whole logical unit and creates consistency verification results. However, it can also identify logical units whose justification score is below a predetermined threshold, find those logical units individually as non-justification statements corresponding to hallucination, and reflect them in the consistency verification results. The predetermined thresholds that serve as the criteria for discriminating response information based on the justification score of the entire logical unit, and the criteria for discriminating consistency information based on the justification score of an individual logical unit, are set as risk indicators by the statistical evaluation unit 17, which will be described later.

[0025] The statistical evaluation unit 17 performs statistical processing on the rationale scores for each logical unit, which are implemented, for example, by the CPU and recorded and stored as statistical information in the memory unit 19. Based on the results of the statistical processing, it performs a statistical evaluation of the input information, chunk information, and generation characteristics of the generated AI, and identifies items that receive a relatively low evaluation. For example, if the items that receive a low evaluation are biased towards a particular field, the input information and chunk information for that particular field are labeled, and the generation characteristics of the generated AI for that field are labeled as receiving a low evaluation. The statistical evaluation unit 17 analyzes the distribution of rationale scores for each logical unit obtained through statistical processing, or the distribution of statistical evaluations, and sets risk indicators that serve as criteria for consistency verification based on the analysis results. Specifically, the statistical evaluation unit 17 calculates the mean, median, variance, etc., of the rationale scores, and if the rationale scores are polarized, it obtains an instability as an analysis result, such as the fact that very accurate answer information can be obtained under certain conditions, but answer information that completely ignores the rationale under other conditions. In other words, this statistical evaluation unit 17 process allows for the identification of hallucination trends, the identification of low-rated items that are likely to occur, and the setting of corresponding risk indicators. For example, for input information in specific fields where hallucination is likely to occur, the risk indicators are set to a high level to establish stricter criteria, while for input information in fields where hallucination is less likely to occur, the risk indicators are set to a low level.

[0026] The output control unit 18 is implemented, for example, by a CPU and an output interface. Based on the consistency verification results of the answer information by the verification unit 16, it modifies the answer information or decides whether or not to output it, and then outputs it to the output unit 12. For example, if the logical unit of non-foundational sentences that does not match the chunk information is included in part, and the consistency verification result is created as partial grounding answer information, the logical unit of non-foundational sentences is modified to match the chunk information by rewriting or other means, and then output to the output unit 12. If the consistency verification result is created as hallucination answer information for the entire logical unit, the answer information is not output as is. Instead, the input control unit 14 is made to adjust the input information or adjust the generation characteristics of the generation AI, so that the input information is recursively fed to the generation AI until hallucination is eliminated. If the answer information is not output as is, a warning message may be displayed to that effect. Furthermore, the output control unit 18 outputs the results of statistical processing and analysis by the statistical evaluation unit 17, as well as the low-evaluation items identified therefrom, as visualization data to the output unit 12, or records the visualization data in the storage unit 19. By looking at the recorded visualization data, the hallucination occurrence trend and low-evaluation items can be easily confirmed.

[0027] The memory unit 19 is primarily implemented by an external storage device and stores programs and information necessary for basic arithmetic control information processing, as well as programs and various related information for causing the computer to execute the input / output control processing and statistical evaluation processing described later. The various information includes input information, chunk information, statistical information, evidence scores, risk indicators, visualization data, etc. The computer of the generated AI information consistency verification device 1 executes each step of each process by reading the programs related to input / output control processing and statistical evaluation processing from the memory unit 19.

[0028] The functions of each component of the generated AI information consistency verification device 1 described above are realized by a computer executing a predetermined program stored in the memory unit 19. However, instead, these functions may be realized by, for example, using a SaaS (Software as a Service) mechanism to download the necessary programs from an external application server or the like when needed.

[0029] The generation AI server 2 includes a communication unit 20, an input receiving unit 21, an answer generation unit 22, a generation output unit 23, and a storage unit 24 as components that enable the device to function when a computer executes a predetermined program.

[0030] The communication unit 20 transmits and receives various types of information with the generated AI information consistency verification device 1 and the external server 3. Specifically, the communication unit 20 receives input y information, prompts, and chunk information transmitted from the generated AI information consistency verification device 1, and transmits the response information generated by the response generation unit 22 to the generated AI information consistency verification device 1 accordingly. The communication unit 20 also receives file data and the like from the external server 3 and inputs it into the response generation unit 22 so that it can refer to the received data.

[0031] The input receiving unit 21 receives input information, prompts, and chunk information received from the generated AI information consistency verification device 1 via the communication unit 20, and inputs the received input information, prompts, and chunk information to the response generation unit 22.

[0032] The answer generation unit 22 performs information processing related to basic arithmetic control, and as a function of the generation AI, for example, it generates answer information corresponding to the input information question based on the input information and prompts, while referring to chunk information along with the learned data. The generated answer information is treated as text data in a natural expression and transmitted to the generation output unit 23.

[0033] The generation output unit 23 outputs the response information generated by the response generation unit 22 to be transmitted to the generated AI information consistency verification device 1 via the communication unit 20.

[0034] The memory unit 24 stores programs and information necessary for basic arithmetic control information processing, as well as various information related to neural network programs, trained data, and language knowledge databases for executing processing related to the generation AI. The computer of the generation AI server 2 executes processing by the generation AI by reading the programs related to the generation AI from the memory unit 24.

[0035] [Information flow via the generated AI information consistency verification device] The flow of information via the generated AI information consistency verification device 1 described above is as illustrated in Figures 3 and 4.

[0036] As shown in Figure 3 as an example, when the generated AI information consistency verification device 1 receives input information such as "How many days of summer vacation can I take?", it searches for vacation-related matters on an external server 3, etc., and extracts chunk information (chunks A to C) from the entire text of the vacation regulations obtained as a search result, "Regarding the vacation system...".

[0037] Next, the Generating AI Information Consistency Verification Device 1 inputs input information and chunk information along with prompts to the Generating AI. Based on the prompts and input information, the Generating AI refers to the chunk information and generates response information. The generated response information, for example, "According to the employment regulations, summer vacation can be taken for 3 days between July and September," is sent to the Generating AI Information Consistency Verification Device 1.

[0038] Next, the generated AI information consistency verification device 1 decomposes the received response information into logical units, calculates the rationale score for each logical unit, and verifies the consistency between the rationale score and the chunk information for each logical unit. In this example, there are no logical units that correspond to non-rational statements because their rationale score is below a predetermined threshold (e.g., 20%), and the average rationale score for all logical units is 80% or higher, so the response information is certified as fully grounded as a result of the consistency verification. Incidentally, if the average rationale score is, for example, between 40% and 80%, the response information is certified as partially grounded, and if the average rationale score is less than 40%, the response information is certified as hallucination.

[0039] Next, the Generating AI Information Consistency Verification Device 1, having determined that there are no logical units of non-supporting sentences and that the response information is fully grounded, displays the response information generated by the Generating AI as is. At this time, the average support score obtained as a result of consistency verification is also displayed along with the response information.

[0040] On the other hand, as shown in Figure 4 for another example, when the generated AI information consistency verification device 1 receives input information such as "By when is the settlement required after receiving the receipt?", it similarly searches for expense-related information on an external server 3, etc., and extracts chunk information such as "Article 12: Expense settlements must be applied for within 30 days from the date of issuance of the receipt" from the entire text of the expense regulations obtained as a search result.

[0041] Next, the Generating AI Information Consistency Verification Device 1 receives the generated response information from the Generating AI by inputting input information and chunk information along with prompts. The generated response information may include, for example, "According to the expense regulations, settlement is required within one week of receipt issuance."

[0042] Next, the generated AI information consistency verification device 1 decomposes the received response information into logical units, calculates the rationale score for each logical unit, and verifies the consistency between the rationale score and the chunk information for each logical unit. In this example, there is a logical unit ("within one week") whose rationale score is below a predetermined threshold (e.g., 20%) and which corresponds to a non-rational statement. The average rationale score for all logical units is 71.3%, which is within the range of 40-80%, so the response information is certified as partially grounded as a result of the consistency verification.

[0043] In this case, the generated AI information consistency verification device 1 modifies the logical units corresponding to non-supporting statements to match the chunk information as rewritten parts, and then outputs and displays the response information. As a result, for example, the response information with the modified content, such as "According to the expense regulations, settlement is required within 30 days of receipt issuance," is displayed, and the average support score after the modification is also displayed along with the response information.

[0044] Although not specifically illustrated, if the response information received from the generating AI is deemed hallucination based on the average of the evidence scores, the generating AI information consistency verification device 1 modifies and adjusts the original input information to make it easier to match with the chunk information. The adjusted input information and chunk information are then input back into the generating AI to generate new response information. For example, if the original input information is "By when is the settlement required after receiving the receipt?", the modified input information would be something like "Within how many days after the receipt is issued is the application required?". Subsequently, the generating AI generates response information through a similar process, and response information without hallucination is displayed.

[0045] [Input / Output Control Processing] As shown in Figure 5, the generated AI information consistency verification device 1 performs input / output control processing in the following steps.

[0046] First, the input unit 11 receives input information from the user (S11).

[0047] Next, the acquisition unit 13 searches for chunk information related to the input information and obtains fragmented chunk information from the search results (S12).

[0048] Next, the input control unit 14 generates a prompt corresponding to the input information and sends the input information and the acquired chunk information along with the prompt to the generation AI server 2, thereby inputting it to the generation AI (S13).

[0049] Next, the communication unit 10 receives the response information generated by the generation AI from the generation AI server 2 (S14).

[0050] Next, the calculation unit 15 extracts logical units from the response information and decomposes the response information into logical units (S15).

[0051] Next, the calculation unit 15 calculates the justification score for each logical unit (S16). By calculating the justification score for all logical units, the calculation unit 15 also calculates the average justification score for the entire response information. The calculated justification scores for each logical unit and the average justification score are aggregated and recorded for the statistical evaluation process described later.

[0052] Next, the verification unit 16 determines whether the average of the evidence scores for all response information is less than the first threshold (e.g., 40%) (S17). If the average of the evidence scores is not less than the first threshold (S17: NO), the verification unit 16 proceeds to S18. On the other hand, if the average of the evidence scores is less than the first threshold (S17: YES), the verification unit 16 proceeds to S20.

[0053] Next, the verification unit 16 determines whether the average of the evidence scores for the entire response information is less than the second threshold (e.g., 80%) (S18). If the average of the evidence scores is not less than the second threshold (S18: NO), the verification unit 16 proceeds to S19. On the other hand, if the average of the evidence scores is less than the second threshold (S18: YES), the verification unit 16 proceeds to S22.

[0054] Next, the verification unit 16 creates consistency verification results showing the basis score and the average basis score for each logical unit, and the output control unit 18 outputs the response information generated by the generation AI directly to the output unit 12 (S19). In such cases, the fully grounded response information is displayed.

[0055] If S17:YES, the verification unit 16 creates consistency verification results showing the rationale scores of each logical unit, including those with relatively low rationale scores, and the average of the rationale scores that fall below the first threshold (S20). Then, the input control unit 14 adjusts the content of the original input information or adjusts the generation characteristics of the generated AI (S21). After that, the input control unit 14 returns to S13 and the series of processes up to S19 are performed. In such cases, the hallucination response information corresponding to the original input information is not displayed, and the response information regenerated by recursive adjustment is displayed as having no hallucination.

[0056] If S18:YES, the verification unit 16 further determines whether the rationale score of a specific logical unit is less than the third threshold (e.g., 20%) (S22). If the rationale score of a specific unit is not less than the third threshold (S22:NO), the verification unit 16 proceeds to S19. In such cases, the response information generated by the generating AI is displayed as partial grounding without any adjustments to the input information or modifications to the response information.

[0057] On the other hand, in S22, if a specific justification score is below the third threshold (S22: YES), the verification unit 16 identifies the logical unit corresponding to that specific justification score as a non-justification sentence (S23). The verification unit 16 then creates a consistency verification result showing the justification score and the average justification score for the non-justification sentence logical unit and other logical units, and the output control unit 18 modifies the response information so that the non-justification sentence logical unit is consistent with the chunk information before outputting it (S24). In such cases, the response information before modification was partially hallucination-based, but after modification, response information with no hallucination is displayed.

[0058] This type of input / output control processing allows for recursive adjustment of input information and modification of response information as needed, based on the consistency verification results of the response information against the chunk information. This makes it possible to display fully grounded response information with as little hallucination as possible.

[0059] [Statistical evaluation processing] As shown in Figure 6, the generated AI information consistency verification device 1 performs statistical evaluation processing in the following steps.

[0060] First, the calculation unit 15 aggregates the basis score and the average basis score for each logical unit (S31).

[0061] Next, the statistical evaluation unit 17 performs statistical processing on the aggregated evidence scores, etc. (S32). As a result of this statistical processing, statistical data such as the mean, median, and variance of the evidence scores are obtained.

[0062] Next, the statistical evaluation unit 17 performs a statistical evaluation of the statistical processing results and identifies, for example, low evaluations in which the average evidence score is generally low (S33).

[0063] Next, the statistical evaluation unit 17 converts the statistical data and low-evaluation items into graphs or other visualization data and records them in the storage unit 19 (S34). With such visualization data, it is possible to easily identify specific fields, etc., related to low-evaluation items that are likely to cause hallucination by referring to them afterward.

[0064] Next, the statistical evaluation unit 17 analyzes the distribution of statistical data and the distribution of statistical evaluations (S35).

[0065] Next, the statistical evaluation unit 17 sets appropriate risk indicators according to the analysis results of each distribution (S36). The risk indicators set here correspond to the first to third thresholds mentioned earlier. With such risk indicators, the first to third thresholds can be variably set according to the statistical evaluation of the evidence score. For example, even in specific areas of response information where hallucination has occurred in the past, a higher threshold can be set as a stricter risk indicator, preventing the repeated display of hallucination response information.

[0066] According to this statistical evaluation process, the threshold values ​​for parameters related to input / output control processing are variably set to be appropriate based on the input information, chunk information, response information, and statistical data obtained as actual results such as the basis score. This eliminates issues such as the repeated display of hallucination response information in a particular field.

[0067] [Effects of the Generated AI Information Integrity Verification Device] According to this Generative AI Information Consistency Verification Device 1, the consistency between chunk information corresponding to user input information and the response information generated by the Generative AI can be verified using a rationale score for each logical unit. Furthermore, the conditions under which hallucination risk occurs can be verified from statistical evidence based on the rationale score, thereby increasing the reliability of the Generative AI, enabling explanations based on the rationale score, and allowing for the continuous avoidance of hallucination.

[0068] Furthermore, according to the generated AI information consistency verification device 1, logical units are extracted from the response information, a justification score for the chunk information is calculated for each extracted logical unit, and logical units whose justification score is less than the third threshold are identified as non-justification sentences and reflected in the consistency verification results. Therefore, by appropriately setting the third threshold, it is possible to reliably find partial logical units that correspond to hallucination.

[0069] Furthermore, according to the generated AI information consistency verification device 1, statistical processing is performed on the basis score for each logical unit, and based on the results of the statistical processing, a statistical evaluation is performed on the input information, chunk information, or generation characteristics of the generated AI, identifying relatively low-rated items, and recording these low-rated items as visualized data. Therefore, specific fields and conditions related to low-rated items that are likely to cause hallucination can be easily grasped by referring to the visualized data obtained as actual results.

[0070] Furthermore, the Generating AI Information Consistency Verification Device 1 analyzes the distribution of evidence scores for each logical unit and the distribution of statistical evaluations obtained through statistical processing, and based on the analysis results, a risk indicator is variably set as a threshold that serves as the criterion for consistency verification. Then, the consistency of the chunk information of the response information is verified based on the evidence score against the set risk indicator (threshold), and the input information is adjusted or the generation characteristics of the Generating AI are adjusted according to the consistency verification results. As a result, the input information can be recursively adjusted and input to the Generating AI until hallucination is eliminated, and ultimately, response information without hallucination can be displayed.

[0071] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Furthermore, the effects described in the embodiments of the present invention are merely a list of the most preferred effects arising from the present invention, and the effects of the present invention are not limited to those described in the embodiments of the present invention. [Explanation of Symbols]

[0072] 1. AI Information Generation and Verification Device 10 Communications Department 11 Input section 12 Output section 13 Acquisition Department 14 Input Control Unit 15 Calculation Section 16 Verification Department 17. Statistical Evaluation Department 18 Output control unit 19 Memory section 2. Generation AI Server 20 Communications Department 21 Input Reception Section 22 Answer generation part 23 Generation Output Unit 24 Memory section 3. External Servers

Claims

1. A generation AI information consistency verification device that performs information consistency verification when generating response information in response to user input information and outputting it, An acquisition unit that acquires chunk information as information units obtained by dividing the aforementioned input information into meaningful fragments, An input control unit that causes the generating AI to generate response information based on the acquired chunk information and the input information, A calculation unit extracts logical units of semantic units from the generated response information and calculates a basis score for the chunk information for each extracted logical unit, A verification unit that, based on the calculated justification score, verifies the consistency of the response information with the chunk information and creates a consistency verification result that indicates the logical units whose justification score is below a predetermined threshold as non-justification statements, Based on the results of the consistency verification of the aforementioned response information, an output control unit determines whether to correct or output the response information and then outputs it. A device for verifying the consistency of generated AI information, equipped with the following features.

2. The system further comprises a statistical evaluation unit that performs statistical processing on the basis score for each logical unit, and based on the results of the statistical processing, performs a statistical evaluation of the input information, the chunk information, or the behavior of the generated AI, and identifies matters that receive a relatively low evaluation, The output control unit visualizes or records the identified relatively low-rated items, as described in claim 1, for the generated AI information consistency verification device.

3. The statistical evaluation unit analyzes the distribution of the basis score for each logical unit obtained by statistical processing, or the distribution of the statistical evaluation, and sets a risk indicator that serves as a criterion for consistency verification based on the analysis results. The verification unit verifies the consistency of the response information with the chunk information based on the basis score for the set risk indicator, The generated AI information consistency verification device according to claim 2, wherein the input control unit adjusts the input information or changes the behavior of the generated AI by parameter adjustment based on the consistency verification result of the response information.

4. A method for verifying the consistency of information when generating AI generates and outputs response information in response to input information from a user, which is performed by a generating AI information consistency verification device, The steps include: obtaining chunk information as information units obtained by dividing the aforementioned input information into meaningful fragments; The steps include: causing the generating AI to generate response information based on the acquired chunk information and input information; The steps include: extracting logical units of semantic units from the generated response information, and calculating a basis score for the chunk information for each extracted logical unit; The steps include: verifying the consistency of the response information with the chunk information based on the calculated justification score, and creating a consistency verification result that indicates the logical units whose justification score is below a predetermined threshold as non-justification statements; Based on the results of the consistency verification of the aforementioned response information, the steps include: correcting the response information or deciding whether or not to output it, and then outputting it; A method for verifying the consistency of generated AI information, including [the specified method].

5. When generating response information in response to input information from a user, the computer of the generating AI information consistency verification device performs information consistency verification, A step of obtaining chunk information as information units obtained by dividing the aforementioned input information into meaningful fragments, A step of causing the generating AI to generate response information based on the acquired chunk information and the input information, The steps include: extracting logical units of semantic units from the generated response information, and calculating a basis score for the chunk information for each extracted logical unit; A step of creating a consistency verification result that, based on the calculated justification score, verifies the consistency of the response information with the chunk information, and indicates the logical units whose justification score is below a predetermined threshold as non-justification sentences. Based on the results of the consistency verification of the aforementioned response information, the step of correcting the response information or deciding whether or not to output it, and then outputting it. A computer-readable computer program for executing a program.

Images