Audit systems, audit methods, and audit programs
The audit system addresses inefficiencies in machine learning-based auditing by using a control unit and generating AI to quickly identify and score risks in document data, enhancing the efficiency and accuracy of auditing processes.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- MIZUHO BANK
- Filing Date
- 2024-11-11
- Publication Date
- 2026-05-21
AI Technical Summary
Existing auditing systems using machine learning are time-consuming and inefficient due to the diversity of events to be detected in document data, making efficient auditing difficult.
An audit system utilizing a control unit connected to a generating AI that determines the risk of audited data by inputting prompts including detection items, and outputs audited data with risk scores based on the AI's responses.
Enables efficient auditing by reducing processing time and improving the detection of relevant events through parallel processing and risk scoring.
Smart Images
Figure 2026084345000001_ABST
Abstract
Description
Technical Field
[0005] ,
[0001] The present disclosure relates to an auditing system, an auditing method, and an auditing program for auditing documents.
Background Art
[0002] In order to suppress information leakage and the like, documents transmitted from companies and the like may be audited (see, for example, Patent Document 1). The information processing apparatus described in Patent Document 1 determines the weight of a morpheme in a model based on a feature amount determined from the result of morphological analysis of learning document data. Further, machine learning is performed to delete the feature amount corresponding to the morpheme whose weight is determined to be less than or equal to a given threshold from the input data of the model. Then, document data including the e-mails of the audited person is acquired. While determining the feature amount of this document data, by inputting the determined feature amount into the learned model, a score representing the degree of relevance between the document data and a given case is calculated.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] In the technique described in Patent Document 1, using machine learning, the degree of relevance of the case of document data can be grasped by a score. However, machine learning takes time and effort. Further, since the events to be detected in the document data are diverse, efficient auditing is difficult.
Means for Solving the Problems
[0005] An audit system that solves the above problems includes a control unit connected to a generating AI that determines the risk of the audited data. The control unit acquires an audited file containing multiple audited data, inputs prompts including detection items for the audited data to the generating AI, and outputs audited data including the detection items according to the risk score of the detection items for each audited data output from the generating AI. [Effects of the Invention]
[0006] This disclosure enables efficient auditing of the audited subject. [Brief explanation of the drawing]
[0007] [Figure 1] This is an explanatory diagram of an audit system according to an embodiment. [Figure 2] This is an explanatory diagram of the hardware configuration of the embodiment. [Figure 3] This is an explanatory diagram of the processing procedure of the embodiment. [Figure 4] This is an explanatory diagram of the task setting screen of the embodiment. [Figure 5] This is an explanatory diagram of the risk allocation setting screen in the embodiment. [Modes for carrying out the invention]
[0008] The following describes one embodiment of the audit system, audit method, and audit program, with reference to Figures 1 to 5. This embodiment assumes a service that audits concerns contained in sent and received emails (audit items). Examples of concerns include information leaks, legal violations, and harassment. As shown in Figure 1, audit system A1 uses user terminals 10, support servers 20, AI systems 30, etc., which are interconnected via a network.
[0009] (Hardware configuration) Figure 2 illustrates the hardware configuration of the information processing device H10, which comprises the user terminal 10, support server 20, and AI system 30. The information processing device H10 includes a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H15. Note that this hardware configuration is just one example, and it can be implemented using other hardware.
[0010] Communication device H11 is an interface that establishes a communication path with other devices and performs data transmission and reception. The input device H12 is a device that receives input from users, etc. The display device H13 is a display, etc., that displays various information.
[0011] The storage device H14 is a storage device that stores data and various programs for executing various functions of the user terminal 10, support server 20, and AI system 30. Examples of storage devices H14 include ROM, RAM, and hard disks.
[0012] The processor H15 uses programs and data stored in the memory device H14 to control the various processes in the user terminal 10, the support server 20, and the AI system 30. Examples of processor H15 include CPUs and MPUs. This processor H15 loads programs stored in ROM, etc., into RAM and executes various processes for each service.
[0013] The processor H15 is not limited to performing all of its operations through software processing. For example, the processor H15 may include dedicated hardware circuits (e.g., application-specific integrated circuits: ASICs) that perform hardware processing for at least some of the operations it performs. In other words, the processor H15 can be configured as follows:
[0014] [1] One or more processors that operate according to a computer program (software) [2] One or more dedicated hardware circuits that perform at least some of the various processes, 〔3〕Circuitry including those combinations The processor includes a CPU and memories such as RAM and ROM. The memories store program codes or instructions configured to cause the CPU to execute processing. The memories, i.e., computer-readable media, include any available media accessible by a general-purpose or dedicated computer.
[0015] (System configuration) Next, the functions of the user terminal 10, the support server 20, the AI system 30, etc. will be described using FIG. 1.
[0016] The user terminal 10 is a computer terminal used by a user (auditor) who uses this service. The support server 20 is a computer system that supports auditing. This support server 20 includes a control unit 21, a detection information storage unit 22, and an audit information storage unit 23.
[0017] The control unit 21 performs processing (including processing such as a management stage, a detection stage, a determination stage, etc.). By executing an audit support program for this purpose, the control unit 21 functions as a management unit 211, a detection unit 212, a determination unit 213, etc.
[0018] The management unit 211 executes processing to acquire various information from the user terminal 10. The detection unit 212 generates a prompt to be input to the AI system 30, uses the AI system 30 to acquire a response, and executes processing to detect a matter of concern.
[0019] The determination unit 213 executes processing to determine the response acquired from the AI system 30. The determination unit 213 holds risk distribution information set in advance in the user terminal 10. Scores are distributed for each detection item in this risk distribution information. For example, in the risk distribution information, scores are set for detection items subdivided such as information related to performance, information related to stock mergers, information related to new stock issues, etc. regarding insider trading.
[0020] The detection information storage unit 22 stores detection item management information for items to be detected by the AI system 30. This detection item management information is pre-registered through a setting process. The detection item management information includes information about the source, destination, and detection item.
[0021] Source information is information used to identify the origin of the information. For example, in the case of email, the domain of the sender's email address is used. Destination information is information used to identify the recipient of the information. For example, in the case of email, the domain of the recipient's email address is used.
[0022] The detection item information consists of items to be detected in the audited case (email in this embodiment), corresponding to the source and recipient. These detection items include, for example, methods and conditions for detecting content of concern in information disseminated via email. For example, they include keywords, phrases, contextual meanings, and samples of the content to be detected.
[0023] The audit information storage unit 23 stores audit target management information, which records the cases to be audited. Audit target management information is registered when audit targets, which are multiple emails compiled in list format, are obtained from the user terminal 10. This audit target management information records information such as the audit target case, recurrence count, detection items, relevant sentences, detection reasons, and risk score.
[0024] The audited item information includes information to identify the email being audited, such as the sender's address, recipient's address, subject, and email body. The "Repeat Count" indicates the nth iteration (the last time iteration) in which the AI system 30 was input and a response was obtained.
[0025] The detection items are the items of findings contained in the email being audited. Examples of detection items include concerns requiring attention. These detection items are identified by the AI system 30.
[0026] The relevant text is an excerpt from the audited email that corresponds to the detected item. This relevant text is identified by AI system 30. The detection reason is the reason for identifying the detection item. This detection reason is generated by the AI system 30.
[0027] The risk score information is the total score of the findings contained in the audited emails. This risk score is calculated according to the findings identified by the AI system 30.
[0028] The AI system 30 is a computer system that utilizes generative AI technology based on a large-scale language model that generates responses to prompts. In this embodiment, the AI system 30 predicts the presence or absence of detection items based on prompts in an audited case and outputs a response regarding the presence or absence of detection items.
[0029] (Audit process) The audit process will be explained using Figure 3. First, the control unit 21 of the support server 20 performs the configuration process (step S10). Specifically, the management unit 211 of the control unit 21 outputs a configuration screen to the user terminal 10.
[0030] As shown in Figure 4, the task settings screen 500 is displayed. On this task settings screen 500, you set the number of concurrent executions, the number of repetitions, the score calculation method, and the AI model. The number of concurrent executions is the number of questions (tasks) entered into the AI system 30 at the same time.
[0031] The number of repetitions is the number of times a response is repeatedly obtained from the AI system 30 for the same case. The scoring method involves calculating a risk score for each case (statistical processing). For example, the calculation method can use the mode, sum, mean, and maximum values of multiple responses obtained from the AI system 30. The AI model is the model (type, version, etc.) used in the AI system 30.
[0032] Furthermore, as shown in Figure 5, the risk allocation setting screen 510 is output. A weighting value is set for each detection item. Here, the weighting of each detection item is set using sliders. The management unit 211 then retrieves the information entered on the settings screen and temporarily stores it in memory.
[0033] Next, the control unit 21 of the support server 20 executes the list acquisition process (step S11). Specifically, the management unit 211 of the control unit 21 acquires the audit target file from the user terminal 10. This audit target file contains information (sender address, recipient address, subject, email body) of multiple cases to be audited (in this case, emails as audit target data) in list format. The management unit 211 then records the acquired audit target file in the audit information storage unit 23.
[0034] Next, the control unit 21 of the support server 20 performs a process to determine the number of concurrent executions (step S12). Specifically, the management unit 211 of the control unit 21 identifies the number of concurrent executions set on the task setting screen 500.
[0035] Next, the control unit 21 of the support server 20 repeats the following process for each case included in the audit target file. In this case, multiple cases are processed in parallel, according to the number of concurrent executions. Here, the control unit 21 of the support server 20 performs the process of identifying the sender and receiver (step S13). Specifically, the management unit 211 of the control unit 21 identifies the domain of the sender address and the domain of the recipient address of the case (email) included in the audited file. In this case, if a case has multiple recipient addresses set, the domains of all recipient addresses are identified.
[0036] Next, the control unit 21 of the support server 20 performs a process to identify detection items according to the sender and receiver (step S14). Specifically, the management unit 211 of the control unit 21 obtains detection item management information corresponding to the domain of the source address (originator) and the domain of the destination address (destination) from the detection information storage unit 22.
[0037] Next, the control unit 21 of the support server 20 repeats the following process for the same case. This process is repeated the number of times set in the task setting screen 500. Here, the control unit 21 of the support server 20 performs prompt input processing according to the detection items (step S15). Specifically, the detection unit 212 of the control unit 21 specifies the AI model set in the task setting screen 500 and inputs a prompt including the detection items along with the audit target case information to the AI system 30. In this prompt, the response includes instructions to output a combination of the detection item (key) and the number of detections (value).
[0038] Next, the control unit 21 of the support server 20 executes the process of acquiring the answer for each detection item (step S16). Specifically, if the processing load is not excessive, the AI system 30 predicts whether the audit target case information contains a detection item in response to a prompt. Furthermore, if a detection item is included, it extracts the relevant sentence and generates the number of detections and the reason for detection. Then, the detection unit 212 of the control unit 21 acquires the answer output from the AI system 30. This answer includes information on the detection item, the number of detections, the relevant sentence, and the reason for detection.
[0039] Next, the control unit 21 of the support server 20 performs a determination process to determine whether or not there is an error (step S17). Specifically, the detection unit 212 of the control unit 21 determines whether or not the response from the AI system 30 contains a processing load error. For example, if the processing load on the AI system 30 is excessive, such as when the number of tokens input to the AI system 30 at the same time exceeds the allowable limit, a processing load error is output.
[0040] If an error is detected (if the answer is "YES" in step S17), the control unit 21 of the support server 20 returns to the process of determining the number of concurrent executions (step S12). In this case, the detection unit 212 of the control unit 21 reduces the number of concurrent executions and re-executes.
[0041] If it is determined that there is no error (i.e., "NO" in step S17), the control unit 21 of the support server 20 performs the process of recording the detected items (step S18). Specifically, the detection unit 212 of the control unit 21 records the acquired response in the audit information storage unit 23, associating it with the audited case and the number of repetitions. Then, the process described above is repeated for the same case until the specified number of repetitions is reached.
[0042] Next, the control unit 21 of the support server 20 performs scoring processing (step S19). Specifically, the determination unit 213 of the control unit 21 obtains weighted values for the detection items included in the responses for each iteration in the risk allocation information. Next, the determination unit 213 uses the weighted values to calculate the total score for the detection items for each iteration in the responses for each iteration.
[0043] Next, the control unit 21 of the support server 20 performs a risk determination process (step S20). Specifically, the determination unit 213 of the control unit 21 calculates a risk score using the score calculation method set in the task setting screen 500. For example, if the score calculation method is "mode," the mode of the total score for each iteration is identified as the risk score. If the score calculation method is "sum," the sum of the total scores for each iteration is calculated as the risk score. If the score calculation method is "average," the average of the total scores for each iteration is calculated as the risk score. If the score calculation method is "maximum," the maximum value of the total score for each iteration is calculated as the risk score. The determination unit 213 then compares the risk score with a reference value. If the risk score exceeds the reference value, it determines that there is a risk.
[0044] Next, the control unit 21 of the support server 20 performs input processing to the list (step S21). Specifically, the determination unit 213 of the control unit 21 records the cases it has determined to have a risk in the audit target file of the audit information storage unit 23. Here, it records information regarding the detected item, the relevant sentence, the reason for detection, and the risk score. Repeat the above process for all cases included in the audited file until completion.
[0045] Next, the control unit 21 of the support server 20 executes the list output process (step S22). Specifically, the management unit 211 of the control unit 21 outputs the audit target files to the display device H13 of the user terminal 10.
[0046] (Operation of this embodiment) The prompt, which includes both the audit target and any findings, is input to the generating AI, allowing the AI to predict whether or not any findings are present in the audit target.
[0047] (Effects of this embodiment) (1) The control unit 21 of the support server 20 executes a configuration process (step S10). This allows the user to configure the task method and weight the detection items. (2) In this embodiment, the control unit 21 of the support server 20 executes the list acquisition process (step S11). This makes it possible to acquire the audit targets all at once. (3) In this embodiment, the control unit 21 of the support server 20 performs a process to determine the number of concurrent executions (step S12). This makes it possible to process multiple audit targets corresponding to the specified number of concurrent executions in parallel at the same time. Therefore, the processing time for multiple audit targets can be shortened.
[0048] (4) In this embodiment, the control unit 21 of the support server 20 performs a process to identify the sender and receiver (step S13) and a process to identify detection items according to the sender and receiver (step S14). As a result, since the concerns change depending on the source and destination of the information, it is possible to identify detection items according to the source and destination.
[0049] (5) In this embodiment, the control unit 21 of the support server 20 performs prompt input processing according to the detection item (step S15) and response acquisition processing for each detection item (step S16). This makes it possible to obtain responses according to the detection item at the audit target.
[0050] (6) In this embodiment, the control unit 21 of the support server 20 repeats the process of acquiring answers for each detection item (step S16) for the same case. This makes it possible to detect detection items in accordance with fluctuations in the AI system 30's answers. Furthermore, by identifying the relevant sentences, it is possible to easily grasp descriptions in the case that require attention.
[0051] (7) In this embodiment, if an error is determined (if the answer is "YES" in step S17), the control unit 21 of the support server 20 returns to the process of determining the number of concurrent executions (step S12). This makes it possible to reduce the number of parallel processes and obtain a response when the load on the AI system 30 is high.
[0052] (8) In this embodiment, the control unit 21 of the support server 20 performs scoring processing (step S19) and risk determination processing (step S20). This makes it possible to determine whether or not a warning is necessary for the detected items.
[0053] (9) In this embodiment, the control unit 21 of the support server 20 performs input processing to the list (step S21) and output processing to the list (step S22). This makes it possible to identify cases that require confirmation in the list.
[0054] This embodiment can be implemented with the following modifications. This embodiment and the following modifications can be combined with each other to the extent that they do not contradict each other technically. In the above embodiment, the control unit 21 of the support server 20 performs prompt input processing according to the detected item (step S15). Here, Retrieval-Augmented Generation (RAG) technology may be used. In this case, cases containing previously detected items are registered in the performance information storage unit. In this case, the control unit 21 of the support server 20 searches the performance information storage unit for similar cases similar to the audit target case. Then, it adds the identified similar cases to the prompt as case examples (knowledge) to be extracted and inputs them to the AI system 30.
[0055] In the above embodiment, email text is used as the document to be audited, but it is not limited to this. It can be applied to information that is transmitted, such as interview records and chat messages. For example, an interview record includes participants (sender and recipient of the information), date and time of the interview, subject, and content of the interview. A chat message includes sender ID, recipient ID, date and time of the chat, and message content. Furthermore, the system is not limited to text; audio and images may also be used. In this case as well, the AI system 30 predicts the information contained in the audio and images and includes them in the audit.
[0056] In the above embodiment, the control unit 21 of the support server 20 performs a process to identify detection items according to the sender and receiver (step S14). The detection items are not limited to those corresponding to the sender and receiver. For example, detection items corresponding to the sender, detection items corresponding to the receiver, or detection items not limited to the sender and receiver may be used.
[0057] In the above embodiment, the control unit 21 of the support server 20 performs the process of acquiring a list (step S11). Alternatively, the audit target cases may be acquired individually, and a determination may be made for each audit target case. In the above embodiment, the control unit 21 of the support server 20 repeatedly inputs the same case to the AI system 30 a set number of times. Alternatively, multiple responses corresponding to the number of repetitions may be simultaneously obtained from the AI system 30.
[0058] In the above embodiment, the control unit 21 of the support server 20 holds the risk allocation information that has been set in advance on the user terminal 10. In this case, the weighting may be changed for at least one of the sender and the receiver.
[0059] In the above embodiment, if an error is detected (if the answer is "YES" in step S17), the control unit 21 of the support server 20 returns to the process of determining the number of concurrent executions (step S12). In this case, the number of concurrent executions is reduced. A constant number may be maintained if it reduces the processing load on the AI system 30.
[0060] In the above embodiment, the control unit 21 of the support server 20 performs scoring processing (step S19). Here, the scores of each audited case information are added together to calculate a total score as a statistical value of the scores of the detection items included in the response. The method of calculating the total score is not limited to addition. [Explanation of Symbols]
[0061] A1...Audit system, 10...User terminal, 20...Support server, 21...Control unit, 211...Management unit, 212...Detection unit, 213...Determination unit, 22...Detection information storage unit, 23...Audit information storage unit, 30...AI system.
Claims
1. An audit system equipped with a control unit that is connected to a generating AI and determines the risk of the data to be audited, The control unit, Obtain the audit file containing multiple audit data items that are subject to audit, Regarding the audited data, a prompt including the detected items is input to the generating AI. An audit system characterized by outputting audit data including the detected items according to the risk score of the detected items for each audit data output from the generating AI.
2. The control unit, From the aforementioned generating AI, multiple responses are obtained for each of the aforementioned audited data. The audit system according to claim 1, characterized in that it calculates a risk score for each of the detected items for each of the audited data by statistical processing of the multiple responses.
3. The audit system according to claim 1 or 2, characterized in that the control unit inputs a prompt to the generating AI that modifies the detection items according to the source and destination of the audited data.
4. The control unit, Enter a prompt that includes multiple of the aforementioned detection items, and obtain a response for each of the aforementioned detection items. The audit system according to claim 1 or 2, characterized in that it calculates a risk score weighted by the responses for each of the aforementioned detection items.
5. The control unit, The audit system according to claim 1 or 2, characterized in that it acquires and outputs the relevant portion corresponding to the detection item in the audited data including the detection item.
6. The audit system according to claim 1 or 2, characterized in that the control unit inputs a plurality of audit target data, each with a simultaneous processing count, from among the audit target data included in the audit target file to the generating AI.
7. The audit system according to claim 6, characterized in that the control unit changes the number of simultaneous processes when it obtains a processing load error from the generated AI.
8. An auditing method using an auditing system that includes a control unit connected to a generating AI and capable of determining the risk of the data to be audited, The control unit, Obtain the audit file containing multiple audit data items that are subject to audit, Regarding the audited data, a prompt including the detected items is input to the generating AI. An auditing method characterized by outputting audited data including the detected items according to the risk score of the detected items for each audited data output from the generating AI.
9. A program for auditing using an audit system that is connected to a generating AI and includes a control unit that determines the risk of the data to be audited, The control unit, Obtain the audit file containing multiple audit data items that are subject to audit, Regarding the audited data, a prompt including the detected items is input to the generating AI. An audit program characterized by functioning as a means to output audit data including the detected items, according to the risk score of the detected items for each audit data output from the generating AI.