System for ensuring the effectiveness of decision-making

Abstract

This solution addresses the problem of decision-makers' approval of proposals generated by AI execution engines becoming a mere formality, and the difficulty of independent judgment due to the AI ​​anchoring effect. [Solution] The decision-making substance assurance system 1000 includes a decision-making material structure presentation unit 1100, a preliminary decision acquisition unit 1200 that acquires preliminary decision data of decision-makers before exposure to AI recommendations, a deviation degree calculation unit 1300 that calculates the deviation degree between the preliminary decision and the AI ​​recommendation, a browsing behavior recording unit 1400 that records browsing behavior to decision-making materials, a decision reason acquisition unit 1500 that acquires the reason for the decision, a substance evaluation unit 1600 that calculates a substance index based on the deviation degree, browsing behavior, and reason for the decision, and a substance record unit 1700 that records the substance record in a format that allows for detection of tampering, and activates additional procedures when the substance index falls below a predetermined threshold.

Description

Technical Field

[0004]

[0001] The present invention relates to decision-making support technology in corporate operations using artificial intelligence (AI). In particular, it relates to a system, method, and program for verifying, recording, and guaranteeing that the approval act of a natural person (hereinafter referred to as the "decision maker") who constitutes the decision-making mechanism 30 for the proposal generated by the AI execution engine 10 is based on a substantial judgment.

[0002] The present invention operates in cooperation with the smart articles data structure disclosed in Japanese Patent Application No. 2026-005527, the approval policy management mechanism disclosed in Japanese Patent Application No. 2026-009336, and the governance rule dynamic application mechanism disclosed in Japanese Patent Application No. 2026-065392, and provides a decision-making substantial guarantee layer located between the AI execution engine 10 and the decision-making mechanism 30.

Background Art

[0003] With the rapid development of AI technology including large language models (LLMs), the opportunities for AI to provide decision-making support in various aspects of corporate operations are increasing. For example, U.S. Patent No. 12254966 (published in March 2025) discloses a system that analyzes doctor-patient conversations in real time, generates medical decision-making support insights using a large language model, and presents them to doctors with priorities based on medical urgency. Such prior art optimizes the quality and timeliness of information provided from AI to humans, assuming that humans make the final judgment.

[0004] However, conventional AI decision support systems, as described above, lack the means to verify the quality of the approval process performed by humans on the information presented by the AI. In other words, the more optimized the recommendations the AI ​​presents, the more likely human decision-makers are to become dependent on the AI's recommendations, creating a risk that the approval process will become merely a formality. Parasuraman and Manzey (2010, Human Factors, Vol. 52, No. 3, pp. 381-410) empirically demonstrated that automation complacency and automation bias occur in the interaction between automated systems and humans, and comprehensively explained that these stem from changes in the allocation of attentional resources. This finding suggests that, even in the context of AI decision support, situations where approvers formally approve AI recommendations without adequately considering them can structurally occur. In particular, when AI systems provide decision support for the operation of a corporation (e.g., management decisions in board meetings, credit decisions in loan reviews, diagnostic support in medical institutions, etc.), it is important to evaluate the substance of the approval process performed by natural persons (directors, reviewers, doctors, etc.) who constitute the decision-making body of the corporation.

[0005] Meanwhile, technologies related to compliance and governance management of AI products have also been proposed. For example, U.S. Patent Application Publication No. 2025 / 0078091 (published in March 2025) discloses a system for continuously monitoring indicators such as bias, fairness, and explainability of AI models and determining corrective strategies.

[0006] However, such technologies monitor the quality of the AI ​​model itself, and do not include the quality of human judgment that is used to approve the AI's suggestions. In other words, conventional technologies have an asymmetry between quality assurance on the AI ​​side and quality assurance on the human judgment side.

[0007] In recent years, with the advancement of AI system implementation in society, there has been a growing social and institutional demand for AI systems to be explainable, transparent, subject to substantial human oversight, and to ensure accuracy and robustness. In particular, there is a recognized need to ensure that human oversight is not merely a formal intervention but is substantial when AI assists or performs decision-making.

[0008] In the academic field, there have been suggestions to introduce intentional friction into the human approval process for AI recommendations, ensuring that approvers have an opportunity to make independent judgments. However, these social demands and academic recommendations do not provide specific system configurations for technically verifying substantive oversight, leaving a gap between the demands and implementation. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Application No. 2026-005527 [Patent Document 2] Japanese Patent Application No. 2026-009336 [Patent Document 3] Japanese Patent Application No. 2026-065392 [Patent Document 4] U.S. Patent No. 12254966 [Patent Document 5] U.S. Patent Application Publication No. 2025 / 0078091 [Non-patent literature]

[0010] [Non-Patent Document 1] Parasuraman, R. and Manzey, DH, "Complacency and Bias in Human Use of Automation: An Attentional Integration", Human Factors, Vol. 52, No. 3, 2010, pp. 381-410

[0011] [Non-Patent Document 2] Regulation (EU) 2024 / 1689 of the European Parliament and of the Council on laying down harmonized rules on artificial intelligence (AI Act), 2024

[0012] [Non-Patent Document 3] Nanda Min Htin, "Redefining the Standard of Human Oversight for AI Negligence", Harvard Journal of Law and Technology Digest, February 2026 [Overview of the Initiative] [Problems that the invention aims to solve]

[0013] In light of the background technology described above, the conventional technology has the following four problems.

[0014] The first challenge is the prevalence of formal approval. Even when a decision-maker approves a proposal generated by the AI ​​execution engine 10 without adequately reviewing the information or considering the reasons for their decision, conventional systems treat this as valid approval. In the system disclosed in U.S. Patent No. 12254966, whether or not a physician has reviewed the AI's proposal is merely an estimation based on the conversation content, and there is no means to quantitatively evaluate the substance of the review.

[0015] The second problem is the AI anchoring effect. When AI recommendations are presented first, the judgment of the decision-maker is dragged towards the AI recommendations, making it difficult to make an independent judgment. Conventional AI decision-making support systems aim to prioritize information presentation and reduce cognitive load, and do not have a configuration to obtain an independent judgment before the decision-maker is exposed to the AI recommendations.

[0016] The third problem is the lack of post-approval verification. The system disclosed in the U.S. Patent Application Publication No. 2025 / 0078091 verifies the output quality of the AI model retrospectively, but does not have a function to retrospectively verify the quality of the human approval act for the output. That is, there is no means to audit whether the approval was based on a substantial judgment after the approval act was made.

[0017] The fourth problem is the ambiguity of attribution. In a pipeline where multiple LLMs are combined in series or in parallel, the contribution of each stage leading to the generation of the final proposal and the degree of the decision-maker's involvement in each stage are opaque, making it difficult to retrospectively track the attribution of the decision-making. The comparison between the monitoring targets of the prior art and those of the present invention will be described later.

[0018] An object of the present invention is to provide a system, a method, and a program that verify, quantify, and record the substance of the approval act of a decision-maker for a proposal generated by an AI execution engine 10 by technical means in order to solve the above problems.

[0019] Specifically, the present invention obtains a preliminary judgment of the decision-maker before exposure to the AI recommendation, calculates the degree of deviation between the preliminary judgment and the AI recommendation, structurally records the browsing behavior of the judgment material and the reasons for the judgment, calculates a substance index integrating these, activates an additional procedure when the index falls below a predetermined threshold, and records a series of processes in a form that can detect tampering.

Means for Solving the Problems

[0020] To solve the above problems, the decision-making substance assurance system 1000 according to the present invention comprises: a decision-making substance structure presentation unit 1100 that structures and presents decision-making substance related to a proposal generated by an AI execution engine 10 or 200; a preliminary decision acquisition unit 1200 that acquires preliminary decision data of a decision-maker regarding the proposal; a deviation degree calculation unit 1300 that calculates the deviation degree between the preliminary decision data and the proposal generated by the AI ​​execution engine 10 or 200; a viewing behavior recording unit 1400 that records the viewing behavior of a decision-maker regarding the decision-making substance; a decision reason acquisition unit 1500 that structurally acquires the decision-maker's reason for judgment; a substance evaluation unit 1600 that calculates a substance index based on the deviation degree, the viewing behavior record and the decision reason; and a substance recording unit 1700 that records a substance record including the substance index in a format that allows for tamper detection.

[0021] If the substance index calculated by the substance evaluation unit 1600 falls below a predetermined threshold, the decision substance assurance system 1000 initiates additional procedures, including requesting further deliberation, withholding approval, escalating to a higher authority, or a combination thereof. [Modes for carrying out the invention]

[0022] Embodiments of the present invention will be described in detail below with reference to the drawings. Note that the following embodiments are not limiting to the present invention, and various modifications are possible within the scope of the technical concept of the present invention.

[0023] Let me explain the overall configuration. Figure 1 is a diagram showing the overall configuration of the decision-making substance assurance system 1000 according to the present invention.

[0024] The decision-making substance assurance system 1000 is positioned between the AI ​​execution engine 10 or 200 and the decision-making mechanism 30, and comprises a decision material structure presentation unit 1100, a preliminary decision acquisition unit 1200, a deviation degree calculation unit 1300, a browsing behavior record unit 1400, a decision reason acquisition unit 1500, a substance evaluation unit 1600, and a substance recording unit 1700. The decision material structure presentation unit 1100 decomposes the output of each LLM and presents it to the decision-maker along with their respective contributions. For example, the contribution of LLM-A, who is responsible for the initial analysis, is calculated to be 30%, the contribution of LLM-B, who is responsible for the risk assessment, is 45%, and the contribution of LLM-C, who is responsible for integrating recommendations, is 25% (see Figure 12).

[0025] The decision-making substance assurance system 1000 receives proposed data from the AI ​​execution engine 10 or 200 via the information sharing platform 40 and presents structured decision-making information to the decision-makers constituting the decision-making mechanism 30. During the decision-making approval process, preliminary decision data is acquired, viewing behavior is recorded, reasons for the decision are acquired, the degree of deviation is calculated, and substance indicators are calculated, and an integrated substance record is generated.

[0026] The structure of the proposed data will be explained. The proposed data generated by AI execution engine 10 or 200 has the following JSON structure. ```json { "proposal_id": "PROP-2026-00142", "effective_at": "2026-04-01T00:00:00Z", "action_type": "investment", "action_subtype": "new_subsidiary_establishment", "policy_id": "POL-2026-003", "ai_recommendation": { "decision": "approve", "confidence": 0.87, "basis_info": { "rules_applied": ["rule_047", "rule_112"], "policy_refs": ["POL-2026-003"], "bylaws_version_id": "BYL-v3.2", "data_sources": ["financial_db", "legal_db", "market_db"], "reasoning_summary": "Based on financial indicators and market analysis, the establishment of the new subsidiary aligns with the business objectives outlined in Article 12 of the Articles of Incorporation and satisfies the conditions of Approval Policy POL-2026-003." }, "risk_info": { "risk_score": 0.34, "risk_factors": ["market_volatility", "regulatory_change"], "mitigation_suggested": ["hedging_strategy", "phased_rollout"] } }, "alternatives": [ { "alternative_id": "ALT-001", "decision": "approve_with_conditions", "conditions": ["phased_investment", "quarterly_review"], "risk_score": 0.28 }, { "alternative_id": "ALT-002", "decision": "defer", "reason": "Waiting for market conditions to stabilize", "risk_score": 0.12 } ] } ```

[0027] The decision-making information structured presentation unit 1100 will now be explained. Figure 2 shows the data flow of the decision-making information structured presentation unit 1100. The decision-making information structured presentation unit 1100 structures the proposal data received from the AI ​​execution engine 10 or 200 as information necessary for decision-makers to make a decision, and controls the order in which it is presented. Specifically, it presents the proposal summary, basis information (basis_info), risk information (risk_info), alternatives, and related articles of incorporation and approval policies hierarchically according to the decision-maker's authority level and the importance of the proposal.

[0028] The decision-making material structure presentation unit 1100 has the function of controlling the order in which information is presented to the decision-maker. This control is performed in cooperation with the preliminary decision acquisition unit 1200, and the timing of AI-recommended presentation is adjusted according to the preliminary decision acquisition pattern (Pattern 1 to Pattern 4, described later).

[0029] The preliminary decision acquisition unit 1200 will now be described. Figure 3 shows the sequence of synchronous preliminary decision acquisition (pattern 1) in the preliminary decision acquisition unit 1200. The preliminary decision acquisition unit 1200 acquires preliminary decision data from the decision-maker. Preliminary decision data is data indicating the decision-maker's judgment or judgment tendency regarding the recommendation generated by the AI ​​execution engine 10 or 200, recorded before or independently of the decision-maker's exposure to the recommendation.

[0030] The acquisition of preliminary decision data is carried out using one of the following four patterns or a combination thereof. Pattern 1 (synchronous) is a pattern in which the decision-maker enters a preliminary decision on the approval screen before the AI ​​recommendation is displayed, and the AI ​​recommendation is displayed after the input is completed, and the degree of deviation is calculated. Pattern 2 (asynchronous, pre-policy type) is a pattern in which policy decisions registered in advance at monthly or quarterly policy meetings are compared with the AI ​​recommendation at a later date. Pattern 3 (asynchronous, staged type) is a pattern in which a rough decision direction is entered at the initial stage of the proposal, and is refined after AI analysis. Pattern 4 (asynchronous, historical type) is a pattern in which the decision-maker's decision patterns from past similar cases are extracted from the database and compared with the new AI recommendation.

[0031] The processing flow for Pattern 1 (synchronous) is described in detail below. First, the decision-making material structure presentation unit 1100 presents the decision-maker with an overview of the proposal and decision-making materials (basis information, risk information, alternatives, relevant articles of incorporation, and approval policy). At this stage, AI recommendations (recommended judgments such as approve, reject, or defer, and confidence values) are not displayed. Next, the preliminary judgment acquisition unit 1200 requests the decision-maker to input a preliminary judgment. The decision-maker selects one of the following: approve, conditionally approve, postpone, or reject, and inputs conditions and brief reasons as necessary. After the input of the preliminary judgment is completed, the decision-making material structure presentation unit 1100 displays the AI ​​recommendation. The deviation calculation unit 1300 calculates the deviation between the preliminary judgment data and the AI ​​recommendation and presents the result to the decision-maker. After confirming the deviation, the decision-maker makes a final decision and inputs the reason for the decision via the decision reason acquisition unit 1500.

[0032] Figure 4 shows the sequence for Pattern 2 (asynchronous, pre-policy type). The processing flow for Pattern 2 is described in detail. Members of the decision-making body 30 register policy decisions for a predetermined period in advance at monthly or quarterly policy meetings, etc. The policy decisions are structured and recorded by categories such as investment policy, risk tolerance, and priority business areas. Later, when the AI ​​execution engine 10 or 200 generates an individual proposal, the preliminary decision acquisition unit 1200 extracts the pre-registered policy decisions corresponding to the attributes of the proposal as preliminary decision data. The deviation calculation unit 1300 calculates the deviation between the extracted policy decisions and the individual AI recommendations.

[0033] Figure 5 shows the sequence for Pattern 3 (asynchronous, staged). The processing flow for Pattern 3 is described in detail. When the consideration of a proposal involves multiple stages, the decision-maker inputs a general direction of judgment in the initial stage (for example, agreeing with the general direction but still considering the detailed conditions). This initial judgment is recorded as preliminary judgment data. Subsequently, when the AI ​​execution engine 10 or 200 performs a detailed analysis and generates a recommendation, the deviation calculation unit 1300 calculates the deviation between the initial judgment and the detailed recommendation. The decision-maker refines their judgment while referring to the deviation and makes a final decision.

[0034] Figure 6 shows the sequence for Pattern 4 (asynchronous, historical type). The processing flow for Pattern 4 is described in detail below. The preliminary decision acquisition unit 1200 extracts the decision patterns of the decision-maker (or the same decision-making mechanism 30) in similar past cases from the database and constructs them as preliminary decision data. Similar cases are identified by calculating the similarity based on the attributes of the proposal (action_type, action_subtype, risk_score, etc.). The deviation calculation unit 1300 calculates the deviation between the extracted past decision patterns and the current AI recommendation. This pattern has the advantage of being able to construct preliminary decision data without imposing an additional input burden on the decision-maker.

[0035] The preliminary decision data has the following JSON structure. ```json { "pre_judgment_id": "PRE-2026-00142-001", "proposal_id": "PROP-2026-00142", "decision_maker_id": "DM-003", "set_by_decision_body_ref": "BOARD-001", "pattern_type": "synchronous", "captured_at": "2026-04-01T09:30:00Z", "ai_recommendation_exposed": false, "preliminary_decision": "approve_with_conditions", "preliminary_conditions": ["quarterly_review", "budget_cap"], "confidence_self_assessed": 0.65, "reasoning_brief": "We support the direction of business expansion, but considering market risks, a phased implementation is desirable." } ```

[0036] The deviation calculation unit 1300 will now be described. Figure 7 shows the calculation flow in the deviation calculation unit 1300. The deviation calculation unit 1300 calculates the deviation between the preliminary judgment data acquired by the preliminary judgment acquisition unit 1200 and the recommendation generated by the AI ​​execution engine 10 or 200. The deviation calculation is performed based on at least the degree of agreement in the direction of judgment, the difference in conditions, and the difference in risk perception. In quantifying the deviation between the preliminary judgment data and the AI ​​recommendation, the deviation calculation unit 1300 is not limited to a single calculation method, but can select or combine appropriate calculation methods depending on the nature of the proposal, the configuration of the decision-making mechanism 30, and the requirements of the approval policy.

[0037] Figure 8 shows an example of the calculation process in the deviation calculation unit 1300. The deviation is calculated by combining, for example, the following three elements. The first element is the degree of agreement in the judgment direction. The degree of agreement between the judgment direction of the preliminary judgment (approve, approve_with_conditions, defer, reject) and the judgment direction of the AI ​​recommendation is converted into a numerical value from 0 to 1 based on a predefined mapping table. For example, 0 represents a perfect match, 0.3 for approve vs. approve_with_conditions, 0.6 for approve vs. defer, and 1.0 for approve vs. reject. The second element is the difference in conditions. The degree of difference in conditions is calculated based on set operations (union, intersection, difference) between the conditions attached to the preliminary judgment and the conditions attached to the AI ​​recommendation. The third element is the difference in risk perception. The absolute value of the difference between the confidence value of the self-assessment in the preliminary judgment and the confidence value of the AI ​​recommendation is calculated.

[0038] The browsing behavior recording unit 1400 will now be described. The browsing behavior recording unit 1400 records the decision-maker's browsing behavior regarding the decision materials presented by the decision material structure presentation unit 1100. The information recorded includes whether each decision material element was viewed, the viewing time, the viewing order, the scroll range, the expansion operation (expanding folded detailed information), and references to links to external materials.

[0039] The browsing behavior log data has the following JSON structure. ```json { "viewing_session_id": "VIEW-2026-00142-001", "proposal_id": "PROP-2026-00142", "decision_maker_id": "DM-003", "session_start": "2026-04-01T09:32:00Z", "session_end": "2026-04-01T09:47:00Z", "total_duration_seconds": 900, "elements_viewed": [ { "element_id": "basis_info", "viewed": true, "duration_seconds": 180, "scroll_depth_percent": 100, "expanded_details": true }, { "element_id": "risk_info", "viewed": true, "duration_seconds": 240, "scroll_depth_percent": 100, "expanded_details": true }, { "element_id": "alternative_ALT-001", "viewed": true, "duration_seconds": 120, "scroll_depth_percent": 80, "expanded_details": false }, { "element_id": "alternative_ALT-002", "viewed": true, "duration_seconds": 90, "scroll_depth_percent": 60, "expanded_details": false }, { "element_id": "bylaws_article_12", "viewed": true, "duration_seconds": 60, "scroll_depth_percent": 100, "expanded_details": false }, { "element_id": "policy_POL-2026-003", "viewed": true, "duration_seconds": 150, "scroll_depth_percent": 90, "expanded_details": true } ], "elements_total": 6, "elements_viewed_count": 6, "external_references_accessed": [ { "reference_type": "financial_report", "reference_id": "FIN-2026-Q1", "access_duration_seconds": 60 } ] } ```

[0040] The decision reason acquisition unit 1500 will now be described. The decision reason acquisition unit 1500 structurally acquires the reasons for a decision made by a decision-maker when making an approval or rejection decision. The reasons for the decision include the selection of a decision category, a free-form explanation of the reasons, and a clear indication of the differences from the AI ​​recommendation.

[0041] The decision reasoning data has the following JSON structure. ```json { "decision_reason_id": "RSN-2026-00142-001", "proposal_id": "PROP-2026-00142", "decision_maker_id": "DM-003", "final_decision": "approve_with_conditions", "decision_categories": ["strategic_alignment", "risk_mitigation"], "free_text_reason": "We approve this as it aligns with our medium- to long-term business expansion strategy. However, considering market fluctuation risks, we are subject to phased investment and quarterly reviews. We do not agree to the AI-recommended immediate full investment and will adopt a phased approach." "deviation_from_ai": { "ai_recommended": "approve", "human_decided": "approve_with_conditions", "deviation_points": ["investment_phasing", "review_frequency"], "deviation_reasoning": "While AI recommendations are based on market stability, we have added conditions to reflect the risk of recent regulatory changes." }, "char_count": 127, "specificity_indicators": { "named_entities": ["Article 12 of the Articles of Incorporation", "POL-2026-003", "Q1 Financial Report"], "quantitative_references": ["quarterly", "gradual"], "conditional_statements": 2 } } ```

[0042] The substance evaluation unit 1600 will now be explained. Figure 9 shows the calculation concept of the substantiality score in the substantiality evaluation unit 1600. The substantiality evaluation unit 1600 calculates a substantiality score (substantiveness_score) that indicates the substantiality of the approved act, based on the deviation degree calculated by the deviation degree calculation unit 1300, the browsing behavior recorded by the browsing behavior recording unit 1400, and the judgment reason acquisition unit 1500.

[0043] The method for calculating the substance index is not limited to a specific algorithm. The substance evaluation unit 1600 receives at least the degree of deviation, records of browsing behavior, and reasons for judgment as input, and outputs a quantitative index indicating whether the approval action is based on a substantive judgment. The selection of the calculation method can be configured according to the size of the corporation, the importance of the decision, regulatory requirements, and operational policies.

[0044] As an example of a method for calculating the realness index, we will explain a method using a weighted linear combination. Figure 10 shows the structure of the weighted linear combination. In this example, the realness index S is calculated by the following formula. S = w1 × D + w2 × V + w3 × R Here, D is the deviation score (0 to 1), V is the viewing completeness score (0 to 1), R is the reasoning completeness score (0 to 1), and w1, w2, and w3 are the weighting coefficients for each score (w1 + w2 + w3 = 1).

[0045] The weighting coefficients are dynamically set according to the importance of the proposal. For example, if the importance is HIGH, w1=0.3, w2=0.4, and w3=0.3 are used, and if the importance is LOW, w1=0.4, w2=0.3, and w3=0.3 are used. This setting means that the completeness of the viewing is given more weight to proposals of higher importance, and whether or not the decision-maker has sufficiently viewed the information has a significant impact on the substance index.

[0046] The browsing completeness score V in the weighted linear combination described above can be calculated, for example, using the following formula: V = (Number of elements viewed / Total number of elements) × min(1.0, Actual browsing time / Expected browsing time). Here, the expected viewing time is the estimated time required to reasonably view each element, which is predetermined based on the amount of information (number of characters, number of figures, etc.) of each element. According to this formula, if all elements are displayed formally but not actually viewed (i.e., the actual viewing time is extremely short), a low viewing completeness score will be calculated.

[0047] The reasoning completeness score R in the weighted linear combination described above is calculated, for example, using a rubric. The evaluation items include whether a judgment category is selected, the number of characters in the free-response section, specificity (number of proper nouns, numerical values, and conditional statements), and whether differences from the AI ​​recommendation are clearly stated. Each item is assigned a score from 0 to 1, and the average or weighted average of these scores is used as the reasoning completeness score.

[0048] As another example of a method for calculating the substance index, we will explain the multi-stage thresholding method. Figure 11 shows the configuration of the multi-stage thresholding method. In this example, individual thresholds are set for each score of deviation, viewing completeness, and reason completeness, and the document is judged to be substance overall only when all scores exceed each threshold. If any score falls below an individual threshold, an additional procedure corresponding to that score is triggered. For example, if only the access completeness score is low, a request for re-examination of the materials is made, and if only the completeness of the reasoning score is low, an additional explanation of the reasoning is made.

[0049] As yet another example of a method for calculating the substance index, we will explain a method using a machine learning model. In this example, the substance index is calculated using a classification or regression model that has been trained on the degree of deviation in past approval actions, browsing behavior records, reasoning for decisions, and subsequent results (ex-post evaluation of the appropriateness of approval) as training data. This example has the advantage of not requiring manual setting of weight coefficients and improving evaluation accuracy as operational data accumulates.

[0050] As yet another example of a method for calculating the substance index, we describe a hybrid approach combining rule-based and exception detection. In this example, the basic substance assessment is performed using a rule-based method, and the detection of anomaly patterns is supplemented by statistical methods or machine learning. For example, by establishing an anomaly detection layer independent of the calculation of individual scores, such as issuing a warning when the viewing time deviates by more than two standard deviations from the average of the same decision-maker in the past, it becomes more difficult to avoid intentional formal approval.

[0051] The substance assessment unit 1600 initiates additional procedures if the calculated substance index falls below a predetermined threshold. These additional procedures include requesting further deliberation, withholding approval, escalating to a higher authority, or a combination thereof.

[0052] The threshold setting for the substantiality index is explained below. The threshold in the substantiality evaluation unit 1600 is dynamically set according to the importance and risk level of the proposal. For example, the threshold is 0.80 when the importance is HIGH and the risk level is HIGH, the threshold is 0.70 when the importance is HIGH and the risk level is MEDIUM, the threshold is 0.60 when the importance is MEDIUM and the risk level is MEDIUM, and the threshold is 0.50 when the importance is LOW. The threshold setting is managed in association with the approval policy (policy_id) in the approval policy management mechanism disclosed in Japanese Patent Application No. 2026-009336, and the threshold is updated when the policy_version is updated.

[0053] The details of the additional procedure will be explained. Figure 13 shows the three-stage flow of the additional procedure. The additional procedure, which is triggered when the substance index falls below the threshold, consists of the following three stages. The first stage involves a request for re-viewing and re-entry. If the completeness of the review or the thoroughness of the reasoning is insufficient, the system requests that previously unviewed information be reviewed or that additional reasoning be provided. The second stage is the request for additional deliberation, which is requested with members of the other 30 decision-making bodies if the degree of discrepancy is high and the substance index falls below the threshold. The third stage is escalation, which involves escalating the proposal to a higher authority if the substance indicators do not recover after further deliberation or if the proposal is of the highest importance level.

[0054] The substance recording unit 1700 will now be described. The substance recording unit 1700 records substance records, including substance indicators calculated by the substance evaluation unit 1600, in a format that allows for detection of tampering. The substance records include preliminary judgment data, deviation degree, browsing behavior record, judgment reason, substance indicators, applied thresholds, and whether or not additional procedures were initiated.

[0055] Figure 18 shows the hash chain structure in the substance record unit 1700. The substance record unit 1700 assigns a hash value to each substance record and forms a chain structure that includes the hash value of the immediately preceding substance record, thereby realizing a format that can be detected for tampering. The case identifier 1010 may include the subject of the decision (e.g., case number, contract ID), the decision-maker (e.g., user ID, job title), and the date and time of the decision (e.g., timestamp).

[0056] The actual record data has the following JSON structure. ```json { "substantiveness_record_id": "SUB-2026-00142-001", "proposal_id": "PROP-2026-00142", "decision_maker_id": "DM-003", "policy_id": "POL-2026-003", "set_by_decision_body_ref": "BOARD-001", "bylaws_version_id": "BYL-v3.2", "pre_judgment_id": "PRE-2026-00142-001", "viewing_session_id": "VIEW-2026-00142-001", "decision_reason_id": "RSN-2026-00142-001", "scores": { "divergence_score": 0.45, "viewing_score": 0.78, "reasoning_score": 0.82, "substantiveness_score": 0.68 }, "threshold": { "applied_threshold": 0.70, "threshold_basis": "risk_level_HIGH", "policy_ref": "POL-2026-003" }, "trigger": { "triggered": true, "action": "additional_review_requested", "escalation_level": "senior_board_member", "triggered_at": "2026-04-01T09:48:00Z" }, "record_integrity": { "hash_algorithm": "SHA-256", "record_hash": "a1b2c3d4e5f6...", "previous_record_hash": "f6e5d4c3b2a1...", "chain_id": "CHAIN-PROP-2026-00142" }, "created_at": "2026-04-01T09:48:01Z" } ``` The case identifier (proposal_id) identifies the case subject to decision-making and, together with the decision-maker (decision_maker_id) and the date and time of the decision (created_at), constitutes a fundamental element of the decision-making substance record data structure.

[0057] The substantiality evaluation index (substantiveness_score) is calculated using a weighted average of the divergence score (divergence_score), viewing score (viewing_score), and reasoning score (reasoning_score). For example, the substantiality score can be calculated as: Substantiveness Score = w1 × (1 - Normalized Divergence) + w2 × Viewing Behavior Score + w3 × Reasoning for Judgment (w1, w2, and w3 are weighting coefficients). This formula makes it possible to quantitatively evaluate the substantiality of approval actions from multiple perspectives.

[0058] Furthermore, the present invention can be realized not only as the decision-making substance assurance system described above, but also as a data structure for recording the substance of decision-making. This data structure includes a case identifier, deviation degree, browsing behavior record, reason for decision, and substance evaluation index, thereby ensuring transparency and traceability of the decision-making process. This data structure may include a tamper-detectable hash value, and the reliability of the record can be further enhanced by forming a hash chain structure (record_hash and previous_record_hash).

[0059] This section explains the guarantee of the decision-maker's freedom to make a final decision. An important point in this invention is that even if the substance index falls below a threshold, the decision-maker can make a final decision to approve or reject after going through additional procedures. In other words, this invention does not have a function to automatically overturn the decision of a decision-maker, but rather ensures ex-post audits and accountability by verifying and recording the substance of the approval process. If approval is made with a low substance index, that fact is recorded in the substance record and can be referenced in ex-post audits.

[0060] This section describes the decomposition display of the LLM pipeline. Figure 12 shows the configuration of the LLM pipeline decomposition display. When the AI ​​execution engine 10 or 200 generates a proposal using a pipeline that combines multiple LLMs in series or parallel, the decision material structure presentation unit 1100 has the function of decomposing and displaying the input, output, and contribution of each LLM processing stage. This allows decision-makers to make decisions after understanding which LLM and which processing stage's output the final recommendation depends on. The deviation calculation unit 1300 can calculate the deviation from the recommendation of the entire pipeline, as well as the partial deviation from the output of each stage.

[0061] The following describes the cooperation with the dynamic AI delegation control unit 1800. Figure 16 shows the dynamic AI delegation control feedback loop. The substantiality records accumulated by the substantiality evaluation unit 1600 are used for dynamic control of the delegation scope to the AI ​​execution engine 10 or 200. Specifically, if the substantiality index for a proposal in a particular category is consistently high, the autonomous execution authority of the AI ​​execution engine 10 or 200 for proposals in that category can be gradually expanded. Conversely, if the substantiality index for a particular category is consistently low, the automatic execution of AI recommendations is stopped, and the delegation scope is reduced to require human approval. This control is performed through updating the allowed_scope and allow_conditions fields of the approval policy management mechanism disclosed in Japanese Patent Application No. 2026-009336.

[0062] The meta-level verification of substance is described below. In order to ensure that the Dynamic AI Delegation Control Unit 1800 itself is based on substance judgment, it is equipped with a meta-level verification function that applies substance evaluation to operations that change the scope of delegation. That is, a series of processes such as obtaining a preliminary judgment, calculating the degree of deviation, recording browsing behavior, obtaining the reason for the judgment, and calculating a substance index are recursively applied to the judgment of expanding or shrinking the scope of delegation.

[0063] This section describes the collaboration with Japanese Patent Application No. 2026-009336. Figure 14 shows the collaboration configuration between the decision-making substance assurance system 1000 and the approval policy management mechanism disclosed in Japanese Patent Application No. 2026-009336. The approval policy management mechanism manages the approval policy (policy_id, policy_version, effective_from, effective_to, allowed_scope, allow_conditions) applied to each proposal. The decision-making substance assurance system 1000 refers to the set_by_decision_body_ref defined in the approval policy to identify the configuration of the decision-making mechanism 30 that should be involved in the approval of the proposal, and applies a substance assessment to each member.

[0064] The substance record generated by the substance record unit 1700 includes the policy_id and set_by_decision_body_ref of the approval policy and is recorded in conjunction with the approval policy version control (policy_version). This makes it possible to track the substance of approval actions performed under a specific approval policy for each policy version.

[0065] This section explains the collaboration with Japanese Patent Application No. 2026-065392. Figure 15 shows the collaboration configuration between the decision-making substance assurance system 1000 and the governance rule dynamic application mechanism disclosed in Japanese Patent Application No. 2026-065392. The responsibility definition 112 of the governance rule dynamic application mechanism defines the authority level of each member of the decision-making mechanism 30. The substance evaluation unit 1600 dynamically sets the threshold of the substance index by referring to the said authority level. A higher threshold is applied to approval actions by members with a higher authority level.

[0066] The substance records generated by the substance record unit 1700 are stored in the evidence management unit 130 of the governance rule dynamic application mechanism. The evidence management unit 130 manages the governance rule application history and substance records in association, enabling analysis of the impact of changes in governance rules on the substance of approved actions.

[0067] This section explains the evaluation of substance in a deliberative body. Figure 17 shows the structure of the evaluation of substance in a deliberative body. When the decision-making mechanism 30 is a deliberative body consisting of multiple members, the substance evaluation unit 1600 calculates a substance index for the deliberative body as a whole, in addition to the individual substance index for each member. The substance index for the deliberative body as a whole is calculated, for example, as the minimum, average, or median of the substance index for each member. Additional procedures are also triggered if the substance index for the deliberative body as a whole falls below a threshold.

[0068] The cognitive state adaptation function will now be explained. The browsing behavior recording unit 1400 can estimate the decision-maker's cognitive state (fatigue, distraction, haste, etc.) from the browsing behavior patterns recorded by the unit, and can be equipped with a function to adaptively adjust the threshold for substance evaluation or the method of presenting judgment materials. For example, if the browsing speed is extremely fast compared to normal, a warning message may be displayed or the threshold may be temporarily raised to prevent formal approval caused by the cognitive state.

[0069] This section describes inertia detection using time-series analysis. The substance evaluation unit 1600 has the function of analyzing the time-series changes of substance indicators across multiple approval actions by the same decision-maker and detecting inertia patterns (patterns in which substance indicators decrease with each successive approval action). If an inertia pattern is detected, the unit prevents the approval action from becoming a mere formality by issuing a warning to the decision-maker or by temporarily raising the threshold for that decision-maker.

[0070] This section describes the generation of regulatory compliance reports. The Substance Record Unit 1700 has the function of generating reports for audit or external reporting purposes based on accumulated substance records. The report is output in a format that includes statistical information on the substance of human supervisory activities (mean, median, minimum, and distribution of substance indicators), the number of times additional procedures were initiated and the reasons for them, and time-series trends. The output format of the report can be configured according to the user's requirements.

[0071] This section will explain the comparison between the monitoring targets of the prior art and the present invention. Figure 19 shows a comparison between the monitoring targets of the prior art and the present invention. In the prior art, the output quality of the AI ​​model (bias, fairness, explainability, etc.) was the monitoring target, but the quality of human approval of the AI's proposals was not included in the monitoring target. The present invention fills this gap and realizes a configuration that covers both AI-side quality assurance and human-side judgment quality assurance. [Brief explanation of the drawing]

[0072] [Figure 1] This figure shows the overall configuration of the decision-making substance assurance system 1000 according to the present invention. [Figure 2] This diagram shows the data flow of the decision-making material structure presentation unit 1100. [Figure 3] This diagram shows the sequence for synchronous preliminary decision acquisition (Pattern 1). [Figure 4] This diagram shows the sequence for asynchronous, pre-policy type preliminary decision acquisition (Pattern 2). [Figure 5] This diagram shows the sequence of asynchronous, staged preliminary decision acquisition (Pattern 3). [Figure 6] This diagram shows the sequence for asynchronous, historical preliminary decision acquisition (pattern 4). [Figure 7] This diagram shows the calculation flow in the deviation degree calculation unit 1300. [Figure 8] This figure shows an example of the calculation process in the deviation degree calculation unit 1300. [Figure 9] This diagram shows the calculation concept of the substantiality index in the substantiality evaluation unit 1600. [Figure 10] This is a diagram showing the structure of a weighted linear combination. [Figure 11] This diagram shows the configuration of a multi-stage thresholding scheme. [Figure 12] This diagram shows the configuration of the LLM pipeline breakdown display. [Figure 13] This diagram shows the three-step flow chart for the additional procedure. [Figure 14] This diagram shows the integration configuration with the approval policy management mechanism disclosed in Japanese Patent Application No. 2026-009336. [Figure 15] This diagram shows the configuration for coordinating with the dynamic application mechanism of governance rules disclosed in Japanese Patent Application No. 2026-065392. [Figure 16] This diagram shows a dynamic AI delegated control feedback loop. [Figure 17] This diagram shows the structure of the substantive assessment in the collegial body. [Figure 18]This figure shows the hash chain structure in the actual recording unit 1700. [Figure 19] This figure shows a comparison between the monitoring targets of the conventional technology and the monitoring targets of the present invention. [Industrial applicability]

[0073] The decision-making substance assurance system 1000 according to the present invention can be used in any situation in which a corporation is operated using an AI execution engine, and is particularly useful in AI support for loan screening in financial institutions, support for determining treatment policies in medical institutions, support for contract screening in legal departments, and support for investment decisions by management. In particular, when an AI execution engine provides decision-making support for the operation of a corporation, it can technically guarantee effective human oversight by evaluating the substance of approvals made by natural persons (directors, reviewers, doctors, etc.) who constitute the corporation's decision-making body, thereby preventing superficial approvals. [Explanation of Symbols]

[0074] 10 AI Execution Engines 30 Decision-making mechanism 40 Information sharing infrastructure 112 Responsibility Definition (Patent Application No. 2026-065392) 130 Evidence Management Department (Patent Application No. 2026-065392) 200 AI execution engine (reference number in Japanese Patent Application No. 2026-009336) 1000 Decision-Making Substance Assurance Systems 1100 Judgment material structured presentation part 1200 Preliminary judgment acquisition unit 1300 Deviation Calculation Unit 1400 Browsing Activity Records Section 1500 Judgment reason acquisition department 1600 Substance Assessment Department 1700 Substance Recording Unit 1800 Dynamic AI Delegation Control Unit

Claims

1. A decision-making substance assurance system comprising: a decision-making substance structuring and presentation unit that structures and presents decision-making substance related to a proposal generated by an AI execution engine to a decision-maker; a preliminary decision acquisition unit that acquires preliminary decision data of the decision-maker regarding the proposal in a recorded state; a deviation degree calculation unit that calculates the deviation degree between the preliminary decision data and the proposal generated by the AI ​​execution engine; a viewing behavior recording unit that records the viewing behavior of the decision-maker regarding the decision-making substance; a decision reason acquisition unit that structurally acquires the decision-maker's reasoning; a substance evaluation unit that calculates a substance index based on the deviation degree, the viewing behavior record and the reasoning; and a substance recording unit that records a substance record including the substance index in a format that allows for tamper detection.

2. The decision-making substance assurance system according to claim 1, wherein the preliminary decision acquisition unit acquires at least one of the following: synchronous preliminary decision data acquired before the decision-maker is exposed to the recommendations of the AI ​​execution engine; pre-policy type preliminary decision data based on pre-registered policy decisions; stage type preliminary decision data acquired during the proposal review stage; and historical type preliminary decision data extracted from decision patterns in similar past cases.

3. The decision-making substance assurance system according to claim 1, wherein the substance evaluation unit initiates an additional procedure including at least one of the following: requesting further deliberation, withholding approval, and escalating to a higher authority, if the substance index falls below a predetermined threshold.

4. The decision-making substance assurance system according to claim 3, wherein the threshold is dynamically set according to at least one of the importance of the proposal, the risk level, the requirements of the approval policy, and the authority level of the decision-maker.

5. The decision-making substance assurance system according to claim 1, wherein the substance evaluation unit analyzes the time-series changes of the substance index over multiple approval actions by the same decision-maker and detects an inertia pattern in which the substance index decreases with each successive approval action.

6. The decision-making substance assurance system according to claim 1, wherein the decision-making material structure presentation unit, when the AI ​​execution engine generates the proposal by a pipeline combining multiple LLMs in series or parallel, decomposes and displays the input, output and contribution for each processing stage of each LLM.

7. The decision-making substance assurance system according to claim 1, further comprising a dynamic AI delegation control unit that dynamically changes the scope of delegation to the AI ​​execution engine based on the substance record accumulated by the substance evaluation unit, wherein the substance evaluation by the preliminary judgment acquisition unit, the deviation degree calculation unit, the browsing behavior record unit, the judgment reason acquisition unit, and the substance evaluation unit is recursively applied to the operation of changing the scope of delegation.

8. The decision-making substance assurance system according to claim 1, wherein the substance record unit includes a policy identifier of an approval policy management mechanism that manages the approval policy and an identifier of a decision-making mechanism in the substance record, and manages the substance record in conjunction with version control of the approval policy.

9. The decision-making substance assurance system according to claim 3, wherein the threshold is dynamically set by referring to the authority level of the decision-maker defined in the responsibility definition of the governance rule dynamic application mechanism, and the substance record is stored in the evidence management unit of the governance rule dynamic application mechanism.

10. The decision-making substance assurance system according to claim 1, wherein, if the decision-maker is a deliberative body consisting of multiple members, the substance evaluation unit calculates a substance index for the deliberative body as a whole, in addition to the individual substance index for each member.

11. The decision-making substance assurance system according to claim 1, wherein the system estimates the cognitive state of the decision-maker from the browsing behavior patterns recorded by the browsing behavior recording unit, and adaptively adjusts the threshold or the method of presenting the decision materials.

12. The decision-making substance assurance system according to claim 1, wherein the substance record unit generates a report for regulatory authorities based on the accumulated substance records.

13. A method for ensuring the substance of a decision, comprising the steps of: structuring and presenting decision-making materials related to a proposal generated by an AI execution engine to a decision-maker; acquiring the decision-maker's preliminary decision data on the proposal in a recorded state; calculating the degree of discrepancy between the preliminary decision data and the proposal generated by the AI ​​execution engine; recording the decision-maker's viewing behavior regarding the decision-making materials; structurally acquiring the decision-maker's reasons for judgment; calculating a substance index based on the degree of discrepancy, the record of viewing behavior, and the reasons for judgment; and recording a substance record including the substance index in a format that allows for detection of tampering.

14. A program that causes a computer to perform the following processes: structuring and presenting decision-making information regarding a proposal generated by an AI execution engine to a decision-maker; acquiring preliminary decision data of the decision-maker regarding the proposal in a recorded state; calculating the degree of discrepancy between the preliminary decision data and the proposal generated by the AI ​​execution engine; recording the decision-maker's viewing behavior regarding the decision-making information; structurally acquiring the decision-maker's reasons for judgment; calculating a substantiality index based on the degree of discrepancy, the record of viewing behavior, and the reasons for judgment; and recording the substantiality record, including the substantiality index, in a format that allows for tamper detection.

15. The decision-making substance assurance system according to claim 1, wherein even if the substance index falls below a predetermined threshold, the decision-maker is permitted to make a final decision of approval or rejection after going through additional procedures, and the decision and the substance index are recorded in the substance record.

16. The decision-making substance assurance system according to claim 1, wherein the substance recording unit is configured to be in a format that allows for detection of tampering by assigning a hash value to the substance recording and forming a chain structure that includes the hash value of the immediately preceding substance recording.

17. The decision-making substance assurance system according to claim 1, wherein the preliminary decision acquisition unit acquires the preliminary decision data before the recommended content of the proposal generated by the AI ​​execution engine is presented to the decision-maker.

18. The decision-making substance assurance system according to claim 1, wherein the substance evaluation unit quantifies the substance index by weighted averaging of the degree of deviation, the browsing behavior, and the reason for the judgment.

19. The decision-making substance assurance system according to claim 1, wherein the substance recording unit records the substance record in a format that allows for detection of tampering.

20. The decision-making substance assurance system according to claim 19, wherein the substance recording unit records the substance record by hash chain.

21. The decision-making substance assurance system according to claim 1, wherein the structured presentation unit for the decision-making materials visualizes and presents the reasons for the AI ​​execution engine's recommendation.

22. The decision-making substance assurance system according to claim 3, wherein, when the substance evaluation unit determines that the substance index falls below a predetermined standard, it notifies a higher decision-maker of the reason why the substance index fell below the standard.

23. The decision-making substance assurance system according to claim 8, wherein the approval policy management unit, which manages the approval policies, updates the approval policies in cooperation with an external policy library.

24. The decision-making substance assurance method according to claim 13, wherein the step of calculating the substance index includes evaluating the substance index lower if the degree of deviation exceeds a predetermined threshold.

25. A data structure that records the substance of decision-making in an environment where an AI execution engine provides decision support, A case identifier that identifies the decision-making case, The degree of discrepancy between the recommendations made by the aforementioned AI execution engine and the preliminary judgments made by decision-makers, Records of the decision-maker's viewing of decision-making materials, The reasons for the decision made by the aforementioned decision-maker, The degree of discrepancy, the browsing behavior, and the substantiality evaluation index calculated based on the reason for the judgment, A data structure for recording the substance of decision-making, characterized by including the following:

26. The aforementioned AI execution engine provides support for decision-making regarding the operation of the corporation. The aforementioned decision-maker is a natural person who constitutes the decision-making body of the aforementioned legal entity. The decision-making substance assurance system according to claim 1.

27. The decision-making substance record data structure according to claim 25, wherein the case identifier includes the subject of the decision, the decision-maker, and the date and time of the decision.

28. The decision-making substance record data structure according to claim 25, wherein the substance evaluation index is calculated by a weighted average of the degree of deviation, the browsing behavior, and the reason for the judgment.

29. The decision-making substance record data structure according to claim 25, wherein the record of the browsing behavior includes the browsing time, browsing duration, and browsing order.

30. The data structure is a decision-making substance record data structure according to claim 25, wherein the data structure includes a tamper-detectable hash value.

Images