A control method, system, and program for evaluation deferral based on unverifiableness under finite operating conditions.

Abstract

Under conditions where the number of observations, time, and computing resources are finite, the evaluation results obtained are Even if it is not possible to fully determine whether or not it is verifiable, the evaluation will be finalized immediately. Conventional methods have risks such as misjudgment, unclear attribution of responsibility, and significant risks in subsequent operations. There was a problem that arose. [Solution] The present invention determines verifiability based on input observation data or trial results, If it is difficult to finalize the evaluation, an evaluation deferral token will be issued to delay the finalization of the evaluation. This invention provides an evaluation confirmation delay control method and system for switching to a limited operating mode. This avoids incorrect evaluations even under finite observation conditions, and after the conditions for re-evaluation are met... This enables accurate and reliable evaluation.

Description

Technical Field

[0001] The present invention relates to a control technique for handling situations where evaluation cannot be verified under constraints such as limited observation resources, number of trials, time, measurement accuracy, etc. In particular, it relates to a method, system, and program for continuing safe and explainable operation while withholding the determination of evaluation results.

Background Art

[0002] In advanced and complex information processing systems, control systems, medical devices, financial systems, AI decision-making, quantum and advanced measurement, etc., the state space of the evaluation target has become large-scale, and situations where performance, safety, reliability, etc. cannot be fully verified under limited observation resources have become the norm. Conventionally, there are many operations of (i) "definitely" handling the estimated value from limited data and operating it, or (ii) implicitly postponing it. However, there has been a problem that it is impossible to explain "why it was determined / why it was postponed" in situations such as accidents, audits, introduction of regulations, and pursuit of responsibility, and the allocation of evaluation and responsibility fails.

Summary of the Invention

Problems to be Solved by the Invention

[0003] To provide a technical framework that enables continuous operation without determining the evaluation result when evaluation cannot be verified under limited operation and observation conditions, and enables future re-evaluation, audit, and explanation.

Means for Solving the Problems

[0004] The evaluation withholding control method according to the present invention comprises the following. (1) Verification impossibility determination: For the evaluation result of the target system, a verification possibility score S is calculated based on the number of observations, reproducibility, measurement error, observation window length, boundary visibility index, etc., and it is determined that verification is impossible when S is less than the threshold value T. (2) Evaluation pending: If it is determined that verification is impossible, the finalization of the evaluation result will be suspended and an "evaluation pending token" will be issued (the token is an identifier that refers to the reason for suspension, conditions, and re-evaluation conditions). (3) Operational switching: Based on the evaluation pending token, the system switches to maintenance mode, degraded mode, audit waiting mode, retry mode, etc. (4) Records: Record the reason for holding, the indicators used, the observation conditions, the calculated score, the threshold, the re-evaluation conditions, and the deadline (expiration conditions). [Effects of the Invention]

[0005] (1) It helps avoid misjudgments and overconfidence based on unverifiable evaluations. (2) At the time of introducing post-audits, regulations, and standards compliance, the basis for determining the evaluation (conditions, indicators, and re-evaluation conditions) can be presented. (3) It is applicable regardless of the field, and improves operational stability and explainability. [Brief explanation of the drawing]

[0006] [Figure 1] This schematic diagram illustrates a processing flow that, under finite observation conditions, determines verifiability based on input observation data or trial results, and either confirms the evaluation or delays the confirmation of the evaluation by issuing an evaluation deferral token. [Figure 2] This block diagram shows an example configuration of an evaluation confirmation delay control system that includes a verifiability determination unit, an evaluation postponement unit, an operation mode switching unit, and a recording unit, and controls whether to confirm or postpone the evaluation. [Modes for carrying out the invention]

[0007] (common definition) • Verifiability score S: A real number ranging from 0 to 1, with a value closer to 1 indicating higher verifiability. • Threshold T: Set according to the operational policy (e.g., T=0.8). • Re-evaluation conditions: Expressed as the number of additional observations N_add, observation window length W_min, error limit E_max, etc. · Conservative mode: Operations that reduce risks, such as setting to the safe side, low speed, and redundancy. · Degraded mode: Operations that limit the impact through function limitation, output suppression, step stop, etc. · Audit pending mode: Operations that prohibit the external presentation of definitive values until external audit and approval are obtained. · Retry mode: Operations that prioritize additional data acquisition and re - execution.

Example

[0008] As an example, the accuracy of the predicted value y estimated by the machine learning model is evaluated using a finite amount of verification data D_val (e.g., 100 samples). (1) Reproducibility R: Calculated from the variance in bootstrap re - estimation (e.g., R = 0.6). (2) Error upper limit E: Confidence interval width (e.g., E = ±12%). (3) Score S: S = w1 x R+w2 x (1 - E / E_ref) (e.g., S = 0.72, T = 0.8). Since S < T, the evaluation is not finalized, an evaluation hold token is issued, and a transition is made to the audit pending mode (prohibiting external presentation). The record includes the size of D_val, R, E, S, T, and re - evaluation conditions (additional samples N_add = 400, E ≤ ±5%).

[0009] An example where the state estimation x of the controlled object becomes unverifiable due to sensor failure or disturbance. When the observation window length W is short (e.g., W = 0.5 seconds), the upper limit of the estimation error cannot be evaluated. When it is determined that S < T, the control gain is switched to the conservative side (gain reduction, speed limit), and the degraded mode is entered. As re - evaluation conditions, record W ≥ 5 seconds, the return of the sensor redundancy system, and the convergence conditions of disturbance estimation.

[0010] In diagnostic support, a situation where there is a high risk of misdiagnosis when making a definitive diagnosis based on only one image + medical interview. The verifiability is calculated from data sufficiency (missing rate of inspection items), reproducibility (secondary read consistency), and error upper limit (uncertainty estimation). In the case of S < T, the definitive diagnosis is withheld, and additional examinations (blood tests, re-imaging, etc.) are recorded as re-evaluation conditions. The operation transitions to the audit-wait (physician confirmation) mode.

[0011] Examples where risk estimation such as VaR becomes unverifiable due to regime changes. The frequency of extreme events cannot be evaluated with short-term data, and the upper limit of the estimation error is large. In the case of S < T, the position limit is reduced (degenerate), and the hedge ratio is adjusted to the conservative side. As re-evaluation conditions, record the observation of an additional period, the condition for the recovery of the correlation structure stability, etc.

[0012] If the validity of the inference result for the input z outside the learning distribution cannot be verified, stop external presentation and automatic execution. The score S is calculated from the OOD distance, reproducibility (ensemble consistency), and error upper limit (uncertainty), and if S < T, it transitions to the human confirmation mode (audit-wait).

Industrial Applicability

[0013] The present invention can be used in all fields where unverifiability becomes an operational problem, such as information processing, control, medical, finance, AI, advanced measurement, etc.

Claims

1. An evaluation suspension control method characterized by determining whether the evaluation results of a target system can be verified under conditions of finite observational resources, and switching the operating mode while suspending the confirmation of the evaluation results if it is determined that verification is not possible.

2. The method according to claim 1, wherein the determination involves calculating a verifiability score based on at least one of the number of observations, reproducibility, measurement error, observation window length, and boundary visibility index, and determining that verification is not possible if the score is below a predetermined threshold.

3. The method according to claim 1 or 2, wherein the operating mode includes at least one of maintenance mode, degraded mode, audit waiting mode, and retry mode, and is selected in response to the determination of unverifiable status or the evaluation being held back.

4. The method according to any one of claims 1 to 3, characterized in that when the determination of the evaluation result is withheld, the indicators used, observation conditions, calculated score, threshold, conditions for future re-evaluation, and deadline or expiration conditions are recorded.

5. An evaluation hold control system comprising an unverifiableness determination unit, an evaluation hold unit, an operation switching unit, and a recording unit, which performs the method described in any one of claims 1 to 4.

6. A program for causing a computer to perform the method described in any one of claims 1 to 4.

Images