A system and method of assessment of risk and accuracy using a (binary) confusion matrix

Abstract

The present disclosure discloses a novel system (100) and method (200) for assessing risk and accuracy utilizing a binary confusion matrix framework. The system (100) integrates modules for input, assessment, evaluation, and output generation. The input module (108) receives input data from a centralized data repository (106) and feeds into an assessment module (110) equipped with pre-trained models, which select between decision and complimentary decision metrics based on problem-specific requirements. An evaluation module (112) subsequently computes performance metrics, focusing on correctness and error types derived from the binary confusion matrix. The present disclosure includes methods for decision making using decision metrics and, additionally using complementary decision metrics for assessment of risk and accuracy with the metrics balanced over the Z factor for domain-specific applications. The system's (100) structured approach facilitates comprehensive evaluations tailored to specific domains, ensuring robust insights into decision-making. This methodological framework of the present disclosure enhances analytical capabilities across diverse applications requiring detailed assessment of risk and accuracy.

Description

[0001] FIELD OF INVENTION

[0002] The present disclosure generally relates to the field of statistics. More particularly, the present disclosure relates to a system and method for performance assessment of accuracy and associated risk using metric s derived from a binary confusion matrix for application in Geographic Information Systems (GIS), Decision Support Systems (DSS), Artificial

[0003] Intelligence (Al) and Machine T .earning (ML) and Expert Systems (HS) software. The embodiments of the present disclosure can be applied in part or entirely for decision- making, comparison of algorithms, datasets, computations, and so on in many other scientific disciplines.

[0004] BACKGROUND

[0005] The background information herein below relates to the present disclosure but is not necessarily prior art.

[0006] A confusion matrix representing the differences between the expectations and die actuals in a matrix is the standard method of determining the accuracies and related performance metrics for various scientific and engineering applications. A confusion matrix in its simplest form is a binary confusion matrix and its related metrics are commonly used for comparison of different methods of classifications, parametric estimations, prediction algorithms, etc. The standard metrics are based on a confusion matrix even though is highly accepted and widely used amongst the different scientific communities, each has its own advantages and problems.

[0007] There are several well-established standard performance metrics based on a binary confusion matrix, such as accuracy, precision, recall, selectivity, specificity, etc. Apart from these the scientific community has time and again come up with defining a new system of metrics to overcome the disadvantages and the problems of some of these metrics, such as Cohen’s Kappa Coefficient, Fl -Score, Matthews Correlation Coefficient (MCC), etc. However, these standard metrics do not provide a method for accessing the risk and accuracies associated with errors (or incorrectness) and truthness (or correctness) as factors for appropriate decision-making.

[0008] Therefore, there is a need for a new set of formulations of performance metric s based on the binary confusion matrix to provide a new system and method of assessing and evaluating the errors, accuracies, and inaccuracies for informed and critical decisionmaking and computations thereon This set of performance metrics from hereon is referred to as decision metrics which additional includes complimentary decision metrics for a given binary confusion matrix

[0009] OBJECTS

[0010] Some of the objects of the present disclosure, which at least one embodiment herein satisfies, are as follows:

[0011] It is an object of the present disclosure tn ameliorate one or more problems of the prior art or to at least provide a useful alternative.

[0012] An object of the present disclosure is to provide a system and method for performance assessment of accuracy and associated risk using metrics derived from a binary confusion matrix.

[0013] Another object of the present disclosure is to provide a novel technique for assessing the risk and accuracies with a new set of performance metrics formulations based on a binary confusion matrix as decision metrics for appropriate use in systems and applications.

[0014] Another abject of the present disclosure is to provide a new complimentary set of performance metrics formulations as complimentary decision metrics so as to augment the decision metrics tor problems and scenarios that are highly sensitive to errors (or incorrectness) and accuracies (or correctness) respectively.

[0015] Yet another abject of the present disclosure is provide a system of performance metrics for assessment of risk and accuracy as decision metrics and complimentary decision metrics built over the software, product, or algorithm which either improves the user experience, improves the computations, and / or the information presented to the user.

[0016] Other objects and advantages of the present disclosure will be more apparent from the following description, which is not intended to limit the scope of the present disclosure.

[0017] SUMMARY

[0018] The present disclosure envisages a system and method for performance assessment of accuracy and associated risk using metrics derived from a binary confusion matrix. The system comprises several interconnected modules to facilitate this assessment process. The input module retrieves binary confusion matrix data from a centralized data repository. This data serves as the foundation for subsequent analysis.

[0019] The assessment module then applies a pre-trained assessment model to the input data, selecting between decision risk metrics and complimentary decision risk metrics, as well as decision accuracy metrics and complimentary decision accuracy metrics with a configurable Z factor. This module may dynamically adjust its approach based on the specific requirements of the problem domain and data.

[0020] The evaluation module employs another pre-trained model to calculate performance metrics. The required decision metrics are computed based on the True Positives, True

[0021] Negatives, False Positives, and False Negatives in the binary confusion matrix The output module synthesizes and presents the computed performance metrics derived from the binary confusion matrix and additional computations thereon.

[0022] The present disclosure facilitates the utilization of the binary confusion matrix, which forms th e basis for assessing the risk and accuracy between actual and decision outcomes for decision-making. Decision metrics compute risk and accuracy based on incorrectness, while complimentary decision metrics compute risk and accuracy based on correctness.

[0023] The present disclosure also envisages a method for assessment of risk, and accuracy using a binary confusion matrix and related decision and complementary decision performance metrics, comprising the following method steps: e receiving, by an input module implemented through a hardware interface device, a set of input data pertaining to a binary confusion matrix for assessment of risk and accuracy from a data repository; e retrieving, by an assessment module, a pre-trained assessment model from the data repository; implementing, by the assessment module, the assessment model on the set of input data received by the input module for assessment by performing assessments on the set of input data based on: o selecting decision risk metrics or complementary decision risk metrics; o selecting decision accuracy metrics or complementary decision accuracy metrics; o choosing the type of assessment as either based on actuals, decisions, predictions, or outcomes, based on the problem and domain of application; and o selecting the Z factor value based on the problem and domain of application; retrieving, by an evaluation module, the pre-trained evaluation model from the data repository and implementing the evaluation model on the input data assessed by the assessment module so as to perform the computations of the decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics; computing, by the evaluation module, the decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics based on the selection and the type of formulated problem; generating, by an output module implemented through a hardware interface device, a solution that includes the computed decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics based on the binary confusion matrix and additional computations; and presenting, by the output module implemented through a hardware interface device, the results of the assessment of risk and accuracy. BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWING

[0024] A system and method for performance assessment of accuracy and associated risk using metrics derived from a binary confusion matrix of the present disclosure will now be described with the help of the accompanying drawing, in which:

[0025] Figure 1 illustrates a block diagram of a system for performance assessment of accuracy and associated risk using metrics derived from a binary confusion matrix, in accordance with an embodiment of the present disclosure;

[0026] Figure 2 illustrates a flow chart of a method for perfbnnance assessment of accuracy and associated risk using metrics derived from a binary confusion matrix, in accordance with an embodiment of the present disclosure;

[0027] Figure 3 illustrates a flow diagram depicting the steps involved in the method for performance assessment of accuracy and associated risk using metrics derived from a binary confusion matrix, in accordance with an embodiment of the present disclosure; and

[0028] Figure 4 illustrates a decision accuracy scenario of a binary decision confusion matrix, in accordance with an embodiment of the present disclosure.

[0029] LIST OF REFERENCE INDICES

[0030] DETAILED DESCRIPTION

[0031] Embodiments, of the present disclosure, will now be described with reference to the accompanying drawing.

[0032] Embodiments are provided so as to thoroughly and fully convey the scope of the present disclosure to the person skilled in the art. Numerous details are set forth, relating to specific components, and methods, to provide a complete understanding of embodiments of the present disclosure. It will be apparent to the person skilled in the art that the details provided in the embodiments should not be construed to limit the scope of the present disclosure. In some embodiments, well-known processes, well-known apparatus structures, and well-known techniques are not described in detail.

[0033] The terminology used, in the present disclosure, is only for the purpose of explaining a particular embodiment and such terminology shall not be considered to limit the scope of the present disclosure. As used in the present disclosure, the forms "a," "an," and "the" may be intended to include the plural forms as well, unless the context clearly suggests otherwise. The terms "including” and “having,” are open-ended transitional phrases and therefore specify the presence of stated features, integers, steps, operations, elements. and / or components, but do not forbid the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The particular order of steps disclosed in the method and process of the present disclosure is not to be construed as necessarily requiring their performance as described or illustrated.

[0034] It is also to he understood that additional or alternative steps maybe employed. When an element is referred to as being "engaged to,” "connected to," or "coupled to" another element, it may be directly engaged, connected, or coupled to the other element. As used herein, the term "and / or” includes any and all combinations of one or more of the associated listed elements.

[0035] A confusion matrix representing the differences between the expectations and the actuals in a matrix is the standard method of determining the accuracies and related performance metrics for various scientific and engineering applications. A confusion matrix in its simplest form is a binary contusion matrix and its related metrics are commonly used for comparison of different methods of classifications, parametric estimations, prediction algorithms, etc. The standard metrics are based on a confusion matrix even though is highly accepted and widely used amongst the different scientific communities, each has its own advantages and problems.

[0036] There are several well-established standard performance metrics based on a binary confusion matrix, such as accuracy, precision, recall, selectivity, specificity, etc. Apart from these the scientific community has time and again come up with defining a new system of metrics to overcome the disadvantages and the problems of some of these metrics, such as Cohen’s Kappa Coefficient, Fl-Score, Matthews Correlation Coefficient (MCC), etc.

[0037] However, these standard metrics do not provide a method for accessing the risk and accuracies associated with errors (or incorrectness) and truthness (or correctness) as factors for appropriate decision-making.

[0038] Therefore, there is a need for a new set of formulations of performance metrics based on the binary confusion matrix to provide a new system (hereinafter referred to as “system 100”) and method (hereinafter referred to as “method 200**) of assessing and evaluating the errors, accuracies, and inaccuracies for informed and critical decision-making and computations thereon. This set of performance metrics from hereon is referred to as decision metrics and complimentary decision metrics. The present disclosure is anew set of formulations in a method for a system of assessment of risk and accuracy in systems and applications based on a binary confusion matrix, in general, it is essential to know the accuracy of the solution to any problem involving computations, algorithms, predictions, classifications, etc. Also, it is sometimes essential to know the inaccuracies in the solutions to problems and evaluate them in terms of the associated risk. The risk and accuracy associated can also he used for comparison in evaluating the difference between foe performance of processes, algorithms, datasets, etc.

[0039] This performance evaluation is usually done using a confusion matrix and related performance metrics A confusion matrix in its simplest state is a binary confusion matrix that comes with a set of well-known performance metrics, however, each of these metrics has its own disadvantages. The present disclosure is a set of new formulations based on a binary confusion matrix, that includes decision risk, decision accuracy, complimentary decision risk, and complimentary decision accuracy to evaluate the risk and accuracy from both the perspectives of actuals and decisions (or predictions or outcomes)

[0040] System 100 will now be described with reference to Figure 1, and method 200 will be described with reference tn Figures 2 - 3.

[0041] Referring to Figure 1 illustrates a block diagram of system 100 for performance assessment of accuracy and associated risk using metrics derived from a binary confusion matrix, in accordance with an embodiment of the present disclosure. System 100 comprises a microprocessor 104, a data repository 106, an input module 108 implemented through aa hhaarrddwwaarere iinntteerrllaaccee ddeevviiccee., an output module 114 implemented through a hardware interlace device, and a central server 116 communicatively connected to the input module 108. The central server H6 is implemented as a microprocessor 104 and includes an assessment module 110, an evaluation module 112.

[0042] The data repository 106 is configured to store a set of pre-defined instructions and pre- trained modules. The processor 104 is communicatively coupled to the data repository

[0043] 106 configured to execute one or more processing units using the set of predefined instructions for performance assessment of accuracy and associated risk using metrics derived from a binary confusion matrix. The one or more modules include foe input module 108, the assessment module 110, the evaluation module 112, and the output module 114. Among other capabilities, the processor 104 may fetch and execute the set of predefined instructions stored in the data repository 106. The functions of the processor 104 may be provided through the use of dedicated hardware as well as hardware capable of executing the set of predefined instructions. In an aspect, the processor 104 may be a processor with a memory 102. The processor may be implemented as microprocessors, microcontrollers, microcomputers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions.

[0044] The input module 108 implemented through the hardware interface device, which when executed by the microprocessor 104 is configured to receive a set of input data from a data repository 106. The input data pertaining to a binary confusion matrix for assessment of risk and accuracy.

[0045] The assessment module 110, which when executed by the microprocessor 104, is configured to retrieve a pre-trained assessment model from the data repository 106 and implement the pre-trained assessment model on the set of input data received by the input module 108 for assessment by performing assessments on the set of input data based on selecting decision risk metrics or complimentary decision risk metrics and decision accuracy metrics or complimentary decision accuracy metrics. The assessment module 110 is further configurable to select the type of assessment as to either assessment based on actuals or based on decisions or predictions or outcomes arrived based on the problem and the domain of application; and select the Z factor value based on the problem and the domain of application.

[0046] The evaluation module 112, which when executed by the microprocessor 104, is configured to retrieve the pre-trained evaluation model from the data repository 106 and implement the pre-trained evaluation model on the input data assessed by the assessment module 110 so as to perform the computations of the decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics. The evaluation module 112 is further configured to compute the decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics based on the selection and the type of formulated problem.

[0047] The output module 114 implemented through the hardware interface device, which when executed by the microprocessor 104, is configured to generate a solution of computed decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics based on the binary confusion matrix and additional computations so as to present the results of the assessment of risk and accuracy.

[0048] Figure 2 illustrates a workflow flow diagram of method 200 in system 100 of assessment of risk and accuracy using a binary confusion matrix and related decision and complimentary decision performance metrics, in accordance with an embodiment of the present disclosure. The structure of a binary confusion matrix is a 2 by-2 matrix that maps the decisions (or predictions, or outcomes) against the actuals. The actual states are true representing the correctness or agreement between the actuals and decisions, and false representing the incorrectness or disagreement between the actuals and decisions The decision states are any binary states of the subject of interest such as positive or negative as used here for illustration.

[0049] Referring to Figure 3 illustrates a flow diagram depicting the steps involved in method

[0050] 200 for performance assessment of accuracy and associated risk using metrics derived from a binary confusion matrix, in accordance with an embodiment of the present disclosure The order in which method 200 is described is not intended to be construed as a limitation, and any number of the described method steps maybe combined in any order to implement method 200, or an alternative method. Furthermore, method 200 may be implemented by processing resource or computing device(s) through any suitable hardware, non-transitory machine-readable medium / instructions, or a combination thereof. The method 200 comprises the following steps: (step 201) receiving, by an input module 108 implemented through a hardware interface device, a set of input data pertaining to a binary confusion matrix for assessment of risk and accuracy from a data repository 106;

[0051] (step 202) retrieving, by an assessment module 110, the pre-trained assessment model from the data repository 106;

[0052] (step 203) implementing, by the assessment module 110, the assessment model oil the set of input datareceived by the input module 108 for assessment by performing assessments on the set of input data based on: o selecting decision risk metrics or complementary decision risk metrics; o selecting decision accuracy metrics or complementary decision accuracy metrics; o choosing the type of assessment as either based on actuals, decisions, predictions, or outcomes, based on the problem and domain of application; and o selecting the Z factor value based on the problem and domain of application" and

[0053] (step 204) retrieving, by an evaluation module 112, the pre-trained evaluation model from the data repository 106 and implementing the evaluation model on the input data assessed by the assessment module 110 so as to perform the computations of the decision risk metrics or complementary decision risk metric s and decision accuracy metrics or complementary decision accuracy metrics:

[0054] (step 205) computing, by the evaluation module 112, the decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics based on the selection and the type of formulated problem;

[0055] (step 206) generating, by an output module 114 implemented through a hardware interface device, a solution that includes the computed of the decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics based on the binary confusion matrix and additional computations; and • (step 207) presenting, by the output module 114 implemented through a hardware interface device, the results of the assessment of risk and accuracy.

[0056] In an embodiment, the decision risk metrics or complementary decision risk metrics include a decision risk positive ( ), a decision risk negative a complementary decision risk positive and a complementary decision risk negative

[0057] In an embodiment, the decision accuracy metrics or complementary decision accuracy metrics include a decision accuracy present (actual) a decision accuracy present (predictor) a decision accuracy absent (actual) a complementary decision accuracy positive and a complementary decision accuracy negative

[0058] In an embodiment, the results of the assessment of risk and accuracy presented by the output module 114 include a total decision accuracy (DA), and a total complementary decision accuracy

[0059] In an embodiment, the binary confusion matrix elements True Positive (TP) and True Negative (TN) along the matrix diagonal represent the measure of correctness when both the actuals and decisions agree with respect to their states respectively.

[0060] In an embodiment, the binary confusion matrix elements False Positive (FP) and False

[0061] Negative (FN) across the matrix diagonal represent the measure of incorrectness when both the actuals and decisions disagree with respect to their states respectively.

[0062] The decision risk positive is computed by the evaluation module 112 using the equation:

[0063] Where, Fc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

[0064] The decision risk negative (DR) is computed by the evaluation module 112 using the equation:

[0065] TP = True Positives;

[0066] FP = False Positives;

[0067] FN = False Negatives; and

[0068] Fc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

[0069] The complementary decision risk positive (DR+Comp) is computed by the evaluation module 112 using the equation:

[0070] Where,

[0071] TP = True Positives;

[0072] TN = True Negatives;

[0073] FP = False Positives; and

[0074] Fc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

[0075] The complementary decision risk negative is computed by the evaluation module 112 using the equation: 5

[0076] Where,

[0077] TP = True Positives;

[0078] TN = True Negatives;

[0079] FN = False Negatives; and

[0080] Fc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

[0081] The decision accuracy present (actual) (DA+Actual) is computed by tide evaluation module 112 using the equation:

[0082] Where,

[0083] TP = True Positives;

[0084] TN = True Negatives;

[0085] FN = False Negatives; and

[0086] Fc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

[0087] The decision accuracy present (predictor) is computed by the evaluation module 112 using the equation:

[0088] TP

[0089] Where,

[0090] TP = True Positives; FP = False Positives; and

[0091] Fc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

[0092] The decisian accuracy absent (actual) (DA+Actual) is computed by the evaluation module 112 using the equation:

[0093] Where,

[0094] TN = True Negatives;

[0095] FP = False Positives;

[0096] FN = False Negatives; and

[0097] Fc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

[0098] The complementaiy decisian accuracy positive ) is computed by the evaluation module 112 using the equation:

[0099] Where,

[0100] TP = True Positives;

[0101] TN = True Negatives;

[0102] FP = False Positives;

[0103] FN = False Negatives; and Fc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

[0104] The complementary decision accuracy negative is computed! by the evaluation module 112 usin g the equation:

[0105] Where,

[0106] TP = True Positives;

[0107] TN = True Negatives;

[0108] FP = False Positives;

[0109] FN = False Negatives; and

[0110] Fc= 8 / 3 factor which is a weighting factor used to balance- the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

[0111] The total decision accuracy (DA) is computed by the evaluation module 112 using the equation:

[0112] Where,

[0113] DA+= Decision Accuracy Positive; and

[0114] DA- = Decision Accuracy Negative.

[0115] The total complementary decision accuracy is computed! by the evaluation module 112 using the equation:

[0116] Where,

[0117] Example:

[0118] Here is a simulated scenario for decision-making to evaluate the predictor's performance nd guide further computations. As shown in Figure 4, it illustrates a decision accuracy cenario of a binary decision confusion matrix. Consider the binary confusion matrix elow, which maps the presence of mineral deposits against the predictions made by nstructions.

[0119] Table 1 Standard and Decision Accuracy Metrics Computations for Scenario A

[0120] Decision Making Uusteations for Computation of Decision Metrics:

[0121] Additionally, Table 1 displays the computations for the safe estimate based on absolute ccuracy and the best estimate based on decision accuracy. This is derived from the nterpretation of actual values and predictor decision accuracy, where the predictor's ecision accuracy of 0.713 exceeds the actual decision accuracy of 0.653. The system's 100 structured approach facilitates comprehensive evaluations tailored to pecific domains, ensuring robust insights into decision-making accuracy and risk assessment. This methodological framework of the present disclosure enhances analytical apabilities across diverse applications requiring precise performance measurement and assessment of outcomes n an embodiment, the present disclosure provides a new method 200 of assessment of isk and accuracy based on a binary confusion matrix On a part, the present disclosure provides a set of metrics based on two different cost functions. In an other part, the new method of the disclosure enables further computations based on decision metrics and omplimentary decision metrics. n an aspect, the present disclosure improves the performance assessments for risk and accuracy based on a binary confusion matrix which seems so for unavailable. n an embodiment, a novel technique for assessing the risk and accuracies with a new set of performance metrics formulations based on a binary confusion matrix in systems and applications are performed by the mathematical expressions implemented in one or more omputer-usable storage media (including but not limited to a disk memory, a CD-ROM, optical memory, and the like) and / or one or more modules that include computer-usable program code. n an embodiment, the mathematical expressions herein are the same as those described above and may be implemented by using a processor. A function of the processing element may be the same as a function of the mathematical expression performed by the processor using machine learning techniques and / or artificial intelligence as described in Figure 2. n an aspect, it can be Learned that the mathematical expressions may perform some or all teps performed by one or more modules in a first mann er , to be specific, by invoking the program stored in the storage element; or may perform some or all steps performed by he device in a second manner, to be specific, by using a hardware integrated Logic circuitn the processing element in combination with instructions; or may certainly perform, by combining the first manner and the second maimer, some or all steps performed by the network device.

[0122] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored on or transmitted over as one or morenstructions or code oonn aa computer-readable medium. Other examples and implementations are within the scope and spirit of the disclosure and appended claims, For example, due to the nature of software, functions described above can be implemented using software executed by a processor, hardware, firmware, hardwiring, or combinations of any of these. Features implementing functions may also be physically located at various positions, including being distributed such that portions of functions are implemented at different physical locations.

[0123] The foregoing description of the embodiments has been provided for purposes of illustration and is not intended to limit the scope of the present disclosure. Individual components of a particular embodiment are generally not limited to that particular embodiment but are interchangeable. Such variations are not to be regarded as a departure from the present disclosure, and all such modifications are considered to be within the scope of the present disclosure.

[0124] TECHNICAL ADVANCEMENTS

[0125] The present disclosure described herein above has several technical advantages including, but not limited to, the realization of a system and method for assessment of risk and accuracy using a binary confusion matrix that:

[0126] • provides a method of assessment of risk and accuracy based on a binary confusion matrix;

[0127] • provides a mathematical formulation for assessment of risk and accuracy based on binary confusion matrix and using decision metrics and complimentary decision metrics that can be directly applied and extended for different applications; and

[0128] • can be directly applied to implementations in software, algorithms, and products of Geographic Information Systems (GIS), Decision Support Systems (DSS), Artificial Intelligence (Al), Machine Learning (ML), and Expert Systems (ES).

[0129] The embodiments herein and the various features and advantageous details thereof are explained with reference to the non-limiting embodiments in the following description. Descrip tions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

[0130] The foregoing description of the specific embodiments so fully reveals the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and / or adapt for various applications such specific embodiments without departing from the generic concept and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the embodiments as described herein.

[0131] The use of the expression “at least” or “at least one” suggests the use of one or more elements or ingredients or quantities, as the use may be in the embodiment of the disclosure to achieve one or more of the desired objects or results.

[0132] While considerable emphasis has been placed herein on the components and component parts of the preferred embodiments, it will be appreciated that many embodiments can be made and that many changes can be made in the preferred embodiments without departing from the principles of the disclosure. These and other changes in the preferred embodiment as well as other embodiments of the disclosure will be apparent to those skilled in the art from the disclosure herein, whereby it is to be distinctly understood that the foregoing descriptive matter is to be interpreted merely as illustrative of the disclosure and not as a limitation

Claims

WE CLAIM1. A system (100) for assessment of risk and accuracy using a binary confusion matrix and related decision and complementary decision performance metrics, said system (100) comprising: an input module (108) implemented through a hardware interface device configured to receive a set of input data from a data repository (106), said input data pertaining to a binary confusion matrix for assessment of risk and accuracy;■ a central server (116) communicatively connected to said input module (108), wherein said central server (116) is implemented as a microprocessor (104) and includes; an assessment module (110) configured to: o retrieve a pre-trained assessment model from the data repository (106) and implement the pre-trained assessment model on the set of input data received by the input module (108) for assessment by performing assessments on the set of input data based on selecting decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics; o select the type of assessment as to either assessment based on actuals or based on decisions or predictions or outcomes arrived based on the problem and the domain of application; and o select the Z factor value based on the problem and the domain of the application; an evaluation module (112) configured to: o retrieve the pre-trained evaluation model from the data repository (106) and implement the pre-trained evaluation model on the input data assessed by the assessment module (110) so as to perform the computations of thedecision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics; and c compute the decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics based on the selection and the type of formulated problem; and• an output module (114) implemented through the hardware interface device configured to generate a solution of computed decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics based on the binary confusion matrix and additional computations so as to present the results of the assessment of risk and accuracy,2. The system (100) as claimed in claim 1, wherein the decision risk metrics or complementary decision risk metrics include a decision risk positive (DR*), a decision risk negative (DR"), a complementary decision risk positive (DR+camp), and a complementary decision risk negative (-3. Thc system (100) as- claimed in claim 1, wherein the decision accuracy metrics or complementary decision accuracy metrics include a decision accuracy present (actual), a decision accuracy present (predictor)a decision accuracy absent (actual)a complementary decision accuracy positive , and a complementary decision accuracy negative4. The system (100) as claimed in claim 1, wherein the results of the assessment of risk and accuracy presented by the output module (114) include a total decision accuracy (DA), and a total complementary decision accuracy5. The system (100) as claimed in claim 2, wherein the decision risk positive (is computed by the evaluation module (112) using the equation;Where,TN = True Negatives;FP = False Positives;FN = False Negatives; andFc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

6. The system (100) as claimed in claim 2, wherein the decision risk negative (DR-) is computed by the evaluation module (112) using the equation:TP = True Positives;FP = False Positives;FN = False Negatives; andFc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

7. The system (100) as claimed in claim. 2, wherein the complementary decision risk positive (DR+comp) is computed by the evaluation module (112) using the equation:Where,TP = True Positives;TN = True Negatives;FP = False Positives; andFc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

8. The system (100) as claimed in claim 2, wherein the complementary decision risk negative is computed by the evaluation module (1.12) using the equation:Where,TP = True Positives;TN = True Negatives;FN = False Negatives; andFc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

9. The system (100) as claimed in claim 3, wherein the decision accuracy present (actual)is computed by the evaluation module (112) using the equation:Where,TP = True Positives;TN = True Negatives;FN = False Negatives; andFc“ 8 / 3 factor which is a weighting factor used to balance 'the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics* 0. The system (100) as claimed in claim 3, wherein the decision accuracy present (predictor) is computed by the evaluation module (112) using theequation:Where.TP “ True Positives;FP - False Positives; andFc- 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

11. The system (WO) as claimed in claim 3, wherein the decision accuracy absent (actual) is computed by the evaluation module (112) using the equation:Where,TN = True Negatives;FP = False Positives;FN = False Negatives; andFc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics,12. The system (100) as claimed in claim 3, wherein the complementary decision accuracy positive is computed by the evaluation module (112) using theequation:Where,TP = True Positives;TN = True Negatives;FP = False Positives;FN = False Negatives; andFc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics,13. The system (100) as claimed in claim 3, wherein the complementary decision accuracy negativeis computed by the evaluation module (112) using the equation:Where,TP = True Positives;TN = True Negatives;FP = False Positives;FN = False Negatives; andFc= 8 / 3 factor which is a weighting factor used to balance the impact of false negatives (FN) and false positives (FP) in the computation of decision accuracy metrics.

4. The system (100) as claimed in claim 4, wherein the total decision accuracy (DA) is computed by the output module (110) using the equation:Where,5. The system (100) as claimed in claim 1, wherein the total complementary decision accuracy (DAcymp) is computed by the output module (114) using the equation;Where,6. The system (100) as claimed in claim 1, wherein said system (100) further comprises:* a data repository (106) configured to store a set of pre-defined instructions and pre-trained modules;* a microprocessor (104) communicatively coupled to the data repository (106) configured to execute one or more processing units using the set of predefined instructions for performance assessment of accuracy and associated risk,wherein one or more m odules include the input module (108), the assessment module (110), the evaluation module (112), and the output module (114).17, A method (200) for assessment of risk and accuracy using a binary confusion matrix and related decision and complementary decision performance metrics, comprising the following method steps: receiving (201), by an input module (108) implemented through a hardware interface device, a set of input data pertaining to a binary confusion matrix for assessment of risk and accuracy from a data repository (106); retrieving (202), by an assessment module (110), a pre-trained assessment model from the data repository (106); implementing (203), by the assessment module (110), the assessment model on the set of input data received by the input module (108) for assessment by performing assessments on the set of input data based on: o selecting decision risk metrics ot complementary decision risk metrics; o selecting decision accuracy metrics or complementary decision accuracy metrics; o choosing the type of assessment as either based on actuals, decisions, predictions, or outcomes, based on the problem and domain of application; and o selecting the Z factor value based on the problem and domain of application; and6 retrieving (204), by an evaluation module (112), the pre-trained evaluation model from the data repository (106), and implementing the evaluation model on the input data assessed by the assessment module (110) so as to perform the computations of the decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics;o computing (205), by the evaluation module (112), the decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics based on the selection and the type of formulated problem; generating (206), by an output module (114) implemented through a hardware interface device, a solution that includes the computed decision risk metrics or complementary decision risk metrics and decision accuracy metrics or complementary decision accuracy metrics based on the binary confusion matrix and additional computations; and presenting (207), by the output module (114) implemented through a hardware interface device, the results of the assessment of risk and accuracy.

Images