Computer-implemented method, information processing system, and computer program (detection of non-inferable data)

Abstract

To provide an approach in which a method, a system and a program product identify a plurality of models for testing a set of data.SOLUTION: Each one of a plurality of models produces one of a plurality of predictions corresponding to one of a plurality of targets. The method, system and program product detect one or more conflicts between the plurality of predictions in response to testing the set of data for each of the plurality of models. The method, system, and program product report an uninferable result of the testing in response to the detecting the one or more conflicts.SELECTED DRAWING: Figure 3

Description

[Background technology]

[0001] Artificial intelligence uses machine learning algorithms to build models based on sample data (training data) and make predictions or judgments about topics without being explicitly programmed to do so. Machine learning algorithms are used in a wide variety of applications where it is difficult or infeasible to develop traditional algorithms to perform the required tasks.

[0002] The accuracy level of a machine learning model is based on its "true positives," "true negatives," "false positives," and "false negatives." A true positive is an outcome where the machine learning model correctly predicts the positive class. A true negative is an outcome where the machine learning model correctly predicts the negative class. A false positive is an outcome where the machine learning model incorrectly predicts the positive class. And a false negative is an outcome where the machine learning model incorrectly predicts the negative class.

[0003] When a machine learning model produces a false positive result, it may be attempting to predict an outcome that is not predictable, referred to herein as "un-inferable." A machine learning model is required to predict a specific outcome, even if the prediction has low confidence. When a system uses multiple machine learning models to arrive at a final result, a user cannot distinguish whether there is a conflict between the individual results of different machine learning models, which subsequently produces a false positive final result. Workaround techniques exist, such as creating an "other" class of results, but these techniques do not work for binary classification. Summary of the Invention [Problem to be solved by the invention]

[0004] According to one embodiment of the present disclosure, a method, system, and program product are provided for identifying multiple models for testing a set of data. [Means for solving the problem]

[0005] Each one of the plurality of models generates one of a plurality of predictions corresponding to one of a plurality of targets. The method, system, and program product detects one or more conflicts between the plurality of predictions in response to testing the set of data against each of the plurality of models. The method, system, and program product reports an uninferable result of the test in response to detecting one or more conflicts.

[0006] According to another embodiment of the present disclosure, a method, system, and program product generates a strong first prediction corresponding to a first target of a plurality of targets using a first one of the models, generates a strong second prediction corresponding to a second target of the plurality of targets using a second one of the models, and then generates a non-inferable result in response to determining that the first target is different from the second target.

[0007] According to yet another embodiment of the present disclosure, a method, system, and program product are provided that determine a strong first prediction based on a first mean + 2 standard deviation area on a first probability curve corresponding to a first model, and a strong second prediction based on a second mean + 2 standard deviation area on a second probability curve corresponding to a second model.

[0008] According to yet another embodiment of the present disclosure, a method, system, and program product is provided for constructing multiple models based on a set of training data. The method, system, and program product calculates, for each of the multiple models, one of multiple model evaluation measures that measures the performance of one of the multiple models. The method, system, and program product then selects a subset of models (K) from the multiple models based on their corresponding model evaluation measures, where the K models include a set of important features.

[0009] According to yet another embodiment of the present disclosure, a method, system, and program product ranks the set of significant features corresponding to the K models, and identifies a set of differential features based on the ranking. For each of the set of distinct features, the method, system, and program product selects one of the set of distinct features and removes the portion of the training data that corresponds to the selected distinct feature.

[0010] The method, system, and program product tests each of the K models against a subset of the training data excluding the removed portion of the training data, selects one of the K models based on the testing, designates the selected K models as one of a set of S models, and utilizes the set of S models during testing of the set of data to detect one or more conflicts.

[0011] According to yet another embodiment of the present disclosure, a method, system, and program product determines a confidence threshold for each of the S models in the set of S models, and then uses the confidence threshold to determine whether one or more of the predictions is a strong prediction.

[0012] According to yet another embodiment of the present disclosure, a method, system, and program product are provided for determining that a plurality of predictions includes a plurality of strong first predictions, each corresponding to a first target among the plurality of targets. The method, system, and program product determines that the plurality of predictions includes a single strong second prediction corresponding to a second target among the plurality of targets. The method, system, and program product then reports an uninferable result in response to determining that the first target is different from the second target.

[0013] The foregoing is a summary and, as such, necessarily contains simplifications, generalizations, and omissions of detail, so that those skilled in the art will appreciate that the summary is merely illustrative and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the present disclosure, as defined solely by the claims, will become apparent in the non-limiting detailed description set forth below. [Brief explanation of the drawings]

[0014] The present disclosure may be better understood, and its numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings. [Figure 1] 1 is a block diagram of a data processing system in which the methods described herein may be implemented. [Figure 2] FIG. 2 provides an extension of the information processing system environment shown in FIG. 1 to illustrate that the methods described herein can be performed on a wide variety of information processing systems operating in a networked environment. [Figure 3] FIG. 1 is an exemplary diagram illustrating a system that generates machine learning models, selects a subset of the machine learning models for conflict analysis, and uses the selected machine learning models to determine whether output results are inferable. [Figure 4] 1 is an exemplary flowchart illustrating steps taken to evaluate predictive models and select the best predictive model for competitive analysis. [Figure 5] 1 is an exemplary flowchart showing steps taken in a leave-one-out cross-validation process to select a group of models for competitive analysis. [Figure 6] 10 is an exemplary flowchart illustrating steps taken during runtime processing to determine whether any strong conflicts arise during conflict analysis. [Figure 7] FIG. 1 is an exemplary diagram illustrating training data including a set of features and targets. [Figure 8]FIG. 10 is an exemplary diagram illustrating confidence thresholds for S models. [Figure 9] FIG. 1 is an exemplary diagram illustrating a predictive model decision tree including strong prediction confidence nodes and weak to medium prediction confidence nodes. [Figure 10] FIG. 10 is an exemplary diagram illustrating various score data model results. DETAILED DESCRIPTION OF THE INVENTION

[0015] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0016] In the following claims, equivalent structure, material, acts, and all means-plus-function or step-plus-function element equivalents are intended to include any structure, material, or act for performing a function in combination with other claimed elements as specifically claimed. The description of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the disclosure. The embodiments have been chosen and described to best explain the principles and practical application of the disclosure and to enable others skilled in the art to understand the disclosure in various embodiments with various modifications to suit the particular use contemplated.

[0017] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions that cause a processor to perform aspects of the present invention.

[0018] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves that record instructions, and any suitable combination of the foregoing. Computer-readable storage medium, as used herein, is not to be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0019] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computing / processing device for storage.

[0020] The computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, etc., and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the present invention.

[0021] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0022] These computer-readable program instructions may be provided to a computer processor or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the computer processor or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium, whereby the instructions can instruct a computer, programmable data processing apparatus or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0023] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of the flowcharts or block diagrams, or both.

[0024] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions, that implement the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be implemented as a single step, or may be executed simultaneously, substantially simultaneously, partially, or fully overlapping in time, or the blocks may even be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions. The following detailed description generally follows from the summary of the disclosure as set forth above and further explains and expands, as needed, on definitions of various aspects and embodiments of the disclosure.

[0025] 1 illustrates an information handling system 100, a simplified example of a computer system capable of performing the computing operations described herein. The information handling system 100 includes one or more processors 110 coupled to a processor interface bus 112. The processor interface bus 112 connects the processors 110 to a northbridge 115, also known as a memory controller hub (MCH). The northbridge 115 connects to system memory 120 and provides a means for the processors 110 to access the system memory. A graphics controller 125 also connects to the northbridge 115. In one embodiment, a peripheral component interconnect (PCI) express bus 118 connects the northbridge 115 to the graphics controller 125. The graphics controller 125 connects to a display device 130, such as a computer monitor.

[0026] Northbridge 115 and Southbridge 135 connect to each other using bus 119. In some embodiments, the bus is a Direct Media Interface (DMI) bus, which transfers data at high speeds in each direction between Northbridge 115 and Southbridge 135. In some embodiments, a PCI bus connects the Northbridge and Southbridge. Southbridge 135, also known as an Input / Output (I / O) Controller Hub (ICH), is a chip that generally implements capabilities that operate at slower speeds than those provided by the Northbridge. Southbridge 135 typically provides various buses used to connect various components. These buses include, for example, PCI and PCI Express buses, ISA buses, System Management Buses (SMBus or SMB), or Low Pin Count (LPC) buses, or combinations thereof. The LPC bus often connects low-bandwidth devices such as Boot ROM 196 and "legacy" I / O devices (which use "super I / O" chips). The "legacy" I / O devices (198) may include, for example, serial and parallel ports, keyboard, mouse, or floppy disk controllers, or a combination thereof. Other components often included within the southbridge 135 include a direct memory access (DMA) controller, a programmable interrupt controller (PIC), and a storage device controller, which connects the southbridge 135 to a non-volatile storage device 185, such as a hard disk drive, using bus 184.

[0027] ExpressCard 155 is a slot that connects hot-pluggable devices to the information handling system. ExpressCard 155 connects to Southbridge 135 using both Universal Serial Bus (USB) and PCI Express buses, thereby supporting both PCI Express and USB connectivity. Southbridge 135 includes USB controller 140, which provides USB connectivity for devices that connect to USB. These devices include a webcam (camera) 150, an infrared (IR) receiver 148, a keyboard and trackpad 144, and a Bluetooth® device 146, which provides a wireless personal area network (PAN). USB controller 140 also provides USB connectivity for a wide variety of other USB-connected devices 142, such as a mouse, a removable non-volatile storage device 145, a modem, a network card, an Integrated Services Digital Network (ISDN) connector, a fax machine, a printer, a USB hub, and many other types of USB-connected devices. Although the removable non-volatile storage device 145 is shown as a USB-connected device, the removable non-volatile storage device 145 can be connected using a different interface, such as a Firewire interface, or the like.

[0028] Wireless local area network (LAN) devices 175 connect to Southbridge 135 via PCI or PCI Express bus 172. LAN devices 175 typically implement one of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards for wireless modulation techniques, all of which use the same protocol for wireless communication between information handling system 100 and another computer system or device. Optical storage devices 190 connect to Southbridge 135 using Serial Analog Telephone Adapter (ATA) (SATA) bus 188. Serial ATA adapters and devices communicate over a high-speed serial link. The Serial ATA bus also connects Southbridge 135 to other forms of storage devices, such as hard disk drives. Audio circuitry 160, such as a sound card, connects to Southbridge 135 via bus 158. Audio circuitry 160 also provides functionality associated with audio hardware, such as audio line-in and optical digital audio input port 162, optical digital output and headphone jack 164, internal speaker 166, and internal microphone 168. Ethernet controller 170 connects to southbridge 135 using a bus such as a PCI or PCI Express bus. Ethernet controller 170 connects information handling system 100 to computer networks such as local area networks (LANs), the Internet, and other public and private computer networks.

[0029] 1 illustrates one information handling system, information handling systems may take many forms. For example, an information handling system may take the form of a desktop, server, portable, laptop, notebook, or other form factor computer or data processing system. In addition, an information handling system may take other form factors, such as a personal digital assistant (PDA), a gaming device, an automated teller machine (ATM), a portable telephone device, a communications device, or other device that includes a processor and memory.

[0030] Figure 2 provides an expansion of the information handling system environment shown in Figure 1 to illustrate that the methods described herein can be performed on a wide variety of information handling systems operating in a networked environment. Types of information handling systems range from small handheld devices, such as handheld computer / cell phone 210, to large mainframe systems, such as mainframe computer 270. Examples of handheld computers 210 include personal entertainment devices, such as personal digital assistants (PDAs), Moving Picture Experts Group Layer-3 Audio (MP3) players, portable televisions, and compact disc players. Other examples of information handling systems include pen or tablet computers 220, laptop or notebook computers 230, workstations 240, personal computer systems 250, and servers 260. Other types of information handling systems not individually shown in Figure 2 are represented by information handling system 280. As shown, various information handling systems can be networked together using computer network 200. Types of computer networks that can be used to interconnect various information handling systems include local area networks (LANs), wireless local area networks (WLANs), the Internet, public switched telephone networks (PSTNs), other wireless networks, and any other network topology that can be used to interconnect information handling systems. Many information handling systems include a non-volatile data store, such as a hard drive or non-volatile memory, or both. The embodiment of the information handling system shown in FIG. 2 includes a separate non-volatile data store (more specifically, server 260 utilizes non-volatile data store 265, mainframe computer 270 utilizes non-volatile data store 275, and information handling system 280 utilizes non-volatile data store 285). The non-volatile data store can be a component external to the various information handling systems or can be internal to one of the information handling systems.Additionally, the removable non-volatile storage device 145 can be shared between two or more information handling systems using various techniques, such as connecting the removable non-volatile storage device 145 to a USB port or other connector on the information handling systems.

[0031] As discussed above, machine learning models are always required to predict outcomes and therefore occasionally produce false positive results. FIGS. 3-10 illustrate a method that can be implemented on an information processing system to determine whether two different machine learning models produce different strong predictions for two different targets. If this occurs, the method produces an uninferable result. As discussed in detail below, the method uses training data to build N models with different parameter settings or model types, or both. For each model, the method calculates a threshold for classifying prediction confidence and selects a set of models (S models) for competitive analysis. The method then analyzes score data using the S models, and if the S models produce strong predictions for different targets, the method outputs an uninferable result.

[0032] FIG. 3 is an exemplary diagram illustrating a system that generates machine learning models, selects a subset of the machine learning models for competitive analysis, and uses the selected machine learning models to determine whether output results are inferable.

[0033] The system 300 uses training data 302 to generate an initial set of predictive models 305, 310, 315, and 320. The system 300 begins a model selection phase 325 using a model evaluation and initial selection stage 330 to select the top-K models 335. During the model evaluation and initial selection stage 330, the system 300 uses a metric to evaluate the classification models, such as using Percent Correction Classification (PCC) or a confusion matrix, or both. Percent Correct Classification (PCC) measures the overall accuracy rate, with all errors having the same weight. A confusion matrix also measures the accuracy rate but distinguishes between errors (e.g., false positives, false negatives, and correct predictions).

[0034] Similarly, the system 300 may use metrics to evaluate regression models, such as R-squared, average error, mean square error (MSE), median error, average absolute error, or median absolute error, or combinations thereof. R-squared produces a goodness-of-fit metric ranging from 0 to 1, with higher values ​​indicating greater coherence and predictive ability of the model. Average error is the numerical difference between predicted and actual values. Mean square error (MSE) may be a preferred approach when there are many outliers in the data. Median error is the average of all differences between predicted and actual values. Mean absolute error is similar to average error, except that the absolute value of the difference balances out outliers in the data. Median absolute error is the average of the absolute differences between predictions and actual observed values. Individual differences have equal weight, allowing large outliers to influence the final evaluation of the model.

[0035] After selecting the top-K models 335, the system 300 then performs a leave-one-out cross-validation stage 340 to determine which of the top-K models 345 should be used in a competitive analysis. FIG. 5 shows the detailed steps of the leave-one-out cross-validation stage 340 and the selection of the S models 345. The confidence threshold calculation stage 350 determines the threshold for each of the S models 345 at which strong predictions begin. The confidence threshold calculation stage 350 may use several techniques to calculate or assign a confidence threshold to a given model. For example, a user may rely on their own domain knowledge to set the confidence threshold, or the confidence threshold calculation stage 350 may calculate the confidence threshold as mean+2std, where "mean" is the average confidence value of the models and "std" is the standard deviation. The system 300 then loads the S models 345 and their corresponding confidence thresholds into the runtime phase 355, shown as models M_1 365, M_2 370, and model M_S 375.

[0036] During runtime phase 355, score data 360 is analyzed by each of S models 365, 370, and 375. Conflict analyzer 380 evaluates the results of the S models and determines output 395. If the outputs of models 365, 370, or 375 produce strong predictions for different targets, such as strong prediction "A" and strong prediction "B," conflict analyzer 380 produces an uninferable result as output 395. For example, if model M_1 365 and model M_2 370 produce strong predictions for target A, but model M_S 375 produces a strong prediction for target B, conflict analyzer 380 outputs an uninferable result (see score data result 1050 in FIG. 10 and corresponding text for further details).

[0037] 4 is an exemplary flowchart showing the steps taken to evaluate predictive models and select the best predictive model for competitive analysis. The processing of FIG. 4 begins at 400, whereupon, at step 410, the process uses training data 302 to build n predictive models.

[0038] In step 420, the process calculates model evaluation metrics and selects the top K models 335. As discussed above, several techniques may be used to evaluate and select the top K models. For example, metrics that can be used to evaluate classification models include percent correct classification (PCC) or a confusion matrix, or both. Metrics that can be used to evaluate a regression model include R-squared, mean error, mean squared error (MSE), median error, mean absolute error or median absolute error, or a combination thereof.

[0039] In predefined process 430, the process performs a leave-one-out cross-validation step for each of the K models and selects the best model (S models) for each leave-one-out feature iteration, resulting in multiple S models (see FIG. 5 and corresponding text for processing details).

[0040] In step 440, the process determines a confidence threshold for each of the S models. For example, data groups with significantly higher confidence are strong, e.g., greater than mean+2*std (see FIG. 8 and corresponding text for further details). In step 450, the process loads the S models 345, along with their corresponding confidence thresholds, into the runtime phase 355, after which the processing of FIG. 4 ends at 495.

[0041] 5 is an exemplary flowchart showing steps taken in a leave-one-out cross-validation process to select a group of models (S models) for competitive analysis. The processing of FIG. 5 begins at 500, whereupon, in step 510, the process identifies the most important features in each of the K models 335, and in step 520, the process identifies a total number (S) of significant differential features. In one embodiment, each of the K models 335 may have a slightly different set of most important features. In this embodiment, assuming the total number of significant differential features is S, the process labels all of the most important features with a number from 1 to S.

[0042] In step 530, the process selects the first significant differential feature ("j"). In step 540, the process extracts the jth feature from the training data. Referring to Figure 7, during the first iteration, where j=1, the process extracts column 700 corresponding to feature X1 in the training data 302.

[0043] In step 550, the process tests each of the K models 335 against the remaining features in the training data. In step 560, the process selects the best (e.g., most accurate) one of the K models for the jth iteration and denote the selected model as M_j S model 345 (e.g., model M_1).

[0044] The process determines whether each of the significant differential features has been processed (j=S) (decision 570). If each of the significant differential features has not been processed (j=S), decision 570 branches to the "no" branch which loops back to select and process the next distinct significant feature. Note that for the next iteration, the process returns the previously extracted data so that only the particular jth column of data is extracted for the next iteration. This loop continues until each of the significant differential features has been selected, at which point decision 570 branches to the "yes" branch which terminates the loop. The process of FIG. 5 then returns to the calling routine (see FIG. 4) at 595.

[0045] 6 is an exemplary flowchart illustrating steps taken during runtime processing to determine whether a model produces an uninferable result. The processing of FIG. 6 begins at 600, whereupon, at step 610, the process receives a set of score data 360. At step 620, the process tests the set of score data against each of the S selected models 345 (e.g., competitive analyzer 380).

[0046] In step 630, the process analyzes the results from the S models to check for strong target prediction conflicts. Referring to Figure 10, score data result 1000 shows a single strong prediction in row 1010 for target A, and therefore score data result 1000 has no conflicts. However, score data result 1050 shows a strong prediction for target A in row 1060, and also a strong prediction for target B in row 1070. Therefore, score data result 1050 has a conflict.

[0047] The process determines whether any strong prediction conflicts exist (decision 640). If any strong prediction conflicts exist, decision 640 branches to the "yes" branch, upon which the process generates output result 395 as an unreasonable result at stage 650, after which the processing of FIG. 6 ends at 660.

[0048] On the other hand, if no strong prediction conflicts exist, decision 640 branches to the "no" branch, upon which, at stage 670, the process generates an output result based on the score data test (e.g., strong inference target A), after which the processing of FIG. 6 ends at 695.

[0049] FIG. 7 is an exemplary diagram illustrating training data 302 including a set of features and targets. FIG. 7 illustrates training data 302, which includes multiple records in rows 1 through n. Each column 700, 710, and 720 is a feature, also referred to as a "predictor." Column 730 is a target column that includes targets for the various rows. The example in FIG. 7 shows that the targets in column 730 are categorical targets. In one embodiment, the targets can be continuous targets or a combination of categorical and continuous targets.

[0050] As discussed herein, the model selection phase 325 uses the training data 302 to create an initial predictive model and also performs a leave-one-out cross-validation step that removes data from one feature column at a time to ultimately select S models 345.

[0051] 8 is an exemplary diagram illustrating probability confidence curves and strong confidence thresholds for S models. Graph 800 shows the probability confidence curve for S model M_1. Graph 800 shows a strong inferable A-class prediction 810 at the mean + 2 standard deviations (confidence threshold 805).

[0052] Graph 820 shows the probability confidence curve for S model M_8. Graph 820 shows a strong inferable B class prediction 840 at the mean + 2 standard deviations (confidence threshold 825). Referring to score data results 1050 in FIG. 10 , if score data 360 produces a strong prediction A from model M_1 and a strong prediction B from model M_8, competitive analyzer 380 determines that output 395 is not inferable. As discussed herein, and comparing confidence threshold 805 to confidence threshold 825, the strong confidence threshold level may be at different locations along different probability curves for different models.

[0053] 9 is an exemplary diagram illustrating a predictive model decision tree including strong prediction confidence nodes and weak-medium prediction confidence nodes. The decision tree 900 corresponds to a predictive model and its decision points.

[0054] During the confidence threshold calculation stage 350, predictive model nodes are analyzed for their individual confidence levels. FIG. 9 shows that nodes 940, 970, and 990 correspond to strong predictive confidence. Therefore, if the corresponding predictive model makes a decision based on these nodes, the predictive model will output a strong predictive inference. Nodes 910, 920, 930, 950, 960, and 980 correspond to weak to medium predictive confidence. Therefore, if the corresponding predictive model makes a decision based on these nodes, this decision is not relevant with respect to conflict analysis, and is relevant when there is no strong conflict between different predictive models (see FIG. 6 and corresponding text for further details).

[0055] FIG. 10 is an exemplary diagram showing various score data model results. Score data result 1000 shows column results for models M1-M8. Each row corresponds to a strong or weak prediction of a particular target. Rows 1010, 1020, and 1030 contain strong prediction results used during conflict analysis. As can be seen, the only row with a strong prediction result is row 1010, where models M1, M4, and M7 all agree on strong prediction A. Therefore, score data result 1000 does not have any strong target prediction conflicts, and output 395 will indicate a strong prediction A inference.

[0056] Meanwhile, score data result 1050 shows strong target prediction conflicts. Row 1060 shows that models M1, M4, and M7 make strong target predictions for target A. Meanwhile, row 1070 shows that model M8 makes a strong target prediction (1075) for target B. Thus, score data result 1050 produces uninferable output, even though the majority of strong predictions are for target A.

[0057] While specific embodiments of the present disclosure have been illustrated and described, it will be apparent to those skilled in the art that, based on the teachings herein, changes and modifications can be made without departing from the present disclosure and its broader aspects. Accordingly, the appended claims are to encompass within their scope all such changes and modifications as are within the true spirit and scope of the present disclosure. It should be further understood that the present disclosure is defined solely by the appended claims. Those skilled in the art will understand that where a specific number of introduced claim elements is intended, such intention will be expressly recited in the claim, and that, absent such recitation, no such limitation exists. By way of non-limiting example and as an aid to understanding, the following appended claims include the use of the introductory phrases "at least one" and "one or more (one or more)" to introduce claim elements. However, the use of such phrases should not be construed as suggesting that the introduction of a claim element by the indefinite article "a" or "an" limits any particular claim containing such introduced claim element to a disclosure containing only one such element, even if the same claim contains the introductory phrase "one or more (one or more)" or "at least one" and an indefinite article such as "a" or "an," and the same applies to the use of definite articles in the claims.

Claims

1. A computer-implemented method comprising: identifying a plurality of models to test on a set of data, each one of the plurality of models generating one of a plurality of predictions corresponding to one of a plurality of targets; detecting one or more conflicts among the plurality of predictions in response to testing the set of data against each of a plurality of models; reporting an unreasonable result of the test in response to detecting the one or more conflicts; Equipped with the plurality of models includes a first model and a second model, and the computer-implemented method further comprises: generating a strong first prediction corresponding to a first target of the plurality of targets using the first model; generating a strong second prediction from the second model corresponding to a second target of the plurality of targets; generating the non-inferable result in response to determining that the first target is different from the second target; Further provided with A computer-implemented method, wherein the strong first prediction is based on a confidence threshold on a first probability curve corresponding to the first model, and the strong second prediction is based on a confidence threshold on a second probability curve corresponding to the second model.

2. A computer-implemented method comprising: identifying a plurality of models to test on a set of data, each one of the plurality of models generating one of a plurality of predictions corresponding to one of a plurality of targets; detecting one or more conflicts among the plurality of predictions in response to testing the set of data against each of a plurality of models; reporting an unreasonable result of the test in response to detecting the one or more conflicts; Equipped with the plurality of models includes a first model and a second model; The computer-implemented method comprises: generating a strong first prediction corresponding to a first target of the plurality of targets using the first model; generating a strong second prediction from the second model corresponding to a second target of the plurality of targets; generating the non-inferable result in response to determining that the first target is different from the second target; Further provided with A computer-implemented method, wherein the strong first prediction is based on a first mean + 2 standard deviation confidence threshold on a first probability curve corresponding to the first model, and the strong second prediction is based on a second mean + 2 standard deviation confidence threshold on a second probability curve corresponding to the second model.

3. constructing the plurality of models based on a set of training data; calculating, for each of the plurality of models, one of a plurality of model evaluation measures that measure the performance of one of the plurality of models; selecting a subset of K models from the plurality of models based on their corresponding model evaluation measures, the subset of K models including a set of important features; The computer-implemented method of claim 1 or 2, further comprising:

4. A method of identifying a plurality of models for testing a set of data, each one of the plurality of models generating one of a plurality of predictions corresponding to one of a plurality of targets; detecting one or more conflicts among the plurality of predictions in response to testing the set of data against each of a plurality of models; reporting an unreasonable result of the test in response to detecting the one or more conflicts; Equipped with constructing the plurality of models based on a set of training data; calculating, for each of the plurality of models, one of a plurality of model evaluation measures that measure the performance of one of the plurality of models; selecting a subset of K models from the plurality of models based on their corresponding model evaluation measures, the subset of K models including a set of important features; The computer-implemented method further comprises:

5. ranking the set of important features corresponding to a subset of the K models; identifying a set of distinctive features based on said ranking; For each of said set of distinctive features: selecting one of the set of distinctive features; removing the portion of the training data that corresponds to the selected differential feature; testing each of the subset of K models against the training data of a subset excluding the removed portion of the training data; selecting one of the subset of K models based on the test; designating the selected subset of K models as one of a set of S models; utilizing the set of S models during said testing of said set of data to detect said one or more conflicts; The computer-implemented method of claim 3 further comprising:

6. determining a confidence threshold for each of the S models in the set of S models; utilizing the confidence threshold to determine whether one or more of the plurality of predictions is a strong prediction; The computer-implemented method of claim 5 further comprising:

7. determining that the plurality of predictions includes a plurality of strong first predictions each corresponding to a first target of the plurality of targets; determining that the plurality of predictions includes a single strong second prediction corresponding to a second target of the plurality of targets; reporting the uninferable result in response to determining that the first target is different from the second target; The computer-implemented method of claim 1 or 2, further comprising:

8. one or more processors; a memory coupled to at least one of the one or more processors; a set of computer program instructions stored in said memory; wherein the set of computer program instructions, when executed by at least one of the one or more processors, acts of identifying a plurality of models for testing a set of data, each one of the plurality of models generating one of a plurality of predictions corresponding to one of a plurality of targets; detecting one or more conflicts among the plurality of predictions in response to testing the set of data against each of a plurality of models; reporting a non-inferable result of the test in response to detecting the one or more conflicts; and Run the plurality of models includes a first model and a second model, and the one or more processors: generating a strong first prediction corresponding to a first target of the plurality of targets using the first model; generating a strong second prediction from the second model corresponding to a second target of the plurality of targets; generating the non-inferable result in response to determining that the first target is different from the second target; and and performing an additional operation having the following structure: An information processing system, wherein the strong first prediction is based on a confidence threshold on a first probability curve corresponding to the first model, and the strong second prediction is based on a confidence threshold on a second probability curve corresponding to the second model.

9. One or more processors; a memory coupled to at least one of the one or more processors; a set of computer program instructions stored in said memory; wherein the set of computer program instructions, when executed by at least one of the one or more processors, acts of identifying a plurality of models for testing a set of data, each one of the plurality of models generating one of a plurality of predictions corresponding to one of a plurality of targets; detecting one or more conflicts among the plurality of predictions in response to testing the set of data against each of a plurality of models; reporting a non-inferable result of the test in response to detecting the one or more conflicts; and Run the plurality of models includes a first model and a second model, and the one or more processors: generating a strong first prediction corresponding to a first target of the plurality of targets using the first model; generating a strong second prediction from the second model corresponding to a second target of the plurality of targets; generating the non-inferable result in response to determining that the first target is different from the second target; and and performing an additional operation having the following structure: An information processing system, wherein the strong first prediction is based on a confidence threshold of a first mean + 2 standard deviations on a first probability curve corresponding to the first model, and the strong second prediction is based on a confidence threshold of a second mean + 2 standard deviations on a second probability curve corresponding to the second model.

10. the one or more processors constructing the plurality of models based on a set of training data; calculating, for each of the plurality of models, one of a plurality of model evaluation metrics that measure the performance of one of the plurality of models; selecting a subset of K models from the plurality of models based on their corresponding model evaluation measures, the subset of K models including a set of important features; 10. The information processing system according to claim 8, further comprising:

11. One or more processors; a memory coupled to at least one of the one or more processors; a set of computer program instructions stored in said memory; wherein the set of computer program instructions, when executed by at least one of the one or more processors, acts of identifying a plurality of models for testing a set of data, each one of the plurality of models generating one of a plurality of predictions corresponding to one of a plurality of targets; detecting one or more conflicts among the plurality of predictions in response to testing the set of data against each of a plurality of models; reporting a non-inferable result of the test in response to detecting the one or more conflicts; and Run the one or more processors constructing the plurality of models based on a set of training data; calculating, for each of the plurality of models, one of a plurality of model evaluation metrics that measure the performance of one of the plurality of models; selecting a subset of K models from the plurality of models based on their corresponding model evaluation measures, the subset of K models including a set of important features; An information processing system that performs additional operations including:

12. the one or more processors ranking the set of important features corresponding to a subset of the K models; identifying a set of distinctive features based on said ranking; For each of said set of distinctive features: selecting one of the set of distinctive features; removing portions of the training data corresponding to the selected differential features; testing each of the subset of K models against the training data of a subset excluding the removed portion of the training data; selecting one of the subset of K models based on the test; designating the selected subset of K models as one of a set of S models; utilizing the set of S models during said testing of said set of data to detect said one or more conflicts; The information handling system of claim 10 , further comprising:

13. the one or more processors determining a confidence threshold for each of the S models in the set of S models; utilizing the confidence threshold to determine whether one or more of the plurality of predictions is a strong prediction; The information handling system of claim 12 , further comprising:

14. the one or more processors determining that the plurality of predictions includes a plurality of strong first predictions each corresponding to a first target of the plurality of targets; determining that the plurality of predictions includes a single strong second prediction corresponding to a second target of the plurality of targets; reporting the uninferable result in response to determining that the first target is different from the second target; and 10. The information processing system according to claim 8, further comprising:

15. A computer program comprising: The computer program is configured to: identifying a plurality of models for testing a set of data, each one of the plurality of models generating one of a plurality of predictions corresponding to one of a plurality of targets; detecting one or more conflicts among the plurality of predictions in response to testing the set of data against each of a plurality of models; and reporting an unreasonable result of the test in response to detecting the one or more conflicts; The plurality of models includes a first model and a second model, and the computer program is configured to provide the information processing system with: generating a strong first prediction corresponding to a first target of the plurality of targets using the first model; generating a strong second prediction from the second model corresponding to a second target of the plurality of targets; generating the non-inferable result in response to determining that the first target is different from the second target; Then run The strong first prediction is based on a confidence threshold on a first probability curve corresponding to the first model, and the strong second prediction is based on a confidence threshold on a second probability curve corresponding to the second model.

16. A computer program comprising: The computer program is configured to: identifying a plurality of models for testing a set of data, each one of the plurality of models generating one of a plurality of predictions corresponding to one of a plurality of targets; detecting one or more conflicts among the plurality of predictions in response to testing the set of data against each of a plurality of models; and reporting an unreasonable result of the test in response to detecting the one or more conflicts; The plurality of models includes a first model and a second model, and the computer program is configured to provide the information processing system with: generating a strong first prediction corresponding to a first target of the plurality of targets using the first model; generating a strong second prediction from the second model corresponding to a second target of the plurality of targets; generating the non-inferable result in response to determining that the first target is different from the second target; Then run The computer program product, wherein the strong first prediction is based on a first mean + 2 standard deviation confidence threshold on a first probability curve corresponding to the first model, and the strong second prediction is based on a second mean + 2 standard deviation confidence threshold on a second probability curve corresponding to the second model.

17. The computer program is configured to: constructing the plurality of models based on a set of training data; calculating, for each of the plurality of models, one of a plurality of model evaluation metrics that measure the performance of one of the plurality of models; selecting a subset of K models from the plurality of models based on their corresponding model evaluation measures, the subset of K models including a set of important features; 17. The computer program according to claim 15 or 16, further comprising:

18. A computer program comprising: The computer program is configured to: identifying a plurality of models for testing a set of data, each one of the plurality of models generating one of a plurality of predictions corresponding to one of a plurality of targets; detecting one or more conflicts among the plurality of predictions in response to testing the set of data against each of a plurality of models; and reporting an unreasonable result of the test in response to detecting the one or more conflicts; The computer program is configured to: constructing the plurality of models based on a set of training data; calculating, for each of the plurality of models, one of a plurality of model evaluation metrics that measure the performance of one of the plurality of models; selecting a subset of K models from the plurality of models based on their corresponding model evaluation measures, the subset of K models including a set of important features; a computer program for further executing the above.

19. The computer program is configured to: ranking the set of important features corresponding to a subset of the K models; identifying a set of distinctive features based on said ranking; For each of said set of distinctive features: selecting one of the set of distinctive features; removing portions of the training data corresponding to the selected differential features; testing each of the subset of K models against the training data excluding the removed portion of the training data; selecting one of the subset of K models based on the test; designating the selected subset of K models as one of a set of S models; utilizing the set of S models during said testing of said set of data to detect said one or more conflicts; 18. The computer program of claim 17, further comprising:

20. The computer program is configured to: determining a confidence threshold for each of the S models in the set of S models; utilizing the confidence threshold to determine whether one or more of the plurality of predictions is a strong prediction; 20. The computer program of claim 19, further comprising:

Images