Machine learning classification of data using decision boundaries

Abstract

Data is classified using machine learning with a decision boundary. Classifier accuracy is improved in a machine learning application by training a machine learning (ML) model including a classifier across classes by determining a set of contributing weighted input points to store values of true positive and true negative predictions. In a subsequent step, input data is provided to the classifier of the machine learning model at runtime and a classification output from the classifier is determined. For the classification output, values input during runtime are compared to input samples stored for training the machine learning model to determine a distance of spread for the classification output. A class is determined from the classification output with the smallest distance and spread. The method can also determine whether the class with the smallest distance and spread is a true positive or a true negative by comparing the class with the smallest distance and spread to the classification output.

Description

BACKGROUND

[0001] The present invention relates generally to machine learning, and in some embodiments to classifier components of neural networks in machine learning applications.

[0002] Machine learning (ML) classifiers are elements used to solve a number of industry problems in a variety of types of a wide range of neural networks. In some examples, two or more class output classifiers and a number of activation functions are used to obtain classifications with clear boundaries. In the field of language processing, such as in chatbots, lower classification accuracy of the result text for a query can result in lower satisfaction of the end user or customer. Users who have a poor experience given the poor classification accuracy can require human operator interaction rather than continuing the conversation with the chatbot. This impacts the level of service desk or expert support automation.

[0003] Classification problems with false positives (FP) or false negatives (FN) can be widespread where improvements can increase the confidence of machine learning (ML) solutions in a number of industries. While a number of approaches have been taken to improve classification accuracy, there is still a need for newer approaches that can improve the accuracy of a number of class problems.

[0004] The next set of problems for a classifier is the confidence of the output that the decision made by the classifier is consistent with the human decision as provided in the training and test data sets. More specifically, is the result a true or false value that it shows a cluster belongs to - is it a true positive or false positive, a true negative or false negative. Knowing whether the final decision at runtime is a false positive or false negative is critical to alerting whether other decision mechanisms are needed to supplement the decision from the classifier, or whether special handling of the input is needed. SUMMARY

[0005] According to embodiments of the present invention, there is provided a computer- implemented method for improving classifier accuracy in machine learning applications, the method comprising training a machine learning (ML) model comprising a classifier across classes by determining weighted input points of a contribution set to store locations and values of true positive and true negative predictions. The method for improving classifier accuracy further comprises providing input data to the classifier of the machine learning model at runtime; and determining a classification output from the classifier. In some embodiments, to determine the classification output, the method compares values input during runtime to samples of input stored for training the machine learning model to determine an average spread distance of the classification output. The method for improving classifier accuracy further determines a class with a minimum distance and spread from the classification output; and can then determine whether the class with the minimum distance and spread is a true positive or a true negative by comparing the class with the minimum distance and spread to an average distance of the classification output, wherein the class is designated as a false positive or a false negative if the class with the minimum distance and spread is less than the average distance and spread of the classification output. By more accurately training the classifier by more accurately detecting true positives and true negatives, processing time of computational hardware performing artificial intelligence and machine learning tasks is reduced. This is particularly suitable for artificial intelligence and machine learning tasks such as machine vision and chatbot applications.

[0006] According to another embodiment of the present invention, there is also described a system for improving classifier accuracy in machine learning applications. The system can comprise: a hardware processor; and a memory storing a computer program product. The computer program product, when executed by the hardware processor, causes the hardware processor to: train a machine learning (ML) model comprising a classifier across classes by determining weighted input points of a contribution set to store locations and values of true positive and true negative predictions. The system for improving classifier accuracy can further: provide input data to the classifier of the machine learning model at runtime; and determine a classification output from the classifier. In some embodiments, to determine the classification output, the method compares values input during runtime to samples of input stored for training the machine learning model to determine an average spread distance of the classification output. The system for improving classifier accuracy can further determine a class with a minimum distance and spread from the classification output; and can then determine whether the class with the minimum distance and spread is a true positive or a true negative by comparing the class with the minimum distance and spread to the classification output, wherein the class is designated as a false positive or a false negative if the class with the minimum distance and spread is less than the average distance and spread of the classification output. By more accurately training the classifier by more accurately detecting true positives and true negatives, processing time of computational hardware performing artificial intelligence and machine learning tasks is reduced. This is particularly suitable for artificial intelligence and machine learning tasks such as machine vision and chatbot applications.

[0007] In yet another embodiment, a computer program product for improving classifier accuracy in machine learning applications is described. The computer program product includes a computer readable storage medium having computer readable program code embodied therein. Program instructions executable by a processor cause the processor to train a machine learning (ML) model comprising a classifier across classes by determining a weighted input point of a contribution set to store locations and values of true positive and true negative predictions. The program instructions for improving classifier accuracy are further for providing input data to the classifier of the machine learning model at runtime; and determining an average spread distance of a classification output from the classifier. In some embodiments, the computer program product determines the classification output by comparing values input during runtime to samples of input stored for training the machine learning model to determine the average spread distance of the classification output. The computer program product for improving classifier accuracy can also determine a class with a minimum distance and spread from the classification output; and then can determine whether the class with the minimum distance and spread is a true positive or a true negative by comparing the class with the minimum distance and spread to the classification output, wherein the class is designated as a false positive or a false negative if the class with the minimum distance and spread is less than the average of the classification output. By more accurately training the classifier by more accurately detecting true positives and true negatives, processing time for computing hardware to perform artificial intelligence and machine learning tasks is reduced. This is particularly suitable for artificial intelligence and machine learning tasks such as machine vision and chatbot applications. BRIEF DESCRIPTION OF DRAWINGS

[0008] The following description will provide details of preferred embodiments with reference to the following drawings, in which:

[0009] Figure 1 is an illustration of an example environment in which the methods and systems of embodiments of the present disclosure can be applied to improve classifier accuracy using a weight nudge method on a decision boundary.

[0010] Figure 2 is a block diagram / flowchart illustrating a method of improving classifier accuracy using a weight nudge method on a decision boundary according to one embodiment of the present disclosure.

[0011] Figure 3 is a block diagram of a system for improving classifier accuracy using a weight nudge method on a decision boundary according to one embodiment of the present disclosure.

[0012] Figure 4 is a generalized diagram of a neural network.

[0013] Figure 5 is a table showing calculations of true positives and false positives.

[0014] Figure 6 is a block diagram illustrating a system according to one embodiment of the disclosure that can incorporate the system shown in FIG. 1. Figure 3 to improve classifier accuracy using a weighted fine-tuning method on decision boundaries.

[0015] Figure 7 A cloud computing environment according to an embodiment of the disclosure is shown.

[0016] Figure 8 An abstraction model layer according to an embodiment of the disclosure is shown. DETAILED DESCRIPTION

[0017] This invention relates generally to machine learning (ML) classifiers in neural network applications. Sensitivity analysis on a classification does not allow for improved confidence in the classification, and sensitivity analysis does not show whether the output is a false positive. Soft k-means provides a weighted distance method. However, soft k-means analysis is only at the output level of each input data point. These methods require retraining of the classifier, essentially moving the class boundary to include new data points where false positives and false negatives occur. Retraining of the classifier is expensive because if many new inputs are close to the boundary, then the classifier must be retrained constantly. Other methods use post-output processing and receiver operating characteristic (ROC) curves.

[0018] The methods, systems, and computer program products of the present disclosure address the issue of whether a classification is a true positive or a false positive, or a true negative or a false negative by addressing the question of whether a new input is located at a distance (di) from an important input point of a sample across various classes of objects, where, starting from the first layer and then proceeding to the layers before the output layer, at each layer, a distance between the input and some class examples is obtained.

[0019] A "class" is a set of enumerated target values of a label. For example, in a binary classification model that detects spam emails, the two classes are "spam" and "not spam." In a multi-class classification model that identifies dog breeds, the classes would be Poodle, Beagle, Pug, etc.

[0020] For each cluster, a set of random images or truly representative images, or a combination, are taken, and the top contributing weighted inputs for each layer are stored, such as the weighted inputs that show a change of 10% or 20% compared to the peak. A "cluster" is a collection of data items that are "similar" to each other and "not similar" to data items in other clusters. This is used for comparison and distance calculations. Cluster categories that show the least distance and proximity factors are the representation of the class compared to cluster categories that show longer distances and proximity spread. This is compared to the output from the classifier, and if the values are the same, then it is a true positive. If there is a cluster difference between the cluster provided by the classifier and the cluster calculated from the layer-by-layer distance, then the output is a false positive.

[0021] Neural network clustering and classification. Neural networks can be thought of as a clustering and classification layer on top of stored and managed data. Neural networks help group unlabeled data based on similarity between example inputs, and when the neural network has a labeled dataset to train on, the neural network classifies the data.

[0022] One consideration is that the machine learning layer on the training corpus already knows the weights, and the impact of all weights is not considered because in the proposed method, system, and computer program product, only the weighted top input points from a set of input points are considered.

[0023] However, when the input is evaluated by a trained machine learning (ML) model as belonging to a class, the weight matrix and output decision threshold are used to calculate the distance di, and assigned to a cluster. The weight matrix is considered to find those weights that have driven the input calculation to a distance di from other images. The input values from those points in the input set that contribute most to the maximum impact are taken from any sample image of the class. These values are replaced in the input data set, and the new class distance and spread and classifier output are taken to determine if such a replacement moves the false positive result to a true positive result, which helps determine which are the input points that are making an impact. This is run on a set of sample images of the input class to determine the number of times the output changes to a true positive, which also yields an answer to whether a particular set of input points is contributing to the false positive classification alone, or if it is spreading over a large portion of the input data set. If it is the latter, then the mapping should produce better true positive matches for other classes. If the same change is obtained in all classes, then either the quality of the input has to be enhanced with other input data sets, or other pre-filtering operations have to be performed on the input data set. This replaces key input data points to determine which changes will have a positive impact on the output classification, moving to a more accurate true positive classification for a class in the set of output classes.

[0024] However, looking at the weight contribution, as it is a linear operation, across the network from input to output till the activation function, the path with the highest contribution can be derived.

[0025] The methods, systems, and computer program products described herein provide a method for determining whether a machine learning (ML) conclusion is a true positive or a false positive belonging to a cluster / class. In some embodiments, the method and system provide a determination of node influence and influence of input values. The method and system provide a determination of various aspects of images / inputs abstracted at each layer in a machine learning environment. In some embodiments, the methods and systems described herein can provide a determination of input quality to recommend more input filtering or new input recommendations to a set of input data in a machine learning (ML) environment. In some embodiments, the method and system can provide a determination of images to obtain false positive outputs across multiple classes, e.g., a robustness factor of a model. In some embodiments, the method and system use samples from training / testing while in supervised mode to determine distance and cluster spread to produce better classification results. The method, system, and computer program products provide a mechanism to reduce churn on the use of automation to help customers.

[0026] In some embodiments, the method, system, and computer program products can address the question of whether a runtime output is a true positive or a false positive. In addition, the method helps address which parts of the input contribute to false positives so that these inputs can be better captured. Referring now to Figures 1-8 The methods, systems, and computer program products of the present disclosure are described in more detail.

[0027] Figure 1 is an illustration of an example environment in which the methods and systems of embodiments of the present disclosure can be applied to improve classifier accuracy using a weight fine-tuning method on decision boundaries. Figure 2 is a block diagram / flowchart illustrating a method of improving classifier accuracy using a weight fine-tuning method on decision boundaries. Figure 3 is a block diagram illustrating a system of improving classifier accuracy using a weight fine-tuning method on decision boundaries that can be employed in the method shown in Figure 2 is a block diagram illustrating a system of improving classifier accuracy using a weight fine-tuning method on decision boundaries that can be employed in the method shown in

[0028] Aspects of the disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0029] These computer readable program instructions can be provided to a processor of a computer or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing device, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0030] Figure 1 Machine learning is shown, where an expert uses their knowledge and time to craft a set of features that can be extracted from an input and can be used to train a classification network to produce an output. In Figure 1 the example shown in FIG. 1, a network is trained to recognize a car from an image, in the sense of machine vision, for example. In one example, the input is an image of a vehicle. The feature extraction step 10 includes extracting features from the input, and the classification step 15 includes training a model to predict a result 20. For example, the machine learning can employ a neural network to recognize whether a car is present or not, from extracted image features. The output is an estimated conclusion, e.g., whether a car is present or not. The method, system, and computer program product improve a machine learning classifier by using a weight fine-tuning method on the decision boundary, by computing the distance of each layer to each sample of each class of data clustering to detect / reduce false positives and false negatives. While Figure 1 One example of machine vision is shown, but the application of the method, system, and computer program product of the present disclosure can include chat solutions, image recognition, and speech recognition. Any solution with a classifier and a clustered output determination - two-class or multi-class clustering driving the classifier - is applicable to the method and system described herein.

[0031] Referring now to Figure 4 , a generalized diagram of a neural network is shown. While a particular structure of an ANN is shown, with three layers and a set of fully connected neurons, it should be understood that this is for illustration purposes only. In practice, the present embodiments can take any suitable form, including any number of layers and any one or more connection patterns between them.

[0032] ANNs exhibit the ability to derive meaning from complex or imprecise data, as well as can be used to extract patterns and detect trends that are too complex to be detected by humans or other computer-based systems. The structure of a neural network is generally known to have input neurons 302 that provide information to one or more "hidden" neurons 304. The connections 308 between the input neurons 302 and the hidden neurons 304 are weighted, and the hidden neurons 304 then process these weighted inputs according to some function in the hidden neurons 304. There can be any number of layers of hidden neurons 304, as well as neurons that perform different functions. There are also different neural network structures, such as convolutional neural networks, maxout networks, etc., that can vary according to the structure and function of the hidden layers and the pattern of weights between the layers. The individual layers can perform specific functions, and can include convolutional layers, pooling layers, fully connected layers, softmax layers, or any other appropriate type of neural network layer. Finally, a set of output neurons 306 accept and process the weighted inputs from the last set of hidden neurons 304.

[0033] This represents a "feedforward" computation, where information propagates from the input neurons 302 to the output neurons 306. When the feedforward computation is complete, the output is compared to the expected output that can be obtained from the training data. Errors related to the training data are then processed in a "backpropagation" computation, where the hidden neurons 304 and the input neurons 302 receive information about the error that is backpropagated from the output neurons 306. Once the backpropagation of error is complete, a weight update is performed, where the weighted connections 308 are updated to take into account the received error. It should be noted that these three modes of operation - feedforward, backpropagation, and weight update - do not overlap with each other. This merely represents one type of ANN computation, and any appropriate form of computation can be used instead. In the present case, the output neurons 306 provide an analysis of whether the packet has been properly processed according to the data provided from the inputs of the IoT sensors.

[0034] To train an ANN, the training data can be divided into a training set and a test set. The training data includes pairs of inputs and known outputs. During training, the inputs of the training set are fed into the ANN using feedforward propagation. After each input, the output of the ANN is compared to the corresponding known output. The difference between the output of the ANN and the known output associated with that particular input is used to generate an error value, which can be backpropagated through the ANN, after which the weight values of the ANN can be updated. This process continues until the pairs in the training set are exhausted.

[0035] After training is complete, the ANN can be tested against a test set to ensure that training has not resulted in overfitting. If the ANN can generalize to new inputs beyond those it has already been trained on, then it is ready for use. If the ANN does not accurately reproduce the known outputs of the test set, additional training data can be needed, or the hyperparameters of the ANN can need to be adjusted.

[0036] ANNs can be implemented in software, hardware, or a combination of the two. For example, each weight 308 can be represented as a weight value stored in computer memory, and the activation function of each neuron can be implemented by a computer processor. The weight values can store any appropriate data value, such as a real number, a binary value, or a value chosen from a fixed number of possibilities, which is multiplied by the relevant neuron output. Alternatively, the weights 308 can be implemented as resistive processing units (RPUs), which generate a predictable current output when an input voltage is applied according to a settable resistance.

[0037] The ANN shown in Figure 1 may be employed in the classification step 15 of the machine learning environment shown in Figure 4 .

[0038] Figure 2 A method of using a weight fine-tuning method on a decision boundary to improve classifier accuracy is shown. Note that the process flow illustrated by the description of the various blocks shown in Figure 2 is not an exclusive list. Many intermediate steps, preparatory steps, and concluding steps can also apply to the methods described herein.

[0039] In some embodiments, a computer-implemented method for using a weight fine-tuning method on a decision boundary to improve classifier accuracy can begin at block 21 of Figure 2 . Block 21 includes training a machine learning (ML) model across binary or multiclass. An artificial intelligence (AI) / machine learning model can be created by training an n-th layer 34 from a neural network as shown in Figure 3 .

[0040] In the testing phase, for each true positive or true negative, a weighted input point is determined in the set of inputs that contributed to the importance value in each layer, so as to store the location and value. True positives (TP) refer to the number of predictions in which the classifier correctly predicted a positive class as positive. True negatives (TN) refer to the number of predictions in which the classifier correctly predicted a negative class as negative.

[0041] The set of contributions is equal to:

[0042] Equation 1: {X n |Y n / Ypeak >0.9}, wherein,

[0043] X n equals the important input into the neuron / AI layer

[0044] Y n equals the weighted input "n" to the neuron / AI in the layer

[0045] Y peak equals the highest or most important value of the weighted input of all neurons in that AI layer.

[0046] At block 22, the method captures some samples in each class, where the top percentile of weighted inputs are stored in a reference database. The reference Figure 3 , the reference database can be provided by a storage of the training data set layer values for each cluster, identified by reference numeral 36.

[0047] The reference Figure 2 , the method can continue to provide input to the classifier at runtime at block 23. The reference Figure 3 , the input layer of the classifier 15 is identified by reference numeral 31. The output layer of the classifier 15 is identified by reference numeral 34. There are multiple layers between the input layer 31 and the output layer (nth layer) 34, such as the 1st layer of the neural network identified by reference numeral 32 and the 2nd layer of the neural network identified by reference numeral 33.

[0048] In a subsequent step, at block 24, the method can determine the classification output from the classifier 15. At runtime stage, for an input, the method compares the input to the stored input samples to determine the distance (scalar value) to all other previous run / sample results. This can be provided by the classifier accuracy determiner engine 39 of the system shown in Figure 3 . From each layer between the input and the output, the classifier accuracy determiner engine 39 receives the highest weighted output, such as the N highest weighted outputs from the 1st layer of the neural network identified by reference numeral 36 and the N highest weighted outputs from the 2nd layer of the neural network identified by reference numeral 37. The stored input samples are provided by the training data set layer values for each cluster, which also have a feed to the determiner engine 39. The classifier accuracy determiner engine 39 can then find the distance and cluster spread.

[0049] The distance and cluster spread can be calculated by the determiner engine 39 using the following formula:

[0050] Formula 2: Distance d n at each X n = sqrt(sqr(Xninput – X nsample )+sqr(V ninput – V nsample ))

[0051] Formula 3: Total distance =∑d n / n, where:

[0052] d n equals the distance of the important input “n”.

[0053] sqrt equals square root.

[0054] sqr equals square.

[0055] V ninput equals the weighted input of element “n”.

[0056] V nsample equals the weighted input of element “n” in the sample under consideration.

[0057] X ninput equals the input of element “n”.

[0058] X nsample equals the input of element “n” in the sample under consideration.

[0059] According to Formula 2 and Formula 3, the determiner engine 38 can determine the class with the smallest distance and spread and compare to the classification output to determine whether it is a true positive or false positive at block 25 of the method shown in Figure 2

[0060] If it is determined at block 25 that a false positive has occurred, the method can continue to calculate the distance of each layer to each sample of each class at block 26 of the method shown in Figure 2 If it is a false positive, the method can continue to select each or some of the input set from the class with the smallest distance and replace the n values in the input set at runtime with those in the sample set at block 27. True positive samples returned by the classification are determined. This shows the set of sensitive points in the input set. Block 28 includes determining the classification accuracy or true positive rate.

[0061] In subsequent processing steps, the method can determine the false positive rate across classes at block 29 in order to determine the strength of the input at block 30 of the method shown in Figure 2 Figure 2 Blocks 25-30 of the method shown in Figure 3 may be formed by the classifier accuracy determiner engine 39 of the system shown in

[0062] In some examples, Figure 2 ​​The method shown can be used to employ machine learning in machine vision applications, such as recognizing objects via machine vision. In other examples, it can employ... Figure 2 The method shown provides answers to questions presented to the chatbot.

[0063] Figure 5 This is a table showing the calculations for true and false positives used to determine the probability of false positives across categories in order to determine the intensity of inputs in machine vision applications. Figure 5 In the example shown, the machine learning application is trying to identify the bag type (e.g., bag type 2) among other types of bag types (e.g., bag 3, bag 5, bag 6, bag 7) and other types of objects.

[0064] Figure 5 The sample calculations shown in the table illustrate the results performed at the first layer of the classifier, as shown regarding... Figures 1-3 The output of the calculation described is as follows: Bag 2 is a false positive for the category "Coat". For the input sample of bags {bag 3, bag 5, bag 6, bag 7}, the output value is between 8.08 and 12.3, while the distance to "Coat" is 40% higher than the maximum bag value. Therefore, the classification as "Coat" is a false positive. Replacing Bag 2 with n values ​​provided from any bag, and having an output value < 12, will show the classification as "bag" instead of "Coat" by the ML model as a true positive.

[0065] System 100 (including a classifier for machine learning model 15, a classifier accuracy determiner 39, and a training dataset layer 38) for improving classifier accuracy in machine learning applications can be integrated via bus 102. Figure 6 The processing system 400 shown includes at least one processor (CPU) 104 (also referred to as a hardware processor) operatively coupled to other components via a system bus 102. A cache 106, a read-only memory (ROM) 108, a random access memory (RAM) 110, an input / output (I / O) adapter 120, a sound adapter 130, a network adapter 140, a user interface adapter 150, and a display adapter 160 are operatively coupled to the system bus 102. As described herein, the bus 102 interconnects multiple components.

[0066] As employed herein, the terms "hardware processor subsystem" or "hardware processor" can refer to a processor, memory, software, or combination thereof that cooperate to perform one or more particular tasks. In useful embodiments, a hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, and / or a separate processor or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem can include one or more on-board memories (e.g., cache, dedicated memory arrays, read-only memory, etc.). In some embodiments, a hardware processor subsystem can include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.) that can be on-board or off-board, or that can be dedicated for use by the hardware processor subsystem.

[0067] In some embodiments, a hardware processor subsystem can include and execute one or more software elements. The one or more software elements can include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0068] In other embodiments, a hardware processor subsystem can include specialized dedicated circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry can include one or more application specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0069] These and other variations of a hardware processor subsystem are contemplated in accordance with embodiments of the present application.

[0070] Figure 6 The system 400 shown in FIG. 1 can further include a first storage device 122 and a second storage device 124 operably coupled to the system bus 102 by the I / O adapter 120. The storage devices 122 and 124 can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid state magnetic device, etc. The storage devices 122 and 124 can be the same type of storage device or different types of storage devices.

[0071] A speaker 132 is operably coupled to the system bus 102 by the sound adapter 130. A transceiver 142 is operably coupled to the system bus 102 by the network adapter 140. A display device 162 is operably coupled to the system bus 102 by the display adapter 160.

[0072] The first user input device 152, the second user input device 154, and the third user input device 156 are operably coupled to the system bus 102 via the user interface adapter 150. The user input devices 152, 154, and 156 can be any of a keyboard, a mouse, a keypad, an image capture device, a motion sensing device, a microphone, a device integrated with at least two of the preceding devices, etc. Of course, other types of input devices can be used while still maintaining the spirit of the present application. The user input devices 152, 154, and 156 can be the same type of user input device or different types of user input devices. The user input devices 152, 154, and 156 are used for inputting and outputting information to and from the system 400.

[0073] Of course, the processing system 400 can also include other elements (not shown) and / or omit certain elements, as readily appreciated by one of ordinary skill in the art, without departing from the spirit of the present application. For example, various other input devices and / or output devices can be included in the processing system 400, as readily appreciated by one of ordinary skill in the art, depending on the particular implementation of the processing system 400. For example, various types of wireless and / or wired input devices and / or output devices can be used. Moreover, additional processors, controllers, memories, etc. can also be employed in various configurations, as readily appreciated by one of ordinary skill in the art. These and other variations of the processing system 400 are readily contemplated by one of ordinary skill in the art in light of the teachings of the present application provided herein.

[0074] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0075] For example, the present disclosure provides a computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code embodied in the medium that is used to provide a plurality of questions from a presentation. In some embodiments, the computer program product is used to improve classifier accuracy in machine learning applications. The computer program product can include a computer-readable storage medium having computer-readable program code embodied in the medium. The program instructions executable by a processor to cause the processor to train a machine learning (ML) model comprising a classifier across classes by determining a weighted input point of a contribution set to store locations and values of true positive and true negative predictions; and provide input data to the classifier of the machine learning model at runtime. The program instructions can also use the processor to determine a classification output from the classifier, wherein for the classification output, the method compares values inputted during runtime to input samples stored for training the machine learning model to determine a spread distance of the classification output. The program instructions can also use the processor to determine a class with a minimum distance and spread from the classification output; and use the processor to determine whether the class with the minimum distance and spread is a true positive or a true negative by comparing the class with the minimum distance and spread to the classification output.

[0076] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0077] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage. Those skilled in the art should appreciate that the network and / or network adapter card can comprise any

[0078] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or any combination of source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as SMALLTALK, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0079] Aspects of the present application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0080] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0081] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0082] The methods of this disclosure can be implemented using a cloud computing environment. Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0083] On-demand self-service: cloud consumers can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically (without human interaction with the service's provider).

[0084] Broad network access: capabilities are available over a network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0085] Resource pooling: the provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to demand. There is a sense of location independence in that the consumer generally has no control or knowledge over the exact location of the provided resources but can be able to specify location at a higher level of abstraction (e.g., country, state, or data center).

[0086] Rapid and flexible: Features can be supplied quickly and flexibly (in some cases automatically) to rapidly expand outward and rapidly release to rapidly expand inward. For consumers, the available features often appear unlimited, and any number of features can be purchased at any time.

[0087] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging measurement capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the service providers and users.

[0088] The service model is as follows:

[0089] Software as a Service (SaaS): The functionality provided to consumers is the use of a provider's applications running on cloud infrastructure. Applications (e.g., web-based email) can be accessed from various client devices via a thin client interface (such as a web browser). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application functionality, with possible exceptions such as limited user-specific application configuration settings.

[0090] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environments.

[0091] Infrastructure as a Service (IaaS): This provides consumers with the capability to offer processing, storage, networking, and other basic computing resources on which they can deploy and run arbitrary software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over the operating system, storage, deployed applications, and possibly limited control over chosen networking components (e.g., host firewalls).

[0092] The deployment model is as follows:

[0093] Private cloud: Cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0094] Community cloud: Cloud infrastructure shared by several organizations and supporting a specific community with common concerns (e.g., mission, security needs, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0095] Public cloud: the cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services.

[0096] Hybrid cloud: the cloud infrastructure is a composition of two or more types of cloud (private, community, or public) that remain unique entities but are bound together, creating the hybrid cloud with capabilities that exceed those of the individual clouds.

[0097] A cloud computing environment is service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure of interconnected nodes, including end points and middle nodes.

[0098] Referring now to the drawing Figure 7 , a schematic cloud computing environment 50 is depicted. As shown, cloud computing environment 50 includes one or more cloud computing nodes 51 with which a cloud consumer can engage with to use the cloud computing nodes 51 can be communication with one or more cloud computing nodes 51. Nodes 110 can communicate with one another. They can be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment 50 to offer infrastructure, platforms and / or software as services with Figure 7 The types of computing devices 54A-N shown in

[0099] Referring now to the drawing Figure 8 , a set of functional abstraction layers provided by cloud computing environment 50 is shown. It should be understood in context Figure 8 The components, layers, and functions shown in

[0100] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61; RISC (Reduced Instruction Set Computer) architecture based servers 62; servers 63; blade servers 64; storage devices 65; and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0101] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71 ; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0102] In one example, management layer 80 can provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing 82 provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources can include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which future usage is anticipated in accordance with an SLA.

[0103] Workloads layer 90 provides examples of functionality for which the cloud computing environment can be utilized. Examples of workloads and functions which can be provided from this layer include: mapping and navigation 91 ; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and Figures 1-5 The described classifier accuracy system 100.

[0104] Reference in the specification to "one embodiment" or "an embodiment" of the application, or other variants thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in an embodiment" or any other variants thereof in various places in the specification are not necessarily all referring to the same embodiment.

[0105] Preferred embodiments of a system for improving classifier accuracy using a weight fine-tuning method on a decision boundary are described, it should be noted that modifications and variations can be made according to the above teachings by those skilled in the art. Accordingly, it should be understood that within the scope of the application, disclosed in the appended claims, modifications can be made to the specific embodiments described. Therefore, aspects of the present application have been described with particularity and detail, with a preferred embodiment being shown. However, it will be apparent that other embodiments can be practiced which are within the spirit and scope of the application. Therefore, the above description should not be construed as limiting, but merely as exemplification of the present application. Those skilled in the art will readily comprehend other aspects of the application as pertains to the application as claimed in the claims.

Claims

1. A computer-implemented method for improving classifier accuracy in machine learning applications, comprising: A machine learning model, including a cross-class classifier, is trained by determining the weighted input points of the contribution set to store the locations and values ​​of true positive and true negative predictions; Receive input data for the classifier of the machine learning model at runtime; The method determines the classification output from the classifier, wherein, for the classification output, the method compares the values ​​input during runtime with input samples stored for training the machine learning model to determine the average diffusion distance of the classification output; The category is determined based on the classification output with the minimum distance and spread; and The category with the minimum distance and diffusion is determined as a true positive or true negative by comparing it with the classification output. If the category with the minimum distance and diffusion is less than the average of the classification output, then that category is designated as a false positive or false negative. The machine learning applications include image recognition applications.

2. The computer-implemented method according to claim 1, wherein, The input data is a query, and the true positive or true negative is the correct answer to the query.

3. The computer-implemented method according to claim 1, wherein, Each of the true positives or true negatives is a category of the response.

4. The computer-implemented method according to claim 3, wherein, The category is either binary or multi-category.

5. The computer-implemented method according to claim 1, wherein determining the weighted input points of the contribution set includes calculating {X} n | Y n / Y peak > 0.9}, where, X n This is equivalent to the crucial input Y entering the neuron / AI layer. n This is equivalent to the weighted input "n" to the neurons / AI in the layer, and Y peak It is equal to the highest or most important weighted input value of all neurons in that AI layer.

6. The computer-implemented method according to claim 1, wherein, The classifier of the machine learning model includes multiple layers that provide a neural network, including an input layer and an output layer, with an intermediate layer between the input layer and the output layer.

7. The computer-implemented method according to claim 1, wherein, The input data is an image extracted from the dataset by the feature extraction element of the machine learning engine for recognition.

8. The computer-implemented method according to claim 1, wherein, Determining the classification output from the classifier includes a determiner engine for each layer between the input and output, wherein the determiner engine receives input samples for storing layer values ​​of the training dataset, and based on comparisons of the sample input with each layer of the training dataset, the determiner engine is able to find distance and cluster diffusion by computation: In each X n Distance d at point n = sqrt(sqr(X ninput -X nsample )+sqr(V ninput -V nsample )),as well as Total distance = Σ d n / n, where: d n The distance equal to the important input "n" sqrt equals the square root. sqr equals square. V ninput The weighted input equal to the element "n" V nsample It is equal to the weighted input of the element "n" in the considered sample. X ninput The input equal to element "n", and X nsample It is equal to the input of element "n" in the sample under consideration.

9. A system for improving classifier accuracy in machine learning applications, comprising: Hardware processor; and A memory for storing a computer program product, which, when executed by the hardware processor, causes the hardware processor to perform the method according to any one of claims 1 to 8.

10. A computer program product for improving classifier accuracy in machine learning applications, the computer program product comprising computer-readable program code executable by a processor to cause the processor to perform the method according to any one of claims 1 to 8.

Images