Learning system and learning method for machine learning model using streaming data as input

Abstract

To maintain the prediction accuracy of a machine learning model using imbalanced streaming data as input.SOLUTION: A system receives, for each data element in imbalanced streaming data input to a model, the class prediction probability of the data element from the model and calculates the uncertainty level of the class conforming to the class prediction probability on the basis of the prediction probability. The system also identifies the uncertainty range corresponding to the class conforming to the class prediction probability from among the uncertainty ranges prepared for each class for each data element, determines whether the uncertainty level is outside the identified uncertainty range, and issues or does not issue a labeling request about the data element according to the determination results. In addition, the system learns the model on the basis of a labeled dataset composed of data elements labeled in response to the issued labeling requests.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention generally relates to training machine learning models using streaming data as input. [Background technology]

[0002] With regard to the processing of machine learning models, for example, the techniques disclosed in Patent Documents 1 and 2 and the technique disclosed in Non-Patent Document 1 are known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP2022-184272 [Patent Document 2] WO2008 / 047835 [Non-patent literature]

[0004] [Non-Patent Document 1] Liu, Weike, et al. “A comprehensive active learning method for multiclass imbalanced data streams with concept drift.”.Knowledge-Based Systems 215 (2021):106778. Summary of the Invention [Problem to be solved by the invention]

[0005] A machine learning model that uses streaming data as input outputs prediction results for each data element in the streaming data.

[0006] Imbalanced data, which is data in which classes are imbalanced, is known. Streaming data, or imbalanced streaming data, is an example of imbalanced data. The prediction accuracy of a machine learning model that receives imbalanced streaming data may be reduced due to at least one of the following reasons (a) and (b): (a) Low probability of existence of target data (e.g., defective product data) detected using a machine learning model. (b) Concept drift, which can occur due to deterioration or replacement of streaming data sources (e.g., surveillance cameras or sensors).

[0007] Neither Patent Documents 1 and 2 nor Non-Patent Document 1 discloses or suggests a technique for solving such problems. [Means for solving the problem]

[0008] The learning system receives, for each data element in imbalanced streaming data input to a machine learning model, a class prediction probability for that data element from the machine learning model, and calculates an uncertainty level for the class according to the class prediction probability based on the predicted probability. For each data element, the learning system identifies an uncertainty range corresponding to the class according to the class prediction probability from among uncertainty ranges prepared for each class and defined as having high uncertainty, performs a range determination to determine whether the uncertainty level is outside the identified uncertainty range, and issues or does not issue a labeling request for the data element depending on the result of the range determination. The learning system trains the machine learning model based on a labeled dataset consisting of data elements labeled in response to the issued labeling requests. [Effects of the Invention]

[0009] According to the present invention, it is possible to maintain the prediction accuracy of a machine learning model that uses imbalanced streaming data as input. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments of the present invention. [Brief explanation of the drawings]

[0010] [Figure 1] 1 shows an overview of a data processing system according to an embodiment. [Figure 2] The flow of processing performed by the learning module is shown below. [Figure 3] The process flow of the uncertainty parameter estimator is shown below. [Figure 4] The flow of processing performed by the uncertainty update module is shown below. [Figure 5A] An example is shown where the class prediction probability falls in the lower margin of the uncertainty range. [Figure 5B] An example is shown where the class prediction probability falls in the upper margin of the uncertainty range. [Figure 6] 1 illustrates an example of a hardware configuration of a data processing system. [Figure 7] An overview of learning and inference in the system shown in Figure 6 is shown. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the following description, an "interface apparatus" may refer to one or more interface devices, which may be at least one of the following: An I / O interface device is one or more I / O (Input / Output) interface devices. The I / O (Input / Output) interface devices are interface devices for at least one of an I / O device and a remote display computer. The I / O interface device for the display computer may be a communications interface device. The at least one I / O device may be a user interface device, for example, either an input device such as a keyboard and a pointing device, or an output device such as a display device. A communication interface apparatus that is one or more communication interface devices. The one or more communication interface devices may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., an NIC and an HBA (Host Bus Adapter)).

[0012] In the following description, "memory" refers to one or more memory devices, which are an example of one or more storage devices, and may typically be a primary storage device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0013] In the following description, a "persistent storage device" may refer to one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device (e.g., an auxiliary storage device), and specifically may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a non-volatile memory express (NVME) drive, or a storage class memory (SCM).

[0014] Also, in the following description, "storage device" may be at least one of memory and persistent storage device.

[0015] Furthermore, in the following description, a "processor" may refer to one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a circuit that is a collection of gate arrays written in a hardware description language that performs some or all of the processing (for example, an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0016] In the following description, functions may be described using terms such as an uncertainty analyzer, a learning module, an uncertainty update module, and an uncertainty parameter estimator. However, these functions may be implemented by one or more computer programs executed by a processor, by one or more hardware circuits (e.g., FPGAs or ASICs), or by a combination thereof. When a function is implemented by a program executed by a processor, the specified processing is performed using a storage device and / or an interface device, etc., as appropriate, and therefore the function may be considered to be at least a part of the processor. Processing described using a function as the subject may also be processing performed by a processor or a device having the processor. A program may be installed from a program source. The program source may be, for example, a program distribution computer or a computer-readable storage medium (e.g., a non-transitory storage medium). The description of each function is merely an example; multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0017] In the following explanation, data that produces an output in response to an input may be described using expressions such as "xxx database," but the data may have any structure (for example, structured data or unstructured data). Therefore, "xxx database" can also be referred to as "xxx data." One database may be divided into two or more databases, or all or part of two or more databases may be one database.

[0018] In the following description, when elements of the same type are described without distinction, common reference symbols are used, and when elements of the same type are described with distinction, reference symbols are used.

[0019] FIG. 1 shows an overview of a data processing system according to an embodiment.

[0020] A data processing system 100 is configured. In this embodiment, the data processing system 100 is a physical computer system (one or more physical computers). However, it may also be a logical computer system (e.g., a virtual machine or a cloud computing service) based on a physical computer system (e.g., a cloud platform). Specifically, for example, as illustrated in FIG. 6 , the data processing system 100 includes a learning server 610 that performs training of an ML (machine learning) model and an inference server 620 that performs inference using the ML model. The learning server 610 includes a network interface 611, storage 612, memory 613, a GPU 614, and a CPU 615 connected thereto. The inference server 620 includes a network interface 621, storage 622, memory 623, a GPU 624, and a CPU 625 connected thereto. The network interfaces 611 and 621 are communicatively connected via a communication network 650 (e.g., the Internet or a WAN (Wide Area Network)). The network interfaces 611 and 612 are examples of interface devices. Storages 612 and 622 are examples of persistent storage devices. GPUs 614 and 624, and CPUs 615 and 625 are examples of processors. In the example shown in Figure 6, learning server 610 and inference server 620 are physically separate servers, but they may also be physically integrated servers.

[0021] 1, the data processing system 100 stores a parameter database 112 and a knowledge database 113. The knowledge database 113 may not be provided.

[0022] The parameter database 112 includes multiple class-dependent uncertainty parameters. A "class-dependent uncertainty parameter" is an uncertainty parameter that depends on the class. For example, suppose there are classes such as "positive" (abnormal) and "negative" (normal). For each class, the class-dependent uncertainty parameters include parameters that define an uncertainty range, such as a lower margin and an upper margin. The lower margin is a range on the lower side from the decision threshold. The upper margin is a range on the upper side from the decision threshold. For each class, the range consisting of the lower margin and the upper margin is the uncertainty range of that class. Instead of the lower margin and the upper margin, a lower threshold and an upper threshold may be used. The uncertainty range of that class may be a range greater than the lower threshold and less than the upper threshold. Note that the "uncertainty range" is a range defined as having high uncertainty. The decision threshold is a threshold for class determination, and therefore may be common to multiple classes.

[0023] The knowledge database 113 includes data of a prior knowledge base created by human engineers. The data may include quantitative data (e.g., the predictive difficulty of each class) or qualitative data (e.g., data on the proportion of each class).

[0024] 1, in the data processing system 100, functions such as an uncertainty analyzer 102, a learning module 103, an uncertainty update module 104, and an uncertainty parameter estimator 105 are realized by, for example, a processor in the data processing system 100 executing a computer program. Some functions such as the uncertainty parameter estimator 105 may not be required.

[0025] Unlabeled data 111 is input to a trained ML model 101. "Unlabeled data" refers to streaming data consisting of multiple data elements that have not yet been labeled (classified). The identity of the data elements in the streaming data may depend on the data source of the streaming data. For example, a data element may be a frame of video data or audio data, or a value detected by a sensor at a certain time. Learning may also be active learning. That is, the ML model 101 may typically be a supervised model, and is trained using a training dataset consisting of labeled data. The ML model 101 may be a neural network, a tree-structured model, or other models.

[0026] The trained ML model 101 outputs a class prediction probability for each data element in the unlabeled data 111. For each data element, the "class prediction probability" is the probability predicted by the ML model 101 for that data element, that is, the probability that the data element belongs to a class. For example, a class prediction probability of "1" means the class "positive," and a class prediction probability of "0" means the class "negative." Therefore, the class prediction probability is a value between 0 and 1. The closer the class prediction probability is to "1," the higher the probability that the class is "negative," and the closer the class prediction probability is to "0," the higher the probability that the class is "positive." The class prediction probability for each data element in the unlabeled data 111 is input to the uncertainty analyzer 102.

[0027] The uncertainty analyzer 102 calculates an uncertainty level for each data element of the unlabeled data 111 based on the input class prediction probability of the data element and a certain period of data including the data element. The maximum and minimum values ​​of the "uncertainty level" may be the same as the class prediction probability, for example, and therefore the "uncertainty level" may be a value between 0 and 1. The uncertainty level may be calculated using a predetermined calculation method (for example, a margin-based or entropy-based calculation method) using the class prediction probability. The calculated uncertainty level is input to the learning module 103.

[0028] The learning module 103 trains the ML model 101. Specifically, the learning module 103 receives the uncertainty level output from the uncertainty analyzer 102 and acquires, from the parameter database 112, a decision threshold, a lower margin, and an upper margin corresponding to the class determined from the uncertainty level. The learning module 103 determines whether to issue a labeling request depending on whether the uncertainty level falls within the uncertainty range. The labeling request may be issued to a predetermined function or computer system that labels data elements, but in this embodiment, it is issued to a user. That is, issuing a labeling request means requesting a user to label the data element. Specifically, for example, issuing a labeling request may involve providing a user interface (UI) that accepts label input to a display device or a user terminal (an information processing terminal (e.g., a client) serving as an input / output console) connected to an interface device of the data processing system 100. The UI may, for example, display data elements and their class prediction probabilities and may include a GUI component that accepts label input for the data elements. When a user assigns a label (class) to a data element in response to a labeling request, the data element becomes a data element (Sample) of labeled data 114 (Hard Samples). The labeled data 114 consists of one or more data elements, each labeled. This labeled data 114 becomes at least a part of the training data set, and the ML model 101 is trained, thereby improving the accuracy of the ML model 101.

[0029] In addition to being used for learning by the ML model 101, the labeled data 114 is input to the uncertainty update module 104. Each data element in the labeled data 114 input to the uncertainty update module 104 may be associated with a labeled class and a class prediction probability obtained from the ML model 101 for that data element. For each data element in the input labeled data 114, the uncertainty update module 104 updates at least one of the decision threshold, lower margin, and upper margin corresponding to that class in the parameter database 112 based on the labeled class and the class prediction probability. This is expected to result in a more appropriate decision threshold, lower margin, and upper margin for that class. Furthermore, the learning module 103 obtains the updated decision threshold, lower margin, and upper margin from the parameter database 112, and these decision threshold, lower margin, and upper margin are used in the processing of the learning module 103, which is expected to further improve the accuracy of the ML model 101.

[0030] The uncertainty parameter estimator 105 may estimate appropriate class-dependent uncertainty parameters based on the knowledge database 113 and reflect the estimated class-dependent uncertainty parameters in the parameter database 112 .

[0031] Below, detailed examples of functions and processes will be explained.

[0032] FIG. 2 shows the flow of processing performed by the learning module 103.

[0033] The learning module 103 is C Enter m corresponding to class C C l and m C u and are obtained from the parameter database 112, and Th C l C <Th C u It is determined whether or not (S201). C ​is the uncertainty level for Class C. Class C is U C It is determined by the result of comparing U with the decision threshold. C If is less than the decision threshold, C = "negative". C If is greater than or equal to the decision threshold, C is "positive". C l is the lower limit of the lower margin corresponding to class C, i.e., the lower threshold. C u is the upper end of the upper margin corresponding to class C, i.e., the upper threshold. If the uncertainty level is outside the uncertainty range, the uncertainty is low, i.e., the prediction by the ML model 101 is likely to be accurate. Conversely, if the uncertainty level is within the uncertainty range, the uncertainty is high, i.e., the prediction by the ML model 101 is likely to be inaccurate.

[0034] If the determination result of S201 is true (S201: True), that is, if the prediction by the ML model 101 is likely to be inaccurate, the learning module 103 requests the user to input labels for the data elements (S206). In other words, a labeling request is made.

[0035] If the determination result of S201 is false (S201: False), the learning module 103 calculates the remaining sampling budget B. B = sampling budget - (number of labeling requests) / (total number of samples). The "sampling budget" is the highest percentage of samples that require labeling. A "sample" is a data element. The "number of labeling requests" is the number of labeling requests that have been issued so far. The "total number of samples" is the total number of data elements that have been issued so far. In other words, the remaining sampling budget B represents the likelihood that issuing a labeling request will be permitted.

[0036] The learning module 103 acquires a random number X from a uniform distribution between 0 and 1 (S203). The learning module 103 determines whether the random number X is greater than the remaining sampling budget B (S204).

[0037] If the determination result of S204 is false (S204: False), that is, if it is determined that the remaining sampling budget is equal to or less than B, the learning module 103 issues a labeling request (S206). The data elements labeled by the user in response to the labeling request are adopted as part of the training dataset for the ML model 101.

[0038] If the determination result of S204 is true (S204: True), that is, if it is determined that the remaining sampling budget B has been exceeded, the learning module 103 does not issue a labeling request (S205). In this case, the class predicted by the ML model 101 is the label of the data element. Since the uncertainty level belongs to the uncertainty range, the data element labeled with the predicted class is not adopted as part of the training dataset of the ML model 101.

[0039] FIG. 3 shows the flow of the process performed by the uncertainty parameter estimator 105.

[0040] The plurality of parameters acquired by the learning module 103 from the parameter database 112 may each be a default value, but the uncertainty parameter estimator 105 may set at least one parameter to an appropriate parameter based on the knowledge database 113. Specifically, the uncertainty parameter estimator 105 estimates the decision threshold H based on the knowledge database 113 (S301), and also estimates the upper margin m of class C. C u and lower margin m C l are estimated based on the knowledge database 113 (S302). The estimated parameters are reflected (stored) in the parameter database 112 by the uncertainty parameter estimator 105.

[0041] More specifically, at least one of the following examples (1) to (3) may be adopted. Furthermore, for any of the following examples (1) to (3), the class-dependent uncertainty parameters may be estimated taking into consideration the current performance of the ML model 101 following the input of the current streaming data. For example, for at least one of the examples (1) to (3), the knowledge database 113 may have a table including a history or statistics of the past lower margins and upper margins for each class, and the uncertainty parameter estimator 105 may estimate the lower margins and upper margins of the class from the table. (1) For at least one class, the knowledge database 113 includes data representing the prediction difficulty of the class. The uncertainty parameter estimator 105 estimates a large uncertainty range based on the prediction difficulty. The uncertainty range may be estimated as either an upper margin or a lower margin. The higher the prediction difficulty, the larger the estimated uncertainty range. This increases the likelihood that the class prediction probability for a class with a high prediction difficulty falls within the uncertainty range, thereby increasing the likelihood of obtaining an accurate label in response to a labeling request. (2) For at least one class, the knowledge database 113 includes data representing the rank of the class. The uncertainty parameter estimator 105 estimates a larger uncertainty range for higher ranks based on the data representing the rank of the class. This increases the likelihood that the class prediction probability for a higher-ranked class falls within the uncertainty range, thereby increasing the likelihood of obtaining an accurate label in response to a labeling request. The rank may be a major rank or a minor rank. (3) The knowledge database 113 includes data representing the distribution of class proportions. The uncertainty parameter estimator 105 may set weights for classes based on the distribution of class proportions and change the uncertainty range based on the weights, or may set the weights in the parameter database 112 as class-dependent uncertainty parameters. The weights may be weights for learning, and the weights for the classes may be taken into account in learning the ML model 101. This allows an appropriate uncertainty range to be prepared for each class according to the distribution of the class proportions.

[0042] FIG. 4 shows the flow of the process performed by the uncertainty update module 104.

[0043] The uncertainty update module 104 performs S401 to S404 for each labeled data element.

[0044] That is, the uncertainty update module 104 obtains the upper margin and lower margin corresponding to the label (class) assigned to the labeled data element from the parameter database 112, and determines whether the class prediction probability of the data element is correct (S401). "The class prediction probability is correct" means that the magnitude relationship between the predicted class prediction probability and the judgment threshold matches the labeled class.

[0045] If the determination result of S401 is true (S401: True), the uncertainty update module 104 narrows the uncertainty range corresponding to the class (accurate class) assigned to the data element. Note that the degree to which the uncertainty range is narrowed may be determined using a predetermined rule (for example, a rule based on a weighted moving average or an exponential moving average).

[0046] If the determination result of S401 is false (S401: False), the uncertainty update module 104 expands the uncertainty range corresponding to the class according to the class prediction probability of the data element (i.e., the incorrect class). Note that the extent to which the uncertainty range is expanded may be determined using a predetermined rule (e.g., a rule based on a weighted moving average or an exponential moving average).

[0047] After S402 or S403, the uncertainty update module 104 updates the decision threshold (S404). The decision threshold does not necessarily need to be updated. The uncertainty update module 104 stores the parameters updated in S402 or S403 and S404 in the parameter database 112. After this, the parameters obtained from the learning module 103 are the updated parameters.

[0048] A specific example of parameter updating will be described with reference to Figures 5A and 5B. Figures 5A and 5B take a certain data element as an example. Assume that the target value (correct answer of class prediction probability) of the data element is "1," and therefore the class "positive" is assigned to the data element.

[0049] FIG. 5A shows an example where the class predicted probability is below the decision threshold and falls in the lower margin of the uncertainty range.

[0050] The class prediction probability is "0.3," which is below the decision threshold "0.5," so the class "negative" is predicted. Since the correct class is "positive," an incorrect prediction was made (S401: False), and the class "negative" is an incorrect class. Furthermore, as shown in FIG. 5A, the class prediction probability "0.3" belongs to the lower margin, which is part of the uncertainty range. Therefore, a labeling request is issued (S201: True and S206 in FIG. 2). In response to the labeling request, the correct class (label) is assigned to the data element.

[0051] 5A, there may be a tendency for the class prediction probability of a data element that should be assigned the correct class "positive" to be low. The class prediction probability of a data element that should be assigned the class "positive" should be a class prediction probability that exceeds the decision threshold.

[0052] Therefore, the uncertainty update module 104 increases the upper margin. Increasing the upper margin may involve decreasing the decision threshold. The amount of change in the upper margin (and decision threshold) may be specified by a user or may be determined according to the difference between the decision threshold (e.g., 0.5) and the actual class prediction probability (e.g., 0.3). When the upper margin is increased, the upper threshold may be increased, and when the decision threshold is decreased, the upper threshold may or may not be changed.

[0053] FIG. 5B shows an example where the class predicted probability is above the decision threshold and falls within the upper margin of the uncertainty range.

[0054] The class prediction probability is "0.7", which is above the decision threshold of 0.5, so the class "positive" is predicted. Since the correct class is "positive", the prediction is correct (S401: True). However, as shown in FIG. 5B, the class prediction probability "0.7" belongs to the upper margin, which is part of the uncertainty range. Therefore, a labeling request is issued (S201: True and S206 in FIG. 2). In response to the labeling request, the correct class (label) is assigned to the data element.

[0055] According to the example shown in FIG. 5B, even if the correct class "positive" is predicted, labeling requests may be issued frequently if left as is.

[0056] Therefore, the uncertainty update module 104 reduces the upper margin. The function of the upper margin may involve increasing the decision threshold. The amount of change in the upper margin (and decision threshold) may be specified by the user or may be determined based on the difference between the decision threshold (e.g., 0.5) and the actual class prediction probability (e.g., 0.7). If the upper margin is reduced, the upper threshold may be reduced, and if the decision threshold is increased, the upper threshold may or may not be changed.

[0057] When the above-described functions and processes are deployed in the hardware configuration shown in FIG. 6, for example, learning and inference are performed according to the flow illustrated in FIG.

[0058] For example, a trained ML model 101 is deployed on the inference server 620. The inference server 620 receives unlabeled data as streaming data from a data source external to the inference server 620, for example, via an API (Application Programming Interface). The inference server 620 inputs the unlabeled data into the ML model 101, thereby outputting a class prediction probability for each data element in the unlabeled data. Data including the class prediction probability for each data element may be accumulated as inference results 730. The inference server 620 may collect these inference results 730 and transmit an inference log 701 based on the collected inference results 730 to the learning server 610. The processing performed by the inference server 620 may be performed by an inference module (not shown). The inference module may be a function implemented by a program executed by a processor in the inference server 620.

[0059] Inference server 620 may also receive, e.g., via an API, labeled data elements after class predicted probabilities have been output. In addition to inference results 730, inference log 701 may include unlabeled data elements and subsequent labeled data elements.

[0060] The inference log 701 input to the learning server 610 may include inference results 730, unlabeled data elements, and labeled data elements.

[0061] The learning server 610 may include a learning network 750. The learning network 750 includes an uncertainty analyzer 102, a learning module 103, an uncertainty update module 104, and an uncertainty parameter estimator 105. The ML model 101 is trained using an inference log 701 and a training dataset including unlabeled data elements and labeled data elements. For example, the ML model 101 may be trained based on the unlabeled data elements themselves, the class prediction probabilities as inference results for the unlabeled data elements, and the labels (classes) of the labeled data elements. The learning network 750 (e.g., the learning module 103) may generate a learning log 702 including data regarding the differences between the ML model 101 at the time of deployment to the inference server 620 and the ML model 101 after training. The learning log 702 may be sent to the inference server 620, and the learning log 702 may be reflected in the ML model 101 in the inference server 620, thereby constructing the ML model 101 after training (after accuracy improvement) in the inference server 620. Instead of the learning log 702, the ML model 101 after training itself may be deployed to the inference server 620.

[0062] Although one embodiment has been described above, this is merely an example for explaining the present invention and is not intended to limit the scope of the present invention to only this embodiment. The present invention can be implemented in various other forms. For example, there may be two or more decision thresholds, and therefore there may be three or more classes and three or more margins. For example, there may be first and second decision thresholds, and for each class, the uncertainty range may be composed of a first margin that is a range below the first decision threshold, a second margin that is a range equal to or greater than the first decision threshold and less than the second decision threshold, and a third margin that is equal to or greater than the second decision threshold.

[0063] The above description can be summarized as follows: The following summary may include supplementary explanations and explanations of variations of the above description.

[0064] As described above, the data processing system 100 or the learning server 610, which is an example of a learning system, includes the uncertainty analyzer 102 and the learning module 103. For each data element in the imbalanced streaming data input to the ML model 101, the uncertainty analyzer 102 receives the class prediction probability of the data element from the ML model 101 and calculates the uncertainty level of the class according to the class prediction probability based on the class prediction probability. For each data element, the learning module 103 identifies an uncertainty range corresponding to the class according to the class prediction probability from among uncertainty ranges prepared for each class and defined as having high uncertainty, and performs range determination (S201) to determine whether the uncertainty level calculated for the class of the data element is outside the identified uncertainty range. Depending on the result of the range determination, the learning module 103 may issue or not issue a labeling request for the data element. Specifically, for example, the learning module 103 issues a labeling request for a data element for which the result of the range determination is false. The learning module 103 may train the ML model 101 based on a labeled dataset consisting of data elements labeled in response to the issued labeling request, thereby maintaining the prediction accuracy of the ML model 101 when the imbalanced streaming data is used as input.

[0065] For data elements for which the result of the range determination is true, the learning module 103 may calculate a remaining sampling budget, which is a threshold, based on the number of labeling requests issued so far relative to the number of data elements so far, obtain a random value from a range of possible uncertainty levels (for example, the same range as the range of possible class prediction probabilities), and perform budget determination (S204) to determine whether the obtained random value is larger than the remaining sampling budget. If the result of the budget determination is false, the learning module 103 issues a labeling request for the data element. If the result of the budget determination is true, the learning module 103 does not issue a labeling request for the data element. This allows the frequency of issuing labeling requests to be kept appropriate.

[0066] The data processing system 100 or the learning server 610, as an example of a learning system, may further include an uncertainty update module 104. A predicted class may be determined based on the magnitude relationship between the class prediction probability and one or more decision thresholds. For each class, the uncertainty range may have two or more margins, which are two or more ranges bounded by one or more decision thresholds. The uncertainty update module 104 may perform a correct answer determination (S401) for each data element labeled with a class in response to a labeling request, determining whether the predicted class, determined based on the magnitude relationship between the class prediction probability of the data element and one or more decision thresholds, matches the labeled class. If the result of the correct answer determination is true, the uncertainty update module 104 may reduce the uncertainty range corresponding to the predicted class (S402). If the result of the correct answer determination is false, the uncertainty update module 104 may expand the uncertainty range corresponding to the predicted class (S403). This allows the size and structure of the uncertainty range to be maintained at an appropriate size and structure, thereby allowing the frequency of issuing labeling requests to be maintained at an appropriate level. More specifically, for example, if the result of the correct answer determination is true, the uncertainty update module 104 may reduce the margin to which the class prediction probability belongs within the uncertainty range corresponding to the predicted class. If the result of the correct answer determination is false, the uncertainty update module 104 may expand the margin to which the class prediction probability belongs within the uncertainty range corresponding to the predicted class.

[0067] The data processing system 100 may be configured to analyze defective products using streaming data from various sensors, for example. While the predictive accuracy of the ML model 101 may be reduced due to the rarity of defective products, this embodiment allows the accuracy of the ML model 101 to be maintained.

[0068] The data processing system 100 may also be configured to predict air pollution levels based on streaming data from a sensor, for example. Over time, degradation of the sensor may cause concept drift, reducing the accuracy of the ML model 101. However, according to this embodiment, the accuracy of the ML model 101 can be maintained even when concept drift occurs. [Explanation of symbols]

[0069] 100: Data Processing System

Claims

1. an uncertainty analyzer and a learning module; The uncertainty analyzer, for each data element in the imbalanced streaming data input to the machine learning model, receiving a class prediction probability for the data element from the machine learning model; Calculating the uncertainty level of the class according to the class prediction probability based on the class prediction probability; The learning module, for each data element, Identifying the uncertainty range corresponding to the class according to the class prediction probability from among the uncertainty ranges prepared for each class and defined as having high uncertainty; performing a range determination that determines whether the calculated uncertainty level for the class of data elements is outside the specified uncertainty range; issuing or not issuing a labeling request for the data element depending on the result of the range determination; The learning module trains the machine learning model based on a labeled dataset consisting of data elements labeled in response to issued labeling requests. Learning system.

2. For data elements for which the result of the range test is true, the learning module: Calculating a threshold remaining sampling budget based on the number of labeling requests issued to date relative to the number of data elements to date; obtaining a random value from a range of possible values ​​for the uncertainty level; performing a budget determination to determine whether the obtained random value is greater than the remaining sampling budget; If the result of the budget determination is false, issuing a labeling request for the data element; If the result of the budget determination is true, the labeling request for the data element is not issued; The learning system of claim 1 .

3. For data elements for which the result of the range determination is false, the learning module issues a labeling request. The learning system of claim 1 .

4. further comprising an uncertainty update module; A predicted class is determined according to a magnitude relationship between the class prediction probability and one or more decision thresholds; For each class, the uncertainty range has two or more margins, which are two or more ranges bounded by the one or more decision thresholds; The uncertainty update module, for each data element labeled with a class in response to a labeling request, performing a correct answer determination as to whether a predicted class determined according to a magnitude relationship between the class prediction probability of the data element and the one or more determination thresholds matches a labeled class; If the result of the correct answer determination is true, reducing the uncertainty range corresponding to the predicted class; If the result of the correct answer determination is false, expand the uncertainty range corresponding to the predicted class. The learning system of claim 1 .

5. If the result of the correct answer determination is true, the uncertainty update module reduces a margin to which the class prediction probability belongs within the uncertainty range corresponding to the predicted class; If the result of the correct answer determination is false, the uncertainty update module expands a margin to which the class prediction probability belongs within the uncertainty range corresponding to the predicted class. The learning system according to claim 4 .

6. an uncertainty parameter estimator; the uncertainty parameter estimator estimates a larger uncertainty range for a class as the prediction difficulty of the class increases based on data representing the prediction difficulty of the class; The uncertainty range of a class is the estimated uncertainty range, The learning system of claim 1 .

7. an uncertainty parameter estimator; The uncertainty parameter estimator estimates a larger uncertainty range based on data representing the rank of a class, the higher the rank. The uncertainty range of a class is the estimated uncertainty range, The learning system of claim 1 .

8. an uncertainty parameter estimator; the uncertainty parameter estimator includes data representing a distribution of class proportions and estimates uncertainty ranges based on the distribution; The uncertainty range of a class is the estimated uncertainty range, The learning system of claim 1 .

9. For each data element in the imbalanced streaming data input to the machine learning model, receiving a class prediction probability for the data element from the machine learning model; Calculating the uncertainty level of the class according to the class prediction probability based on the class prediction probability; For each such data element: Identifying the uncertainty range corresponding to the class according to the class prediction probability from among the uncertainty ranges prepared for each class and defined as having high uncertainty; performing a range determination that determines whether the calculated uncertainty level for the class of data elements is outside the specified uncertainty range; issuing or not issuing a labeling request for the data element depending on the result of the range determination; training the machine learning model from a labeled dataset consisting of data elements labeled in response to issued labeling requests; A learning method that uses computers.

Images