Computer-implemented method, computer-implemented system, and computer program (classification based on imbalanced data sets)

Abstract

To provide a computer-implemented method, computer-implemented system and computer program product for classification based on an imbalanced dataset.SOLUTION: A method comprises: generating a plurality of predictive models based on a plurality of positive samples included in an imbalanced dataset, where the amount of the plurality of positive samples is less than or equal to the amount of a plurality of negative samples included in the imbalanced dataset, and where each sample in the plurality of positive samples and the plurality of negative samples includes a plurality of parameters; identifying a plurality of respective influential parameter groups from the plurality of parameters for the plurality of positive samples; and determining a final predictive model based on the plurality of predictive models and the plurality of influential parameter groups, where the final predictive model is used to classify a sample into a positive or negative type.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to classification, and more particularly, to methods, systems, and computer program products for generating predictive models based on imbalanced datasets. [Background technology]

[0002]

[0003] Nowadays, classification techniques are widely used in various work environments. For example, samples containing multiple parameters of a computing system may be collected to predict whether the computing system will fail. A predictive model may be generated in advance based on an imbalanced dataset containing multiple samples of the computing system collected at a previous time point.

[0003] Current classification techniques have the problem of being inadequate in determining the number of failed positive cases, especially when the positive rate is less than, say, 0.01%, because the results are too small to determine the positive rate. Additionally, current solutions suffer from unsatisfactorily high false positive rates and incorrectly predicted negative cases due to inaccurate patterns learned by the predictive model. Summary of the Invention [Problem to be solved by the invention]

[0004] A computer-implemented method, a computer-implemented system, and a computer program product for classification based on an imbalanced dataset are provided. [Means for solving the problem]

[0005] According to embodiments of the present disclosure, a computer-implemented method is provided that can be implemented by one or more processors. In the method, one or more processors respectively generate a plurality of predictive models based on a plurality of positive samples included in an imbalanced dataset, the amount of the plurality of positive samples being less than or equal to the amount of a plurality of negative samples included in the imbalanced dataset, and each sample in the plurality of positive samples and the plurality of negative samples includes a plurality of parameters. The one or more processors respectively identify a plurality of influential parameter sets for the plurality of positive samples from the plurality of parameters. The one or more processors determine a final predictive model based on the plurality of predictive models and the plurality of influential parameter sets, and the final predictive model is used to classify samples into either a positive type or a negative type. These embodiments enable positive models to be generated based on an imbalanced dataset in an accurate and effective manner.

[0006] According to another embodiment of the present disclosure, there is provided a computer-implemented system comprising a computer processor coupled to a computer-readable memory unit, said memory unit containing instructions that, when executed by said computer processor, perform the above-described method.

[0007] According to another embodiment of the present disclosure, there is provided a computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by an electronic device to cause the electronic device to perform the operations of the method described above.

[0008] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of several embodiments of the present disclosure in the accompanying drawings, in which like references generally refer to like components in the embodiments of the present disclosure. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a cloud computing node according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating a cloud computing environment in accordance with an embodiment of the present disclosure. [Figure 3] FIG. 3 is a diagram illustrating abstraction model layers according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of an imbalanced dataset for generating a predictive model according to an embodiment of the present disclosure. [Figure 5] FIG. 5 illustrates an example of generating a predictive model based on an imbalanced dataset, according to an embodiment of the present disclosure. [Figure 6] FIG. 6 illustrates an example of generating a predictive model based on positive samples in an imbalanced dataset, according to an embodiment of the present disclosure. [Figure 7] FIG. 7 illustrates an example of generating a model set of a target sample according to an embodiment of the present disclosure. [Figure 8] FIG. 8 illustrates an example procedure for updating a model set based on the difference between a set of influential target parameters and a set of influential parameters, according to an embodiment of the present disclosure. [Figure 9] FIG. 9 illustrates an exemplary flowchart of a method for generating a final predictive model based on an imbalanced dataset, according to an embodiment of the present disclosure. [Figure 10] FIG. 10 illustrates an exemplary flowchart of a method for classifying a target sample based on a final prediction model according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Some embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings, which illustrate embodiments of the present disclosure, although the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments disclosed herein.

[0011] Although this disclosure includes detailed descriptions of cloud computing, implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present disclosure can be practiced in conjunction with any other type of computing environment now known or developed in the future.

[0012] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0013] The characteristics are as follows:

[0014] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time or network storage, automatically as needed, without the need for human interaction with the service provider.

[0015] Broad network access: Computing power is available over the network and can be accessed through standard mechanisms, facilitating use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, PDAs).

[0016] Resource Pooling: Computing resources from a provider are pooled and offered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated based on demand. Consumers generally have no control or knowledge of the exact location of the resources they are provided with, so there is a sense of location independence. However, consumers may be able to identify their location at a higher level of abstraction (e.g., country, state, data center).

[0017] Rapid Elasticity: Computing capacity can be provisioned quickly and elastically, sometimes automatically, to instantly scale out and quickly released to instantly scale in. To the consumer, the computing power available to provision often appears unlimited, and can be purchased at any time and in any quantity.

[0018] Metered Services: Cloud systems leverage measurement capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts) to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of utilized services.

[0019] The service model is as follows:

[0020] Software as a Service (SaaS): The functionality offered to the consumer is the availability of a provider's applications running on a cloud infrastructure. The applications can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., webmail). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, except for limited user-specific application configuration settings.

[0021] Platform as a Service (PaaS): The capability offered to consumers is to deploy applications they create or acquire onto a cloud infrastructure using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of their hosting environment.

[0022] Infrastructure as a Service (IaaS): The functionality provided to consumers is the provisioning of processors, storage, networking, and other basic computing resources on which the consumer can deploy and run any software, including operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, and deployed applications, and in some cases partial control over some network components (e.g., host firewalls).

[0023] The deployment model is as follows:

[0024] Private Cloud: This cloud infrastructure is dedicated to a specific organization and can be managed by that organization or a third party, and can exist on-premise or off-premise.

[0025] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by those organizations or a third party and can exist on-premises or off-premises.

[0026] Public cloud: This cloud infrastructure is available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0027] Hybrid cloud: This cloud infrastructure combines two or more cloud models (private, community, or public), each of which retains its inherent nuances but is bound by standards or specific technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0028] A cloud computing environment is a service-oriented environment that emphasizes statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0029] An example cloud computing node is illustrated schematically in Figure 1. It should be noted that cloud computing node 10 is merely an example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the present disclosure described herein. In any event, cloud computing node 10 may implement and / or perform any of the functions described above.

[0030] Cloud computing node 10 includes a computer system / server 12 or a portable device such as a communications device, which is operable with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, suitable for use with computer system / server 12 include personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0031] The computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be stored in both local and remote computer system storage media, including memory storage devices.

[0032] 1, computer system / server 12 in cloud computing node 10 is depicted as a general-purpose computer device. Examples of components of computer system / server 12 include one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components, including system memory 28, to processor 16.

[0033] Bus 18 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port (AGP), and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0034] Computer system / server 12 typically includes a variety of computer system-readable media, which can be any available media that can be accessed by computer system / server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

[0035] The system memory 28 may include a computer system-readable medium such as a volatile memory, such as RAM 30 or cache memory 32, or both. The computer system / server 12 may also include other removable / non-removable and volatile / non-volatile computer system-readable media. As an example, the storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, commonly referred to as a "hard drive"). Also, although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a floppy disk) and an optical disk drive for reading from and writing to a removable, non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these examples, each may be connected to the bus 18 by one or more data medium interfaces. As further shown and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present disclosure.

[0036] As an example, a program / utility 40 having a set (at least one) of program modules 42 may be stored in memory 28, as may an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of embodiments of the present disclosure.

[0037] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, or display 24, one or more devices that allow a user to interact with the computer system / server 12, or any device (e.g., a network card, modem, etc.) that allows the computer system / server 12 to communicate with one or more other computer devices, or any combination thereof. Such communication may occur via an input / output (I / O) interface 22. The computer system / server 12 may also communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. Note that other hardware and / or software components (not shown) may be used with the computer system / server 12, such as microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data archive storage systems, etc.

[0038] FIG. 2 illustrates an exemplary cloud computing environment 50. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10, with which local computing devices used by cloud consumers (e.g., PDAs or cell phones 54A, desktop computers 54B, laptop computers 54C, or automobile computer systems 54N, or combinations thereof) can communicate. The nodes 10 can communicate with each other. The nodes 10 can be physically or virtually grouped (not shown) in one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof. This enables the cloud computing environment 50 to provide infrastructure, platform, or software as a service, or combinations thereof, for which cloud consumers are not required to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-N illustrated in FIG. 2 are exemplary only, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network-addressable connection (e.g., using a web browser), or both.

[0039] Next, a set of functional abstraction layers provided by the cloud computing environment 50 (FIG. 2) is shown in FIG. 3. It should be understood in advance that the components, layers, and functions shown in FIG. 3 are merely exemplary, and embodiments of the present disclosure are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0040] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage device 65, and network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0041] The virtualization layer 70 provides an abstraction layer from which virtual entities such as virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75 can be provided.

[0042] By way of example, the management layer 80 may provide the following functionality: Resource provisioning 81 enables dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. By way of example, these resources may include application software licenses. Security enables identification and verification of cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables allocation and management of cloud computing resources so that requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 enables advance arrangement and procurement of anticipated future cloud computing resources required in accordance with SLAs.

[0043] The workload layer 90 provides examples of functionality that can be utilized in a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and classification processing 96.

[0044] It should be noted that the classification process 96 according to an embodiment of the present disclosure may be implemented by the computer system / server 12 of FIG. 1 . The classification process 96 may be implemented in various types of work environments. For example, the classification process 96 may be used to generate a predictive model for determining whether a computing system will experience a fault based on a historical dataset. Here, a fault refers to an abnormal state of the computing system that requires further support from a technical engineer. However, because computing systems rarely experience faults and operate normally most of the time, the dataset will contain only a small number of positive samples (where the computing system experiences an abnormality) compared to a large number of negative samples (where the computing system operates normally).

[0045] An overview of an imbalanced dataset will be described with reference to FIG. 4. FIG. 4 illustrates a diagram 400, which is an example of an imbalanced dataset for generating a predictive model according to an embodiment of the present disclosure. In FIG. 4, an imbalanced dataset 410 includes a plurality of positive samples 412 and a plurality of negative samples 414, but the amount of the plurality of positive samples 412 is much smaller than the amount of the plurality of negative samples 414. For example, a dataset may include 10,000 samples, of which 9,500 samples are negative samples and only 500 samples are positive samples. In this case, if a predictive model is generated directly from the imbalanced dataset, the accuracy of the predictive model will not be satisfactory.

[0046] Several solutions for imbalanced classification have been proposed. One of them is to generate a balanced dataset from an imbalanced dataset using oversampling and undersampling. Another method is to use the Synthetic Minority Oversampling Technique (SMOTE) to simulate positive samples and generate a balanced dataset. A predictive model can then be trained using the balanced dataset. However, these methods require complex procedures and a large amount of work, and the generated predictive model has low accuracy.

[0047] In view of the above drawbacks, an embodiment of the present disclosure may provide a solution for generating a predictive model. An outline of an embodiment of the present disclosure will be described with reference to FIG. 5. FIG. 5 illustrates an example of generating a predictive model based on an imbalanced dataset according to an embodiment of the present disclosure. In FIG. 5, an imbalanced dataset 410 may be used to generate a final predictive model for classifying samples into either positive or negative types. Here, the imbalanced dataset 410 may include only a small number of positive samples 412, ..., 510 and a large number of negative samples 414. Each sample may include multiple parameters. In an environment for predicting failures in a computing system, the parameters may include multiple aspects of the computing system. An example data structure of a sample is shown in Table 1.

[0048] [Table 1]

[0049] Here, the positive sample may include parameters when the computing system fails (where the “failure” parameter is set to “true”), and the negative sample may include parameters when the computing system is in a good state (where the “failure” parameter is set to “false”). As shown in FIG. 5 , individual predictive models may be generated based on the individual positive samples. Specifically, a predictive model 520 is generated for the positive sample 412, ..., and a predictive model 530 is generated for the positive sample 510. Furthermore, an influence predictor analysis may be performed on the positive samples and the corresponding predictive models to identify influential parameter sets for the positive samples. Referring to FIG. 5 , an influential parameter set 522 may be identified for the positive sample 412, an influential parameter set 532 may be identified for the positive sample 510, and so on. Furthermore, a final predictive model 540 may be generated based on the multiple predictive models 520, ..., and 530 and the multiple influential parameter sets 522, ..., and 532. In these embodiments, a prediction model is generated based on a positive sample, so the information in the positive sample can be fully extracted to construct the final prediction model, thereby significantly improving the accuracy level of the final prediction model.

[0050] Hereinafter, with reference to FIG. 6 , details of generating multiple models based on multiple positive samples in an imbalanced dataset will be described. In some embodiments of the present disclosure, a balanced training dataset may be generated based on each positive sample in the imbalanced dataset 410. FIG. 6 shows a diagram 600, which is an example of generating a predictive model based on positive samples in an imbalanced dataset according to an embodiment of the present disclosure. In FIG. 6 , a balanced training dataset 610 is generated for the positive samples 412 based on the imbalanced dataset 410. Here, the training dataset 610 includes at least one positive training sample 620 and multiple negative training samples 630. Furthermore, the predictive model 520 may be generated based on existing machine learning techniques. For example, an initial predictive model may be obtained and trained based on the training samples in the training dataset 610.

[0051] The training samples of the training dataset 610 can be obtained in various ways. In some embodiments, the negative training samples may be selected directly from the plurality of negative samples 414 in the imbalanced dataset 410. Alternatively, or in addition, or both, the positive samples 412 may be modified to obtain the negative training samples. For example, one or more parameters of the positive samples 412 may be adjusted to simulate the negative training samples. Given that CPU utilization is an important parameter for detecting failures in a computing system, the higher the CPU utilization, the more likely a failure will occur in the computing system. Therefore, the CPU utilization of the positive samples 412 may be adjusted to a lower value within a normal range to simulate the negative training samples.

[0052] In some embodiments, a threshold (such as 1000 or another value) may be defined for the total number of samples in the balanced training data set 610. At this point, the amount of negative training samples may be determined to be 1000 / 2=500. Some of the negative training samples may be selected from the negative samples in the imbalanced data set 410, and some of the negative training samples may be generated by modifying the positive samples 412. These embodiments allow negative training samples to be obtained in an easy and effective manner.

[0053] In some embodiments of the present disclosure, the positive samples 412 may be obtained directly from the imbalanced dataset 410 and serve as the positive training samples 620. Because the training dataset 610 should be a balanced dataset, the amount of positive training samples should be similar to the amount of negative training samples. To account for the scarcity of positive training samples, the positive training samples may be assigned a weight based on the ratio of the number of multiple negative training samples to the number of at least one positive training sample. Continuing with the above example, since the number of negative training samples 630 is 500 (half of the total number "1000"), there is only one positive training sample 620, and the weight may be set to 500. These embodiments do not require complex procedures for simulating a large number of positive training samples, but instead can directly obtain positive training samples in a simple and effective manner.

[0054] In some embodiments, the training dataset 610 may include one or more positive training samples, and therefore, more positive training samples may be generated based on the positive samples 412. For example, the CPU usage in the positive samples 412 may be slightly increased to simulate another positive training sample. Furthermore, the acquired positive training samples 620 and negative training samples 630 may be added to the training dataset 610 for the training procedure. Thus, embodiments of the present disclosure do not require complex procedures, and the workload for generating a training dataset may be reduced. In some embodiments, the balanced training dataset 610 may be used to train an initial predictive model. After the training procedure, the predictive model 520 may be automatically generated.

[0055] The above paragraphs merely provide an example procedure for generating a predictive model 520 for a positive sample 412. Other positive samples may be subjected to similar processing to obtain multiple predictive models 520, ..., and 530. Once the predictive model 520 is generated, an influence predictor analysis may be performed on the predictive model to identify a set of influential parameters for the positive sample 412. The influence predictor analysis is intended to analyze the relationship between a sample and a predictive model generated based on the sample. Specifically, from multiple parameters, an influential set of parameters that cause the sample to be classified as a positive type may be identified. For example, the parameters (CPU usage, memory usage, ...) from Table 1 above may be identified as the influential set of parameters 522 for the positive sample 412.

[0056] In some embodiments, a set of influential parameters may be identified for each positive sample, where the set of influential parameters includes a set of parameters among the plurality of parameters that cause the positive sample to be classified as a positive type. Specifically, the Local Interpretable Model Agnostic Explanations (LIME) method may be used to identify the set of influential parameters. Details of the LIME method will be omitted below. To date, the LIME method has become a mature technology, and various LIME tools have been provided for determining the set of influential parameters. In some embodiments, the positive sample and the predictive model are input into the LIME tool, which then outputs a set of influential parameters associated with the positive sample and the predictive model. Here, the set of influential parameters includes a set of parameters among the plurality of parameters that cause the positive sample to be classified as a positive type. Thus, all predictive models may be processed using the LIME tool, such that an influential parameter set 522 is identified for the positive sample 412 based on the predictive model 520, and an influential parameter set 532 is identified for the positive sample 510 based on the predictive model 530.

[0057] Furthermore, a final prediction model 540 may be generated based on multiple prediction models 520, ..., 530 and multiple influential parameter sets 522, ..., 532. Here, the final prediction model may be used to classify samples into either positive or negative types. In these embodiments, multiple prediction models generated based on multiple positive samples may provide more knowledge necessary for prediction. Because the multiple prediction models are constructed based on different training datasets, the multiple prediction models may reflect various aspects for prediction. That is, the final prediction model may contain rich knowledge that fully reflects all factors for determining failure. Here, the multiple prediction models may work together to classify additional samples, and thus the accuracy level of the final prediction model may be significantly improved.

[0058] In some embodiments, the final prediction model 540 may be updated over time. For example, more positive samples may be collected by monitoring multiple parameters during operation of the computing system. As more positive samples are added to the imbalanced dataset 410, more predictive models may be generated based on the more positive samples, and more influential parameters may be identified for the more predictive models. The final prediction model may then be updated based on the more predictive models and the more influential parameters. According to these embodiments, the final prediction model 540 may be updated based on more positive samples, such that the accuracy level of the final prediction model 540 may be gradually increased.

[0059] In some embodiments, the final predictive model 540 may be used to classify incoming samples. This will be described in more detail below with reference to FIG. 7, which illustrates an example diagram 700 for generating a model set for a target sample according to an embodiment of the present disclosure. In FIG. 7, a target sample 710 may be input to the final predictive model 540, and the data structure of the target sample 710 is identical to the data structure of the training samples used to generate the final predictive model 540. In the final predictive model 540, the target sample 710 may be input to each of multiple predictive models 520, ..., 530. Here, the multiple predictive models 520, ..., 530 may operate independently and output multiple prediction types 720, ..., 730. For example, the predictive model 520 may output a negative type 720, ..., the predictive model 530 may output a positive type 730.

[0060] Furthermore, a model set 740 may be created based on the results from multiple predictive models 520, ..., 530. Here, if a prediction type is determined to indicate a positive type, the predictive model corresponding to the positive type may be added to the model set 740. That is, the predictive model(s) that output the positive prediction type(s) may be added to the model set 740. In FIG. 7, because the predictive model 530 outputs a positive prediction type, the predictive model 530 is added to the model set 740 (represented as predictive model 530'). Here, the amount of models in the model set 740 is the same as the amount of positive types output by the final predictive model 540. In these embodiments, the model set 740 may be used as a basis for classifying the target sample 710.

[0061] In some embodiments of the present disclosure, if the model set 740 is empty, it indicates that all the predictive models 520, ..., 530 classify the target sample 710 as a negative type. At this point, the target sample 710 may be classified as a negative type. Compared to existing solutions that use only one predictive model, in this embodiment, all the predictive models 520, ..., 530 output a negative type, which increases the confidence of the final classification type.

[0062] In some embodiments, if the model set 740 is not empty (i.e., contains one or more predictive models that output a positive type), the predictive model(s) in the model set may be subjected to further processing. Referring to FIG. 8 , this figure illustrates an example procedure 800 for updating a model set based on differences between influential target parameters and influential parameters, according to an embodiment of the present disclosure. In FIG. 8 , influential target parameters may be identified from multiple parameters for a target sample 710. Here, the LIME method may be performed on the predictive model 530 to identify influential target parameters 810 for the target sample 710. Furthermore, the influential parameters 532 associated with the predictive model 530 may be obtained from the final predictive model 540.

[0063] Here, the influential parameter set 532 includes parameters that cause the positive sample 510 to be classified as a positive type. Here, the influential parameter set is an index for measuring whether the predictive model 530 is suitable for classifying subsequent samples. If the influential parameter set for the subsequent sample is similar to the influential parameter set 532, it means that the predictive model 530 is suitable; if not, it means that the predictive model 530 is not suitable. Therefore, to update the model set 740, a difference 820 may be determined between the influential target parameter set 810 and the influential parameter set 532. In some embodiments, if the difference 820 exceeds a predefined threshold, it indicates that the positive type 730 is output based on different parameters other than those included in the influential parameter set 532, and therefore the output of the predictive model 530 is unreliable and should be rejected. Otherwise, if the difference 820 is below a predefined threshold, it indicates that the positive type 730 is output based on parameters similar to those included in the influential parameter group 532, and therefore the output of the predictive model 530 is reliable, so it is desirable to adopt the predictive model.

[0064] In some embodiments, the threshold may be determined based on the overlap between the influential target parameter set 810 and the influential parameter set 532. If the influential target parameter set 810 and the influential parameter set 532 completely overlap, the output of the predictive model 530 is adopted. An overlap rate (e.g., 90% or some other value) may be defined as the threshold, and if the overlap rate between the two sets exceeds 90%, the output may be reliable; otherwise, the output may be unreliable and rejected. Further, continuing with the above example of determining a fault in a computing system, if the influential parameter set 532 includes 10 parameters while the influential target parameter set 810 includes 8 parameters (only 5 of which are included in the influential parameter set 532), the difference 820 may be below the threshold, and therefore the predictive model 530 may be removed from the model set 740. In another example, if the difference 820 exceeds the threshold, the predictive model 530 may be retained in the model set 740.

[0065] Although the above paragraphs describe processing for only one predictive model 530, other predictive models in the model set 740 may be processed in a similar manner so that the model set 740 is updated. After all predictive models have been processed according to FIG. 8, the updated model set 740 may be used to determine the final type of the target sample 710. If the updated model set is empty, this indicates that all reliable predictive models believe that the target sample 710 belongs to the negative type, and therefore the target sample 710 may be classified as the negative type. Alternatively, if the updated model set is not empty, this indicates that one or more reliable predictive models believe that the target sample 710 belongs to the positive type. Therefore, the target sample may be classified as the positive type. In these embodiments, all predictive models that output unreliable results are removed from the model set 740, and thus the model set 740 includes only one or more reliable predictive models. This may improve the reliability of the final output.

[0066] While the above paragraphs describe embodiments in a computing system environment, embodiments of the present disclosure may alternatively, additionally, or both implement a medical environment, an image processing environment, etc. In a medical environment, an imbalanced dataset may include positive samples associated with individuals with cancer and negative samples associated with healthy individuals, and thus a predictive model may be generated based on the imbalanced dataset. In an image processing environment, an imbalanced dataset may include positive samples associated with photos of unqualified products and negative samples associated with photos of qualified products, and thus a predictive model may be generated based on the imbalanced dataset. These embodiments enable accurate and effective predictive models to be generated from imbalanced datasets in a variety of work environments.

[0067] The present disclosure may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium having stored thereon computer-readable program instructions for causing a processor to perform aspects of the present disclosure.

[0068] For details regarding embodiments of the present disclosure, please refer to FIG. 9 . FIG. 9 illustrates an exemplary flowchart of a method 900 for generating a final predictive model based on an imbalanced dataset according to an embodiment of the present disclosure. Here, method 900 may be implemented by one or more processors. In block 910, multiple predictive models are generated based on multiple positive samples included in the imbalanced dataset, where the amount of the multiple positive samples is less than or equal to the amount of multiple negative samples included in the imbalanced dataset, and each sample of the multiple positive samples and negative samples includes multiple parameters. In block 920, multiple influential parameter sets for the multiple positive samples are identified from the multiple parameters, respectively. In block 930, a final predictive model is determined based on the multiple predictive models and the multiple influential parameter sets, and the final predictive model is used to classify samples into either a positive type or a negative type.

[0069] In some embodiments, a predictive model in the plurality of predictive models is generated based on positive samples in the plurality of positive samples. Specifically, a training dataset is generated based on the positive samples, the training dataset including at least one positive training sample and multiple negative training samples. The predictive model is generated by training an initial predictive model using the training dataset.

[0070] In some embodiments, the plurality of negative training samples are obtained by either selecting a negative sample from the plurality of negative samples and changing a positive sample to a negative sample, and adding the plurality of negative training samples to a training dataset.

[0071] In some embodiments, at least one positive training sample is obtained based on the positive samples, and then a weight is assigned to the at least one positive training sample based on the ratio of the amount of the plurality of negative training samples to the amount of the at least one positive training sample.

[0072] In some embodiments, each positive sample in the plurality of positive samples is processed with respect to identifying one of a plurality of influential parameter sets, specifically, an influential parameter set is identified for the positive sample, the influential parameter set including parameters in the plurality of parameters that cause the positive sample to be classified as a positive type.

[0073] In some embodiments, the final predictive model is updated in response to determining that additional positive samples have been added to the imbalanced dataset, specifically, generating an additional predictive model based on the additional positive samples, identifying additional influential parameters for the additional predictive model from the plurality of parameters, and updating the final predictive model based on the additional predictive model and the additional influential parameters.

[0074] In some embodiments, when a target sample is received, a plurality of prediction types are respectively determined for the target sample based on the plurality of prediction models included in the final prediction model, and in response to determining that all of the plurality of prediction types indicate a negative type, the target sample is classified as a negative type.

[0075] In some embodiments, in response to determining that at least one prediction type among the plurality of prediction types indicates a positive type, a model set is generated, the model set including at least one prediction model corresponding to the at least one prediction type. Influential parameter sets associated with the prediction models in the model set are determined. Next, influential target parameter sets are identified from the plurality of parameters for the target sample. Further, the model set is updated based on a difference between the influential parameter set and the influential target parameter set, and then the target sample is classified based on the updated model set.

[0076] In some embodiments, in response to a determination that the difference exceeds a predefined threshold, the predictive model is removed from the model set, and in response to a determination that the difference is equal to or less than a predefined threshold, the predictive model is retained in the model set.

[0077] In some embodiments, in response to determining that the updated model set is empty, the target sample is classified as a negative type, and in response to determining that the updated model set is not empty, the target sample is classified as a positive type.

[0078] For details on classifying samples, please refer to FIG. 10. FIG. 10 illustrates an exemplary flowchart of a method 1000 for classifying a target sample based on a final predictive model according to an embodiment of the present disclosure. In block 1002, a model set is set to empty, and in block 1004, it is determined whether there are one or more predictive models that output a positive type for the target sample. If the result is "YES," the method 1000 proceeds to block 1010; if the result is "NO," the method 1000 proceeds to block 1020. In block 1010, the predictive models that output a positive type for the target sample are added to the model set. In block 1020, if the model set is empty, the method 1000 proceeds to block 1022 and outputs a negative type. On the other hand, if the model set is not empty, the method 1000 proceeds to block 1030. In block 1030, a set of target parameters that are influential to the models in the model set is determined. Next, at block 1040, a difference between the influential target parameters and the influential parameters associated with the model is determined. If the difference exceeds a threshold at block 1050, method 1000 proceeds to block 1054, where the model is removed from the model set. If the difference is less than or equal to the threshold, method 1000 proceeds to block 1052, where a positive type is output.

[0079] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, by way of example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, mechanically encoded devices having instructions recorded on punch cards or ridge-in-groove structures, or the like, and suitable combinations thereof. Computer-readable storage devices, as used herein, should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0080] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computer / processing device. Alternatively, they can be downloaded to an external computer or external storage device via a network (e.g., the Internet, a LAN, a WAN, or a wireless network, or a combination thereof). The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computer / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computer / processing device for storage.

[0081] The computer-readable program instructions for carrying out the operations of the present disclosure can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the "C" programming language and similar programming languages. The computer-readable program instructions can be executed entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to customize the electronic circuitry for purposes of carrying out aspects of the present disclosure.

[0082] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. Each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0083] The computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of such computer or other programmable data processing apparatus, form means for performing the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams. The computer-readable program instructions may also be stored on a computer-readable storage medium capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, whereby the computer-readable storage medium having instructions stored thereon constitutes an article of manufacture including instructions for performing aspects of the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams.

[0084] Computer-readable program instructions may also be loaded into a computer, other programmable device, or other device and a series of operational steps executed on the computer, other programmable device, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable device, or other device perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.

[0085] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for performing a specific logical function. In some implementations, the functions depicted in the blocks may be performed in an order different from that depicted in the figures. For example, two blocks shown in succession may actually be accomplished as a single step, executed simultaneously or substantially simultaneously, executed in a partially or fully overlapping manner, or executed in reverse order, depending on the functionality involved. Note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs a specific function or operation, or by a combination of dedicated hardware and computer instructions.

[0086] While various embodiments of the present disclosure have been described by way of example, they are not intended to be exhaustive or limited to these embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles, practical applications, or technical improvements to commercially recognized technologies of the embodiments, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. generating, by one or more processors, a plurality of predictive models based on a plurality of positive samples included in the imbalanced dataset, wherein an amount of the plurality of positive samples is less than or equal to an amount of a plurality of negative samples included in the imbalanced dataset, and each sample of the plurality of positive samples and the plurality of negative samples includes a plurality of parameters; identifying, by one or more processors, a set of influential parameters from the plurality of parameters for the plurality of positive samples, each of the plurality of influential parameters causing the plurality of predictive models to classify an associated one of the plurality of positive samples as a positive type; determining, by one or more processors, a final forecast model that includes at least a portion of the plurality of forecast models; inputting, by one or more processors, the samples into the final predictive model such that the final predictive model classifies the samples into a positive or negative type, the final predictive model comparing the plurality of influential parameters to a set of influential target parameters as part of the classification; 11. A computer-implemented method comprising:

2. generating the plurality of predictive models obtaining, by one or more processors, a training dataset based on positive samples, the training dataset including at least one positive training sample and a plurality of negative training samples; generating the plurality of predictive models by training, with one or more processors, an initial predictive model on the training dataset; The computer-implemented method of claim 1 , comprising:

3. Obtaining the training data set includes: obtaining, by one or more processors, the plurality of negative training samples, the plurality of negative training samples being selected from the group constituting negative samples from the plurality of negative samples and modifying the positive samples to negative samples; adding, by one or more processors, the plurality of negative training samples to the training data set; The computer-implemented method of claim 2 , comprising:

4. Obtaining the training data set includes: obtaining, by one or more processors, the at least one positive training sample based on the positive sample; assigning, by one or more processors, a weight to the at least one positive training sample based on a ratio of an amount of the plurality of negative training samples to an amount of the at least one positive training sample; The computer-implemented method of claim 2 , comprising:

5. generating, by one or more processors, a further predictive model based on the further positive samples in response to determining that further positive samples are to be added to the imbalanced dataset; identifying, by one or more processors, a group of further influential parameters from the plurality of parameters for the further predictive model; updating, by one or more processors, the final forecast model based on the further forecast model and the further set of influential parameters; The computer-implemented method of claim 1 , further comprising:

6. In response to determining that a target sample is received, determining, by one or more processors, a plurality of prediction types for the target sample based on the plurality of prediction models included in the final prediction model; classifying, by one or more processors, the target sample into the negative type in response to determining that all of a plurality of prediction types indicate the negative type; The computer-implemented method of claim 1 , further comprising:

7. generating, by one or more processors, a plurality of predictive models based on a plurality of positive samples included in the imbalanced dataset, wherein an amount of the plurality of positive samples is less than or equal to an amount of a plurality of negative samples included in the imbalanced dataset, and each sample of the plurality of positive samples and the plurality of negative samples includes a plurality of parameters; identifying, by one or more processors, a set of influential parameters from the plurality of parameters for the plurality of positive samples, each of the plurality of influential parameters causing the plurality of predictive models to classify an associated one of the plurality of positive samples as a positive type; determining, by one or more processors, a final prediction model comprising at least a portion of the plurality of prediction models, wherein the final prediction model classifies samples into a positive type or a negative type; generating, by one or more processors, a model set including at least one predictive model corresponding to the at least one prediction type in response to determining that at least one prediction type among a plurality of prediction types indicates the positive type; determining, by one or more processors, influential parameters associated with predictive models in the model set; identifying, by one or more processors, a group of target parameters that have an influence on the target sample from the plurality of parameters; updating, by one or more processors, the model set based on differences between the influential parameters and the influential target parameters; classifying, by one or more processors, the target sample based on the updated model set; and 11. A computer-implemented method comprising:

8. Updating the model set includes: removing, by one or more processors, the predictive model from the model set in response to determining that the difference exceeds a predefined threshold; The computer-implemented method of claim 7, comprising:

9. Updating the model set includes: maintaining, by one or more processors, the predictive model in the model set in response to determining that the difference is less than or equal to a predefined threshold; The computer-implemented method of claim 7, comprising:

10. Classifying the target sample based on the updated model set includes: classifying, by one or more processors, the target sample into the negative type in response to determining that the updated model set is empty; The computer-implemented method of claim 7, comprising:

11. Classifying the target sample based on the updated model set includes: classifying, by one or more processors, the target sample into the positive type in response to determining that the updated model set is not empty; The computer-implemented method of claim 7, comprising:

12. 1. A computer-implemented system comprising a computer processor coupled to a computer-readable memory unit, the memory unit containing instructions that, when executed by the computer processor, perform a method, the method comprising: generating, by one or more processors, a plurality of predictive models based on a plurality of positive samples included in the imbalanced dataset, wherein an amount of the plurality of positive samples is less than or equal to an amount of a plurality of negative samples included in the imbalanced dataset, and each sample of the plurality of positive samples and the plurality of negative samples includes a plurality of parameters; identifying, by one or more processors, a set of influential parameters from the plurality of parameters for the plurality of positive samples, each of the plurality of influential parameters causing the plurality of predictive models to classify an associated one of the plurality of positive samples as a positive type; determining, by one or more processors, a final forecast model that includes at least a portion of the plurality of forecast models; inputting, by one or more processors, the samples into the final predictive model such that the final predictive model classifies the samples into a positive or negative type, the final predictive model comparing the plurality of influential parameters to a set of influential target parameters as part of the classification; 1. A computer-implemented system comprising:

13. generating the plurality of predictive models obtaining, by one or more processors, a training dataset based on positive samples, the training dataset including at least one positive training sample and a plurality of negative training samples; generating the plurality of predictive models by training, with one or more processors, an initial predictive model on the training dataset; The computer-implemented system of claim 12, comprising:

14. Obtaining the training data set includes: obtaining, by one or more processors, the plurality of negative training samples, the plurality of negative training samples being selected from the group constituting negative samples from the plurality of negative samples and modifying the positive samples to negative samples; adding, by one or more processors, the plurality of negative training samples to the training data set; The computer-implemented system of claim 13 , comprising:

15. Obtaining the training data set includes: obtaining, by one or more processors, the at least one positive training sample based on the positive sample; assigning, by one or more processors, a weight to the at least one positive training sample based on a ratio of an amount of the plurality of negative training samples to an amount of the at least one positive training sample; The computer-implemented system of claim 13 , comprising:

16. In response to determining that a target sample is received, determining, by one or more processors, a plurality of prediction types for the target sample based on the plurality of prediction models included in the final prediction model; classifying, by one or more processors, the target sample into the negative type in response to determining that all of a plurality of prediction types indicate the negative type; The computer-implemented system of claim 12 further comprising:

17. A method for generating, by one or more processors, a model set including at least one predictive model corresponding to at least one prediction type of a plurality of prediction types; updating, by one or more processors, the model set; classifying, by one or more processors, target samples based on the updated model set; and further comprising Classifying the target sample based on the updated model set includes: classifying the target sample into the negative type in response to determining that the updated model set is empty; The computer-implemented system of claim 12, comprising:

18. A method for generating, by one or more processors, a model set including at least one predictive model corresponding to at least one prediction type of a plurality of prediction types; updating, by one or more processors, the model set; classifying, by one or more processors, target samples based on the updated model set; and further comprising Classifying the target sample based on the updated model set includes: classifying the target sample into the positive type in response to determining that the updated model set is not empty; The computer-implemented system of claim 12, comprising:

19. A computer program comprising program instructions executable by an electronic device to cause the electronic device to perform a method, the method comprising: generating, by one or more processors, a plurality of predictive models based on a plurality of positive samples included in the imbalanced dataset, wherein an amount of the plurality of positive samples is less than or equal to an amount of a plurality of negative samples included in the imbalanced dataset, and each sample of the plurality of positive samples and the plurality of negative samples includes a plurality of parameters; identifying, by one or more processors, a set of influential parameters from the plurality of parameters for the plurality of positive samples, each of the plurality of influential parameters causing the plurality of predictive models to classify an associated one of the plurality of positive samples as a positive type; determining, by one or more processors, a final forecast model that includes at least a portion of the plurality of forecast models; inputting, by one or more processors, the samples into the final predictive model such that the final predictive model classifies the samples into a positive or negative type, the final predictive model comparing the plurality of influential parameters to a set of influential target parameters as part of the classification; a computer program comprising:

20. In response to determining that a target sample is received, determining, by one or more processors, a plurality of prediction types for the target sample based on the plurality of prediction models included in the final prediction model; classifying, by one or more processors, the target sample into the negative type in response to determining that all of a plurality of prediction types indicate the negative type; 20. The computer program of claim 19, further comprising:

21. generating, by one or more processors, a model set including at least one predictive model corresponding to the at least one prediction type in response to determining that at least one prediction type among the plurality of prediction types indicates the positive type; determining, by one or more processors, influential parameters associated with predictive models in the model set; identifying, by one or more processors, a group of target parameters from the plurality of parameters that have an influence on the target sample; updating, by one or more processors, the model set based on differences between the influential parameters and the influential target parameters; classifying, by one or more processors, the target sample based on the updated model set; and 21. The computer program of claim 20, comprising:

22. generating the plurality of predictive models obtaining, by one or more processors, a training dataset based on positive samples, the training dataset including at least one positive training sample and a plurality of negative training samples; generating the plurality of predictive models by training, with one or more processors, an initial predictive model on the training dataset; 20. The computer program of claim 19, comprising:

23. Obtaining the training data set includes: obtaining, by one or more processors, the plurality of negative training samples, the plurality of negative training samples being selected from the group constituting negative samples from the plurality of negative samples and modifying the positive samples to negative samples; adding, by one or more processors, the plurality of negative training samples to the training data set; 23. The computer program of claim 22, comprising:

24. Obtaining the training data set includes: obtaining, by one or more processors, the at least one positive training sample based on the positive sample; assigning, by one or more processors, a weight to the at least one positive training sample based on a ratio of an amount of the plurality of negative training samples to an amount of the at least one positive training sample; 23. The computer program of claim 22, comprising:

Images