Information processing device, information processing system, and information processing method
By partitioning and adding noise to high variance subsets in small datasets, the method enhances the robustness and reliability of machine learning models, addressing the challenge of poor predictive performance on small sample sizes.
Patent Information
- Application Number
- JP2023079271
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-22
AI Technical Summary
Existing machine learning models struggle with poor predictive performance and high variance in results when trained on small sample sizes, leading to unreliable outcomes.
A method involving a first set of machine learning models to identify a high variance subset, partition the data into true positives, false positives, true negatives, and false negatives, add noise to copied partitions, and train a second set of models on these modified partitions to generate a final analysis result, reducing outcome variance.
This approach enhances the robustness and reliability of machine learning models on small datasets, producing accurate and reliable results by training on modified data subsets with added noise, thereby improving predictive performance.
Smart Images

Figure 2024163549000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing system. [Background technology]
[0002] In recent years, machine learning techniques have been developed for application in a wide range of fields. In machine learning, training data based on known cases is input to a computer. The computer analyzes the training data to learn a model that generalizes the relationship between factors (sometimes called explanatory or independent variables) and outcomes (sometimes called dependent or dependent variables). This model can then be used to predict outcomes for unknown cases. As an example, it is possible to generate a model that predicts the survival of patients who receive different medical interventions based on data including past medical intervention history and patient characteristics for similar patients.
[0003] Previously, techniques have been devised to improve the performance of machine learning techniques. For example, Patent Document 1 discloses the following technique: "A revised support vector machine (SVM) classifier is provided to distinguish between true keywords and false positives based on output from a keyword spotting component of a speech recognition system. The SVM operates on a reduced set of feature dimensions, which are selected based on their ability to distinguish between true keywords and false positives. Further, support vector pairs are combined to create a reduced set of reweighted support vectors. These techniques result in an SVM that can be operated using reduced computing resources, thus improving system performance." [Prior art documents] [Patent documents]
[0004] [Patent Document 1] U.S. Patent No. 9,600,231 Summary of the Invention [Problem to be solved by the invention]
[0005] In machine learning, it is desirable for the generated model to have high accuracy, i.e., the ability to accurately predict the outcome of unknown cases (sometimes referred to as predictive performance). Predictive performance increases with increasing learnable information content in the dataset of the model generalizing the relationships between factors, but is often achievable using relatively large sample sizes. Traditionally, datasets based on small sample sizes can lead to poor predictive performance, with each result having a large variability in outcomes. Due to the high variance between different outcomes, such results may be unreliable and not suitable for drawing conclusions or insights.
[0006] US Patent Publication No. 2009 / 0139996 proposes a technique for using an SVM classifier for keyword recognition. More specifically, the technique of US Patent Publication No. 2009 / 0139996 relates to reducing the size of a feature set to facilitate the operation of an SVM in an environment with limited computing resources. However, US Patent Publication No. 2009 / 0139996 does not relate to improving the predictive performance of a machine learning model in a dataset with a small sample size.
[0007] Therefore, it is an object of the present disclosure to provide information processing techniques for generating machine learning models with high robustness capable of producing reliable results on datasets with small sample sizes. [Means for solving the problem]
[0008] A representative example of the present disclosure is an information processing device including a processor and a memory including computer-implementable instructions executable by the processor, the instructions including: a variance analysis unit configured to: generate a set of analysis results by processing an original dataset with a first set of machine learning models; and, based on the set of analysis results, identify a high variance subset of the original dataset including a first set of subjects (e.g., patients, machine parts, etc.) associated with the first analysis results that reach a variance threshold; and a partition configured to: partition the first set of subjects of the high variance subset into a first data partition and a second data partition; and generate a third data partition that is a copy of the first data partition and a fourth data partition that is a copy of the second data partition. a modification unit configured to generate a modified third data partition by modifying the third data partition (e.g., by adding noise) and a modified fourth data partition by modifying the fourth data partition; and a result generation unit configured to train a second set of machine learning models using the first data partition, the second data partition, the modified third data partition, and the modified fourth data partition, generate second analysis results by processing the original dataset with the second set of machine learning models, and generate final analysis results for the first set of subjects of the high variance subset by aggregating and classifying the first analysis results and the second analysis results. Effect of the Invention
[0009] According to the present disclosure, it is possible to provide information processing techniques for generating machine learning models with high robustness capable of producing reliable results on datasets with small sample sizes.
[0010] Problems, configurations, and advantages other than those mentioned above will become apparent from the following description of embodiments for carrying out the present invention. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 illustrates an exemplary computing architecture for implementing embodiments of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an exemplary hardware configuration of an information processing system according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 illustrates examples of low variance and high variance analysis results, according to an embodiment of the present disclosure. [Figure 4] FIG. 13 illustrates an example of variance reduction analysis results, according to an embodiment of the present disclosure. [Diagram 5] 1 is a flowchart illustrating an information processing method according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates an exemplary logical configuration of an information processing system for identifying datasets associated with high variance results, according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates an exemplary logical configuration of an information processing system for generating analytical results for a highly variance dataset, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] In this specification, embodiments of the present invention will be described with reference to the drawings. It should be noted that the embodiments described in this specification are not intended to limit the present invention according to the claims, and it should be understood that each of the elements and their combinations described with respect to the embodiments are not strictly necessary to implement aspects of the present invention.
[0013] Various aspects are disclosed in the following description and in the associated drawings. Alternative aspects may be devised without departing from the scope of the present disclosure. Additionally, well-known elements of the present disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of the present disclosure.
[0014] The words "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustrative example." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or advantageous over other aspects. Similarly, the phrase "aspects of the disclosure" does not require that all aspects of the disclosure include the discussed feature, advantage, or characteristic of operation.
[0015] Furthermore, many aspects are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be appreciated that the various actions described herein can be performed by specific circuitry (e.g., an application specific integrated circuit (ASIC)), program instructions executed by one or more processors, or a combination of both. In addition, the sequences of actions described herein can be considered to be generally embodied within any form of computer-readable storage medium storing a corresponding set of computer instructions that, when executed, can cause an associated processor to perform the functions described herein. Thus, various aspects of the present disclosure may be embodied in many different forms, all of which are contemplated to be within the subject matter set forth in the claims.
[0016] Detailed descriptions of the embodiments of the present disclosure are described herein with reference to the drawings.
[0017] Referring now to the drawings, Figure 1 is a schematic block diagram of a computer system 100 according to an embodiment for implementing various embodiments of the present disclosure. The mechanisms and apparatus of the various embodiments disclosed herein are equally applicable to any suitable computing system. The main components of the computer system 100 include one or more processors 102, memory 104, terminal interface 112, storage interface 113, I / O (input / output) device interface 114, and network interface 115, all of which are communicatively coupled, directly or indirectly, for inter-component communication via memory bus 106, I / O bus 108, bus interface unit 109, and I / O bus interface unit 110.
[0018] Computer system 100 may include one or more general purpose programmable central processing units (CPUs) 102A and 102B, generally referred to herein as processors 102. In embodiments, computer system 100 may include multiple processors, although in particular embodiments computer system 100 may alternatively be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include one or more levels of on-board cache.
[0019] In an embodiment, memory 104 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing or encoding data and programs. In a particular embodiment, memory 104 represents the entire virtual memory of computer system 100 and may further include virtual memory of other computer systems coupled to computer system 100 or connected via a network. While memory 104 can be conceptually viewed as a single monolithic entity, in other embodiments memory 104 is a more complex configuration, such as a hierarchy of caches and other memory elements. For example, memory may exist in multiple levels of caches, which may be further divided by function, whereby one cache holds instructions and another cache holds non-instruction data used by the processor. Memory may also be distributed and associated with different CPUs or sets of CPUs, as known in any of a variety of so-called non-uniform memory access (NUMA) computer architectures.
[0020] Memory 104 may store all or a portion of various programs, modules, and data structures for handling data transfers as described herein. For example, memory 104 may store information processing application 150. In an embodiment, information processing application 150 may include instructions or statements that are executed on processor 102 or interpreted by instructions or statements that are executed on processor 102 to perform functions as described further below. In certain embodiments, the information processing application 150 is implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices in place of or in addition to a processor-based system. In embodiments, the information processing application 150 may include data in addition to instructions or statements. In certain embodiments, a camera, sensor, or other data input device (not shown) may be provided in direct communication with the bus interface unit 109, the processor 102, or other hardware of the computer system 100. In such a configuration, the need for the processor 102 to access the memory 104 and the information processing application 150 may be reduced.
[0021] Computer system 100 may include a bus interface unit 109 that handles communication between processor 102, memory 104, display system 124, and I / O bus interface unit 110. I / O bus interface unit 110 may be coupled to I / O bus 108 to transfer data to and from various I / O units. I / O bus interface unit 110 communicates with a number of I / O interface units 112, 113, 114, and 115, also known as I / O processors (IOPs) or I / O adapters (IOAs), via I / O bus 108. Display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both types of data to display device 126. Additionally, computer system 100 may include one or more sensors or other devices configured to collect and provide data to processor 102. By way of example, computer system 100 may include biometric sensors (e.g., collecting heart rate data, stress level data), environmental sensors (e.g., collecting humidity data, temperature data, pressure data), motion sensors (e.g., collecting acceleration data, movement data), etc. Other types of sensors are possible. The display memory may be dedicated memory for buffering video data. Display system 124 may be coupled with a display device 126, such as a standalone display screen, a computer monitor, a television, a tablet, or the display of a handheld device. In one embodiment, the display device 126 may include one or more speakers for rendering audio. Alternatively, the one or more speakers for rendering audio may be coupled to the I / O interface unit. In an alternative embodiment, one or more of the functions provided by the display system 124 may be implemented in an integrated circuit that also includes the processor 102. Additionally, one or more of the functions provided by the bus interface unit 109 may be implemented in an integrated circuit that also includes the processor 102.
[0022] The I / O interface unit supports communication with various storage and I / O devices. For example, the terminal interface unit 112 supports the connection of one or more user I / O devices 116, which may include user output devices (such as a video display device, speakers, and / or a television receiver) and user input devices (such as a keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing device). A user may use a user interface to manipulate the user input devices to provide input data and commands to the user I / O devices 116 and the computer system 100, and may also receive output data via the user output devices. For example, the user interface may be presented via the user I / O devices 116, such as displayed on a display device, played through speakers, or printed by a printer.
[0023] Storage interface 113 supports the connection of one or more disk drives or direct access storage devices 117 (typically rotating magnetic disk drive storage devices, but may alternatively be other storage devices including arrays of disk drives or solid state drives such as flash memory that appear as a single mass storage device to a host computer). In some embodiments, storage device 117 may be implemented by any type of secondary storage device. The contents of memory 104, or any portion thereof, may be stored in storage device 117 and retrieved from storage device 117 as needed. I / O device interface 114 provides an interface to any of a variety of other I / O devices, or other types of devices such as printers or fax machines. Network interface 115 provides one or more communication paths from computer system 100 to other digital devices and computer systems, which may include, for example, one or more networks 130.
[0024] 1 illustrates a particular bus structure providing direct communication paths between processor 102, memory 104, bus interface 109, display system 124, and I / O bus interface unit 110, in alternative embodiments, computer system 100 may include different buses or communication paths that may be arranged in any of a variety of forms, such as hierarchical, star, or web configurations, multiple hierarchical buses, parallel and redundant paths, or point-to-point links in any other suitable type of configuration. Additionally, while I / O bus interface unit 110 and I / O bus 108 are shown as single respective components, computer system 100 may actually include multiple I / O bus interface units 110 and / or multiple I / O buses 108. Although multiple I / O interface units are shown isolating I / O bus 108 from the various communication paths running to the various I / O devices, in other embodiments some or all of the I / O devices are directly connected to one or more system I / O buses.
[0025] In various embodiments, computer system 100 is a multi-user mainframe computer system, a single-user system, or a server computer or similar device with little or no direct user interface, but which receives requests from other computer systems (clients). In other embodiments, computer system 100 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable type of electronic device.
[0026] Next, an exemplary hardware configuration of an information processing system according to an embodiment of the present disclosure will be described with reference to FIG.
[0027] 2 is a diagram illustrating an exemplary hardware configuration of an information processing system 200 according to an embodiment of the present disclosure. The information processing system 200 relates to an information processing system configured to identify datasets associated with high variance analysis results and generate highly robust machine learning models capable of generating high confidence results for datasets having small sample sizes.
[0028] 2, an information processing system 200 according to an embodiment of the present disclosure includes an information processing device 210, a communication network 250, and a user terminal 260. The information processing device 210 and the user terminal 260 may be communicatively connected via the communication network 250. Here, the communications network 250 may include a local area network (LAN) connection, the Internet, a wide area network (WAN) connection, a metropolitan area network (MAN) connection, and the like.
[0029] The information processing device 210 is an apparatus for identifying datasets associated with high variance analysis results and generating a robust machine learning model capable of generating reliable results on datasets with small sample sizes, and mainly includes a memory 220, a storage unit 230, a processor 244, and an input / output unit 246, as shown in Fig. 2. In an embodiment, the information processing device 210 may be implemented using a computer system such as the computer system 100 shown in Fig. 1.
[0030] The memory 220 is a memory for storing an information processing application 150 for implementing functions of the information processing technique according to the embodiment of the present disclosure. As shown in Fig. 2, the information processing application 150 may include a variance analysis unit 222, a partition unit 224, a modification unit 226, a labeling unit 228, and a result generation unit 229. Each of the variance analysis unit 222, the partition unit 224, the modification unit 226, the labeling unit 228, and the result generation unit 229 may be implemented as a software module that includes the information processing application 150.
[0031] The variance analysis unit 222 is a functional unit for identifying a dataset associated with a high variance analysis result. More specifically, the variance analysis unit 222 may generate a set of analysis results by processing an original dataset with a first set of machine learning models (e.g., the first set of machine learning models 233) and identify, based on the set of analysis results, a high variance subset 232 from the original dataset that includes a first set of subjects associated with a first analysis result that reaches a variance threshold.
[0032] The partition unit 224 is a functional unit for dividing the high variance dataset into partitions. More specifically, the partition unit 224 may divide the first set of subjects of the high variance subset 232 into a first data partition and a second data partition, and generate a third data partition that is a copy of the first data partition and a fourth data partition that is a copy of the second data partition. In an embodiment, the first set of subjects of the high variance subset 232 may be divided into true positive, false positive, true negative, or false negative data partitions based on the relationship between the analysis results generated by the first set of machine learning models 233 and the ground truth.
[0033] The modifying unit 226 is a functional unit for modifying (e.g., adding noise) a particular data partition generated by the partitioning unit 224. More specifically, the modifying unit 226 may generate a modified third data partition by modifying the third data partition, and a modified fourth data partition by modifying the fourth data partition.
[0034] The labeling unit 228 is a functional unit for assigning outcome labels to the partitions generated by the partitioning unit 224. In an embodiment, the labeling unit 228 may assign outcome labels of true positive or false positive (or true negative or false negative) to the modified third data partition and the modified fourth data partition.
[0035] The result generating unit 229 is a functional unit for training machine learning models based on the data partitions generated by the partition unit, and using the machine learning models to generate analysis results. More specifically, the result generating unit 229 may train a second set of machine learning models (e.g., the second set of machine learning models 234) using the first data partition, the second data partition, the modified third data partition, and the modified fourth data partition, generate second analysis results by processing the original data set with the second set of machine learning models, and generate final analysis results by aggregating and classifying the first analysis results and the second analysis results.
[0036] The storage unit 230 is a unit for storing various data and information used in implementing aspects of the present disclosure. The storage unit 230 may include a collection of hard disk drives, solid state drives, flash memory, cloud, storage, etc. As shown in FIG. 2, the storage unit 230 may include an original dataset 231, a high variance subset 232, a first set of machine learning models 233, and a second set of machine learning models 234.
[0037] The original dataset 231 is a dataset used as part of a machine learning task. The original dataset 231 may include a set of subjects that may be classified into one or more categories based on characteristics associated with each subject, where "subject" refers to an entity in the original dataset. As described herein, all or a portion of the original dataset 231 may include a set of subjects associated with a small sample size (e.g., a set where the measured number of data points does not reach a predetermined threshold).
[0038] As an example, the original dataset 231 may include a dataset related to quality of life information and medical intervention information about patients and can be used for machine learning tasks to classify patients (i.e., subjects) into groups based on life expectancy probability as determined using features such as patient characteristics or demographics (e.g., age, gender, race, income level, social determinants of health), laboratory results (e.g., hemoglobin, albumin, bilirubin, creatinine, sodium, alkaline phosphatase, etc.), diagnostic results (e.g., renal failure, diabetes, cancer, heart failure, high blood pressure, etc.), medical intervention history (past treatments, medications, etc.), etc.
[0039] As another example, original dataset 231 may include a dataset related to equipment maintenance and may be used for machine learning tasks to classify machine parts (i.e., entities) into groups based on probability of failure as determined using features such as machine part characteristics (e.g., device type, material, manufacturing data, age), device measurement or monitoring results (e.g., temperature, humidity, voltage, pressure), diagnostic results (e.g., voltage or current spikes, structural damage due to high pressure, overheating), maintenance history (e.g., past inspections, part replacements, software updates), etc. It should be noted that the original data set 231 is not particularly limited herein and other types of data may also be used.
[0040] The high variance subset 232 is a portion of the original dataset 231 that is identified as being associated with high variance. In an embodiment, the high variance subset 232 may include a first set of subjects selected from the original dataset. The first set of machine learning models 233 is a first group of machine learning models trained based on the original dataset 231. The second set of machine learning models 234 is a second group of machine learning models trained based on the modified training dataset generated based on the high variance subset 232.
[0041] The processor 244 is a processing unit for executing processing instructions for the various software modules and functional units included in the information processing application 150 stored in the memory 220 .
[0042] The input / output unit 246 is a unit for facilitating communication between the information processing device 210 and external sources, such as a user terminal 260. In an embodiment, the input / output unit 246 may be configured to generate and provide a graphical user interface on the user terminal 260 to facilitate input of data and user commands and to display status information and results.
[0043] User terminal 260 is a device usable by a user (e.g., a client) of information processing device 210. In an embodiment, user terminal 260 may be used to request analysis of a set of data (e.g., original data set 231) by information processing device 210 and to determine the results of this analysis. By way of example, user terminal 260 may be implemented using a personal computer, a tablet computer, a smartphone, or other computing device.
[0044] According to the information processing system 200 shown in FIG. 2, it is possible to provide an information processing technique for generating a machine learning model having high robustness capable of generating highly reliable results on a dataset having a small sample size.
[0045] As described herein, aspects of the present disclosure relate to the recognition that when a predictive model is used to process a data set based on a small sample size, the resulting analytical results may be associated with high variance between outcomes. Here, variance refers to the error rate or degree of uncertainty associated with a particular analytical result. Next, with reference to Figures 3 and 4, an example of outcome variance is described with respect to sample analytical results.
[0046] FIG. 3 illustrates examples of low variance analysis results 310 and high variance analysis results 320 for patient survival rates for a particular medical intervention. In the low variance analysis results 310 and high variance analysis results 320, the vertical axis represents survival rates (percentage) and the horizontal axis represents time survived after the medical intervention. As shown in FIG. 3, line graph 312 for low variance analysis result 310 is associated with a variance range 315, and line graph 322 for high variance analysis result 320 is associated with a variance range 325. These variance ranges 315 and 325 indicate the margin of error for patient survival rates as a function of the number of years since the medical intervention.
[0047] As can be seen by comparing the low variance analysis results 310 and the high variance analysis results 320, the range of variance 315 for the low variance analysis results 310 is less than 10%, while the range of variance 325 for the high variance analysis results is close to 60%. As explained herein, due to this high variance between different outcomes, results such as the high variance analysis results are unreliable and unsuitable for drawing conclusions or insights. Accordingly, aspects of the present disclosure relate to generating machine learning models with high robustness that can generate reliable results on datasets with small sample sizes.
[0048] 4 illustrates an example of a variance reduction analysis result 410 according to an embodiment of the present disclosure. The variance reduction analysis result 410 illustrates an analysis result generated by using information processing techniques according to an embodiment of the present disclosure on the same data set used to create the high variance analysis result 320 shown in FIG.
[0049] As described with reference to Figure 3, in the variance reduction analysis results 410, the vertical axis represents the survival rate (percentage) and the horizontal axis represents the time survived after the medical intervention. As shown in Figure 4, the line graph 412 of the variance reduction analysis results 410 is associated with a variance range 415. The variance range 415 indicates the margin of error for the survival rate of the patient as a function of the number of years since the medical intervention.
[0050] 3, the variance range 415 of the variance reduction analysis result 410 is reduced by approximately less than 10% relative to the high variance analysis result 320. Due to this variance reduction range, the variance reduction analysis result 410 can be used to facilitate drawing more accurate conclusions or insights than the high variance analysis result 320.
[0051] Next, an information processing method according to an embodiment of the present disclosure will be described with reference to FIG.
[0052] 5 is a flowchart illustrating an information processing method 500 according to an embodiment of the present disclosure. The information processing method 500 illustrated in FIG. 5 is a method for identifying a dataset associated with high variance analysis results, generating a robust machine learning model, and generating analysis results for a dataset with a small sample size. The information processing method 500 may be performed by various functional units of the information processing device 210 illustrated in FIG. 2.
[0053] Initially, in step S505, the analysis of variance unit 222 generates a set of analysis results by processing an original dataset (e.g., the original dataset 231 shown in FIG. 2) using a first set of machine learning models (e.g., the first set of machine learning models 233 shown in FIG. 2). The first set of machine learning models may be machine learning models trained to perform a classification task on the original dataset. As described herein, the original dataset may be a dataset designated for use as part of a machine learning classification task. For example, the original dataset may include a set of subjects that may be classified into one or more categories based on features associated with each subject. Here, a "subject" refers to an entity in the original dataset. As an example, the original dataset may include personal quality of life, medical characteristics, and medical intervention information for patients (e.g., subjects). As described herein, all or a portion of the subjects of the original dataset may be associated with a small sample size. For example, patients associated with a medical intervention that is rarely performed may be associated with a small sample size.
[0054] The first set of machine learning models may process all or a portion of the original dataset to generate a set of analysis results in which the set of subjects of the original dataset are classified into categories. The set of analysis results may include a first analysis result that corresponds to the first set of subjects of the original dataset. As an example, if the original dataset includes personal medical characteristics and medical intervention information for patients, a first set of machine learning models may generate a set of analysis results in which each patient is sorted into a category based on their probability of survival for a particular medical intervention.
[0055] Next, in step S510, the variance analysis unit 222 identifies a high variance subset (e.g., the high variance subset 232 shown in FIG. 2) that includes a first set of subjects from the set of subjects of the original dataset associated with the first analysis result that reaches the variance threshold based on the set of analysis results generated in step S505. As described herein, variance refers to the error or degree of uncertainty associated with a particular analysis result. This variance may occur due to the sensitivity of the first set of machine learning models to small variations in the original dataset and may be high when the sample size is relatively small. In addition, the variance threshold refers to a criterion or standard that defines the boundary between high variance outcomes and low variance outcomes for the set of analysis results. In an embodiment, the variance threshold may be determined based on the nature of the machine learning task and may be defined in terms of the number of standard deviations, a percentage thereof, etc. As an example, if the variance threshold is "30%", subjects associated with analysis results that have 30% or more variance between outcomes may be identified as reaching the variance threshold.
[0056] Next, in step S515, the partition unit 224 divides the first set of subjects of the high variance subset 232 identified in step S510 into two or more data partitions. In an embodiment, the partition unit 224 may divide the first set of subjects of the high variance subset 232 based on the accuracy rate of the first analysis result generated for the first set of subjects in step S505 by comparing the first analysis result against the ground truth result of the original dataset. More specifically, for each subject of the first set of subjects, the partition unit 224 may compare the result shown for that subject in the first analysis result with the ground truth result associated with the original dataset that shows the actual accurate outcome for that subject, and sort the first set of subjects into different partitions associated with true positive results, false positive results, true negative results and false negative results based on the relationship between the first analysis result and the ground truth result. For convenience of explanation, an exemplary configuration is described herein where a first subset of the first set of subjects associated with true positive results is placed in a first data partition, and a second subset of the first set of subjects associated with false positive results is placed in a second data partition, however, the disclosure is not limited herein and a configuration is also possible where a third subset of the first set of subjects associated with true negative results is placed in a third data partition, and a fourth subset of the first set of subjects associated with false negative results is placed in a fourth data partition. In this manner, by dividing the high variance subset 232 into partitions for true positive, false positive, true negative, and false negative results, subjects with better machine learning classification performance can be managed separately from subjects with worse machine learning classification performance, such that different amounts of noise can be added to particular partitions, as described below.
[0057] Next, in step S520, partition unit 224 generates a copy of each data partition created in step S515. More specifically, if a first data partition associated with a true positive result and a second data partition associated with a false positive result are created in step S515, partition unit 224 may create a third data partition that is a copy of the first data partition and a fourth data partition that is a copy of the second data partition.
[0058] Next, in step S525, the modification unit 226 modifies the copied data partitions created in step S515 to create modified data partitions. Here, the modification unit 226 can modify the copied data partitions by adding noise to the set of features of each data partition to distort or modify the values of those features. Different amounts of noise can be added to each partition. For example, the modification unit 226 can generate a modified third data partition by adding a first set of noise to the third set of features of the third data partition, and can generate a modified fourth data partition by adding a second set of noise to the fourth set of features of the fourth data partition. The amount of noise added to the set of features of the copied data can be determined based on a noise level criterion. As described below, the noise level criterion can be determined based on the performance of the second set of machine learning models on sample datasets having various amounts of noise. In addition, the modified third data partition and the modified fourth data partition can be assigned labels by the labeling unit 228, as described below.
[0059] Next, in step S530, the result generating unit 229 trains a second set of machine learning models (e.g., the second set of machine learning models 234 shown in FIG. 2) using the first and second data partitions generated in step S515 and the modified third and fourth data partitions generated in step S525. Here, the second set of machine learning models are untrained machine learning models having the same type and configuration as the first set of machine learning models. By training the second set of machine learning models using the first and second data partitions including data from the original dataset together with the modified third and fourth data partitions modified to have skewed features, the robustness of the second set of machine learning models with respect to small sample sizes may be increased.
[0060] Next, in step S535, the result generation unit 229 generates second analytical results by processing the first set of subjects of the high variance subset 232 identified in step S510 using the second set of machine learning models trained in step S530. In an embodiment, the second set of machine learning models may be configured to perform the same machine learning task on the same data as in step S505. In this way, in addition to the first analytical results generated for the first set of subjects by the first set of machine learning models, a second analytical result for the same first set of subjects may be generated by the second set of machine learning models trained to increase robustness with respect to small sample sizes.
[0061] Next, in step S540, the result generating unit 229 may generate a final analysis result by aggregating the first analysis result generated in step S505 and the second analysis result generated in step S535 and classifying (e.g., reclassifying and re-stratifying) the result. Here, the first analysis result and the second analysis result may be aggregated using any conventional machine learning result aggregation technique, and the result is not particularly limited herein. Thus, by combining the first analysis result generated for the first set of subjects with the second analysis result generated for the same first set of subjects by a second set of machine learning models trained to be more robust with respect to small sample sizes, it is possible to obtain a more accurate final analysis result with an increased accuracy rate relative to the first analysis result.
[0062] According to the information processing method 500, it is possible to provide an information processing technique for generating a machine learning model with high robustness that can generate reliable results on a data set with a small sample size. More specifically, when a second set of machine learning models is trained based on a modified version of an original data set in which features are distorted by noise, the second set of machine learning models can be trained to recognize features that are most salient to a classification task and robust with respect to small sample sizes. In this way, it is possible to obtain machine learning analysis results with high reliability and high accuracy even on small sample sizes where conventional machine learning techniques produce high variance results.
[0063] Next, with reference to FIG. 6, an exemplary logical configuration of an information processing device for identifying data sets associated with high variance results will be described.
[0064] 6 illustrates an exemplary logical configuration of an information processing device 210 for identifying a dataset associated with high variance results, according to an embodiment of the present disclosure. In an embodiment, the functions performed by the information processing device 210 to identify a dataset associated with high variance results may substantially correspond to steps S505-S510 of the information processing method 500 of FIG. 5. In addition, the functions performed by the information processing device 210 to identify a dataset associated with high variance results may be performed by a variance analysis unit 222 (not shown in FIG. 6).
[0065] Initially, the analysis of variance unit 222 obtains an original dataset 231. As described herein, the original dataset 231 may be a dataset designated for use as part of a machine learning classification task. For example, the original dataset 231 may include a set of subjects that may be classified into one or more categories based on features associated with each subject. In an embodiment, the original dataset 231 may be created and designated by a user for use as part of a machine learning classification task.
[0066] The analysis of variance unit 222 then generates the machine learning dataset 605 by performing feature selection 602 on the original dataset 231. More specifically, the analysis of variance unit 222 may generate the machine learning dataset 605 by identifying a set of salient features in the original dataset determined to be relevant to the machine learning task to be performed and extracting subjects and corresponding salient features for those subjects. In an embodiment, for a given machine learning task, the set of salient features may be pre-determined by a user. Generally, the more salient the features selected for inclusion in the machine learning dataset 605, the higher the accuracy rate achievable by the machine learning model, but the more salient the features in the machine learning dataset 605, the more computing resources are required for the execution of the machine learning task. Thus, the type and number of salient features for inclusion in the machine learning dataset may be selected by a user or administrator based on the nature of the machine learning task.
[0067] The variance analysis unit 222 then performs partition generation 604 on the machine learning dataset 605 to divide the machine learning dataset 605 into a set of development partitions 606 and a set of evaluation partitions 608. In an embodiment, the set of development partitions 606 and the set of evaluation partitions 608 may be mutually exclusive with respect to one another, i.e., none of the data included in the set of development partitions 606 is included in the set of evaluation partitions 608, and none of the data included in the set of evaluation partitions 608 is included in the set of development partitions 606. Furthermore, each individual partition of the set of development partitions 606 and each individual partition of the set of evaluation partitions may be mutually exclusive with respect to other individual partitions of the set. In certain embodiments, each individual partition of the set of development partitions 606 and each individual partition of the set of evaluation partitions 608 may include a single subject and its associated features. In certain embodiments, each individual partition of the set of development partitions 606 and each individual partition of the set of evaluation partitions may include multiple subjects and their associated features.
[0068] In an embodiment, the variance analysis unit 222 may randomly divide the machine learning dataset 605 into a set of development partitions 606 and a set of evaluation partitions 608. This division may be based on a predetermined ratio. For example, a randomly selected combination of data that includes 80% of the data in the machine learning dataset 605 may be used as the set of development partitions 606, and the remaining 20% of the data in the machine learning dataset may be used as the set of evaluation partitions 608.
[0069] Next, the variance analysis unit 222 may train the first set of machine learning models 233 to perform a classification task (e.g., risk classification of patient survivability, component failure, etc.) using the set of development partitions 606. Here, the first set of machine learning models 233 may be a machine learning model ensemble including a plurality of untrained predictive models. In an embodiment, the variance analysis unit 222 may train each individual machine learning model of the first set of machine learning models 233 using a single partition of the set of development partitions 606. Here, the first set of machine learning models 233 may be trained using an existing training method, and the training method is not particularly limited herein. In this manner, the first set of machine learning models 233 may be trained to perform a classification task using the set of development partitions 606.
[0070] The variance analysis unit 222 may then perform a classification task on the set of assessment partitions 608 using the first set of trained machine learning models 233. Each machine learning model in the set of machine learning models 233 may process a single partition of the set of assessment partitions 608 to generate a set of analysis results 620. In an embodiment, an analysis result may be generated for each subject included in the set of assessment partitions 608. The set of analysis results 620 may indicate the subject's probability of corresponding to a particular category or phenomenon. Based on their calculated probabilities, the subjects may be assigned a classification score (e.g., a risk score) and organized into groups (e.g., risk groups). Furthermore, as described herein, each analysis result in the generated set of analysis results 620 may be associated with a variance range that indicates the uncertainty in the outcome of that analysis result for that particular subject.
[0071] The variance analysis unit 222 may then analyze the set of analysis results 620 to identify a high variance subset of the original data set 231 that includes a first set of subjects associated with a first analysis result 625 that reaches the variance threshold, where the variance threshold refers to a standard or standard that defines a boundary between high variance outcomes and low variance outcomes for a set of analysis results. In an embodiment, the variance threshold may be determined based on the nature of the machine learning task and may be defined in terms of a number of standard deviations, a percentage thereof, or the like. As an example, if the variance threshold is "30%", subjects associated with analysis results that have a variance range of 30% or more may be identified as reaching the variance threshold.
[0072] In this manner, the variance analysis unit 222 according to an embodiment of the present disclosure can identify subjects associated with high variance results. As described herein, those high variance subjects (e.g., the first set of subjects and associated first analysis results 625) can be used to facilitate the generation of a machine learning model with high robustness capable of generating reliable results in a dataset having a small sample size.
[0073] Next, with reference to FIG. 7, an exemplary logical configuration of the information processing device 210 for generating analytical results for a highly variance data set will be described.
[0074] 7 is a diagram illustrating an exemplary logical configuration of an information processing system 200 for generating an analysis result for a high variance dataset according to an embodiment of the present disclosure. In an embodiment, the functions performed by the information processing device 210 to generate an analysis result for a high variance dataset may substantially correspond to steps S515 to S540 of the information processing method 500 illustrated in FIG. 5. In addition, the functions performed by the information processing device 210 to generate an analysis result for a high variance dataset may be performed by the partition unit 224, the modification unit 226, the labeling unit 228, and the result generation unit 229 after the subject associated with the high variance result is identified by the function of the variance analysis unit 222 as described with reference to FIG. 6.
[0075] First, the partition unit 224 divides the first set of subjects of the high variance subset 232 identified by the variance analysis unit 222 (described with reference to FIG. 6 ) into a first data partition 707 and a second data partition 709. Here, the partition unit 224 may divide the first set of subjects of the high variance subset 232 based on the accuracy rate of the first analysis result 625 generated for the first set of subjects of the high variance subset 232 with respect to the ground truth result corresponding to the first set of subjects. More specifically, for each subject of the first set of subjects in the high variance subset 232, the partition unit 224 may compare the result indicated for that subject by the first analysis result 625 with the ground truth result associated with the first set of subjects indicating the actual accurate outcome for that subject, and may sort the first set of subjects of the high variance subset 232 into different partitions associated with true positive results, false positive results, true negative results, and false negative results based on the relationship between the first analysis result 625 and the ground truth result.
[0076] For convenience of explanation, an exemplary configuration is described herein where a first subset of the first set of subjects associated with true positive results is placed in the first data partition 707 and a second subset of the first set of subjects associated with false positive results is placed in the second data partition 709, however, the disclosure is not limited herein and a configuration is also possible where a third subset of the first set of subjects associated with true negative results is placed in the third data partition and a fourth subset of the first set of subjects associated with false negative results is placed in the fourth data partition.
[0077] Next, the partition unit 224 generates copies of the first data partition 707 and the second data partition 709. More specifically, the partition unit 224 may create a third data partition 708 that is a copy of the first data partition 707 and a fourth data partition 710 that is a copy of the second data partition 709.
[0078] Then, the modifying unit 226 modifies the third data partition 708 to create a modified third data partition 711, and modifies the fourth data partition 710 to create a modified fourth data partition 712. Here, the modifying unit 226 may modify the third data partition 708 and the fourth data partition 710 by adding noise to the set of features of each data partition to distort or modify the values of those features. The amount of noise added to the set of features of the copied data may be determined based on a noise level criterion.
[0079] The noise level criterion is information indicating a desired amount of noise to be added to the set of features of each data partition. Here, the "desired amount of noise" refers to the maximum amount of noise that can be added to the features of a particular subject without resulting in a result label (e.g., true positive, false positive, true negative, false negative) for the subject change. This is because by adding more noise to the set of features while still maintaining the same result, the second set of machine learning models 234 trained using the modified dataset can be more robust with respect to noise while still recognizing the salient features of the dataset that correspond to the particular analysis result.
[0080] In an embodiment, the modification unit 226 may be configured to use the machine learning models (e.g., the second set of machine learning models 234 and another test set of machine learning models) to successively process modified sample data sets in which increasing amounts of noise have been added to the features, and identify the maximum amount of noise that can be added to the features such that sample prediction results generated for the modified sample data sets meet a predetermined accuracy threshold (e.g., a minimum accuracy threshold). In this manner, the maximum amount of noise that can be added to the features while maintaining a desired accuracy threshold can be identified and set as the noise level reference. Further, based on this noise level criterion, a different amount of noise may be added to each data partition. As an example, a larger amount of noise may be added to the third data partition 708 associated with true positive results, and a smaller amount of noise may be added to the fourth data partition 710 associated with false positive results. In this manner, an appropriate amount of noise necessary to suitably obscure the features of each data partition may be applied to each data partition individually.
[0081] Then, the labeling unit 228 may assign outcome labels to the modified third data partition 711 and the modified fourth data partition 712. Here, the outcome labels refer to the predicted relationship between the analysis results generated based on the modified third data partition 711 and the modified fourth data partition 712 (e.g., true positive, false positive, true negative, false negative) and the ground truth results. Because the analysis results for the modified third data partition 711 and the modified fourth data partition 712 should theoretically be the same as the original data partitions (e.g., the first data partition 707 and the second data partition 709), in an embodiment, the labeling unit 228 may assign a true positive outcome label to the modified third data partition 711 and a false positive outcome label to the modified fourth data partition 712.
[0082] However, aspects of the present disclosure relate to the recognition that due to noise added to the data partitions 707, 709, there is no guarantee that the outcomes generated for the modified data partitions 711, 712 correspond to the outcomes of the data partitions 707, 709. Thus, in an embodiment, the labeling unit 228 may calculate a similarity between the modified data partitions 711, 712 and the original data partitions 707, 709, and assign an outcome label to the modified data partitions 711, 712 based on the similarity of the modified data partitions 711, 712 to the original data partitions 707, 709.
[0083] More specifically, the labeling unit 228 may calculate a first similarity between a first subset of subjects in the first data partition 707 and a third subset of subjects in the modified third data partition 711, and assign a true positive result label to the third subset of subjects if the first similarity reaches a similarity threshold, and assign a false positive result label to the third subset of subjects if the first similarity does not reach the similarity threshold. Similarly, the labeling unit 228 may calculate a second similarity between a second subset of subjects in the second data partition 709 and a fourth subset of subjects in the modified fourth data partition 712, and assign a false positive result label to the fourth subset of subjects if the second similarity reaches a similarity threshold, and assign a true positive result label to the fourth subset of subjects if the second similarity does not reach the similarity threshold. In this way, more accurate outcome labels can be assigned to the modified data partitions 711, 712, and training of the second set of machine learning models 234 can be facilitated. Here, the similarity between the modified data partitions 711, 712 and the original data partitions 707, 709 can be calculated using Euclidean distance, cosine similarity, or other conventional similarity algorithms.
[0084] Next, the result generating unit 229 uses the modified training dataset 713 including the first data partition 707, the second data partition 709, the modified third data partition 711, and the modified fourth data partition 712 to train the second set of machine learning models 234 (e.g., the second set of machine learning models 234 shown in FIG. 2). Here, the second set of machine learning models 234 is an untrained machine learning model having the same type and configuration as the first set of machine learning models. By training the second set of machine learning models 234 using the first and second data partitions 707, 709 including data from the original dataset together with the modified third and fourth data partitions 711, 712 modified to have skewed features, the robustness of the second set of machine learning models 234 with respect to small sample sizes may be increased.
[0085] The result generation unit 229 then generates second analysis results 715 by processing the first set of subjects in the high variance subset 232 using the second set of machine learning models 234. In an embodiment, the second set of machine learning models 234 may be configured to perform the same machine learning tasks on the first set of subjects in the high variance subset 232 as the first set of machine learning models 233 performed on the set of evaluation partitions 608, as described in FIG.
[0086] The result generation unit 229 may then generate a final analysis result 720 by aggregating the first analysis result 625 generated by the first set of machine learning models 233 and the second analysis result 715 generated by the second set of machine learning models 234 and classifying (e.g., reclassifying and re-stratifying) the results, where the first analysis result and the second analysis result may be aggregated using any conventional machine learning result aggregation technique, which is not particularly limited herein.
[0087] According to the functionality of the information processing system 200 described with reference to Figure 7, a second set of machine learning models is trained on the modified training dataset that includes the original high variance dataset along with a version of the high variance dataset modified by noise. In this way, the second set of machine learning models can be trained to be robust with respect to datasets with small sample sizes, and machine learning analysis results with reduced outcome variance can be generated.
[0088] As described herein, in machine learning, it is preferable that the accuracy of the generated model, i.e., its ability to accurately predict the outcome of unknown cases (sometimes referred to as predictive performance), is high. This predictive performance usually increases as the size of the dataset (i.e., sample size) increases. Traditionally, datasets based on small sample sizes may lead to poor predictive performance, such as each result having a large variability in outcomes. Due to the high variance between different outcomes, such results may be unreliable and not suitable for drawing conclusions or insights.
[0089] In embodiments, aspects of the disclosure relate to generating a set of analysis results for an original dataset using a first set of machine learning models, and using the set of analysis results to identify a subset of the original dataset that meets a variance threshold (e.g., a high level of variance between outcomes). In this manner, a subset of the original dataset associated with high variance may be identified.
[0090] Further aspects of the present disclosure relate to splitting the identified high variance subset into separate data partitions. These data partitions may classify portions of the high variance subset into true positive, false positive, true negative, or false negative categories based on the relationship between the analysis results generated by the first set of machine learning models and the ground truth. By splitting the high variance subset into true positive, false positive, true negative, and false negative partitions, the data set portions with higher machine learning classification performance may be managed separately from the data set portions with lower machine learning classification performance. Each generated partition may be copied and noise may be added to each copied data partition. The noise added to each copied data partition may be determined individually for each partition. The copied data partitions with added noise may be merged with the original data partition to generate an altered training dataset.
[0091] This modified dataset can be used to train a second set of machine learning models. In this way, the second set of machine learning models can be trained to be robust for datasets with small sample sizes. The trained second set of machine learning models can be used to generate analytical results (e.g., second analytical results) for the high variance subset extracted from the original dataset, which can be aggregated with the analytical results generated by the first set of machine learning results to generate a final analytical result with reduced outcome variance.
[0092] Therefore, it is possible to provide an information processing technique for generating a machine learning model with high robustness that can generate reliable results on a data set with a small sample size. More specifically, when a second set of machine learning models is trained based on a modified version of an original data set in which features are distorted by noise, the second set of machine learning models can be trained to recognize features that are most salient to a classification task and robust with respect to small sample sizes. In this way, it is possible to obtain machine learning analysis results with high reliability and high accuracy even on small sample sizes where conventional machine learning techniques produce high variance results.
[0093] The present invention may be a system, method, and / or computer program product, which may include computer-readable storage medium(s) having computer-readable program instructions for causing a processor to perform aspects of the present invention.
[0094] A computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanical encryption devices such as punch cards or grooved ridge structures with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not itself be construed as being a primary signal, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted by an electrical wire.
[0095] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.
[0096] The computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / operations specified in the flowchart and / or block diagram blocks. These computer-readable program instructions may further be stored on a computer-readable storage medium that can cause the computer, programmable data processing apparatus, and / or other apparatus to function in a particular manner, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture containing instructions that implement aspects of the functions / operations specified in the flowchart and / or block diagram blocks.
[0097] The above computer readable program instructions may further be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to create a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / operations specified in the flowchart and / or block diagram blocks.
[0098] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing a specialized logical function. In some alternative implementations, the functions described in the blocks may occur in a different order than the order described in the drawings. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may possibly be executed in reverse order, depending on the functionality involved. It will also be appreciated that each block of the block diagrams and / or flowchart illustrations, as well as combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that executes a specialized function or operation, or executes a combination of dedicated hardware and computer instructions.
[0099] Although the above relates to exemplary embodiments of the present invention, other and further embodiments of the present invention may be devised without departing from the basic scope of the present invention, the scope of which is determined by the following claims. The description of various embodiments of the present disclosure has been provided for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to explain the principles of the embodiments, practical applications or technical improvements of the technology existing in the market, or to enable others with ordinary skill in the field to which the disclosure belongs to understand the embodiments disclosed herein.
[0100] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the various embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural as well, unless the context clearly indicates otherwise. The terms "set," "group," "bundle," and the like are intended to include one or more. Furthermore, it will be understood that the terms "comprise" and / or "comprising" as used herein specify the presence of stated features, wholes, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof. In the above detailed description of exemplary embodiments of the various embodiments, reference has been made to the accompanying drawings, in which like numerals indicate like elements, which form a part hereof, by way of example, certain exemplary embodiments are shown and in which the various embodiments may be practiced. The above embodiments have been described in sufficient detail to enable one skilled in the art to practice the embodiments, but other embodiments may be used and logical, mechanical, electrical, and other changes may be made without departing from the scope of the various embodiments. Numerous specific details have been set forth in the above description to provide a thorough understanding of the various embodiments. However, the various embodiments may be practiced without those specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail so as not to obscure the embodiments.
[0101] As described herein, aspects of the present disclosure relate to the following aspects:
[0102] (Aspect 1) A processor; a memory containing computer-implementable instructions executable by said processor; An information processing device comprising: The computer-implementable instructions cause the processor to: generating a set of analytical results by processing the original dataset with a first set of machine learning models; identifying a high variance subset of the original data set that includes a first set of subjects associated with a first analysis result that meets a variance threshold based on the set of analysis results; and a variance analysis unit configured to perform partitioning the first set of subjects of the highly variance subset into a first data partition and a second data partition; generating a third data partition that is a copy of the first data partition and a fourth data partition that is a copy of the second data partition; A partition unit configured to: a modification unit configured to generate a modified third data partition by modifying the third data partition and a modified fourth data partition by modifying the fourth data partition; training a second set of machine learning models using the first data partition, the second data partition, the modified third data partition, and the modified fourth data partition; generating a second analysis result by processing the original dataset with the second set of machine learning models; generating a final analysis result for the first set of subjects of the high variance subset by aggregating and classifying the first analysis result and the second analysis result; a result generating unit configured to: An information processing device that functions as a
[0103] (Aspect 2) The modification unit is generating the modified third data partition by adding noise to a third set of characteristics of the third data partition based on a noise level criterion; generating the modified fourth data partition by adding noise to a fourth set of characteristics of the fourth data partition based on the noise level criterion; 2. The information processing device of claim 1, configured to:
[0104] (Aspect 3) The modification unit is generating a set of sample prediction results by iteratively processing the modified sample data set with increasing amounts of noise using the second set of machine learning models; identifying a subset of sample prediction results from said set of sample prediction results that meet a predetermined accuracy threshold; determining as the noise level metric a maximum amount of noise added to the modified sample data set associated with the subset of sample prediction results that meet the predetermined accuracy threshold; 3. The information processing device of claim 2, configured to:
[0105] (Aspect 4) The partition unit includes: evaluating the accuracy of the first analysis result by comparing the first analysis result against ground truth results for the original dataset; partitioning a first subset of the first set of subjects associated with true positive results into the first data partition; partitioning a second subset of the first set of subjects associated with false positive results into the second data partition; 4. The information processing device of any one of aspects 1 to 3, configured to perform the following:
[0106] (Aspect 5) a labeling unit for assigning true positive or false positive outcome labels to the modified third data partition and the modified fourth data partition; 5. The information processing device of any one of aspects 1 to 4, further comprising:
[0107] (Aspect 6) The labeling unit comprises: calculating a first similarity between a first subset of subjects of the first data partition and a third subset of subjects of the modified third data partition; assigning a true positive result label to the third subset of subjects if the first similarity reaches a similarity threshold; assigning a false positive result label to the third subset of subjects if the first similarity does not reach a similarity threshold; calculating a second similarity between a second subset of subjects of the second data partition and a fourth subset of subjects of the modified fourth data partition; assigning a false positive result label to the fourth subset of subjects if the second similarity reaches a similarity threshold; assigning a true positive result label to the fourth subset of subjects if the second similarity does not reach a similarity threshold. An information processing device according to embodiment 5. [Explanation of symbols]
[0108] 150 Information Processing Applications 200 Information Processing Systems 210 Information Processing Devices 220 Memory 222 Analysis of Variance Unit 224 Partition Unit 226 Modification Unit 228 Labeling Unit 229 Result Generation Unit 230 Storage Unit 231 original dataset 232 High Variance Subset 233 First set of machine learning models 234 Second set of machine learning models 244 processors 246 Input / Output Unit 250 Communication Network 260 User Terminals
Claims
1. A processor; a memory containing computer-implementable instructions executable by said processor; An information processing device comprising: The computer-implementable instructions cause the processor to: generating a set of analytical results by processing the original dataset with a first set of machine learning models; identifying a high variance subset of the original data set that includes a first set of subjects associated with a first analysis result that meets a variance threshold based on the set of analysis results; and a variance analysis unit configured to perform partitioning the first set of subjects of the highly variance subset into a first data partition and a second data partition; generating a third data partition that is a copy of the first data partition and a fourth data partition that is a copy of the second data partition; A partition unit configured to: a modification unit configured to generate a modified third data partition by modifying the third data partition and a modified fourth data partition by modifying the fourth data partition; training a second set of machine learning models using the first data partition, the second data partition, the modified third data partition, and the modified fourth data partition; generating a second analysis result by processing the original dataset with the second set of machine learning models; generating a final analysis result for the first set of subjects of the high variance subset by aggregating and classifying the first analysis result and the second analysis result; a result generating unit configured to: An information processing device that functions as a
2. The modification unit is generating the modified third data partition by adding a first amount of noise to a third set of characteristics of the third data partition based on a noise level criterion; generating the modified fourth data partition by adding a second amount of noise to a fourth set of characteristics of the fourth data partition based on the noise level criterion; The information processing device of claim 1 , configured to:
3. The modification unit is generating a set of sample prediction results by iteratively processing the modified sample data set with increasing amounts of noise using the second set of machine learning models; identifying a subset of sample prediction results from said set of sample prediction results that meet a predetermined accuracy threshold; determining as the noise level metric a maximum amount of noise added to the modified sample data set associated with the subset of sample prediction results that meet the predetermined accuracy threshold; The information processing device of claim 2 , configured to:
4. The partition unit includes: evaluating the accuracy of the first analysis result by comparing the first analysis result against ground truth results for the original dataset; partitioning a first subset of the first set of subjects associated with true positive results into the first data partition; partitioning a second subset of the first set of subjects associated with false positive results into the second data partition; The information processing device of claim 1 , configured to:
5. a labeling unit for assigning true positive or false positive outcome labels to the modified third data partition and the modified fourth data partition; The information processing device of claim 1 further comprising:
6. The labeling unit comprises: calculating a first similarity between a first subset of subjects of the first data partition and a third subset of subjects of the modified third data partition; assigning a true positive result label to the third subset of subjects if the first similarity reaches a similarity threshold; assigning a false positive result label to the third subset of subjects if the first similarity does not reach a similarity threshold; calculating a second similarity between a second subset of subjects of the second data partition and a fourth subset of subjects of the modified fourth data partition; assigning a false positive result label to the fourth subset of subjects if the second similarity reaches a similarity threshold; assigning a true positive result label to the fourth subset of subjects if the second similarity does not reach a similarity threshold.
6. An information processing device according to claim 5.
7. An information processing device; A user terminal; An information processing system comprising: The information processing device is A processor; and a memory containing computer-implementable instructions executable by said processor, said computer-implementable instructions causing said processor to: generating a set of analytical results by processing the original dataset with a first set of machine learning models; identifying a high variance subset of the original data set that includes a first set of subjects associated with a first analysis result that meets a variance threshold based on the set of analysis results; and a variance analysis unit configured to perform partitioning the first set of subjects of the highly variance subset into a first data partition and a second data partition; generating a third data partition that is a copy of the first data partition and a fourth data partition that is a copy of the second data partition; A partition unit configured to: a modification unit configured to generate a modified third data partition by modifying the third data partition and a modified fourth data partition by modifying the fourth data partition; training a second set of machine learning models using the first data partition, the second data partition, the modified third data partition, and the modified fourth data partition; generating a second analysis result by processing the original dataset with the second set of machine learning models; generating a final analysis result for the first set of subjects of the high variance subset by aggregating and classifying the first analysis result and the second analysis result; outputting the final analysis result to the user terminal; a result generating unit configured to: An information processing system that functions as a
8. 1. An information processing method executed by a computer, the information processing method comprising: generating a set of analytical results by processing the original dataset with a first set of machine learning models; identifying a high variance subset of the original data set based on the set of analysis results, the high variance subset including a first set of subjects associated with a first analysis result that meets a variance threshold; partitioning the first set of subjects of the highly variance subset into a first data partition and a second data partition; generating a third data partition that is a copy of the first data partition and a fourth data partition that is a copy of the second data partition; generating a set of sample prediction results by iteratively processing the modified sample data set with increasing amounts of noise using a test set of the machine learning model; identifying, from said set of sample prediction results, a subset of sample prediction results that meets a predefined accuracy threshold; determining as a noise level metric a maximum amount of noise added to the modified sample data set associated with the subset of sample prediction results that meet the predetermined accuracy threshold; generating a modified third data partition by adding a first amount of noise to a third set of characteristics of the third data partition based on the noise level criterion; generating a modified fourth data partition by adding a second amount of noise to a fourth set of characteristics of the fourth data partition based on the noise level criterion; training a second set of machine learning models using the first data partition, the second data partition, the modified third data partition, and the modified fourth data partition; generating a second analysis result by processing the original dataset with the second set of machine learning models; generating a final analysis result for the first set of subjects of the high variance subset by aggregating and classifying the first analysis result and the second analysis result; An information processing method comprising:
Citation Information
Patent Citations
Model shrinking for embedded keyword spotting
US9600231B1
