Systems and methods for training machine learning-based models

Abstract

A system and method for training a machine learning (ML)-based model by at least one processor may include receiving an initial dataset including a plurality of annotated data samples; training at least one ML-based first-level model to perform a first-level task based on the initial dataset; calculating, for each data sample based on the training of the at least one ML-based first-level model, at least one feature representing a value of its contribution to training an ML-based second-level model to perform a second-level task; omitting a subset of data samples from the initial dataset based on the at least one feature to obtain a target dataset; and training an ML-based second-level model to perform the second-level task based on the target dataset.

Description

[Technical Field]

[0001] The present invention relates generally to machine learning and artificial intelligence, and more particularly to techniques for preparing training datasets used in machine-learning (ML) processes. [Background technology]

[0002] As is known, developing a mathematical model that can learn from data and make predictions about the data is a common goal of machine learning. In particular, supervised and semi-supervised machine learning involve model training using a so-called "training dataset" (or "teacher dataset") and testing using an "inference dataset." The term "training dataset" generally refers to a set of annotated data samples, where the annotations provide associations between the data samples and multiple classes. In other words, the annotated data samples represent pairs of input and output vectors (or scalars) for a machine learning model. The model iteratively analyzes the data samples in the training dataset to generate results, which are then compared with target results (corresponding to the annotations of the data samples in the training dataset). Based on this comparison, the supervised learning algorithm determines the optimal combination of variables that provides the highest predictive reliability. Finally, a well-trained model should exhibit sufficiently reliable results when analyzing unknown data.

[0003] Therefore, the quality of the training dataset can reasonably be considered an important aspect of machine learning. When examining and analyzing a training dataset for a particular machine learning task, various problems can be identified and considered to be solved.

[0004] For example, uneven distribution of classes in a dataset, called data imbalance, is considered one of the most common problems in classification machine learning tasks. When the data is highly imbalanced, the trained model is likely to suffer from poor prediction accuracy for the minority class. There are many methods known from the prior art that aim to deal with imbalanced classes, such as random oversampling and undersampling, cluster centroid-based majority undersampling technique (CCMUT), synthetic minority oversampling technique (SMOTE), and applying a higher penalty for misclassification of the minority class.

[0005] However, in addition to the problem of training dataset quantity, there is also the problem of quality. As is known, different data samples in the same dataset can have different degrees of influence on the training process. It is important to find an appropriate combination of data samples for the training dataset in order to develop the model's ability to generalize and accurately distinguish data. Therefore, when solving the data imbalance problem by reducing a certain amount of data samples from the majority class, it is important not to lose valuable data samples and to retain redundant data samples. However, existing undersampling techniques either insufficiently or not at all consider the expected contribution of each data sample to the subsequent training process. As a result, ML-based models trained on datasets achieved by such methods lack reliability, especially when considering the contribution to the training process that each initial dataset can potentially provide. Summary of the Invention

[0006] Therefore, there is a need for a system and method for training an ML-based model that incorporates an improved process for balancing the training dataset, thereby improving the training dataset quality and, consequently, the reliability of the trained ML-based model performance. More specifically, there is a need to create a data balancing method that provides for the omission of data samples based on an assessment of their expected contribution to the following training process.

[0007] In order to overcome the shortcomings of the prior art, the following invention is provided.

[0008] In a general aspect, the present invention relates to a method for training a machine learning (ML)-based model by at least one processor, the method including: receiving an initial dataset including a plurality of annotated data samples; training at least one ML-based first-level model to perform a first-level task based on the initial dataset; calculating, for each data sample based on the training of the at least one ML-based first-level model, at least one feature representing a value of its contribution to training an ML-based second-level model to perform a second-level task; omitting a subset of data samples from the initial dataset based on the at least one feature to obtain a target dataset; and training an ML-based second-level model to perform the second-level task based on the target dataset.

[0009] In another general aspect, the invention relates to a system for training an ML-based model, the system including: a non-transitory memory device having a module of instruction code stored therein; and at least one processor associated with the memory device and configured to execute the module of instruction code, wherein upon execution of the module of instruction code, the at least one processor is configured to: receive an initial dataset including a plurality of data samples; train at least one ML-based first-level model to perform a first-level task based on the initial dataset; calculate, for each data sample based on the training of the at least one ML-based first-level model, at least one feature representing a value of a contribution to training of an ML-based second-level model to perform a second-level task; and omit a subset of the data samples from the initial dataset based on the features to obtain a target dataset for training the ML-based second-level model to perform the second-level task.

[0010] In some embodiments, training the at least one ML-based first-level model, computing the at least one feature, and omitting a subset of the data samples are performed iteratively.

[0011] In some embodiments, the annotated data samples include annotations that provide associations between the data samples and a plurality of classes, and the method further includes selecting at least one class from the plurality of classes based on an amount of data samples associated with each class in the initial dataset.

[0012] In some embodiments, the at least one processor is further configured to select at least one class of the plurality of classes based on an amount of data samples associated with each class in the initial data set.

[0013] In some embodiments, training the at least one ML-based first-level model includes dividing the initial dataset into a training portion of data samples and an inference portion of data samples; training the at least one ML-based first-level model to perform the first-level task based on the training portion; and inferring the at least one trained ML-based first-level model on the inference portion to perform the first-level task based on the training of the at least one ML-based first-level model.

[0014] In some embodiments, the at least one processor is configured to train the at least one ML-based first level model by: dividing the initial dataset into a training portion of data samples and an inference portion of data samples; training the at least one ML-based first level model to perform the first level task based on the training portion; and inferring the at least one trained ML-based first level model on the inference portion to perform the first level task based on the training of the at least one ML-based first level model.

[0015] In some embodiments, the division of the initial dataset into a training portion of data samples and an inference portion of data samples is performed by having a random proportion of data samples of each class of multiple classes in the training portion or the inference portion.

[0016] In some embodiments, the first level task includes classification of data samples of the initial dataset by a plurality of classes, and at least one feature includes a confidence value representing an association of one or more data samples of the inference portion with their respective associated classes in the inference result, and the method further includes selecting a subset of data samples from the inference portion based on a set of skipping conditions, the skipping conditions including (i) association of the selected data samples with at least one selected class in the inference result, and (ii) selection of the data samples based on the calculated confidence value.

[0017] In some embodiments, the at least one processor is further configured to select a subset of data samples from the inference portion based on a set of omission conditions, the omission conditions including (i) association of the selected data samples with at least one selected class in the inference result, and (ii) selection of the data samples based on the calculated confidence value.

[0018] In some embodiments, selecting the data sample based on the calculated confidence values ​​includes selecting the data sample having (a) the highest scoring confidence value and (b) a confidence value above a predetermined threshold.

[0019] In some embodiments, the at least one processor is further configured to select, based on the calculated confidence values, the data sample as having (a) the highest-scoring confidence value and (b) a confidence value above a predetermined threshold.

[0020] In some alternative embodiments, the at least one ML-based first-level model includes multiple ML-based first-level models.

[0021] In some alternative embodiments, training the at least one ML-based first level model includes training a plurality of ML-based first level models, and selecting the data sample based on the calculated confidence values ​​includes selecting the data sample based on a function of the calculated confidence values ​​of particular data samples in the plurality of first level models.

[0022] In yet another alternative embodiment, the first level task includes clustering the selected at least one class and forming, in the clustering results, a set of clusters of data samples associated with the selected at least one class, and the at least one feature includes, in the inference results, a distance of one or more data samples from a centroid of a cluster of the set of clusters to which the one or more data samples are associated, and the method further includes selecting a subset of data samples from the inference portion based on a set of skipping conditions, the skipping conditions including selecting data samples based on the calculated distance.

[0023] In some alternative embodiments, the at least one processor is further configured to select a subset of data samples from the inference portion based on a set of omission conditions, the omission conditions including selecting data samples based on calculated distances.

[0024] In some embodiments, selecting a data sample based on the calculated distance includes selecting a data sample that has (a) the shortest distance to the centroid of the cluster to which the data sample pertains and (b) a distance that is less than a predetermined threshold.

[0025] In some alternative embodiments, selecting data samples based on the calculated distances includes, for at least one cluster, determining a range of distances corresponding to the densest distribution of data samples within the cluster, and selecting at least one data sample having a distance from the determined range.

[0026] In some embodiments, the method further includes calculating, for the selected at least one class, an omission percentage representing a percentage of data samples to be omitted from the initial dataset, and selecting a subset of data samples from the inference portion based on a set of omission conditions, including selecting data samples up to the calculated omission percentage.

[0027] In some embodiments, the second level task involves classification of data samples of the incoming dataset according to multiple classes. [Brief explanation of the drawings]

[0028] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of this specification. However, the invention, both as to organization and method of operation, together with its objects, features, and advantages, may best be understood by reference to the following detailed description when read in connection with the accompanying drawings. [Figure 1] FIG. 1 is a block diagram depicting a computing device that may be included in a system for training an ML-based model, according to some embodiments. [Figure 2] FIG. 1 is a block diagram depicting a system for training an ML-based model, according to some embodiments. [Figure 3A] 10A-10C are a series of diagrams illustrating examples of selecting and omitting a subset of data samples from the inference portion of a training dataset, according to some alternative embodiments. [Figure 3B] 10A-10C are a series of diagrams illustrating examples of selecting and omitting a subset of data samples from the inference portion of a training dataset, according to some alternative embodiments. [Figure 3C] 10A-10C are a series of diagrams illustrating examples of selecting and omitting a subset of data samples from the inference portion of a training dataset, according to some alternative embodiments. [Figure 4] FIG. 1 is a flow diagram depicting a method for training an ML-based model, according to some embodiments.

[0029] It will be understood that for simplicity and clarity of illustration, elements illustrated in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. DETAILED DESCRIPTION OF THE INVENTION

[0030] Those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. Accordingly, the foregoing embodiments are to be considered in all respects illustrative rather than limiting of the invention described herein. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

[0031] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For clarity, descriptions of the same or similar features or elements may not be repeated.

[0032] For example, although embodiments of the invention are not limited in this respect, discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” “establishing,” “analyzing,” “checking,” “choosing,” “selecting,” “omitting,” “training,” etc. may refer to operations and / or processes of a computer, computing platform, computing system, or other electronic computing device that manipulates and / or transforms data represented as physical (e.g., electronic) quantities in the computer's registers and / or memory into other data similarly represented as physical quantities in the computer's registers and / or memory, or other information non-transitory storage medium that may store instructions for performing an operation(s) and / or process(es).

[0033] Although embodiments of the invention are not limited in this respect, the terms "plurality" and "a plurality" as used herein may include, for example, "multiple" or "two or more." The terms "plurality" or "a plurality" may be used throughout this specification to describe two or more components, devices, elements, units, parameters, etc. The term "set," as used herein, may include one or more items.

[0034] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Additionally, the described method embodiments, or some of their elements, may occur or be performed simultaneously, contemporaneously, in parallel, or in an iterative manner.

[0035] In some embodiments of the present invention, the ML-based model may be an artificial neural network (ANN).

[0036] A neural network (NN) or artificial neural network (ANN), e.g., a neural network implementing machine learning (ML) or artificial intelligence (AI) functions, may refer to an information processing paradigm that may include nodes called neurons organized into layers with links between them. The links may transfer signals between neurons and may be associated with weights. NNs may be configured or trained for specific tasks, e.g., pattern recognition or classification. Training a NN for a specific task may involve adjusting these weights based on examples. Each neuron in an intermediate or final layer may receive an input signal, e.g., a weighted sum of output signals from other neurons, and may process the input signal using a linear or nonlinear function (e.g., an activation function). The results of the input and intermediate layers may be transferred to other neurons, and the results of the output layer may be provided as the output of the NN. Typically, neurons and links in a NN are represented by mathematical structures, such as activation functions and matrices of data elements and weights. A processor, e.g., a CPU or a graphics processing unit (GPU), or a dedicated hardware device, may perform the relevant calculations.

[0037] It will be apparent to those skilled in the art that various ML-based models may be implemented without departing from the essence of the present invention. It should also be understood that in some embodiments, the ML-based model may be a single ML-based model or a set (ensemble) of ML-based models that collectively achieve the same functionality as a single one. Therefore, in view of the scope of the present invention, the above-described variations should be considered equivalent.

[0038] In some respects, the following description of the claimed invention is provided with respect to the task of solving the problem of imbalanced data in a training dataset. The imbalance problem arises when an annotated dataset has a significantly unequal distribution of data samples among designated classes. For the sake of illustrative clarity, such particular objectives and embodiments of the invention directed thereto are provided, and these embodiments are not intended to limit the scope of protection claimed by the present invention. Those skilled in the art will understand that implementations of the claimed invention according to such tasks are provided as non-exclusive examples, and that other practical implementations may be covered by the claimed invention, such as any implementation utilizing the claimed method to omit data samples, whether the purpose of such omissions is to solve the problem of imbalanced data or different.

[0039] As is known, artificial intelligence and machine learning techniques have proven to be indispensable tools for solving tasks in which the connections and dependencies between input data and target output data are complex and uncertain, and are therefore ubiquitously implemented. Accordingly, the proposed invention incorporates the use of ML techniques to evaluate the expected contribution of each data sample in an annotated dataset into the following training process. The claimed invention thereby proposes an approach for defining a subset of data samples in the training dataset that are to some extent redundant and therefore can be reasonably omitted. As a result, such incorporation of ML techniques into the process of preparing a training dataset provides for the achievement of an improved technical effect by increasing the quality of the training dataset and, therefore, increasing the reliability of the trained ML-based model performance.

[0040] Reference is now made to FIG. 1, which is a block diagram illustrating a computing device that may be included in one embodiment of a system for training an ML-based model, according to some embodiments.

[0041] Computing device 1 may include a processor or controller 2, which may be, for example, a central processing unit (CPU) processor, chip, or any suitable computing or calculation device, an operating system 3, a memory device 4, instruction code 5, a storage system 6, input devices 7, and output devices 8. Processor 2 (or, as the case may be, one or more controllers or processors across one or more units or devices) may be configured to perform methods described herein and / or to perform or operate as various modules, units, etc. More than one computing device 1 may be included in a system according to embodiments of the present invention, and one or more computing devices 1 may function as components of a system according to embodiments of the present invention.

[0042] Operating system 3 may be or include any code segment (e.g., similar to instruction code 5 described herein) designed and / or configured to perform tasks involving coordinating, scheduling, arbitrating, supervising, controlling, or otherwise managing the operation of computing device 1, such as scheduling the execution of software programs or tasks, or enabling software programs or other modules or units to communicate. Operating system 3 may be a commercial operating system. Note that operating system 3 may be any component, for example, in some embodiments, a system may include a computing device that does not require or include an operating system 3.

[0043] Memory device 4 may be or include, for example, random-access memory (RAM), read-only memory (ROM), dynamic RAM (DRAM), synchronous DRAM (SD-RAM), double data rate (DDR) memory chips, flash memory, volatile memory, non-volatile memory, cache memory, buffer, short-term storage unit, long-term storage unit, or other suitable memory or storage unit. Memory device 4 may be or include a plurality of possible different memory units. Memory device 4 may be a computer or processor non-transitory readable medium or a computer non-transitory storage medium, such as RAM. In one embodiment, a non-transitory storage medium, such as memory device 4, a hard disk drive, or another storage device, may store instructions or code that, when executed by a processor, can cause the processor to perform the methods described herein.

[0044] The instruction code 5 may be any executable code, such as an application, a program, a process, a task, or a script. The instruction code 5 may be executed by the processor or controller 2, perhaps under the control of the operating system 3. For example, the instruction code 5 may be a standalone application or an API module that may be configured to train an ML-based model, as described further herein. For clarity, although a single item of instruction code 5 is shown in FIG. 1 , systems according to some embodiments of the present invention may include multiple executable code segments or modules similar to the instruction code 5 that may be loaded into the memory device 4 and cause the processor 2 to perform the methods described herein.

[0045] Storage system 6 may be or include, for example, flash memory as known in the art, memory within or embedded in a microcontroller or chip as known in the art, a hard disk drive, a CD-Recordable (CD-R) drive, a Blu-ray disc (BD), a universal serial bus (USB) device, or other suitable removable and / or fixed storage unit. Various types of input and output data may be stored in storage system 6 or loaded from storage system 6 to memory device 4, where it may be processed by processor or controller 2. In some embodiments, some of the components shown in FIG. 1 may be omitted. For example, memory device 4 may be a non-volatile memory having the storage capacity of storage system 6. Thus, although shown as a separate component, storage system 6 may be embedded in or included in memory device 4.

[0046] Input device(s) 7 may be or include any suitable input device, component, or system, such as a detachable keyboard or keypad, a mouse, etc. Output device(s) 8 may include one or more (possibly detachable) displays or monitors, speakers, and / or any other suitable output device. Any applicable input / output (I / O) devices may be connected to computing device 1, as indicated by blocks 7 and 8. For example, a wired or wireless network interface card (NIC), a universal serial bus (USB) device, or an external hard drive may be included in input device(s) 7 and / or output device(s) 8. It will be appreciated that any suitable number of input devices 7 and output devices 8 may be operably connected to computing device 1, as indicated by blocks 7 and 8.

[0047] Systems according to some embodiments of the present invention may include components such as, but not limited to, multiple central processing units (CPUs) or any other suitable general-purpose or specific processors or controllers (e.g., similar to element 2), multiple input units, multiple output units, multiple memory units, and multiple storage units.

[0048] Referring now to FIG. 2, a system 10 for training an ML-based model is shown, according to some embodiments.

[0049] According to some embodiments of the present invention, system 10 may be implemented as software modules, hardware modules, or any combination thereof. For example, system 10 may be or include a computing device, such as element 1 of FIG. 1 . Furthermore, system 10 may be adapted to execute one or more modules of instruction code (e.g., element 5 of FIG. 1 ) to request, receive, analyze, calculate, and generate various data. As described in further detail herein, system 10 may be adapted to execute one or more modules of instruction code (e.g., element 5 of FIG. 1 ) to evaluate the expected contribution of each data sample of the annotated dataset to a subsequent training process, calculate at least one feature of the data samples, omit a subset of the data samples based on the feature, train an ML-based model, etc.

[0050] As shown in Figure 2, arrows may represent the flow of one or more data elements into and out of system 10 and / or between modules or elements of system 10. Some arrows have been omitted from Figure 2 for clarity.

[0051] In some embodiments, system 10 may be logically divided into two levels: a first level directed to ML data preprocessing (e.g., solving the problem of imbalanced data) and a second level directed to training an ML-based target model (ML-based second-level model 20) based on the first-level preprocessed data.

[0052] In some embodiments, the first level of the system 10 may include the following modules of an instruction code (e.g., instruction code 5 of the computing device 1 as shown in FIG. 1 ): a dataset analysis and partitioning module 30, a first level training module 40, and an abbreviation module 50.

[0053] The dataset analysis and segmentation module 30 may be configured to request and receive an initial dataset 61A (eg, extracted from a database 60A provided by a third-party data supplier 60).

[0054] The initial dataset 61A can include multiple annotated data samples. The annotated data samples may include annotations that provide associations between the data samples and multiple classes, as is commonly done in supervised or semi-supervised machine learning implementations. The initial dataset 61A can have a variety of structures and contents depending on the particular ML task to be solved (e.g., speech recognition, email spam detection, plant species classification, etc.). The objectives of the claimed invention are not constrained by the particular structure or content of the annotated dataset, and it will be clear to one skilled in the art what structure or content they may have, as such annotated datasets are considered to be a general aspect of machine learning.

[0055] In some embodiments, the dataset analysis and partitioning module 30 may be further configured to analyze the received initial dataset 61A to determine whether the received initial dataset 61A has an imbalanced data problem, i.e., whether it has a significantly unequal distribution of data samples among classes. The dataset analysis and partitioning module 30 may be further configured to select at least one class (e.g., a majority class) of the plurality of classes based on the amount of data samples associated with each class in the initial dataset 61A determined in the results of the analysis.

[0056] The dataset analysis and partitioning module 30 may further be configured to calculate, for at least one selected class (e.g., the majority class), an omission percentage, which may represent the percentage of data samples that must be omitted from the initial dataset 61A in order to equalize the distribution of data samples among the classes.

[0057] The dataset analysis and segmentation module 30 may further be configured to generate a skip setting 30A that includes information about the selected at least one class and the calculated skip percentage.

[0058] The dataset analysis and partitioning module 30 may be further configured to partition the initial dataset 61A into a training portion of data samples 31A and an inference portion of data samples 31B. In some embodiments, this partitioning may be performed by having a random ratio of data samples of each class of multiple classes in the training portion 31A or the inference portion 31B. In some alternative embodiments, the ratio may be predetermined. In yet another alternative embodiment, the dataset analysis and partitioning module 30 may be further configured to calculate the ratio as a function of such ratio within the initial dataset 61A when considered as a whole. The ratio of the sizes of the training portion 31A and the inference portion 31B may also vary as commonly used, e.g., 70% for the training portion 31A and 30% for the inference portion 31B.

[0059] In some embodiments, first-level training module 40 may be configured to train at least one ML-based first-level model (e.g., first-level model 70) to perform the first-level task based on initial dataset 61A.

[0060] Thus, in some embodiments, first-level training module 40 may be configured to receive training portion 31A and inference portion 31B. First-level training module 40 may be further configured to train at least one ML-based first-level model (e.g., first-level model 70) to perform a first-level task based on the training portion (e.g., training portion 31A). First-level training module 40 may be further configured to infer at least one trained ML-based first-level model (e.g., first-level model 70) on the inference portion (e.g., inference portion 31B) to perform a first-level task based on the training of the at least one ML-based first-level model (e.g., first-level model 70).

[0061] In some embodiments, first-level training module 40 may be further configured to calculate, based on the training of at least one ML-based first-level model (e.g., first-level model 70), for each data sample, at least one feature representing a value of its contribution to subsequent training of an ML-based second-level model (e.g., second-level model 20) for performing the second-level task. Thus, in the result of the inference, first-level model 70 may be configured to output characterized inference portion 70A of the data sample, including the data sample of inference portion 31B with the values ​​of the above-calculated features.

[0062] In some embodiments, the skipping module 50 may be configured to receive the characterized inference portion 70A and the skipping configuration 30A. The skipping module 50 may be further configured to select a subset of data samples from the characterized inference portion 70A based on a set of skipping conditions, including selection of data samples up to a calculated skip percentage, and association of the selected data samples with at least one selected class (e.g., a majority class), as specified by the skipping configuration 30A.

[0063] The omission module 50 may be further configured to omit a subset of data samples from the characterized inference portion 70A of the initial dataset 61A based on at least one feature specified in the characterized inference portion 70A. The omission module 50 may be further configured to obtain a target dataset 51A, which is a portion of the initial dataset 61A remaining after the omission. The obtained target dataset 51A may be further used in a second-level process of the system 10, as further described herein, or may be transferred to and used by a third-party system.

[0064] In some embodiments, system 10 may be configured to train at least one ML-based first-level model (e.g., first-level model 70) via first-level training module 40, compute at least one feature via first-level training module 40, and iteratively omit subsets of data samples from characterized inference portion 70A via omission module 50. Depending on the particular embodiment, all three processes (the training, computing, and omission) may be included in the iterative process together, separately, or in any combination thereof. It should be understood that in such embodiments, any of the training, computing, and omission processes may provide intermediate results in intermediate iterations of the iterative process and a final (target) result in a final iteration.

[0065] In particular, the dataset analysis and partitioning module 30 may be configured to divide the calculated omission percentage into intermediate omission percentages to distribute the omission process proportionally to the number of iterations and obtain the required number of data samples that are omitted overall. The dataset analysis and partitioning module 30 may be further configured to generate an omission setting 30A that includes information about the calculated omission percentage of data samples that are omitted overall and the omission percentage of data samples that are omitted in each iteration. The omission module 50 may be further configured to select a subset of data samples from the characterized inference portion 70A based on a set of omission conditions, including selecting data samples up to the calculated intermediate omission percentage, as specified by the omission setting 30A. The omission module 50 may be further configured to omit a subset of data samples from the characterized inference portion 70A of the initial dataset 61A according to each intermediate iteration to obtain an intermediate reduced inference portion 50A. In the next iteration, the dataset analysis and partitioning module 30 may be further configured to receive the intermediate reduced inference portion 50A of the previous iteration and replace the inference portion 31B of the previous iteration with the intermediate reduced inference portion 50A, thereby obtaining an intermediate reduced initial dataset (not shown).

[0066] The dataset analysis and partitioning module 30 may be further configured to perform, at each intermediate iteration, a random partitioning of the intermediate, reduced initial dataset into a training portion of data samples 31A and an inference portion of data samples 31B. As a result, at each iteration, the first-level model 70 is trained and inferred on a respective portion of data samples with a combination of data samples that changes from iteration to iteration. Thus, by achieving more reliable selection and omission of data samples across the initial dataset 61A, additional technical effects may be provided.

[0067] It should be understood that the iterative process is not limited by incorporating only the training, calculation, and omission steps described above, and thus may further include other processes either related or unrelated to the objectives of the claimed invention.

[0068] In some embodiments, the second level of the system 10 may include the following modules of the instruction code (e.g., instruction code 5 of the computing device 1 as shown in FIG. 1): a dataset splitting module 80 and a second-level training module 90.

[0069] In some embodiments, the dataset splitting module 80 may be configured to receive the target dataset 51 A. The dataset splitting module 80 may be further configured to split the target dataset 51 A into a training portion of data samples 80A and an inference portion of data samples 80B.

[0070] In some embodiments, second-level training module 90 may be configured to train an ML-based second-level model (e.g., second-level model 20) to perform a second-level task based on target dataset 51A.

[0071] More specifically, the second-level training module 90 may be configured to receive the training portion 80A and the inference portion 80B. The second-level training module 90 may be further configured to train an ML-based second-level model (e.g., second-level model 20) to perform a second-level task based on the training portion (e.g., training portion 80A) of the target dataset 51A. The second-level training module 90 may be further configured to infer the ML-based second-level model (e.g., second-level model 20) trained on the inference portion (e.g., inference portion 80B) of the target dataset 51A to perform a second-level task based on the training of the ML-based second-level model (e.g., second-level model 20). By inferring, the performance of the second-level model (e.g., second-level model 20) may be evaluated.

[0072] It will be apparent to those skilled in the art that the present invention is not limited by the incorporation of any particular type of ML-based model (e.g., Convolutional Neural Network (CNN), Multilayer Perceptron, Naive Bayes algorithm, Support Vector Machine (SVM) algorithm, etc.), any particular known ML method (e.g., Reinforcement Learning, Backpropagation, Stochastic Gradient Descent (SGD), etc.), or any particular ML task (classification, regression, clustering, etc.).

[0073] The essence of the present invention can be generalized by the idea of ​​utilizing machine learning techniques to optimize an annotated dataset for further use in the following machine learning process. More specifically, the essence of the present invention is based on the idea that training an ML-based model (e.g., first-level model 70) on a dataset (e.g., initial dataset 61A) may provide specific information characterizing the data samples of this dataset with respect to their expected contribution in, or usefulness for, training another ML-based model (e.g., second-level model 20).

[0074] It should therefore be understood that the combination of these models (e.g., first-level model 70 and second-level model 20) itself is essential to the purposes of the claimed invention, rather than any particular implementation of this combination that explicitly defines each or any of the ML-based models utilized. Thus, the particular implementation of the combination of ML-based models may vary from extrapolated examples where the first-level model 70 and the second-level model 20 are the same, to examples where they are completely different, or even where an ensemble of different first-level models 70 is used.

[0075] However, to keep this description sufficiently illustrative and non-abstract, the present invention will be further described by providing three alternative, non-exclusive embodiments, which will be described in detail with reference to Figures 3A-3C, respectively.

[0076] 3A-3C include a series of diagrams depicting examples of selecting and omitting a subset of data samples from an inference portion (e.g., inference portion 31B) of initial data set 61A according to respective alternative embodiments. The diagrams show the distribution of data samples in inference portion 31B in a feature space defined by features A and B.

[0077] According to the annotations provided in the initial dataset 61A, the data samples are divided into two classes. The first class 610A (or 610B or 610C, respectively) is marked by a triangle, and the second class 611A (or 611B or 611C, respectively) is marked by a circle. As can be seen, class 610A (or 610B or 610C, respectively) contains a significantly larger number of data samples than class 611A (or 611B or 611C, respectively) and may therefore be selected by the data analysis and partitioning module 30 as the majority class that requires the omission of a certain percentage of data samples to solve the data imbalance problem.

[0078] The first particular embodiment is further described with reference to FIG. 3A.

[0079] According to a first particular embodiment, the first-level model 70 may be a classification model, and the first-level task may include classifying data samples of the initial dataset 61A according to multiple classes (e.g., classes 610A and 611A). As a result of training the first-level model 70, the first-level training module 40 may be configured to calculate a decision boundary 700A, as shown in the series of figures. The decision boundary 700A may define a division of the feature space into classes, as can be performed by the trained first-level model 70.

[0080] In this embodiment, the second-level model 20 may also be a classification model. The second-level task may involve classifying data samples of an incoming dataset according to multiple classes (e.g., classes 610A and 611A).

[0081] In a first particular embodiment, first-level training module 40 may be configured to calculate the at least one feature, which may include a confidence value representing the association of one or more data samples of inference portion 31B with their respective associated classes (e.g., classes 610A and 611A) in the results of inference of first-level model 70. With respect to the provided illustration, it should be understood that the farther a particular data sample is located from decision boundary 700A, the higher the confidence value it will have in the results of inference.

[0082] In a first particular embodiment, the omission module 50 may be further configured to select a subset of data samples (e.g., subset 612A) from the inference portion 31B based on a set of exclusion conditions including association of the selected data samples with at least one selected class (e.g., class 610A specified by the exclusion configuration 30A) in the inference results and selection of the data samples based on calculated confidence values. More specifically, the omission module 50 may be further configured to select data samples having (a) the highest-scoring confidence value and (b) a confidence value above a predetermined threshold (e.g., threshold 701A indicated by a line located a certain distance from the decision boundary 700A). In some embodiments, the dataset analysis and segmentation module 30 may be configured to predefine (either manually or automatically) the threshold 701A and include information about it in the omission configuration 30A.

[0083] Thus, as can be seen in the series of figures, the omission module 50 selects a subset 612A containing three data samples, selected because these data samples relate to the selected majority class 610A and are located further from the decision boundary 700A and above the threshold 701A. The final figure in the series depicts the contents of a reduced inferred portion formed as a result of the omission of subset 612A by the omission module 50. The reduced inferred portion is further included in the target dataset 51A.

[0084] As can be seen, this particular embodiment can assume that the data sample with the highest confidence value calculated during training of one classification model (e.g., first level model 70) has negligible impact on the training of another classification model (e.g., second level model 20)'s ability to classify incoming data samples by class (e.g., classes 610A and 611A).

[0085] The second particular embodiment is further described with reference to FIG. 3B.

[0086] According to a second particular embodiment, the first-level model 70 may be a clustering model. The first-level task may include clustering at least one selected class (e.g., majority class 610B) and forming a set of clusters 700B of data samples associated with the selected at least one class (e.g., class 610B) in the clustering results. The first-level training module 40 may be configured to calculate the location of the centroid 701B of the cluster 700B in the feature space in the inference results of the first-level model 70.

[0087] In this embodiment, the second-level model 20 may be a classification model. The second-level task may involve classifying data samples of an incoming dataset according to multiple classes (e.g., classes 610B and 611B).

[0088] In a second particular embodiment, the first-level training module 40 may be configured to calculate the at least one feature, the feature comprising, in the results of inference of the first-level model 70, the distance of one or more data samples of the inference portion 31B from the centroid of a cluster of the set of clusters to which the one or more data samples relate (e.g., the centroid 701B of each cluster 700B).

[0089] In a second particular embodiment, the omission module 50 may be further configured to select a subset of data samples from the inference portion 31B based on a set of omission criteria, including selecting data samples based on calculated distances. More specifically, the omission module 50 may be further configured to select data samples having (a) the shortest distance to the centroid of the data sample's associated cluster, and (b) a distance less than a predetermined threshold (not shown). In some embodiments, the dataset analysis and segmentation module 30 may be configured to predefine the threshold (either manually or automatically) and include information about it in the omission configuration 30A. In the illustrated example, the selected data samples are marked by having a dashed closed curve.

[0090] Thus, as can be seen in the series of figures, the omission module 50 selects a subset including four data samples (one from each cluster 700B) that are selected because they are located closer to the respective centroids 701B of the clusters 700B compared to the other data samples, and their respective distances to the centroids 701B are less than a predetermined threshold. The final figure in the series represents the content of the reduced inferred portion resulting from the omission of the subset by the omission module 50. The reduced inferred portion is further included in the target dataset 51A.

[0091] As can be seen, this particular embodiment can assume that the data sample with the shortest distance to the centroid 701B of its associated cluster 700B is the most definable member of the respective cluster and, therefore, of the majority class (e.g., class 610A) overall. Therefore, such data samples have negligible impact on training the ability of a classification model (e.g., second-level model 20) to classify incoming data samples by class (e.g., classes 610A and 611A). As can be seen in the last diagram in the series of Figures 3B, the reduced inference portion has a more uniform distribution of data samples in feature space than inference portion 31B.

[0092] In yet another alternative embodiment, the omission module 50 may be configured to determine, for at least one cluster (e.g., for at least one cluster 700B), a range of distances to the respective cluster centroid 701B that corresponds to the densest distribution of data samples within the respective cluster 700B. The omission module 50 may be further configured to select at least one data sample having a distance from the determined range. In other words, in such an embodiment, the selection of data samples for further omission is made from the region of feature space having the densest distribution of data samples.

[0093] This particular embodiment may assume that the density of the data sample distribution in the feature space does not significantly contribute to training the ability of some classification model (e.g., second-level model 20) to classify incoming data samples by class (e.g., classes 610B and 611B). Therefore, reducing the density with respect to the majority class (e.g., class 610B) but maintaining the diversity of the data sample distribution in the feature space may be considered an effective approach to balancing the data.

[0094] A third particular embodiment will now be described with reference to FIG. 3C, which includes a series of diagrams illustrating a third example of the selection and omission of a subset of data samples from the inferred portion 31B of the initial data set 61A.

[0095] According to a third particular embodiment, the system 10 includes multiple first-level models, e.g., three different first-level models 70. The first-level training module 40 may be further configured to train multiple ML-based first-level models 70. A series of diagrams for each particular first-level model 70 are provided, each on a separate row.

[0096] According to a third particular embodiment, the first-level models 70 may be classification models, and the first-level task may include classifying data samples of the initial dataset 61A according to multiple classes (e.g., classes 610C and 611C). The first-level training module 40 may be further configured to calculate decision boundaries 701C, 702C, and 703C, respectively, in the results of training of each first-level model 70. The decision boundaries 701C, 702C, and 703C define the division of the feature space into classes as can be performed by each first-level model 70 in the results of training.

[0097] In this embodiment, the second-level model 20 may also be a classification model. The second-level task may involve classifying data samples of an incoming dataset according to multiple classes (e.g., classes 610C and 611C).

[0098] In a third particular embodiment, the first-level training module 40 may be configured to calculate the at least one feature, which includes a confidence value representing the relevance of one or more data samples of the inference portion 31B to their respective associated classes (e.g., classes 610C and 611C) in the inference results of each first-level model 70.

[0099] With respect to the diagram provided, it should be understood that the further a particular data sample is located from the respective decision boundary 701C, 702C, or 703C, the higher the confidence value in the resulting inference.

[0100] In a third particular embodiment, for each first-level model 70, the skipping module 50 may be further configured to select a subset of data samples (e.g., subsets 612C, 613C, and 614C, respectively) from the inference unit 31B based on a set of skipping conditions. The set of skipping conditions may include association of the selected data samples with at least one selected class (e.g., majority class 610C as specified by the skipping setting 30A) in the inference results and selection of the data samples based on calculated confidence values. More specifically, for each first-level model 70, the skipping module 50 may be further configured to select data samples having (a) the highest-scoring confidence value and (b) confidence values ​​above a predetermined threshold (e.g., thresholds 704C, 705C, and 706C indicated by respective lines positioned a certain distance from the respective decision boundaries 701C, 702C, or 703C). In some embodiments, the dataset analysis and segmentation module 30 may be configured to predefine (either manually or automatically) the above thresholds 704C, 705C and 706C and include information about them in the default settings 30A.

[0101] The omission module 50 may be further configured to determine the selection of a subset of data samples from the inference portion 31B by effecting the selection of the data samples based on a function of the calculated confidence values ​​of particular data samples in the plurality of first-level models 70. For example, the omission module 50 may apply a conjunction to subsets 612C, 613C, and 614C to obtain a resulting subset (not shown) that includes only data samples that appear in each of subsets 612C, 613C, and 614C. It should be understood that the claimed invention is not limited to using only the conjunction as the function. Any other suitable function may be used to determine the resulting subset of data samples for further omission.

[0102] Thus, the abbreviation module 50 selects a resulting subset including two data samples, selected because they relate to the selected majority class 610C, lie further from the decision boundaries 701C, 702C, and 703C than the others, lie above the respective thresholds 704C, 705C, and 706C, and are included in each of the selected subsets 612C, 613C, and 614C. The final diagram in the series represents the content of the reduced initial portion obtained as a result of the abbreviation module 50 abbreviating the resulting subsets. The reduced inferred portion is further included in the target dataset 51A.

[0103] As can be seen, this particular embodiment can assume that the data samples with the highest confidence values ​​calculated during training of multiple classification models (e.g., first-level model 70) have a negligible impact on the ability of training another classification model (e.g., second-level model 20) to classify incoming data samples by class (e.g., classes 610C and 611C).

[0104] Referring now to FIG. 4, a flow diagram is presented depicting a method for training an ML-based model by at least one processor, according to some embodiments.

[0105] As shown in step S1005, at least one processor (e.g., processor 2 of FIG. 1) may be executed to receive an initial dataset (e.g., initial dataset 61A) including a plurality of annotated data samples, the annotated data samples including annotations providing associations between the data samples and a plurality of classes (e.g., classes 610A and 611A). Step S1005 may be executed by dataset analysis and segmentation module 30 (as described with reference to FIGS. 2 and 3A-3C).

[0106] As shown in step S1010, at least one processor (e.g., processor 2 of FIG. 1) may select at least one class of a plurality of classes (e.g., classes 610A and 611A) based on the amount of data samples associated with each class in an initial dataset (e.g., initial dataset 61A). Step S1010 may be performed by dataset analysis and segmentation module 30 (as described with reference to FIGS. 2 and 3A-3C).

[0107] As shown in step S1015, at least one processor (e.g., processor 2 of FIG. 1) may calculate, for at least one selected class (e.g., class 610A), an omission percentage representing the percentage of data samples to be omitted from the initial dataset (e.g., initial dataset 61A). Step S1015 may be performed by dataset analysis and segmentation module 30 (as described with reference to FIGS. 2 and 3A-3C).

[0108] As shown in step S1020, at least one processor (e.g., processor 2 of FIG. 1) may perform a partitioning of an initial dataset (e.g., initial dataset 61A) into a training portion of data samples (e.g., training portion 31A) and an inference portion of data samples (e.g., inference portion 31B), where the partitioning is performed by having a random proportion of data samples of each class of multiple classes (e.g., classes 610A and 611A) in the training portion (e.g., training portion 31A) or the inference portion (e.g., inference portion 31B). Step S1020 may be performed by dataset analysis and partitioning module 30 (as described with reference to FIGS. 2 and 3A-3C).

[0109] As shown in step S1025, at least one processor (e.g., processor 2 of FIG. 1) may perform training of at least one ML-based first-level model (e.g., first-level model 70) to perform a first-level task based on the training portion (e.g., training portion 31A). Step S1025 may be performed by first-level training module 40 (as described with reference to FIGS. 2 and 3A-3C).

[0110] As shown in step S1030, at least one processor (e.g., processor 2 in FIG. 1) may perform inference of at least one trained ML-based first-level model (e.g., first-level model 70) for an inference portion (e.g., inference portion 31B) to perform a first-level task based on the training of the at least one ML-based first-level model (e.g., first-level model 70). Step S1030 may be performed by first-level training module 40 (as described with reference to FIGS. 2 and 3A-3C).

[0111] As shown in step S1035, at least one processor (e.g., processor 2 of FIG. 1) may perform, for each data sample, calculation of at least one feature representing a value of contribution to training of an ML-based second-level model (e.g., second-level model 20) for performing the second-level task. Step S1035 may be performed by first-level training module 40 (as described with reference to FIGS. 2 and 3A-3C).

[0112] As shown in step S1040, at least one processor (e.g., processor 2 of FIG. 1) may perform omission of a subset of data samples (e.g., subset 612A) from the initial dataset based on at least one feature to obtain a target dataset (e.g., target dataset 51A), where the subset of data samples (e.g., subset 612A) is selected from the inference portion (e.g., characterized inference portion 70A) based on a set of omission conditions including association of the selected data samples with at least one selected class (e.g., class 610A) in the inference result and selection of data samples up to a calculated omission percentage. Step S1040 may be performed by omission module 50 (as described with reference to FIGS. 2 and 3A-3C).

[0113] As shown in step S1045, at least one processor (e.g., processor 2 of FIG. 1) may perform training of an ML-based second-level model (e.g., second-level model 20) to perform a second-level task based on a target dataset (e.g., target dataset 51A). Step S1045 may be performed by second-level training module 90 (as described with reference to FIGS. 2 and 3A-3C).

[0114] As can be seen from the description provided, the claimed invention represents a system and method for training an ML-based model that incorporates an improved process for balancing a training dataset, thereby improving training dataset quality and, consequently, reliability of the trained ML-based model performance. More specifically, the claimed invention includes a data balancing method that provides for the omission of data samples based on an assessment of their expected contribution to the following training process.

[0115] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Furthermore, all equations described herein are intended as examples only, and other or different equations may be used. Additionally, some of the described method embodiments or elements thereof may occur or be performed at the same time.

[0116] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents may occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the invention.

[0117] Various embodiments have been presented, each of which may, of course, include features from the other embodiments presented, and embodiments not specifically described may include various features described herein.

Claims

1. 1. A method for training a machine learning (ML) based model by at least one processor, the method comprising: receiving an initial dataset including a plurality of annotated data samples, the plurality of annotated data samples including annotations providing associations of the data samples with a plurality of clauses; training at least one ML-based first-level model to perform a first-level task based on the initial dataset; calculating, for each data sample based on training the at least one ML-based first-level model, at least one feature representing a value of a contribution to training an ML-based second-level model for performing a second-level task; omitting a subset of data samples from the initial dataset based on the at least one feature to obtain a target dataset; and training the ML-based second-level model to perform the second-level task based on the target dataset, the method further comprising selecting at least one class from the plurality of classes based on an amount of data samples associated with each class in the initial dataset.

2. 2. The method of claim 1, wherein training the at least one ML-based first-level model, computing the at least one feature, and omitting the subset of data samples are performed iteratively.

3. Training the at least one ML-based first-level model includes: dividing the initial data set into a training portion of data samples and an inference portion of data samples; training the at least one ML-based first-level model to perform the first-level task based on the training portion; and inferring the at least one trained ML-based first-level model for the inference portion to perform the first-level task based on the training of the at least one ML-based first-level model.

4. 4. The method of claim 3, wherein the division of the initial data set into a training portion of the data samples and an inference portion of the data samples is performed by having a random proportion of the data samples of each class of the plurality of classes in the training portion or the inference portion.

5. the first level task includes classifying the data samples of the initial data set according to the plurality of classes; the at least one feature includes a confidence value representing an association between the one or more data samples of the inference portion and each associated class of the one or more data samples of the inference portion in the result of the inference; The method comprises:

4. The method of claim 3, further comprising selecting the subset of data samples from the inference portion based on a set of omission conditions, the omission conditions including: (i) association of the selected data samples with the selected at least one class in an inference result; and (ii) selection of the data samples based on the calculated confidence value.

6. 6. The method of claim 5, wherein the selection of the data samples based on the calculated confidence values ​​includes the selection of the data samples having (a) the highest scoring confidence values ​​and (b) confidence values ​​above a predetermined threshold.

7. training at least one ML-based first-level model includes training a plurality of ML-based first-level models; 6. The method of claim 5, wherein the selecting of the data samples based on the calculated confidence values ​​comprises selecting the data samples based on a function of the calculated confidence values ​​of particular data samples in the plurality of first-level models.

8. the first level task includes clustering the selected at least one class and forming, in a clustering result, a set of clusters of the data samples associated with the selected at least one class; the at least one feature includes, in the result of the inference, a distance of the one or more data samples from a centroid of a cluster of the set of clusters to which the one or more data samples are associated; The method comprises: The method of claim 3 , further comprising selecting the subset of data samples from the inference portion based on a set of skipping conditions, the skipping conditions including selecting the data samples based on the calculated distance.

9. 9. The method of claim 8, wherein the selection of the data samples based on the calculated distances comprises selecting the data samples having (a) a shortest distance to the centroid of the cluster to which the one or more data samples pertain, and (b) a distance less than a predetermined threshold.

10. The selection of the data samples based on the calculated distances comprises: determining, for at least one cluster, a range of distances corresponding to the densest distribution of the data samples within the cluster; and selecting at least one data sample having said distance from said determined range.

11. The method of claim 3 , wherein the second level task comprises classification of data samples of an incoming dataset according to the plurality of classes.

12. The method comprises: calculating an omission percentage for the selected at least one class, the omission percentage representing the percentage of data samples that are omitted from the initial data set; 4. The method of claim 3, further comprising: selecting the subset of data samples from the inferred portion based on a set of omission conditions, the omission conditions including selecting the data samples up to the calculated omission percentage.

13. 1. A system for training an ML-based model, the system comprising: a non-transitory memory device in which a module of instruction code is stored; and at least one processor associated with the memory device and configured to execute the module of instruction code, wherein upon execution of the module of instruction code, the at least one processor: receiving an initial dataset including a plurality of data samples, the plurality of data samples including annotations that provide associations of the data samples with a plurality of classes; training at least one ML-based first-level model to perform a first-level task based on the initial dataset; Based on training the at least one ML-based first-level model, calculate, for each data sample, at least one feature representing a value of a contribution to training an ML-based second-level model for performing a second-level task; and omitting a subset of data samples from the initial dataset based on the features to obtain a target dataset for training the ML-based second-level model to perform the second-level task, wherein the at least one processor is further configured to select at least one class from the plurality of classes based on an amount of data samples associated with each class in the initial dataset.

14. The at least one processor dividing the initial data set into a training portion of data samples and an inference portion of data samples; training the at least one ML-based first-level model to perform the first-level task based on the training portion; and inferring the at least one trained ML-based first-level model for the inference portion based on the training of the at least one ML-based first-level model to perform the first-level task.

15. the first level task includes classifying the data samples of the initial data set according to the plurality of classes; the at least one feature includes a confidence value representing an association between the one or more data samples of the inference portion and each associated class of the one or more data samples of the inference portion in the result of the inference; The at least one processor 15. The system of claim 14, further configured to select the subset of data samples from the inference portion based on a set of omission conditions, the omission conditions including: (i) association of the selected data samples with the selected at least one class in an inference result; and (ii) selection of the data samples based on the calculated confidence value.

16. 16. The system of claim 15, wherein the at least one processor is configured to select the data sample based on the calculated confidence values ​​as having (a) a highest-scoring confidence value and (b) a confidence value above a predetermined threshold.

17. 16. The system of claim 15, wherein the at least one ML-based first level model comprises a plurality of ML-based first level models, and wherein the selection of the data sample based on the calculated confidence value comprises selecting the data sample based on a function of the calculated confidence value of a particular data sample in the plurality of first level models.

18. the first level task includes clustering the selected at least one class and forming, in a clustering result, a set of clusters of the data samples associated with the selected at least one class; the at least one feature includes, in the result of the inference, a distance of the one or more data samples from a centroid of a cluster of the set of clusters to which the one or more data samples are associated; The at least one processor 15. The system of claim 14, further configured to select the subset of data samples from the inference portion based on a set of omission conditions, the omission conditions including selecting the data samples based on the calculated distance.

Images