Iterative training of machine learning models

Abstract

A computer-implemented method, a computer program product, and a computer system are provided. The method includes training a machine learning model using an initial set of training data samples, receiving new training data samples, and predicting labels for the new training data samples. The method also includes, in response to determining that a predicted quality value for the predicted label of the new training data sample is below a predefined quality value, adding the new training data sample to the initial set, thereby constructing an expanded training data set. The method also includes retraining the machine learning model using the expanded training data set.

Description

[Technical field]

[0001] The present invention relates generally to the field of machine learning, and more specifically to the iterative training of machine learning models. [Background technology]

[0002] The subfield of Artificial Intelligence (AI) and Machine Learning (ML) is a growing field in science, in addition to enterprise computing. New algorithms are constantly being designed to achieve better predictions from machine learning systems, or to improve the training of the underlying machine learning models, or both.

[0003] Traditional machine learning models are typically based on machine learning algorithms that use a fixed set of training data samples, together with their respective annotations, to build a general model that allows prediction of annotations for new, previously unseen data samples.

[0004] For example, a deep learning model for image classification is trained on a known set of input images and associated image classification labels. Then, during the production phase as well as during the testing phase, this model is used to predict image class labels for new images that have not been seen before. Of course, other data types instead of images can also be used as input to a machine learning system.

[0005] The input data samples can be generated by data generators. These can include real-world sensors, such as photo or video cameras or both, microphones, or any other sensors, humans or data simulators, such as text descriptions. Data generators can also include artificial systems oriented to generate data samples of a certain predefined type. Typically, traditional data recognition methods operate on a fixed number of training data samples that form a training set of data samples.

[0006] US Patent Application Publication No. 2014 / 0058983A1 discloses a mechanism for training and classifying data. The method includes receiving a data set having at least a first annotation and at least a second annotation. The first annotation and the second annotation represent features in the data set. The method also includes determining a first identifier from the first annotation and a second identifier from the second annotation, and associating the first identifier and the second identifier to generate a joint identifier.

[0007] U.S. Patent Application Publication No. 2016 / 0078361A1 discloses obtaining an identification of a data source used to train a linear predictive model that generates predictions using respective parameters assigned to a plurality of features derived from observational records of a data source.

[0008] However, there is a continuing need for better continuous learning mechanisms to address the availability of new annotated and unannotated training data. Summary of the Invention

[0009] According to one aspect of the present invention, a computer-implemented method, a computer program product, and a computer system are provided. The following paragraphs discuss the computer-implemented method, specific features, but the features discussed may also be generally applicable to computer program products and computer systems. The method may include training a machine learning model using an initial set of training data samples, receiving new training data samples, and predicting a label for the new training data samples. The method may also include, in response to determining that a prediction quality value for the predicted label of the new training data sample is below a predefined quality value, adding the new training data sample to the initial set, thereby constructing an expanded training data set, and retraining the machine learning model using the expanded training data set. The method may also include repeating the receiving, predicting, adding, and retraining until a stopping condition is met.

[0010] The proposed computer-implemented method may provide numerous benefits, technical effects, contributions or improvements, or combinations thereof.

[0011] The method allows a method for training or retraining a machine learning system without a large amount of annotated training data samples. In some cases, annotations may not be required at all. The system can - during the training phase - continuously receive potentially unannotated real-world data (e.g., from a camera) to improve the overall predictive quality of the system and method. This can allow a significant reduction in manual manipulation of the training data - e.g., manual addition of labels or annotations or both. This can speed up the training process of neural networks or other machine learning systems or both.

[0012] Specifically, the feature of selecting new training data samples that exhibit low quality prediction values ​​can be used to improve the overall machine learning system prediction quality. This is achieved by attempting to recheck the predicted classes under artificially altered conditions, i.e., operations performed on the received new training data samples. In doing so, a combined prediction quality value (from different prediction cycles for the same new training data sample) can be generated to improve the recheck and originally low prediction quality.

[0013] The method can be advantageously used on documents containing text and graphics (e.g., pdf {Portable Document Format} documents or otherwise scanned documents in pixel format) to achieve better interpretability of graphics or figures within the document (e.g., plots, bar charts, xy-curves, etc.). The documents can include scientific documents, construction documents, maintenance documents, patent documents, etc.

[0014] The method can also be advantageously used to train machine learning models that use computer graphics simulators (e.g. for computer games) to achieve better object detection, object segmentation, or object instance segmentation results.

[0015] According to one advantageous embodiment of the method, a stopping condition can be met if a predefined training time value is reached, if a predefined number of training cycles have been performed, if a predefined number of new training data samples have been processed, or if a predefined summary prediction quality condition is met. Thus, the continuous or iterative learning process can be stopped, or at least interrupted, if too few prediction results with low prediction quality, i.e. below a predefined threshold, are generated. If no stopping condition is defined or if none of the above defined conditions are met, the continuous or iterative learning process can operate without restrictions. This allows the prediction accuracy to be continuously increased.

[0016] Further, a stopping condition can be met when it is determined that no additional new training data samples have a prediction quality below a predefined quality within a predefined time period, which can be, for example, an hour, a day, a predefined operating time period within a day, a week, or any suitable time interval.

[0017] According to a further advantageous embodiment, the method also includes, in response to determining that the predicted quality value for the predicted label of the new training data sample is below a predefined quality value, revalidating the predicted label of the new training data sample by applying a self-supervised learning method to the new training data sample. This self-supervised learning method may include operations on the received data sample (e.g., an image), such as rotating it (by a predefined angle or series of angles), flipping it (e.g., along a horizontal, vertical, diagonal or any other line), or puzzling and reshuffling it. After such operations, the prediction process may be repeated and, if the same class is predicted in the first instance, the annotation is revalidated. In general, new data samples will generate the same poor quality data (or better) over and over again under different conditions. Thereby, the objective is to improve reliability by building a combined predicted quality value (e.g., an average or weighted average) that increases over a series of operations on the sample data. It should also be noted that this step is not required if annotated sample data are available.

[0018] The self-supervised learning method may include at least one method selected from the group consisting of: rotating the new training data samples by a predefined rotation angle and predicting the rotation angle, thereby allowing the predicted rotation angle to be used as annotation, flipping the new training data samples by a predefined flip angle and predicting the flip angle (as well as other possible non-flipping angles), re-coloring the new training data samples and predicting the class of the resulting re-colored training data samples, or puzzle-solving the new training data samples and predicting the original positions of the tiles of the new training data samples, or combinations thereof. For re-coloring, one or more color channels may be blocked or otherwise altered (e.g., decreasing or increasing color intensity).

[0019] According to another optional embodiment, the method also includes: triggering an alarm event in response to determining that the predicted labels of the received new training data samples cannot be improved - i.e., the combined confidence level value is no longer good -, which can be used to trigger a supervised learning process, including manual annotation by a human annotator.

[0020] It should be noted that embodiments of the present invention are described with reference to different subject matter. In particular, some embodiments are described with reference to method type claims, while other embodiments are described with reference to device type claims. However, unless otherwise noted, a person skilled in the art will be able to gather from the above and following description any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matter, in particular between features of method type claims and between features of device type claims, which are considered to be disclosed in this disclosure. The above-defined and further aspects of the present invention will be apparent from and will be elucidated with reference to the examples of embodiments described below, without the present invention being limited thereto. Preferred embodiments of the present invention will be described with reference to the following figures, which are for illustrative purposes only: [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a flow chart illustrating a method for training a machine learning model according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a flow chart illustrating another method for training a machine learning model according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a block diagram of a machine learning model training system according to an embodiment of the present invention. [Figure 4] FIG. 4 is a block diagram of a computing system 400 suitable for executing program code associated with the methods of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] In the context of this description, the following conventions, terms or expressions, or combinations thereof, may be used:

[0023] The term "and / or" is inclusive or; for example, A, b, "and / or" C means that at least one of A, or b, or C is true and applicable.

[0024] The term "machine learning model" may refer to the process of building a model that can be used for machine learning. Machine learning is herein to be understood as the scientific study of algorithms and specific models that computer systems can use to perform specific tasks without the use of explicit instructions, i.e., without the use of procedural programming. Machine learning is a subcategory of artificial intelligence. Typically, models are trained on a set of known, labeled samples, e.g., to classify or otherwise classify unknown data samples into trained classifications. Examples of underlying techniques for ML learning models are artificial neural networks, random forest systems, or decision trees, or combinations thereof. The system can be used for classification tasks or regression tasks. In an embodiment of the present invention, the requirement for annotated samples for training and development of ML models is significantly reduced.

[0025] The term "initial set of training data samples" may refer to a number of data samples used for the first training of a machine learning model. This initial set may include annotated or unannotated samples. It should be noted that typically, the initial set of training data samples may be annotated. However, there are cases where a fully unannotated data set may be used as the initial set of training data samples. In any case, it is highly advantageous to have at least a minimized set of annotated data samples to define the underlying classes for prediction.

[0026] The term “new training data samples” may refer to data samples, e.g., images, that have not been previously used to train a machine learning model.

[0027] The term "prediction" can refer to the activity of a machine learning model. A predicted class - specifically a portion of a predefined number of predicted classes - can be generated for a new, previously unseen data sample based on an underlying trained machine learning model (e.g., a classification model). In the case of using a machine learning model for better interpreting text documents with graphics and images, the output prediction can also be expressed as a set of bounding boxes (e.g., at least one), a segmentation mask, a set of keypoints / landmarks, or other high-level semantic representation of the graphics and / or images embedded in the text, which are available in pixel information only (e.g., in a PDF).

[0028] The term "stopping criterion" may refer to a condition under which a method or process flow is terminated.

[0029] The term "sample data generator" may refer to any artificial data generator, such as a computer graphics simulator, a camera or microphone, or other data generating sensor.

[0030] The term "real-world sensor" can refer to any signal-generating device that provides data from a real-world scene. Such devices can be cameras for still images or for generating a continuous data stream (e.g., video cameras), microphones for acoustic or audio data, news feeds or other sources of written information (e.g., including graphics or other images), chemical analysis systems, etc.

[0031] The term "summary prediction quality metric" can refer to a numerical value that represents a prediction value for the label or annotation of the same data sample using variations of the data sample. Thus, an input sample can be used as an input to predict a class; the input sample can then be manipulated in a predefined manner(s) - for example, the image can be rotated or flipped - and the prediction can be repeated. This can generate different summary prediction qualities for the same predicted class. Several such predictions can be combined to construct a summary prediction quality metric or summary prediction quality value (also referred to as a "combined confidence level value") - for example, to construct an average prediction quality value or a weighted average prediction quality value.

[0032] The term "recolorization" can refer to the process of changing the color scheme for a given image. For one or more color channels of an image, certain colors of the underlying color model may be increased or decreased. Typically, one or more of the color channels may be set to zero intensity. This process is also known as false color rendering or false color representation.

[0033] Additionally, a machine learning model can be trained in a way to predict one image color channel for one or more given image channels for recolorization. For example, one can have an RBG image consisting of three color channels (red, blue, and green). Only a two-channel image can be used as input to the ML model, and the model can predict the remaining color channels, and the prediction error can be determined by comparing the predicted channel with the original channel. This technique can assist in training ML models using unlabeled data, since no annotation is required for this task. Thus, this technique can be considered similar to "puzzling".

[0034] The term "puzzling" refers to the process of cutting an image into pieces or tiles (regular cuts or irregular asymmetric cuts) and reshuffling the pieces. The machine learning system can then predict the class again. As an example, an image can be cut into 9 pieces and, for example, the top left snippet can be replaced with the bottom center snippet, the top left snippet can be replaced with the center left snippet, etc.

[0035] The term "deep learning system" can refer, for example, to a neural network with multiple layers that can be fully or only partially connected. To qualify as a deep learning system, the number of layers (including input and output layers) should be much more than three. A typical deep learning system can include 10 or 50 layers (or 200 or more). These layers can be convolutional layers and pooling layers that can be organized into certain groups.

[0036] The term “hyper-parameters” may refer to a set of parameters that describe the architecture of a machine learning model, such as the number of layers in an artificial neural network, the type of layers used, the length of the input data vector, the number of classes for prediction, the type of connections between layers (fully or partially connected), and the mode of operation of the machine learning system.

[0037] Another important parameter is the learning rate. The learning rate can mean a scale factor for the residual vector during backpropagation inside the neural network. This parameter is quite sensitive and requires fine tuning. Often there is a specific "schedule" that specifies how to change the learning rate during training.

[0038] In the following, a detailed description of the drawings is provided. All indications in the drawings are schematic. First, a flow chart illustrating a computer-implemented method for iterative training of a machine learning model is provided. Then, an embodiment of a machine learning model training system as well as further embodiments are described.

[0039] Figure 1 is a flow chart illustrating a method 100 for training a machine learning model according to an embodiment of the present invention. The method includes training 102 a machine learning model defined by a set of hyper-parameters, whereby an initial set of training data samples is used. Typically, a small number of annotated data samples to define a few classes are needed to start training. However, other cases are possible that operate only on unannotated data.

[0040] The method 100 includes receiving 104 new training data samples and predicting 106 labels or annotations for the new training data samples. These at least one new training data sample may be annotated or unannotated. However, embodiments may be more effective for unannotated new data samples, since other training procedures may be preferable for already annotated data samples. However, it should be noted that in some embodiments, the method 100 may work for annotated and unannotated new training data samples.

[0041] New training data samples can be collected during normal operation of the machine learning system without an initial attempt to use them as new training data, which can then be trained on the basis of the predictive behavior of the underlying machine learning system, for those received samples that give predictions with, for example, low prediction quality marks.

[0042] On the other hand, in some embodiments, the new training data received may be dedicated to an initial training procedure, and the received data samples will not be the result of use of a machine learning system.

[0043] The method 100 also includes, in response to determining that the predicted quality value for the label prediction of the new training data sample is below a predefined quality value, expanding the initial set of training data samples, thereby constructing an expanded training data set 108. The method 100 also includes repeating the training, receiving, predicting, and expanding on the expanded training set until a stopping criterion is met, thereby iteratively, i.e., continuously, training the machine learning model 110.

[0044] 2 is a flow chart illustrating another method 200 for training a machine learning model according to an embodiment of the present invention. As a first step, an initial number of training data samples is determined 202. Then, new or training data samples are received 204. These data samples are typically generated artificially or by a real-world data generator (not shown). Optionally (shown as a dashed box in the flow chart), an automatic annotation process takes place 206.

[0045] If it is determined 208 that a trained model is still not available—case “N”—all received samples are added to the training data set 210. If a trained model is already available—case “Y”—the data evaluator 306 evaluates or determines the quality of predicted annotations for newly received training data samples 212 and adds 214 new training data samples with low predicted quality values ​​to the training data set.

[0046] Then, it is determined whether a stopping criterion is met in 216. If this is the case - case "Y" - the training process is stopped ("Stop"). If the stopping criterion is not met - case "N" - it is determined 218 whether the expected number of new training data samples has been reached. If this is the case - case "Y" - the model is trained with the new training data samples and automatically (predicted) annotations 220. The process then loops back to the start or to the step of determining whether there are any other newly received training data samples with low prediction quality (i.e. below a predefined prediction quality value).

[0047] Several modules can support the process described herein: A data generator has the ability to generate new annotated data samples. In a general form, the data generator generates a continuous stream of data samples, which may be annotated or not. One implementation can be a computer graphics simulator. In this case, the stream of annotated data samples is generated by a computer graphics simulator. This simulator would use an open set of virtual environments or virtual worlds. The data generator has access to virtual instances and their properties within this simulated environment and can generate data samples in the form of color images, image masks, depth images, point clouds, regular maps, and other representations available to the simulator.

[0048] On the other hand, in some embodiments, the data generator can be a real-world sensor, such as a security camera, with the optional capability to generate annotations. Thus, the data stream generator can be thought of as a black box system that allows requesting new data samples with arbitrary data annotations.

[0049] The model trainer 302 can then be thought of as a machine learning system that takes as input a set of data samples, with or without annotations, and generates a model capable of predicting annotations for new data samples. For example, the model trainer can be represented as a deep machine learning system that takes data samples, (optionally) annotations, and a set of hyper-parameters that specify how to train the model.

[0050] The "Data Evaluator" 306 as a block or module is capable of taking data samples, ground truth data annotations (optionally) and the trained model as inputs to generate expected annotations as outputs. The Data Evaluator 306 is also capable of estimating (predicting or judging) the quality of the predicted annotation(s). This quality estimation can be judged by comparing the ground truth and the predicted annotations. Another way of estimating the quality of a prediction is the score of the prediction estimated by the model itself.

[0051] 3 is a block diagram of a system 300 for training a machine learning model according to an embodiment of the present invention. The system 300 includes training means, e.g., a model trainer 302, for training a machine learning model using an initial set of training data samples. It also includes receiving means, e.g., a receiver 304, for receiving new training data samples of a plurality of new training data samples.

[0052] The system 300 also includes a prediction means - for example in the form of a data evaluator 306 - for predicting a label for a new training data sample. It may also include an extension means - in particular an extension module 308 - that is activatable based on a positive result of the determination means for determining that a predicted quality value for the label prediction of the new training data sample is below a predefined (predicted) quality value, for extending the initial set and thereby building an extended training data set.

[0053] The system 300 also includes a triggering means, specifically a trigger module 310, adapted to trigger the triggering means, the receiving means, the predicting means, and the determining means using the expanded training data set until a stopping criterion is met, thereby iteratively providing continuous training of the machine learning model.

[0054] The embodiments of the present invention may be implemented in conjunction with virtually any type of computer, regardless of the platform suitable for storing or executing program code or a combination thereof. Figure 4 is a block diagram of a computing system 400 suitable for executing program code associated with the proposed method.

[0055] Computing system 400 is only one example of a suitable computer system, and whether or not computer system 400 can implement and / or perform any of the above-described functions is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention described herein. There are components in computing system 400 that operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments or configurations or combinations thereof that may be suitable for use with computing system 400 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor systems, microprocessor-based systems, set-top boxes, PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices. Computing system 400 may be described in the general context of computer system-executable instructions, such as program modules, executed by computing system 400. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computing system 400 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in storage media including memory storage devices of both local and remote computer systems.

[0056] As shown in Figure 4, computing system 400 is shown in the form of a general purpose computing device. Components of computing system 400 include, but are not limited to, one or more processors or processing units 402, a system memory 404, and a bus 406 that couples various system components, including system memory 404, to processing unit 402. Bus 406 may represent any of several types of one or more bus structures, including a memory bus or memory controller, a peripheral bus, a graphics accelerator port, and a processor or local bus using any of a variety of bus architectures. By way of example, and without limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro-Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standard Association (VESA) local bus, and a Peripheral Interconnect (PCI) bus. Computing system 400 typically includes a variety of computer readable media. Such media can be any available media that can be accessed by computing system 400 and includes both volatile and nonvolatile media, removable and non-removable media.

[0057] The system memory 404 may include computer readable media such as volatile memory, such as random access memory (RAM) 408, or cache memory 410, or both. The computing system 400 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 412 may provide for reading from and writing to a non-removable, non-volatile magnetic medium (not shown and typically referred to as a "hard drive"). Although not shown, a removable, non-volatile magnetic disk drive (e.g., a "floppy disk") and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such an example, each may be coupled to the bus 406 by one or more media interfaces. As further illustrated and described below, the memory 404 may include at least one program product having a set (at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0058] The programs / utilities include a set (at least one) of program modules 416, which by way of example and not limitation, may include an operating system, as well as one or more application programs, other program modules, and program data stored in memory 404. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a network environment. The program modules 416 generally perform the functions and / or methodologies of embodiments of the present invention as described herein.

[0059] The computing system 400 may also communicate with one or more external devices 418, such as a keyboard, pointing device, display 420, which may allow a user to interact with the computing system 400 or any device (e.g., network card, modem, etc.) that allows the computing system 400 to communicate with one or more computing devices, or both. Such communication may occur through an input / output (I / O) interface 414. Additionally, the computing system 400 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (Internet), or combinations thereof, through a network adapter 422. As shown, the network adapter 422 communicates with other components of the computing system 400 through a bus 406. It should be understood that other hardware and / or software components, not shown, could be used in combination with the computing system 400. Examples include, but are not limited to, microcode, device drivers, redundant processing units, and external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0060] Additionally, the machine learning model training system 300 can be attached to a bus system 406.

[0061] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications or variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the present embodiments, practical applications, or technical improvements over the art found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0062] The present invention may be embodied as a system, method, or computer program product, or combination thereof, including a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform features of the present disclosure.

[0063] The medium can be electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems for transmission media. Examples of computer readable medium can include semiconductor or solid state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read only memory (ROM), rigid magnetic disk, and optical disk. Current examples of optical disk include compact disk read only memory (CD-ROM), compact disk read / write (CD-R / W), DVD and Bul-Ray®-Disk.

[0064] A computer readable medium may be a tangible device capable of holding and storing a plurality of instructions for use by an instruction execution device. A computer readable medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electro-magnetic storage device, a semiconductor storage device, or any suitable combination thereof. Non-limiting examples of computer readable medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a punch card, or a mechanically encoded device having protruding structures within a groove having instructions recorded therein, and any suitable combination thereof. As used herein, a computer-readable recording medium is not to be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves such as wave guides or other communications media (e.g., light pulses passing through fiber optic cable), or electrical signals communicated through wires.

[0065] The computer programs described herein can be downloaded from a computer readable recording medium into each computing / processing device, or can be downloaded to an external computer or external recording device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, and combinations thereof. The network can include copper cables, fiber optics, wireless networks, routers, firewalls, switches, gateway computers, and edge servers, and combinations thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions into a computer readable recording medium in each computing / processing device for storage.

[0066] Computer readable program instructions for performing operations of the present invention may be either source code or object code written in any combination of programming languages, including assembler instructions, Instruction Set Architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or one or more procedural programming languages, such as Smalltalk®, object-oriented programming languages ​​such as C++, the “C” programming language, or similar programming languages. The computer readable program instructions may execute entirely on the user computer, partially on the user computer as a stand-alone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user computer through any type of network, including a local area network (LAN), a wide area network (WAN), or the connection may be to an external computer (e.g., through an Internet service provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute the computer readable program instructions and use state information of the computer readable program instructions to personalize the electronic circuitry to perform features of the invention.

[0067] Aspects of the invention described herein have been described with reference to flowchart instructions and / or block diagrams of methods, apparatus (systems), and computer readable storage media and computer program products according to embodiments of the invention. It will be understood that any combination of flowchart illustrations and / or block diagrams and / or blocks in flowchart illustrations and / or block diagrams can be implemented by computer readable program instructions.

[0068] These computer readable program instructions can be provided to a general purpose computer, a special purpose computer, or other processor or other programmable data processing device to produce a machine, which when executed by the processor of the computer or other programmable data processing device produces means for implementing the functions / operations specified in the block or blocks of the flowcharts and block diagrams, or combinations thereof. These computer readable program instructions that direct a computer, programmable data processing device, and other device, or combinations thereof, to function in a particular manner can also be stored on a computer readable recording medium, and the computer readable recording medium having instructions stored therein constitutes an article of manufacture containing instructions that implement the functional / operational features specified in the block or blocks of the flowcharts and block diagrams, or combinations thereof.

[0069] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device and cause a computer implemented process to perform a series of operational steps on the computer, other programmable apparatus, or other device to implement the functions / acts specified in the blocks of the flowcharts and block diagrams or in a combination thereof.

[0070] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this respect, the flowcharts or block diagrams may represent modules, segments, or portions of instructions, which include one or more executable instructions for implementing a particular logical function(s) (or functions). In some alternative implementations, the functions described in the blocks may be performed other than as illustrated. For example, two blocks shown in succession may actually be executed simultaneously, depending on the functionality involved, or the blocks may sometimes be executed in the reverse order. Also, the illustration of the blocks in the block diagrams and / or the illustration of the flowcharts or combinations thereof may be implemented by a system based on special purpose hardware that performs a particular function or operation or executes specific purpose hardware and computer instructions.

[0071] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, when the terms "comprise", "comprising", or both are used herein, they should be understood to specify the presence of the stated features, integers, steps, operations, elements, or components, or combinations thereof, but not to exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups, or combinations thereof.

[0072] Equivalents of all means or steps of corresponding structures, materials, acts, means-plus-function elements in the following claims are intended to include any structure, material, or acts for performing a function in combination with other claimed elements as specifically claimed. The description of various embodiments of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present disclosure. The embodiments have been selected and described in order to best explain the principles and practical applications of the embodiments and to enable others skilled in the art to understand various embodiments with various modifications as may be suitable for the particular use envisaged.

[0073] Some embodiments of the invention provide a computer-implemented method for iterative training of a machine learning model, the method including: training a machine learning model using an initial set of training data samples; receiving new training data samples; predicting labels for the new training data samples; in response to determining that a predicted quality value for a previously predicted label of the new training data sample is below a predefined quality value, adding the new training data sample to the initial set, thereby constructing an expanded training data set; and repeating the training, receiving, predicting, and expanding with the expanded training data set until a stopping criterion is met, thereby iteratively training the machine learning model.

[0074] In some embodiments, receiving includes receiving new training data samples from a sample data generator or from a real-world sensor.

[0075] In some embodiments, the stopping criteria is met when a predefined training time value is reached, or when a predefined number of training cycles have been performed, or when a predefined number of new training data samples have been processed, or when a predefined summary prediction quality condition is met, or when it is determined that no additional new training data samples have a prediction quality below a predefined quality within a predefined time.

[0076] In some embodiments, the method further includes: in response to determining that the predicted quality value of the unannotated new training data sample is below a predefined quality value, revalidating associated predicted labels of the new training data sample by applying a self-supervised learning method to the new training data sample to increase a combined confidence level value for labels whose predicted quality value is below the predefined quality value.

[0077] In some embodiments, the self-supervised learning method includes at least one selected from the group consisting of rotating the new training data samples by a predefined rotation angle and predicting the rotation angle, flipping the new training data samples by a predefined flip angle and predicting the flip angle, re-coloring the new training data samples and predicting a class of the resulting re-colored training data samples, puzzle-fusing the new training data samples and predicting original positions of tiles of the new training data samples.

[0078] In some embodiments, the repeating step is triggered only if the expanded training data set is increased by a predefined number or a predefined percentage of new training data samples.

[0079] In some embodiments, the machine learning model uses a deep machine learning system.

[0080] In some embodiments, the method further includes triggering an alarm event in response to determining that the predictive quality of received new training data samples cannot be improved.

[0081] In some embodiments, when real-world sensors are used that generate related data samples having two substantially equivalent distances from each other and different angles from the real-world object, the method further includes using the related data samples to adapt the machine learning model in a manner such that identical prediction results and identical prediction quality values ​​are generated for both related data samples.

[0082] In some embodiments, the initial set of training data samples includes annotated samples.

[0083] Some embodiments of the present invention provide a system for iterative training of a machine learning model, the system including: training means for training a machine learning model using an initial set of training data samples; receiving means for receiving new training data samples; predicting means for predicting labels for the new training data samples; expanding means activatable in response to a positive result of the determining means for determining that a predicted quality value for the predicted label of the new training data sample is below a predefined quality value, the expanding means expanding the initial set, thereby constructing an expanded training data set; and triggering means adapted to trigger the training means, the receiving means, the predicting means, and the determining means using the expanded training data set until a stopping criterion is met, thereby iteratively providing for successive training of the machine learning model.

[0084] In some embodiments, the system also includes a sample data generator or real-world sensor for generating new training data samples.

[0085] In some embodiments, the stopping criteria is met when a predefined training time value is reached, or when a predefined number of training cycles have been performed, or when a predefined number of new training data samples have been processed, or when a predefined summary prediction quality condition is met, or when it is determined that no additional new training data samples have a prediction quality below a predefined quality within a predefined time.

[0086] In some embodiments, the system includes a revalidation means for, in response to determining that the predicted quality value of a new unannotated training data sample is below a predefined quality value, revalidating associated predicted labels by applying a self-supervised learning method to the new training data sample to increase a combined confidence level value for labels whose predicted quality value is below the predefined quality value.

[0087] In some embodiments, the self-supervised learning method includes at least one capability selected from the group consisting of: rotating new training data samples by a predefined rotation angle and predicting the rotation angle, flipping new training data samples by a predefined flip angle and predicting the flip angle, re-coloring the new training data samples and predicting a class of the resulting re-colored training data samples, puzzlerizing the new training data samples and predicting original positions of tiles of the new training data samples.

[0088] In some embodiments the triggering means is adapted to be activated only if the expanded training data set is increased by a predefined number or a predefined percentage of new training data samples.

[0089] In some embodiments, the machine learning model uses a deep machine learning system.

[0090] In some embodiments, the system also includes an alarm means for triggering an alarm event if it is determined that the predictive quality of the received data samples cannot be improved.

[0091] In some embodiments, when real-world sensors are used that generate related data samples having two substantially equivalent distances from each other and different angles from the real-world object, the system further uses the related data samples to adapt the machine learning model in a manner such that the same prediction result and the same prediction quality value are generated for both related data samples.

[0092] Some embodiments of the invention provide a computer program product for iterative training of a machine learning model, the computer program product including a computer-readable storage medium having program instructions embodied therein, the stored program instructions being executable by one or more computing systems or controllers to cause the one or more computing systems to: train a machine learning model using an initial set of training data samples, receive new training data samples, predict labels for the new training data samples, in response to determining that a prediction quality value for the predicted labels of the new training data samples is below a predefined quality value, expand the initial set, thereby constructing an expanded training data set, and iteratively train the machine learning model by repeating the training, receiving, predicting, and expanding with the expanded training data set until a stopping criterion is met.

Claims

1. 1. A computer-implemented method performed by a computer, comprising: Training a machine learning model using an initial set of training data samples; receiving new training data samples; predicting labels for the new training data samples; and in response to determining that a predicted quality value for the predicted label of the new training data sample is below a predefined quality value; adding the new training data samples to the initial set, thereby constructing an extended training data set; revalidating the predicted labels of the new training data samples by applying a self-supervised learning method to the new training data samples; retraining the machine learning model using the expanded training data set; to carry out 3. A computer-implemented method.

2. moreover, repeating the receiving, predicting, adding, and retraining until a stopping condition is met. The computer-implemented method of claim 1 , comprising:

3. The stopping condition is met when a predefined training time value is reached. The computer-implemented method of claim 2 .

4. The stopping condition is met when a predefined number of training cycles have been performed. The computer-implemented method of claim 2 .

5. The stopping condition is met when a predefined number of new training data samples have been processed. The computer-implemented method of claim 2 .

6. The stopping condition is met when it is determined that no additional new training data samples have a predicted quality value below the predefined quality value within a predefined time. The computer-implemented method of claim 2 .

7. The self-supervised learning method includes: rotating the new training data samples by a predefined rotation angle and predicting the rotation angle; flipping the new training data samples at a predefined flip angle and predicting the flip angle; recoloring the new training data samples and predicting the class of the resulting recolored training data samples; Puzzling the new training data samples and predicting original positions of tiles of the new training data samples; The method includes at least one method selected from the group consisting of 10. The computer-implemented method of claim 1.

8. moreover, triggering an alarm event in response to determining that the predicted labels of the new training data samples cannot be improved.

10. The computer-implemented method of claim 1.

9. A computer-implemented method executed by a computer, comprising: Training a machine learning model using an initial set of training data samples; receiving new training data samples; predicting labels for the new training data samples; in response to determining that a predicted quality value for the predicted label of the new training data sample is below a predefined quality value, adding the new training data sample to the initial set, thereby constructing an extended training data set; retraining the machine learning model using the expanded training data set; and repeating the receiving, predicting, adding, and retraining until a stopping condition is met. wherein the stopping condition is met if a predefined summary prediction quality condition is met.

10. A computer executable program for causing a computer system to carry out the computer implemented method according to claims 1 to 9.

11. 1. A computer system, comprising: Training a machine learning model using an initial set of training data samples; receiving new training data samples; predicting labels for the new training data samples; and in response to determining that a predicted quality value for the predicted label of the new training data sample is below a predefined quality value; adding the new training data samples to the initial set, thereby constructing an extended training data set; revalidating the predicted labels of the new training data samples by applying a self-supervised learning method to the new training data samples; retraining the machine learning model using the expanded training data set; and to carry out A computer system that executes the above.

12. moreover, repeating the receiving, predicting, adding, and retraining until a stopping condition is met.

12. The computer system of claim 11, comprising:

13. The self-supervised learning method includes: rotating the new training data samples by a predefined rotation angle and predicting the rotation angle; flipping the new training data samples at a predefined flip angle and predicting the flip angle; recoloring the new training data samples and predicting the class of the resulting recolored training data samples; Puzzling the new training data samples and predicting original positions of tiles of the new training data samples; At least one selected from the group consisting of 12. The computer system of claim 11.

14. moreover, triggering an alarm event in response to determining that the predicted labels of the new training data samples cannot be improved.

12. The computer system of claim 11.

15. A computer system, comprising: Training a machine learning model using an initial set of training data samples; receiving new training data samples; predicting labels for the new training data samples; in response to determining that a predicted quality value for the predicted label of the new training data sample is below a predefined quality value, adding the new training data sample to the initial set, thereby constructing an extended training data set; retraining the machine learning model using the expanded training data set; and repeating the receiving, predicting, adding, and retraining until a stopping condition is met. and wherein the stopping condition is met if a predefined summary prediction quality condition is met.

Images