Iterative training of computer models for machine learning

Abstract

The present disclosure relates to a computer receiving a current training dataset. A first fraction of the training dataset includes synthetic training data and a second fraction of the training dataset includes real-life training data. The real-life training data is user-defined data and the synthetic training data is system-defined data. A machine learning based engine is trained, which may be performed iteratively by using the current training dataset. At each iteration or a subset of iterations, the training dataset is updated by adding real-life training data, thereby increasing the second fraction and decreasing the first fraction of the synthetic training data in the updated training dataset.

Description

Technical Field

[0001] The present invention relates to the field of digital computer systems, and more specifically, to a method for training a machine learning-based engine.

Background Art

[0002] Clerical records are records for which a given matching process cannot determine whether the records are duplicate records of each other, and therefore whether the records should be merged, or whether one or more records should be considered non-matching and therefore kept separate from each other. These clerical records may require user intervention to more closely examine the values of the data records. Despite tremendous efforts to automate and improve the record matching process, the number of such clerical records has been continuously increasing (for example, it can reach millions of clerical records). As a result, a large portion of clerical records remain unprocessed for very long periods of time, during which inconsistent data may be used in system configuration.

Summary of the Invention

[0003] Various embodiments provide a method, a computer system and a computer program product for training a machine learning-based engine described by the subject matter of the independent claims. Advantageous embodiments are described in the dependent claims. Embodiments of the present invention can be freely combined with each other as long as they are not mutually exclusive.

[0004] In one aspect of the present invention, a computer implementation method for training a machine learning-based engine includes receiving a current training dataset. A first fraction of the current training dataset includes synthetic training data, and the remaining second fraction of the training dataset includes real-life training data. The real-life training data is user-defined data, and the synthetic training data is system-defined data. The method includes iteratively training the machine learning-based engine using the current training dataset, the training dataset being updated in each iteration or in each iteration of a subset of iterations, the update being done by adding real-life training data, thereby increasing the second fraction of real-life training data and decreasing the first fraction of synthetic training data in the updated training dataset.

[0005] In related embodiments, a machine learning-based engine is trained to determine whether two data records are duplicates of each other, and this method further includes comparing records in a database using the trained machine learning-based engine.

[0006] In a related embodiment, a machine learning-based engine is used to compare database records when the predictive accuracy of the currently trained machine learning-based engine has not increased compared to the predictive accuracy of the trained machine learning-based engine in the last iteration.

[0007] In a related embodiment, a machine learning-based engine is used to compare records in a database when the first fraction is zero.

[0008] In related embodiments, the method further includes reducing the synthetic training data in each iteration or in each iteration of a subset of iterations, thereby further reducing the first fraction of synthetic training data in the updated training dataset.

[0009] In related embodiments, the reduction of synthetic training data is either absolute or relative.

[0010] In related embodiments, repeated reduction of synthetic training data includes gradually reducing the amount of synthetic training data.

[0011] In related embodiments, the amount of synthetic training data is reduced to the point where training is performed exclusively with respect to real-life training data.

[0012] In related embodiments, the level of reduction of the synthetic training data used for training is dynamically adjusted based on at least one predictive quality metric.

[0013] In the relevant embodiment, the second fraction is zero with respect to the first run of training the machine learning-based engine.

[0014] In a related embodiment, a machine learning-based engine is a machine learning-based matching engine for finding duplicates in a database, and the training dataset contains labeled records. Records in the synthetic training data are labeled by a rule-based matching engine based on comparisons of records by the rule-based matching engine.

[0015] In related embodiments, rule-based matching engines operate using deterministic matching, probabilistic matching, or both.

[0016] In related embodiments, labeling synthetic training records includes using the default configuration of a rule-based matching engine.

[0017] In another aspect of the present invention, a computer program product includes a computer-readable storage medium on which program instructions are implemented. These program instructions are executable by the computer to cause the computer to perform a function, the function including receiving a current training dataset, wherein a first fraction of the current training dataset includes synthetic training data, and the remaining second fraction of the training dataset includes real-life training data, the real-life training data being user-defined data, and the synthetic training data being system-defined data; and the function including iteratively training a machine learning-based engine using the current training dataset, wherein in each iteration or in each iteration of a subset of iterations, the training dataset is updated, the update being done by adding real-life training data, thereby increasing the second fraction of real-life training data and decreasing the first fraction of synthetic training data in the updated training dataset.

[0018] In another aspect of the present invention, a system for training a machine learning-based engine includes a computer system, the computer system includes a computer processor, a computer-readable storage medium, and program instructions stored on the computer-readable storage medium, the program instructions being executable by the processor to cause the computer system to perform the following functions: receiving a current training dataset, where a first fraction of the current training dataset includes synthetic training data, and the remaining second fraction of the training dataset includes real-life training data, where the real-life training data is user-defined data and the synthetic training data is system-defined data; and repeatedly training a machine learning-based engine using the current training dataset, where in each iteration or in each iteration of a subset of iterations the training dataset is updated, the update is performed by adding real-life training data, thereby increasing the second fraction of the updated training dataset and decreasing the first fraction of the synthetic training data.

[0019] Hereinafter, embodiments of the present invention will be described in more detail, merely as examples, with reference to the drawings, including the following figures. [Brief explanation of the drawing]

[0020] [Figure 1] This figure shows a computer system as an example of this subject. [Figure 2A] This is a flowchart illustrating a method for training a machine learning-based engine, using an example from this subject. [Figure 2B] This plot illustrates the reduction process of synthetic data, using an example from this subject. [Figure 3] This is a flowchart illustrating a method for matching data records in a dataset, using an example from this subject. [Figure 4]It is a flow diagram of a method for training a machine learning-based engine according to an example of the present subject matter. [Figure 5] It is a flow diagram of a method for inferring by a machine learning-based engine according to an example of the present subject matter. [Figure 6] It is a diagram showing a computerized system suitable for carrying out one or more method steps included in the present subject matter. DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

[0021] Descriptions of various embodiments of the present invention are presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best describe the principles of the embodiments, practical applications, or technical improvements over commercially available technologies, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0022] During data generation and processing by computer systems such as Master Data Management (MDM) systems, data may be stored and analyzed by the computer system's users. While the amount of stored data can increase rapidly over time, the results of analysis may increase very slowly. However, customers may need to use the stored data as quickly as possible to improve the computer system's functionality. In particular, customers may need to label stored data based on analysis results, for example, to use labeled data as training data for training machine learning models. This could, for example, provide customers with a systematic tool that can identify duplicate data within the computer system based on data already stored by the computer system. This subject can solve this problem by using labeled synthetic data in combination with accumulated current labeled real-world data. For example, to bootstrap new customers of a product that lacks available real-world training data, synthetic training data can be generated by labeling data solely using a computer system. This synthetic training data is based on system-based analysis, not on user analysis results. This subject can enable the incremental training of models based on the progress of processing in a computer system. The training dataset used is regularly updated with real-world data, while intermediate training results can still be used with high reliability to perform inference. For example, a client can collect more and more real-world training data over time during the stewardship process. This can allow for a gradual reduction in the amount of synthetic data during the training process. Real-world data is sometimes called natural data. Real-world training data can be defined, for example, by the user, such as a steward, and may be labeled.A user can analyze data and label the data by themselves based on their own analysis. In contrast, synthetic training data is defined by the system and not defined by the user, and thus is system-defined data (e.g., system-labeled data). For example, synthetic training data may include data labeled by a computer system based on analysis performed by the computer system. Such labeling may be performed automatically, for example.

[0023] The present subject matter can enable active learning in which training data generation is controlled within an iterative training process. As used herein, the term "active learning" is used to refer to the active generation of training data according to the present method for training a machine learning-based engine. This may enable iterative addition of valuable records, informative records, or valuable and informative records to a training set. A training dataset may include multiple entries, each entry including a labeled data point. A data point represents multiple data records, and a label indicates whether the multiple records of the data point are duplicate records. The label may have, for example, a value "same" indicating that the records of the data point are duplicate records, or a value "different" indicating that the records of the data point are not duplicate records.

[0024] A machine learning-based engine can be trained to output a classification result for a given input data point. The classification result may include an indication of one or more classes related to a probability that the input data point belongs to each of the one or more classes. For example, a higher probability for the class "same" corresponds to a higher level of matching between the records of the data point, and vice versa. Also, a higher probability for the class "different" corresponds to a lower level of matching between the records of the data point, and vice versa.

[0025] Updating the training dataset can be performed in each iteration of at least a portion of the training iterations of the machine learning-based engine. For example, the training dataset can be updated in each iteration of the training of the machine learning-based engine. In one example, the training dataset can be updated in each iteration of all the training iterations of the machine learning-based engine. Alternatively, the training dataset can be updated in each iteration of a subset of the training iterations of the machine learning-based engine. A subset of iterations can be a selected set of training iterations of the machine learning-based engine. For example, the training dataset can be updated in each selected iteration of the training of the machine learning-based engine, and the training dataset can be updated repeatedly after training the machine learning-based engine N times, for example, if N=10, the updated new training dataset can be used for the 11th training, the 21st training, and so on.

[0026] Machine learning-based engines can include machine learning (ML) models. The term “machine learning” (ML) refers to computer algorithms used to extract useful information from training datasets by building probabilistic models (called machine learning models or “predictive models”) in an automated manner. To make predictions or decisions without being explicitly programmed to perform that task, machine learning algorithms build mathematical models based on sample data known as “training data.” Machine learning can be performed using supervised or unsupervised learning, learning algorithms such as clustering, classification, linear regression, reinforcement algorithms, and self-learning. Machine learning may be based on a variety of techniques such as clustering, classification, linear regression, support vector machines, and neural networks. For example, a “model” or “predictive model” may be a data structure or program such as a neural network, support vector machine, decision tree, or Bayesian network. This model is fitted to predict unmeasured values ​​(e.g., which tags correspond to a given token) from other known values, or to predict or select an action to maximize future rewards, or both.

[0027] According to one embodiment, a machine learning-based engine is trained to determine whether two database records are duplicates of each other. This method further includes comparing the database records using the trained machine learning-based engine. For example, inference can be performed using the trained machine learning-based engine resulting from each iteration.

[0028] A data record, or record, is a collection of related data items, such as a specific user's name, date of birth, and hierarchy. A record represents an entity, which refers to a user, object, or concept about which information is stored in the record. The terms "data record" and "record" are interchangeable. Data records can be stored in a graph database as relational entities, and each record can be assigned to a node or vertex of the graph, having characteristics that are attribute values ​​such as name and date of birth. In another example, a data record can be a record in a relational database.

[0029] This subject can be advantageous because it allows for accurate classification of data points, while simultaneously saving processing resources by utilizing active learning techniques that use improved data for training.

[0030] According to one embodiment, a machine learning-based engine is used to compare records in a database if the predictive accuracy of the currently trained machine learning-based engine has not increased compared to the predictive accuracy of the trained machine learning-based engine in the last iteration. This can ensure that the most accurate classification result is obtained based on the currently available data. For example, since the trained machine learning-based engine may have learned to classify data points well, this can enable it to predict the class of data points with a very high probability. This accuracy can be determined using a test set, which may be different from the training set. The test set may include data points labeled by a user, e.g., real-world data. The trained machine learning-based engine can classify (or predict the class of) the data points in the test set, and the fraction of data points that are correctly classified can be called the predictive accuracy.

[0031] According to one embodiment, a machine learning-based engine is used to compare database records when a first fraction is zero. That is, inference can be performed using only an engine trained on real-world data. This can be particularly advantageous when the accuracy required for prediction is very high.

[0032] According to one embodiment, the method further includes reducing the synthetic training data in each iteration, thereby further reducing the first fraction of the synthetic training data.

[0033] For example, the current training dataset is

number

number

number

number

number

number

[0034] According to one embodiment, the reduction of synthetic training data is absolute or relative. For example, the number of data points in the synthetic training data.

number

number

number

[0035] According to one embodiment, the reduction of synthetic training data includes gradually reducing the amount of synthetic data. For example, the reduction fraction x% may gradually increase, for example, x%=10% in the first reduction, x%=15% in the second reduction, and so on.

[0036] According to one embodiment, the level of reduction of the synthetic data used for training is dynamically adjusted based on at least one prediction quality metric. This metric can be, for example, the accuracy of the predictions of the trained engine. If the accuracy in the current iteration changes slightly compared to the last iteration, the reduction for the next iteration can be made smaller than the last reduction. For example, the first reduction could be x%=10% based on this metric, the second reduction could be x%=5% based on this metric, and so on.

[0037] According to one embodiment, the amount of synthetic data is reduced to the point where the training process is performed exclusively on real-world data.

[0038] In one embodiment, the second fraction is zero during the first run of training. A second fraction of zero means that the data has not yet been labeled by the user. This allows the training to be used from the very beginning, for example, immediately after the computer system is delivered to the customer.

[0039] According to one embodiment, the machine learning-based engine is a machine learning-based matching engine for finding duplicates in a database, the training dataset contains labeled records, and the records in the synthetic training data are labeled by a rule-based matching engine based on the comparison of records by the rule-based matching engine.

[0040] According to one embodiment, the rule-based matching engine operates using deterministic matching, probabilistic matching, or both.

[0041] In one embodiment, the generation of synthetic training data (e.g., labeling records) by a rule-based matching engine is performed using the default configuration of the rule-based matching engine. In another example, synthetic training data can be generated for different configurations of the rule-based matching engine. This can make it possible to obtain a large training set that can be used with high reliability to train a machine learning-based engine.

[0042] Figure 1 shows an exemplary computer system 100. The computer system 100 may be configured to perform, for example, master data management or data warehousing or both, and for example, the computer system 100 may enable a data deduplication system. The computer system 100 includes a data integration system 101 and one or more client systems or data sources 105. The client system 105 may include a computer system (such as the one described with reference to Figure 6). The data integration system 101 can control access (such as read and write access) to the database system 103.

[0043] The client system 105 can communicate with the data integration system 101 via a network connection, including, for example, a wireless local area network (WLAN) connection, a wide area network (WAN) connection, a local area network (LAN) connection, or a combination thereof.

[0044] The data integration system 101 can process records received from the client system 105 and store those data records in the database system 103. The data records stored in the database repository 103 may have a predetermined data structure, such as a data table with numerous columns and rows. This predetermined data structure may include a set of attributes (for example, each attribute representing a column in the data table). In another example, data records can be stored in a graph database as relational entities. The predetermined data structure may include a graph structure where each record is assigned to a node in the graph.

[0045] For example, a client system 105 may be configured to provide or create data records. Each client system 105 may be configured to send the created data records to the data integration system 101 for storage on the database system 103. For example, a client system 105 may be configured to provide records in XML, JSON, or other format that allows attributes to be associated with corresponding attribute values, where in XML, at least a portion of the attributes are associated with their respective values.

[0046] In one example, the data integration system 101 may import data records from the client system 105 using one or more extract-transform-load (ETL) batch processes, via hypertext transport protocol ("HTTP") communication, or via other types of data exchange. The data integration system 101 or the client system 105, or both, may be associated with, for example, a personal computer (PC), a server, a mobile device, or a combination thereof.

[0047] Each data record received by the data integration system 101 from the client system 105 may or may not have all the values ​​of a set of attributes. For example, a data record may have values ​​for a subset of attributes in the set of attributes, and not for the remaining attributes. In one example, after being stored in the repository 103, the remaining attributes that do not have values ​​may be left empty. In other words, the records provided by the client system 105 have different completeness levels. This completeness is the ratio of the number of attributes in a data record that contain data values ​​to the total number of attributes in the set of attributes.

[0048] The data integration system 101 may be configured to process records in the database system 103 using one or more algorithms. For example, the data integration system 101 may include a probabilistic matching engine 107 configured to compare or match records and determine whether the compared records are duplicates of each other. The results of this matching can be used to generate synthetic training data 117 that can be stored in the database system 103. The synthetic data 117 includes labeled data points. A data point refers to two or more records. The labels of data points can be "same" or "different," meaning that the records of the data point represent the same entity or different entities, respectively. For example, a data point may include records of the same person X or records belonging to different people. In other words, a labeled data point is a data point associated with a class (e.g., "same" or "different") derived from the classification of that data point. The synthetic training data 117 is called by this name because it is generated by the probabilistic matching engine 107, not by a user. Therefore, the synthetic training data 117 is called system-defined data. The synthetic training data 117 can be automatically created, for example, by a probabilistic matching engine 107.

[0049] The data integration system 101 may further include an MDM engine 109. The MDM engine 109 is configured to communicate with the user of the data integration system 101 via a user interface 110. In particular, the MDM engine 109 is configured to prompt the user to provide instructions on whether two records are duplicate records or not. This prompt can be performed in different contexts. In a first example, the user may be prompted to compare records as a clerical task. In a second example, the user may be prompted to compare records as part of an analysis. In a third example, the user may be prompted to double-check previous matching results. In all cases, the user can provide a class of one or more data points, for example, via the UI 110. As a result, another training data called real-life training data or natural training data 119 can be provided. The real-life training data 119 includes data points labeled by the user and is therefore called user-defined training data. The real-life training data 119 is stored in the database system 103. Figure 1 shows how real-life training data 119 are acquired in different contexts using the first, second, and third examples.

[0050] The data integration system 101 may further include a machine learning service 111, which includes an ML predictor 112 and an ML model builder 113. The ML model builder 113 is configured to train an ML model using training data 117 or 119 or both, in order to provide a trained model configured to predict whether the records of input data points are the same or different. The ML model can be stored, for example, in an ML model storage 114. The ML predictor 112 is configured to perform record matching using the stored ML model, for example, upon request from the MDM engine 109.

[0051] Figure 2A is a flowchart of a method for training a machine learning-based engine, as an example of the subject of this paper. For illustrative purposes, this method may be implemented in a manner in which the trained machine learning-based engine classifies records in the computer system 100 shown in Figure 1, but is not limited to this embodiment. This is because the machine learning-based engine can also be used to perform other classifications. This method can be performed, for example, by a data integration system 101.

[0052] In step 201, the ML model builder 113 uses, for example, the current training dataset.

number

number

number

number

number

number

number

number

number

number

number

number

[0053] Step 203: Current training dataset

number

[0054] After training the machine learning-based engine, or while training the machine learning-based engine, in step 205, additional real-life training data can be added to the existing real-life training data 119. For example, suppose a user performed an additional cleric task that gives new user-labeled data points. As a result, the increased number of new data points

number

number

number

number

[0055] For example, updated training sets

number

number

number

number

number

number

number

[0056] In another example, updated training sets

number

number

number

number

number

number

[0057] In step 203 of the next iteration / repetition of training, the updated training dataset from step 205 is used.

number

number

[0058] The method in Figure 2A can be performed automatically, for example. Figure 2B shows an example of the gradual evolution of the first fraction of synthetic data (indicated by the area of ​​points) and the second fraction of real-world data (indicated by the area of ​​lines) derived from the method in Figure 2A. This figure shows, for example, day 0 of this plot, in the case where a customer starts working with the MDM system and there is no labeled training data generated through data stewardship yet. At this point, synthetic training data can be created using a rule-based PME engine. This data is used to train a machine learning-based engine. During this stewardship process, more and more real-world data is acquired over time. This training process can always be performed on the entire set of real-world data. Initially, there is not enough data to train a high-quality model. A subset of synthetic data can be combined with real-world data for training. Over time, the amount of synthetic data is reduced to the point where the training process is performed exclusively on real-world data.

[0059] Figure 3 is a flowchart of a method for matching data records in a dataset, as an example of the subject matter. For illustrative purposes, this method may be implemented in the computer system 100 shown in Figure 1, but is not limited to this embodiment. This method can be performed, for example, by a data integration system 101.

[0060] Steps 301 to 305 correspond to steps 201 to 205 in Figure 2A, respectively. The method in Figure 3 further includes a query step 307 for determining whether the ML activation condition has been met. The ML activation condition may include that the number of times the machine learning-based engine has been trained is greater than a threshold. In another example, the ML activation condition may include that the predictive accuracy of the machine learning-based engine is greater than a minimum accuracy value. In yet another example, the ML activation condition may include that the machine learning-based engine has been trained successfully.

[0061] The trained machine learning-based engine can receive one or more input data points in step 309, each input data point containing a number of records. In step 311, the trained machine learning-based engine can provide a prediction for each input data point whether the record at that data point is a duplicate record or not.

[0062] Figure 4 is a flowchart of a method for training a machine learning-based engine, as an example of the subject of this paper. For illustrative purposes, this method may be implemented in a manner in which the trained machine learning-based engine matches or compares records in the computer system 100 shown in Figure 1, but is not limited to this embodiment. This is because the machine learning-based engine can also be used to perform other classifications. This method can be performed, for example, by a data integration system 101.

[0063] In step 401, synthetic training data can be generated by assigning labels to the data using the rule-based engine 107. In step 403, real-life training data can be generated by assigning labels to the data using a data steward. In step 405, the machine learning-based engine can be trained using the synthetic and real-life training data. Training can be repeated as shown in Figure 4, and as more real-life training data is generated for training the machine learning-based engine, the use of synthetic training data in training the machine learning-based engine can be reduced absolutely or relatively.

[0064] Figure 5 is a flowchart illustrating a method for inference using a machine learning-based engine, as an example of this topic.

[0065] In step 501, a new training process for the machine learning-based engine can be started. In step 502, all natural training data can be retrieved and divided into a training set and a test set. In step 503, the last percentage of synthetic training data used during the last training process can be retrieved. It can be determined whether the percentage of synthetic data is 0% (step 504). If the percentage of synthetic data is 0%, in step 505, the machine learning-based engine can be trained on the defined training set and tested on the test set. Then, in step 506, the machine learning-based engine can be started as a new model, for example, to perform inference.

[0066] If the percentage of synthetic data is not 0%, in step 507, the synthetic data used during the last training process can be retrieved and added to the training set. In step 508, the machine learning-based engine can be trained on a defined training set (Model A) and tested on a test set. In step 509, the training set can be reduced by removing some of the synthetic data. In step 510, the machine learning-based engine can be trained on a defined training set (Model B) and tested using a test set. It can be determined whether the accuracy of the machine learning-based engine has increased compared to the machine learning-based engine trained in step 508 (step 511). The reason Model A and Model B have different names is that they refer to different states of the same machine learning-based engine that has been trained. However, in practice, the data used to train Model A is different from the data used to train Model B, so the trainable parameter values ​​of the two Models A and B may be different, and both Models A and B are configured to classify records as duplicates or not duplicates. If the accuracy has increased, steps 509 through 511 can be repeated. If accuracy does not improve, in step 512, the percentage of synthetic training data used in this training process can be saved. In step 513, the synthetic training data used in this training process can be saved. Then, in step 506, the machine learning-based engine can be started. This accuracy can be determined, for example, by inferring or testing models A and B using pre-classified records of the test set. The percentage of accurately classified records in the pre-classified records may be an indicator of accuracy. The pre-classified records of the test set may be different from those of the training dataset and may be classified by a user, for example, a steward.

[0067] Figure 6 shows a typical computerized system 600 suitable for carrying out at least a portion of the method steps included in this disclosure.

[0068] The methods described herein are understood to be at least partially non-interactive and to be automated by a computerized system such as a server or embedded system. However, in exemplary embodiments, the methods described herein can be implemented in a (partially) interactive system. These methods can further be implemented by software 612, 622 (including firmware 622), hardware (processor) 605, or a combination thereof. In exemplary embodiments, the methods described herein are implemented in software as executable programs and run by a dedicated or general-purpose digital computer such as a personal computer, workstation, minicomputer, or mainframe computer. Thus, the most common system 600 includes a general-purpose computer 601.

[0069] In exemplary embodiments, with respect to the hardware architecture, as shown in Figure 6, the computer 601 includes a processor 605, memory (main memory) 610 coupled to a memory controller 615, and one or more input or output (I / O) devices (or peripherals) or input and output (I / O) devices (or peripherals) 10, 645 communicably coupled via a local input / output controller 635. The input / output controller 635 may be, but is not limited to, one or more buses or other wired or wireless connections known in the art. The input / output controller 635 may have additional elements to enable communication, such as controllers, buffers (caches), drivers, repeaters, and receivers, which have been omitted for simplicity. Furthermore, this local interface may include address connections, control connections, or data connections, or a combination thereof, to enable proper communication between the above components. As described herein, the I / O devices 10, 645 may generally include generalized cryptographic cards or smart cards known in the art.

[0070] The processor 605 is a hardware device for executing software, particularly software stored in memory 610. The processor 605 can be a custom-made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with computer 601, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally, a device for executing software instructions.

[0071] Memory 610 may include one or a combination of volatile memory elements (e.g., random access memory (RAM such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM)). Note that memory 610 may have a distributed architecture in which various components are located remotely from one another but can be accessed by the processor 605.

[0072] The software in memory 610 may include one or more separate programs, each of which includes an ordered list of executable instructions for performing logical functions, in particular functions included in embodiments of the present invention. In the example in Figure 6, the software in memory 610 includes instruction 612, for example, instructions for managing a database such as a database management system.

[0073] The software in memory 610 typically also includes a suitable operating system (OS) 611. The OS 611 essentially controls the execution of other computer programs, such as software 612 for carrying out the methods described herein.

[0074] The methods described herein may take the form of a source program 612, an executable program 612 (object code), a script, or other entities including a set of instructions 612 to be executed. When it is a source program, the program needs to be translated by a compiler, assembler, interpreter, etc., which may or may not be contained in memory 610, so that it can function properly in conjunction with the OS 611. Furthermore, these methods may be written as an object-oriented programming language having classes of data and methods, or as a procedural programming language having routines, subroutines, or functions, or a combination thereof.

[0075] In exemplary embodiments, a conventional keyboard 650 and mouse 655 may be coupled to the input / output controller 635. Other output devices, such as I / O device 645, may include input devices, such as, but are not limited to, a printer, scanner, microphone, etc. Finally, I / O devices 10, 645 may further include devices that transmit both input and output, such as, but are not limited to, a network interface card (NIC) or modulator / demodulator (for accessing other files, devices, systems, or networks), radio frequency (RF) or other transceiver, telephone interface, bridge, router, etc. I / O devices 10, 645 may be generalized cryptographic cards or smart cards known in the art. System 600 may further include a display controller 625 coupled to the display 630. In exemplary embodiments, system 600 may further include a network interface for coupling to a network 665. Network 665 may be an IP-based network for communication over a broadband connection between computer 601 and external servers, clients, etc. Network 665 transmits and receives data between computer 601 and an external system 30 which may be included to carry out some or all of the steps of the method discussed herein. In exemplary embodiments, network 665 may be a managed IP network managed by a service provider. Network 665 may be implemented wirelessly, using wireless protocols and technologies such as WiFi, WiMAX, etc. Network 665 may also be a packet-switched network such as a local area network, wide area network, metropolitan area network, internet network, or other similar type of network environment.Network 665 may be a fixed wireless network, a wireless local area network (WLAN), a wireless wide area network (WWAN), a personal area network (PAN), a virtual private network (VPN), an intranet, or other suitable network system, and may include equipment for sending and receiving signals.

[0076] If the computer 601 is a PC, workstation, or intelligent device, the software in memory 610 may further include a Basic Input / Output System (BIOS) 622. The BIOS is a set of basic software routines that initialize and test the hardware at startup, start the OS 611, and assist in the transfer of data between hardware devices. The BIOS is stored in ROM so that it can be executed when the computer 601 is started.

[0077] When computer 601 is operating, processor 605 is configured to execute software 612 stored in memory 610, exchange data with memory 610, and generally control the operation of computer 601 according to the software. The methods and OS 611 described herein are read by processor 605, possibly buffered within processor 605, and then executed, in whole or in part, usually in part.

[0078] As shown in Figure 6, when the systems and methods described herein are implemented in software 612, those methods may be stored on a computer-readable medium such as storage 620 for use by a computer-related system or method, or for use in conjunction with a computer-related system or method. Storage 620 may include disk storage such as HDD storage.

[0079] The subject matter of this disclosure may include: Aspect 1 of this disclosure is a computer implementation for training a machine learning-based engine, which includes receiving a current training dataset, wherein a first fraction of the current training dataset includes synthetic training data, and the remaining second fraction of the training dataset includes real-life training data, where the real-life training data is user-defined data and the synthetic training data is system-defined data; and iteratively training the machine learning-based engine using the current training dataset, wherein in each iteration (where iteration means a repeat of training) or in each iteration of a subset of iterations, the training dataset is updated, the update being done by adding real-life training data, thereby increasing the second fraction of real-life training data and decreasing the first fraction of synthetic training data in the updated training dataset.

[0080] In a second aspect of the above method, a machine learning-based engine is trained to determine whether two data records are duplicates of each other, and the method further includes comparing the records in the database using the trained machine learning-based engine.

[0081] In a third aspect of the above method, a machine learning-based engine is used to compare database records if the predictive accuracy of the currently trained machine learning-based engine has not increased compared to the predictive accuracy of the trained machine learning-based engine in the last iteration.

[0082] In a fourth aspect of the above method, in aspect 3, a machine learning-based engine is used to compare database records when the first fraction is zero.

[0083] A fifth aspect of the above method, including any of the preceding second to third aspects, reduces the synthetic training data in each iteration or in each iteration of a subset of iterations, thereby further reducing the first fraction of synthetic training data in the updated training dataset.

[0084] A sixth aspect of the above method includes the fifth aspect, wherein the reduction of synthetic training data is absolute or relative.

[0085] A seventh aspect of the above method includes the fifth or sixth aspect, which comprises repeated reduction of synthetic training data, thereby gradually reducing the amount of synthetic data.

[0086] The eighth aspect of the above method includes any of the fifth through seventh aspects, and reduces the amount of synthetic data to such an extent that the training is performed exclusively with respect to real-life training data.

[0087] A ninth aspect of the above method includes the first through eighth preceding aspect being that the second fraction is zero with respect to the first run of training the machine learning-based engine.

[0088] A tenth aspect includes the method described above, wherein the machine learning-based engine is a machine learning-based matching engine for finding duplicates in a database, the training dataset includes labeled records, and the records of the synthetic training data are labeled by a rule-based matching engine based on comparisons of records by the rule-based matching engine.

[0089] In the eleventh aspect of the above method, the tenth aspect includes the rule-based matching engine operating using deterministic matching, probabilistic matching, or both.

[0090] In the twelfth aspect of the above method, either the tenth or eleventh aspect may include the generation of synthetic training data using the default configuration of the rule-based matching engine.

[0091] In a thirteenth aspect of the above method, any of the preceding fifth through twelfth aspects may include the level of reduction of the synthetic data used for training being dynamically adjusted based on at least one predictive quality metric.

[0092] In another embodiment of the present disclosure, a computer program product includes a computer-readable storage medium on which computer-readable program code is implemented. This computer-readable program code is configured to carry out the method described above.

[0093] In another embodiment of the present disclosure, a computer system may include training a machine learning-based engine. This computer system may include receiving a current training dataset, the first fraction of which consists of synthetic training data, the remaining second fraction of which consists of real-life training data, the real-life training data being user-defined data, and the synthetic training data being system-defined data, and iteratively training a machine learning-based engine using the current training dataset, wherein in each iteration or in each iteration of a subset of iterations, the training dataset is updated, the update of which adds real-life training data, thereby increasing the second fraction of the updated training dataset and decreasing the first fraction of synthetic training data, and iteratively training.

[0094] In one aspect of the present disclosure, a computer implementation for training a machine learning-based engine includes receiving a current training dataset. A first fraction of the training dataset includes synthetic training data, and the remaining second fraction of the training dataset includes real-life training data, where the real-life training data is user-defined data and the synthetic training data is system-defined data. The method includes iteratively training the machine learning-based engine using the current training dataset. In each iteration (or repetition) of at least a fraction of the iterations, the training dataset is updated by adding real-life training data, thereby increasing the second fraction of the updated training dataset and decreasing the first fraction of synthetic training data.

[0095] In another aspect of this disclosure, a computer implementation for training a machine learning-based engine includes receiving a current training dataset. A first fraction of the current training dataset includes synthetic training data, and the remaining second fraction of the training dataset includes real-life training data, where the real-life training data is user-defined data and the synthetic training data is system-defined data. The method includes iteratively training the machine learning-based engine using the current training dataset, the training dataset being updated in each iteration or in each iteration of a subset of iterations, the update being done by adding real-life training data, thereby increasing the second fraction of real-life training data and decreasing the first fraction of synthetic training data in the updated training dataset.

[0096] In another aspect of this disclosure, a computer program product includes a computer-readable storage medium on which computer-readable program code is implemented, and the computer-readable program code is configured to perform the operation of the method according to the prior embodiments.

[0097] In another aspect of this disclosure, a computer system for training a machine learning-based engine includes receiving a current training dataset, wherein a first fraction of the training dataset includes synthetic training data, and the remaining second fraction of the training dataset includes real-life training data. The real-life training data is user-defined data, and the synthetic training data is system-defined data. The system includes iteratively training the machine learning-based engine using the current training dataset, wherein in each iteration (or repetition) of at least a fraction of the iterations the training dataset is updated, by adding real-life training data, thereby increasing the second fraction of the updated training dataset and decreasing the first fraction of synthetic training data.

[0098] The present invention may be a system, method, or computer program product, or a combination thereof, at a level of technical detail that can be integrated. This computer program product may include (one or more) computer-readable storage media having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0099] This computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. This computer-readable storage medium can be, for example, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or a suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random-access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically coded devices such as punch cards or raised structures in grooves on which instructions are recorded, and suitable combinations thereof. The computer-readable storage media used herein should not be interpreted as themselves transient signals, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating within a waveguide or other transmission medium (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0100] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to the corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. This network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers those computer-readable program instructions for storage in the corresponding computer-readable storage medium within the respective computing / processing device.

[0101] The computer-readable program instructions for performing the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or configuration data for an integrated circuit, or they may be source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the C programming language or similar programming languages. These computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or remote server. In the last scenario above, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or this connection may be made to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, to carry out aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute the computer-readable program instruction by personalizing the electronic circuit using the state information of the computer-readable program instruction.

[0102] In this specification, aspects of the present invention will be described with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in those flowcharts or block diagrams, or both, and combinations of blocks in those flowcharts or block diagrams, or both, can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions can be provided to a computer processor or a processor of another programmable data processing device forming a machine, such that the instructions executed by the computer processor or the processor of the other programmable data processing device generate means to perform the functions / operations specified in the blocks of these flowcharts or block diagrams or both. These computer-readable program instructions can further be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device or other device or a combination thereof to function in a particular manner, such that the computer-readable storage medium in which the instructions are stored contains a product containing instructions that perform the functions / operations specified in the blocks of these flowcharts or block diagrams or both.

[0104] These computer-readable program instructions can further be loaded onto a computer, other programmable device, or other device in such a manner that these instructions, executed on the computer, other programmable device, or other device, perform the functions / operations specified in the blocks of these flowcharts or block diagrams or both, in order to cause a series of operational steps on the computer, other programmable device, or other device to produce a process to be performed by the computer.

[0105] The flowcharts and block diagrams in the figures illustrate the architecture, functions, and operations of possible embodiments of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in those flowcharts or block diagrams may represent a module, segment, or portion of instructions containing one or more executable instructions that perform a specified logical function. In some alternative embodiments, the functions shown in the blocks may be executed in an order different from the order shown in the figures. For example, two blocks shown consecutively may actually be executed as a single step, simultaneously, substantially simultaneously, or partially or completely overlapping in time, or depending on the functions they contain, those blocks may sometimes be executed in reverse order. It should also be noted that each block in those block diagrams or flowcharts or both, and combinations of blocks in those block diagrams or flowcharts or both, may be implemented by a hardware-based dedicated system that performs a specified function or operation or executes a combination of dedicated hardware and computer instructions.

Claims

1. A method for training a machine learning-based engine, which is performed by a computer, wherein the method is Receiving the current training dataset, wherein a first fraction of the current training dataset includes synthetic training data, and the remaining second fraction of the training dataset includes real-life training data, wherein the real-life training data is user-defined data, and the synthetic training data is system-defined data, and The process involves repeatedly training the machine learning-based engine using the current training dataset, updating the training dataset in each iteration or in each iteration of a subset of the iterations by adding additional real-life training data to the training dataset, thereby increasing the second fraction of the real-life training data and decreasing the first fraction of the synthetic training data in the updated training dataset. Methods that include...

2. The method according to claim 1, wherein the machine learning-based engine is trained to determine whether two data records are duplicates of each other, and the method further comprises comparing records in a database using the trained machine learning-based engine.

3. The method according to claim 2, wherein the machine learning-based engine is used to compare the records in the database if the predictive accuracy of the currently trained machine learning-based engine has not increased compared to the predictive accuracy of the trained machine learning-based engine in the last iteration.

4. The method according to claim 2, wherein the machine learning-based engine is used to compare the records in the database when the first fraction is zero.

5. The method according to claim 1, further comprising reducing the synthetic training data in each iteration or in each iteration of a subset of the iterations, thereby further reducing the first fraction of the synthetic training data in the updated training dataset.

6. The method according to claim 5, wherein the reduction of the synthetic training data is absolute or relative.

7. The method according to claim 5, wherein the repeated reduction of the synthetic training data includes gradually reducing the amount of synthetic training data.

8. The method according to claim 5, wherein the amount of synthetic training data is reduced to such an extent that the training is performed exclusively with respect to real-life training data.

9. The method according to claim 5, wherein the level of reduction of the synthetic training data used for training is dynamically adjusted based on at least one predictive quality metric.

10. The method according to claim 1, wherein the second fraction is zero with respect to the first run of training the machine learning-based engine.

11. The method according to claim 1, wherein the machine learning-based engine is a machine learning-based matching engine for finding duplicates in a database, the training dataset includes labeled records, and the records of the synthetic training data are labeled by a rule-based matching engine based on a comparison of the records by the rule-based matching engine.

12. The method according to claim 11, wherein the rule-based matching engine operates using deterministic matching, probabilistic matching, or both.

13. The method according to claim 11, wherein labeling the synthetic training records includes using the default configuration of the rule-based matching engine.

14. A computer program for causing a computer to perform the method described in any one of claims 1 to 13.

15. A computer system for training a machine learning-based engine, wherein the computer system includes a computer processor, a computer-readable storage medium, and a computer program stored on the computer-readable storage medium, and the computer program is trained by the processor to perform the following functions on the computer system, namely A function to receive the current training dataset, wherein a first fraction of the current training dataset includes synthetic training data, and the remaining second fraction of the training dataset includes real-life training data, wherein the real-life training data is user-defined data, and the synthetic training data is system-defined data, and the receiving function, The function of repeatedly training the machine learning-based engine using the current training dataset, and updating the training dataset by adding additional real-life training data to the training dataset in each iteration or in each iteration of a subset of the iterations, thereby increasing the second fraction in the updated training dataset and decreasing the first fraction of the synthetic training data. A system that is executable to perform a task.

16. The system according to claim 15, wherein the machine learning-based engine is trained to determine whether two data records are duplicates of each other, and the system further comprises comparing records in a database using the trained machine learning-based engine.

17. The system according to claim 16, wherein the machine learning-based engine is used to compare the records in the database if the predictive accuracy of the currently trained machine learning-based engine has not increased compared to the predictive accuracy of the trained machine learning-based engine in the last iteration.

18. The system according to claim 16, wherein the machine learning-based engine is used to compare the records in the database when the first fraction is zero.

19. The system according to claim 15, further comprising reducing the synthetic training data in each iteration or in each iteration of a subset of the iterations, thereby further reducing the first fraction of the synthetic training data in the updated training dataset.

20. The system according to claim 19, wherein the reduction of the synthetic training data is absolute or relative.

Images