Shuffled Gradient Method for Training Machine Learning Models with Big Data, Computer-Implemented Method, Computing Device, and Non-Transient Computer-Readable Storage Medium (Shuffled Gradient Method for Training Machine Learning Models with Big Data)

Abstract

To provide a computer execution method, a computing device and a storage medium for shuffling-type gradient for training a machine learning model by big data by using stochastic gradient descent (SGD) in machine learning related to the big data.SOLUTION: A training operation of a machine learning model uniformly distributes a data sample or coordinate update of training data at random, and calculates a scheme with no shuffle and a shuffle scheme obtained by combining training data by using a stochastic gradient descent (SGD) algorithm. A combined calculation is switched to the execution of only the shuffle scheme from the scheme with no shuffle on the basis of one or more prescribed references and the training of the machine learning model with the training data on the basis of the combined scheme with no shuffle and shuffle scheme.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates generally to techniques for applying stochastic gradient methods in machine learning, and more particularly to techniques for using stochastic gradient methods to train machine learning models with big data. [Background technology]

[0002] Stochastic gradient methods used today have varying degrees of success in solving problems related to training convex and non-convex models in machine learning. Gradient descent measures the degree of change of a variable in response to a change in another variable. The direction of descent is the gradient of a functional component of a loss function whose output is a partial derivative of a set of input parameters.

[0003] Stochastic gradient descent (SGD) is one of several gradient methods used in machine learning. SGD attempts to find a global minimum by adjusting the network configuration after each training point. Rather than finding the full gradient, SGD reduces error by approximating the gradient of randomly selected training samples. The randomly selected training samples result from randomly shuffling the dataset and systematically processing the samples. SGD is a popular method for training machine learning models, and because SGD performs more frequent updates, it often converges faster than batch training (e.g., using full gradients). In addition, SGD can efficiently utilize advanced computing architectures such as GPUs, and distributed computing with gradient estimation can be performed in parallel. This method is popular because the random nature of SGD allows it to escape local minima and often find a global solution (e.g., training non-convex DNN models).

[0004] When performing SGD to train a machine learning model, the training data is often shuffled to compute gradients that are no longer independently sampled from the training data set, and the data may be split across multiple machines depending on their size, and a predetermined number of iterations may be performed on the reshuffled data.

[0005] In machine learning involving big data, the training process can take a very long time when the number of data is large and the complete data cannot be stored in a single device. Therefore, there is room for improving the learning rate scheme of the SGD method to speed up the training of machine learning models. Summary of the Invention [Problem to be solved by the invention]

[0006] In machine learning involving big data, the training process can take a very long time when the number of data is large and the complete data cannot be stored in a single device. Therefore, there is room for improving the learning rate scheme of the SGD method to speed up the training of machine learning models. [Means for solving the problem]

[0007] According to one embodiment, a computer-implemented method for shuffled gradients for training machine learning models with big data using stochastic gradient descent (SGD) involves uniformly randomly shuffling data samples or coordinate updates of the training data. distributionThe method includes the steps of: using a stochastic gradient descent (SGD) algorithm to perform a combined no-shuffle and shuffle scheme on the training data; switching from performing the combined no-shuffle and shuffle scheme to performing only the shuffle scheme based on one or more predetermined criteria; and training a machine learning model based on the pure shuffle scheme and the combined no-shuffle and shuffle scheme. This hybrid randomized shuffle gradient scheme results in novel and significantly improved convergence rates over existing work under standard assumptions in the non-convex case, and convergence rates in the strongly convex case match those of best-known cases for solving finite-sum minimization problems up to a constant factor without imposing bounded gradient conditions.

[0008] In one embodiment, the data samples or coordinate updates in the training data are big data. Big data is particularly problematic, and the hybrid randomized shuffling scheme provides a more efficient way to train on such large amounts of data.

[0009] In one embodiment, the SGD algorithm used to implement the combined no-shuffle and shuffle scheme includes at least one of the Adagrad, Adam, RMSProp, or SGD Momentum algorithms, each of which is particularly suitable for use in the hybrid randomized shuffle scheme of the present invention.

[0010] In one embodiment, the one or more predetermined criteria include a learning rate, and the switch from the no-shuffle scheme to the pure shuffle occurs based on the value of the learning rate of the machine learning algorithm proceeding, providing a more efficient way to increase the convergence rate without getting stuck in a local minimum.

[0011] In one embodiment, the one or more predetermined criteria include an error rate of the machine learning, and a switch to a pure shuffling scheme occurs according to the error rate of the machine learning, resulting in a more efficient process with fewer errors.

[0012] In one embodiment, the one or more predetermined criteria include a learning rate rule for the unshuffled scheme and the shuffled scheme, and the method further includes providing one or more learning rate rules that determine one or more of the predetermined theoretical complexity or practical performance of the SGD algorithm. The learning rules can be used to provide practical performance of the SGD algorithm and to improve the convergence rate.

[0013] In one embodiment, the learning rate rule involves a decreasing learning rate scheme and achieving maximum convergence rate results in a non-convex supervised learning model. This feature achieves an enhanced convergence rate.

[0014] In one embodiment, the machine learning model is trained for a deep neural network (DNN) application, which typically requires complex training, and configuring the machine learning model to train the DNN facilitates this training.

[0015] In one embodiment, the machine learning model is trained for one or more of linear regression and binary classification operations, and improved performance of linear regression and binary operations is facilitated.

[0016] In one embodiment, a computing device including a configurable machine learning framework for training machine learning models using shuffled gradient operations on big data includes a processor and a memory coupled to the processor, the memory storing instructions causing the processor to perform acts including generating a machine learning framework including a combined no-shuffle and shuffled scheme using a stochastic gradient descent (SGD) algorithm. The machine learning framework is further configured to receive a predetermined criterion for switching from the combined no-shuffle and shuffled scheme operations to performing only the shuffled scheme, and to train a machine learning model based on the combined no-shuffle and shuffled scheme operations and only the shuffled scheme. This hybrid randomized shuffled gradient scheme results in a novel and significantly improved convergence rate over existing work under standard assumptions in the non-convex case, and a convergence rate in the strongly convex case that matches best-known cases for solving finite-sum minimization problems up to a constant factor without imposing a bounded gradient condition.

[0017] In one embodiment, the computing device is configured to receive a user selection of a predetermined criterion for switching from a no-shuffle scheme to a shuffle-only scheme, the switching providing a more efficient way of obtaining a convergence rate.

[0018] In one embodiment, the machine learning framework is configured to perform a combined shuffling and no-shuffling scheme from one or more SGD algorithms, including Adagrad, Adam, RMSProp, or SGD Momentum, each of which is particularly suited for use in the hybrid randomized shuffling scheme of the present invention.

[0019] In one embodiment, the predetermined criteria for switching from a no-shuffle scheme to a shuffle-only scheme includes an error rate.

[0020] In one embodiment, the predetermined criteria for switching from a no-shuffle scheme to a shuffle-only scheme includes a learning rate.

[0021] In one embodiment, the machine learning framework is configured to prompt at least one of an error rate or a learning rate as a predetermined criterion for switching from a no-shuffle scheme to a shuffle-only scheme.

[0022] In one embodiment, a computing device is configured to perform shuffled gradients for training machine learning models with big data using stochastic gradient descent (SGD). The computer-implemented method includes a processor and a memory coupled to the processor. The memory stores data samples or coordinate updates uniformly randomly. distribution and performing a combined no-shuffle and shuffle scheme of data samples or coordinate updates. Switching from the no-shuffle scheme to only the shuffle scheme is based on one or more predetermined criteria. Training a machine learning model with data samples or coordinate updates is more efficient using the switching operation.

[0023] In one embodiment, the instructions cause the processor to perform additional acts including switching to a shuffle-only operation according to a learning rate of the machine learning system, which provides a more efficient operation.

[0024] In one embodiment, the instructions cause the processor to perform additional acts including switching to a shuffle-scheme-only operation according to a learning rate threshold of a machine learning algorithm, the learning rate being a means of indicating whether the operation is progressing as planned or whether additional intervention can be performed.

[0025] In one embodiment, a computer-implemented method for determining a shuffled gradient algorithm for training a machine learning model with big data uses a stochastic gradient descent (SGD) algorithm, which uniformly randomly selects data samples or coordinate updates for the training data. distribution There is a step to combine the no-shuffle scheme and the shuffle scheme into a framework to randomly select non-convex logistic regression. distribution The learning rate scheme is reduced to obtain the maximum known convergence rate result per sample taken.

[0026] In one embodiment, the machine learning model is trained for a deep neural network (DNN) application. Deep neural networks typically require complex training, which is facilitated by configuring the machine learning model to train the DNN.

[0027] In one embodiment, the machine learning model is trained for linear regression and binary classification operations. Training the machine learning model in such operations facilitates improved operation of linear regression and binary operations.

[0028] In one embodiment, a non-transitory computer-readable storage medium tangibly embodying computer-readable program code having computer-readable instructions that, when executed, cause a computing device to perform a shuffled gradient algorithm for training machine learning models with big data using stochastic gradient descent (SGD). A shuffled gradient algorithm uniformly randomizes data samples or coordinate updates. distributionThe method includes the steps of: performing a combined no-shuffle and shuffled scheme of samples or coordinate updates; switching from the no-shuffle scheme to the shuffled scheme only based on a predetermined criterion; and training a learning model with data samples or coordinate updates based on the shuffled scheme only and the combined no-shuffle and shuffled schemes. This hybrid randomized shuffled gradient scheme results in novel and significantly improved convergence rates over existing work under standard assumptions in the non-convex case; the convergence rate in the strongly convex case matches the best-known case up to a constant factor without imposing a bounded gradient condition, enabling the solution of finite-sum minimization problems.

[0029] In one embodiment, the one or more predetermined criteria include a learning rate rule. The shuffled gradient method further includes providing one or more learning rate rules that determine one or more of a predetermined theoretical complexity and practical performance of the SGD algorithm. The learning rules provide the operation of the SGD algorithm for use with an improved convergence rate.

[0030] In one embodiment, providing a learning rate rule includes obtaining a decreasing learning rate scheme and maximizing convergence rate results in a non-convex supervised learning model, where maximizing convergence rate is an advantage of utilizing a learning rate rule.

[0031] In one embodiment, the machine learning model is trained for one or more of linear regression and binary classification operations. This training of the machine learning model provides advantages, particularly in solving non-convex problems.

[0032] These and other features will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0033] [Figure 1] Diagram of the matrix completion problem for a movie recommender system. [Figure 2]1 illustrates a supervised learning scenario in which a predictive model is trained. [Figure 3] We provide an algorithmic architectural overview of the hybrid-randomized shuffle-gradient scheme consistent with the exemplary embodiment. [Figure 4] 1 is a functional overview of the algorithm for a hybrid-randomized shuffled gradient scheme consistent with an exemplary embodiment. [Figure 5] 10 illustrates machine learning models with different variations in training loss consistent with an example embodiment. [Figure 6] 10 is a graph illustrating why a shuffling scheme using stochastic gradient descent consistent with an exemplary embodiment is preferable for big data. [Figure 7] Some advantages of computer-implemented methods consistent with exemplary embodiments are illustrated. [Figure 8] A comparison of some functions of the regular scheme against a shuffle scheme consistent with the exemplary embodiment is shown. [Figure 9] 1 is a flowchart illustrating operations for training a machine learning model with shuffled gradients consistent with an example embodiment. [Figure 10] FIG. 1 is a functional block diagram of a computer hardware platform capable of communicating with agents in performing collaborative tasks consistent with an illustrative embodiment. [Figure 11] 1 illustrates an exemplary cloud computing environment consistent with illustrative embodiments. [Figure 12] 1 illustrates a set of functional abstraction layers provided by a cloud computing environment consistent with an illustrative embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0034] The drawings are exemplary embodiments. The drawings do not illustrate all embodiments. Other embodiments may be used in addition to or in place of the embodiments shown. Obvious or unnecessary details may be omitted to save space or for more effective illustration. Some embodiments may be practiced with additional components or steps, or without all of the components or steps shown, or both. The appearance of the same numerals in different drawings refers to the same or similar components or steps.

[0035] Overview In the following detailed description, by way of example, numerous specific details are set forth in order to provide a thorough understanding of the relevant teachings. However, it should be understood that the present teachings may be practiced without such details. In other instances, well-known methods, procedures, components, or circuits, or combinations thereof, are described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

[0036] As used herein, a function is considered to be convex if a line segment joining any two points on the graph of the function can be located above the graph between those two points. If there is a set of points on or above the graph of a function, such points may form a convex set.

[0037] A convex function may have multiple optimal solutions or may have no optimal solution (eg, a linear function), but a strongly convex function is a convex function that has a unique optimal solution.

[0038] A function is considered to be strongly convex if there exists a quadratic lower bound on the growth of the function with parameter m>0. That is, there exists a parameter m>0 such that for all points x, y the following inequality exists: f(y)>=f(x)+<\f(x),xy>+(m / 2) * If ||yx||^2 holds, then f(x) is strongly convex.

[0039] As used herein, the term "non-convex problem" generally refers to an optimization problem that has multiple local optima. The term "convex problem" generally refers to an optimization problem whose objective function is a convex function. A feasible set is a convex set, and when a function mapping of any subset is performed, the subset is convex if its domain is convex. f(x) is convex if the following inequality holds for all points x, y: f(y)>=f(x)+<\f(x),xy>.

[0040] The proposed architecture is a hybrid randomized shuffled gradient scheme. The architecture disclosed herein covers numerous variants, including randomized reshuffle, single permutation, and periodic and incremental gradient schemes. In particular, at least two different settings are considered: strongly convex and non-convex problems. One of the improvements of this disclosure includes the use of novel non-asymptotic and asymptotic convergence rates for the general class of shuffled gradient methods in both non-convex and strongly convex problems by introducing a novel learning rate scheme. For strongly convex problems, the proposed algorithm is particularly useful for linear regression and binary classification.

[0041] For non-convex problems, the convergence rates of the computer-implemented methods and systems disclosed herein are significantly improved over existing work under standard assumptions. Thus, the computer-implemented methods and systems disclosed herein provide improvements in the field of finite reduction and approximation problems. For non-convex problems, the proposed algorithms are particularly useful for use in training deep neural networks. Additionally, the computer-implemented methods and systems disclosed herein provide improvements in computational efficiency, as the computations are faster and more accurate than known approaches (especially for non-convex problems), thereby freeing up computer resources and providing a technical improvement in using less computing power to solve such non-convex problems.

[0042] For strongly convex problems, the convergence rate coincides with the best-known rate up to a constant factor without imposing bounded gradient conditions. The effect of the learning rate is also discussed herein via examples of non-convex logistic regression and neural network training. The results suggest several appropriate choices for the learning rate. In one example, the best learning rate is

[0043]

number

[0044] and n t is the number of steps.

[0045] Big data can pose challenges for computational methods such as SGD because the number of sample points is often very large. SGD requires a large number of gradient calculations proportional to the number of sample points. Figure 1 shows a matrix 105 for a movie recommender system. The numbers in the boxes are user ratings for specific movies, ranging from 1 to 5. The matrix shows movies 115 for users 110. The number of users 110 is very large (e.g., millions), and the number of movies 115 is in the thousands. Group 150 shows the decomposition of matrix M 155, as well as U 160 and V 165, and their loss functions. Given a small number of entries in the observation matrix 155, the goal is to complete the matrix by approximating the unseen entries. A ranking operation 170 is also shown.

[0046] Referring to Figure 1, the loss function for learning U and V is

[0047]

number

[0048] where matrix M is an input matrix with some missing entries, and an unknown matrix is ​​used to approximate the input matrix M, and K ij , λ U , λ V are the model parameters.

[0049] Additionally, there are finite sum minimization problems that cover a wide range of convex and non-convex models in machine learning and statistical learning, including multi-kernel learning, logistic regression, conditional random fields, and neural networks. In particular, finite sum minimization problems involve performing empirical risk minimization. Because the actual data distribution is unknown, empirical risk minimization (EMR) is used to evaluate the performance of algorithms that are implemented (with risk).

[0050] Example architecture Figure 2 illustrates a supervised learning scenario 200 in which a predictive model is learned. Referring to Figure 2, at 205 it is shown that training data is typically provided for a supervised learning task. In this example, the data is D n ={(x1,y1),…,(x n ,y n )}, where x and y are data values. The goal is to find a predictive model h w The goal is to learn (x). Examples of linear regression and logistic regression are shown. Linear regression describes the relationship between one or more independent variables (x) and a dependent variable (y). The logistic regression formula also shows a statistical analysis technique used to predict data values ​​based on past observations.

[0051] Still referring to Figure 2, linear regression and logistic regression utilize model parameters w, and empirical risk minimization is a widely used optimization framework, as shown at 225. The performance of the algorithms on a set of known training data is measured using a widely used optimization framework, (Formula 1)

[0052]

number

[0053] where w is the model parameter l: loss function h(x i ;w):Prediction model n: number of training data x i ,i=1,…,n: training input y i ,i=1,...,n: training labels.

[0054] FIG. 3 provides an architecture overview 300 of the hybrid randomized shuffle gradient scheme algorithm consistent with an exemplary embodiment.

[0055]

number

[0056] It is shown that a random selection 315 (s∈[n]) is performed, followed by a random selection 315 (s∈[n]). A variation from a single batch (box 320) can be achieved by using mini-batches (box 325). Mini-batches can be used when there are more computing resources to distribute.

[0057] As shown, different formulas are applied to determine whether the problem is strongly convex or not. Regarding the decision of when to switch between shuffled and unshuffled, the scheme is based on the relationship t<\sqrt{n} between the number of iterations and the number of samples. If t<\sqrt{n}, the unshuffled scheme is used, and if t>=\sqrt{n}, the shuffled scheme is used.

[0058] Figure 4 is a functional overview 400 of the hybrid randomized shuffle gradient scheme algorithm consistent with an exemplary embodiment. Referring to Figure 4, the conventional operation is as follows:

[0059]

number

[0060] However, in accordance with the present disclosure, rather than a uniformly random sample, 405 shows the following:

[0061]

number

[0062] In generating arbitrary permutations as shown above, σ (t) (1),…,σ (t) σ of order (n) (t) Since samples from all over the world are used, there is virtually no possibility of missing data.

[0063] In 415, we show that all data is used in one training iteration, and in 425 we determine the step size based on whether we have a strongly convex or non-convex problem.

[0064] Figure 5 illustrates machine learning models 500 with different variations in training loss consistent with an example embodiment. In Figure 5, it can be seen that at 505 there is a machine learning graphical representation with low training loss and low variance in the solution. At 515 there is a large training loss and large variance for representation 505. Figure 5 illustrates the contours of the objective function, an example in 2D dimensional space where coordinates x and y correspond to the x-axis and y-axis.

[0065] FIG. 6 shows graphs of two recommender systems consistent with exemplary embodiments. From graph 600, it can be seen that the shuffled SGD method is preferred for big data due to error, consistent with exemplary embodiments. Recommender system graphs 605, 615 show the root mean square error (RMSE) versus time. In the first graph 605, time is shown in two-tenths of a time interval. In the second graph 615, time is shown in one-hour intervals. Both of these graphs show that the RMSE is lower with shuffling than without shuffling.

[0066] Figure 7 illustrates some advantages 700 of the computer-implemented method of the present disclosure. As shown in Figure 7, there is faster convergence with machine learning models at >95% accuracy. At 705, it is noted that samples are uniformly used to achieve more accurate modeling. For various datasets listed, the architectures 715 (FFN feedforward neural network, CNN convolutional neural network) and training accuracy 725 are shown.

[0067] FIG. 8 shows a comparison 800 of several functions of a regular scheme 805 versus a shuffled scheme 825 consistent with an exemplary embodiment. While the regular scheme 805 involves uniform sampling selected at random, the shuffled scheme involves the generation of n permutations, an incremental gradient operation, one shuffle, and a reshuffle operation. Advantages of shuffling include providing a scenario where there is virtually no possibility of missing data. There is also no possibility of uneven use of sampled data, since each sample is used once in each training iteration. Shuffling also advantageously provides the ability to escape stuck points with multiple local minima.

[0068] Process Example In conjunction with the foregoing overview of exemplary architectures, it may be useful to now consider a high-level discussion of example processes. To that end, in conjunction with FIGS. 1-8, FIG. 9 depicts a flowchart 900 illustrating various aspects of a computer-implemented method consistent with exemplary embodiments. Process 900 is illustrated as a collection of logically ordered blocks that represent a sequence of operations that may be performed in hardware, software, or a combination thereof. In the software context, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, etc. that perform a function or implement an abstract data type. Within each process, the order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks may be combined in any order and / or performed in parallel to implement the process.

[0069] FIG. 9 is a flowchart 900 illustrating training a machine learning model with shuffled gradients consistent with an example embodiment.

[0070] Referring now to FIG. 9, in operation 905, the data samples or coordinate updates of the training data are uniformly randomly generated. distribution There is an action that makes it random. distribution The data sample helps ensure that the sample is not disproportionately considered.

[0071] A combined no-shuffle and shuffle scheme of training data using a stochastic gradient descent (SGD) algorithm is performed at operation 910. This combination provides the advantages of both schemes, including the advantages of the regular scheme (no-shuffle scheme), such as faster convergence when the current solution is far from the optimal solution, and the advantages of the shuffle scheme, such as faster convergence when the current solution is close to the optimal solution.

[0072] At operation 915, there is a switch from the combined no-shuffle and shuffle schemes to a pure shuffle scheme based on one or more predetermined criteria. The predetermined criteria may be a learning rate, or an error rate, or a combination of the two, or some other metric. A pure shuffle may provide fewer errors, especially in training models with big data.

[0073] At 920, a machine learning model is trained with the training data based on performing a pure shuffle scheme and a combined no-shuffle and shuffle scheme.

[0074] Examples of specifically configured computing devices Figure 10 provides a functional block diagram illustration 1000 of a computer hardware platform. In particular, Figure 10 illustrates a specifically configured network or host computer platform 1000 that may be used to implement the method illustrated in Figure 9. The computer platform 1000 may include a central processing unit (CPU) 1004, a hard disk drive (HDD) 1006, random access memory (RAM) and / or read-only memory (ROM) 1008, a keyboard 1010, a mouse 1012, a display 1014, and a communication interface 1016 connected to a system bus 1002. The HDD 1006 may include a data store.

[0075] In one embodiment, the HDD 1006 is capable of carrying out various processes, such as the hybrid randomized shuffle module 1020 discussed hereinabove, and includes storage of programs configured to manage the overall process.

[0076] The no-shuffle module 1025 is configured to perform a gradient descent operation with no-shuffle operation.

[0077] The shuffle module 1030 is configured to perform only shuffling, and the shuffle module 1030 switches from the no-shuffle module 1025 when one or more predetermined criteria are met. A threshold learning rate, or a threshold error rate, may trigger the switch to pure shuffle module 1030 operation.

[0078] The machine learning module 1035 is configured to train machine learning models used, for example, in training deep neural networks (DNNs).

[0079] The machine learning framework generator module 1040 is configured to generate a framework in which an optimization process that determines the maximum convergence rate can occur. The machine learning framework generator module 1040 can be preloaded with user-selected data or can prompt for such data.

[0080] The training data module 1045 contains data samples of big data used to train machine learning models.

[0081] Cloud Platform Examples As noted above, functionality related to environmental and ecological optimization methods may involve cloud computing. While the present disclosure includes detailed descriptions of cloud computing as described later in this specification, it should be understood that implementations of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of the present invention may be practiced in conjunction with any other type of computing environment now known or later developed.

[0082] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models. The features are as follows:

[0083] On-demand self-service: Cloud consumers can unilaterally configure computing capacity, such as server time and network storage, automatically as needed without requiring human interaction with the service provider.

[0084] Wide network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate usage by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs).

[0085] Pooling of resources: A provider's computing resources are pooled for offering to multiple consumers using a multi-tenant model, with various physical and virtual resources dynamically allocated and reallocated according to demand. Consumers typically have no control or knowledge of the exact location of the resources provided, although a sense of location exists in that location may be specified at a more abstract level (e.g., country, state, or data center).

[0086] Rapid scalability: Capacity can be quickly and elastically provisioned, sometimes automatically, to instantly scale out and quickly released to instantly scale in. To the consumer, the capacity available for provisioning often appears infinite and can be purchased at any time and in any quantity.

[0087] Measured Services: Cloud systems automatically control and optimize resource utilization by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource utilization can be monitored, controlled, and reported, providing transparency to both providers and consumers of utilized services. The service model is as follows:

[0088] Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, except possibly for limited user-specific application configuration settings.

[0089] Platform as a Service (PaaS): The ability offered to consumers is to deploy applications they create or acquire, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application hosting environment configuration.

[0090] Infrastructure as a Service (IaaS): The capability provided to the consumer is to provide processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating system, storage, deployed applications, and possibly limited control over select networking components (e.g., host firewalls). The deployment model is as follows:

[0091] Private Cloud: Cloud infrastructure is operated exclusively for an organization. It can be managed by the organization or a third party and can reside on-premise or off-premise.

[0092] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). The cloud infrastructure can be managed by these organizations or by a third party and can reside on-premise or off-premise.

[0093] Public Cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0094] Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain unique entities but are tied together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that allow for data and application portability.

[0095] Cloud computing environments are service-oriented and emphasize statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0096] Referring now to FIG. 11 , an exemplary cloud computing environment 1100 utilizing cloud computing is illustrated. As shown, the cloud computing environment 1100 includes a cloud 1150 having one or more cloud computing nodes 1110 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 1154A, a desktop computer 1154B, a laptop computer 1154C, or an automotive computer system 1154N, or any combination thereof, may communicate. The nodes 1110 may communicate with each other. They may be physically or virtually grouped in one or more networks (not shown), such as a private, community, public, or hybrid cloud, or any combination thereof, as described herein. This enables the cloud computing environment 1100 to provide infrastructure, platform, or software, or any combination thereof, as a service without the cloud consumer having to maintain resources on their local computing device. It is understood that the types of computing devices 1154A-N shown in FIG. 11 are intended to be illustrative only, and that the computing node 1110 and cloud computing environment 1150 may communicate with any type of computerized device over any type of network and / or network-addressable connection (e.g., using a web browser).

[0097] Referring now to Figure 12, a set of functional abstraction layers 1200 provided by cloud computing environment 1100 (Figure 11) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 12 are intended to be exemplary only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0098] Hardware and software layer 1260 includes hardware and software components. Examples of hardware components include mainframe 1261, RISC (reduced instruction set computer) architecture-based server 1262, server 1263, blade server 1264, storage device 1265, and network and networking components 1266. In some embodiments, software components include network application server software 1267 and database software 1268.

[0099] The virtualization layer 1270 provides an abstraction layer within which virtual entities such as virtual servers 1271, virtual storage 1272, virtual networks including virtual private networks 1273, virtual applications and operating systems 1274, and virtual clients 1275 may be provided.

[0100] In one example, management layer 1280 may provide the functions described below. Resource provisioning 1281 provides dynamic procurement of computing and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 1282 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and tasks, as well as protection of data and other resources. User portal 1283 provides consumers and system administrators with access to the cloud computing environment. Service level management 1284 provides allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and fulfillment 1285 provides advance arrangement and procurement of cloud computing resources for anticipated future needs in accordance with SLAs.

[0101] The workload layer 1290 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 1291, software development and lifecycle management 1292, virtual classroom instructional delivery 1293, data analytics processing 1294, transaction processing 1295, and uniformly randomizing data samples using combined no-shuffle and shuffle schemes, as well as pure shuffle schemes, as described herein. distribution The hybrid randomized shuffle module 1296 is configured to train the learning model by:

[0102] conclusion In this work, a novel unified convergence analysis framework for shuffled gradient methods is developed to solve (P) in both non-convex and strongly convex settings and to apply it to different shuffle variants. Some of the contributions are summarized as follows:

[0103] (a) For solving a non-convex problem (P), we prove that a general shuffled gradient algorithm has a convergence rate of O(1 = T2 = 3) for constant step-size iterations and ~O(1 = T2 = 3) for decreasing step-size iterations, where T:=K=n are the number of iterations. The convergence rates are significantly improved over the O(1 = T1 = 2) convergence rate of incremental gradient methods. These convergence rates are the first improved non-asymptotic convergence rates of SGD with shuffling under both constant and decreasing learning rates under standard assumptions.

[0104] (b) Establish asymptotic convergence to a stationary point under a decreasing learning rate scheme. We achieve the best performance among various variants in both theory and practice with a learning rate _t = O(1 / t^{1 / 3}), where 't' is the learning iteration counter. Our learning rate is close to a "scheduled" learning rate, i.e., it is constant at each learning iteration t and decreases with respect to 't'.

[0105] (c) It can be shown that the convergence rate of O(1 / T^2) for iterative training of our general shuffled gradient scheme in the strongly convex case can be obtained without imposing the "gradient boundedness" and / or Lipschitz Hessian assumptions. In addition, our analysis does not require the convexity of each component function, as in some existing studies. Our convergence rate can also be viewed as O(1 / t^2) for any 1 <= t <= T, without fixing T a priori, as in other studies.

[0106] The description of various embodiments of the present teachings has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications, or technical improvements to technology found in the market, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0107] While the foregoing describes what is considered to be the best mode and / or other examples, it is understood that various modifications may be made therein, that the subject matter disclosed herein may be embodied in various forms and examples, and that the present teachings may be applied to numerous uses, only some of which are described herein. It is intended that the following claims claim any and all applications, modifications, and variations that are within the true scope of the present teachings.

[0108] The components, steps, features, objects, benefits, and advantages described herein are merely exemplary. Neither their discussion nor their intended purpose is to limit the scope of protection. While various advantages are discussed herein, it will be understood that not all embodiments necessarily include all advantages. Unless otherwise specified, all measurements, values, estimates, positions, magnitudes, sizes, and other specifications described herein, including the claims that follow, are approximate and not exact. They are intended to have a reasonable range consistent with the functions to which they pertain and customary practices in the art to which they pertain.

[0109] Numerous other embodiments are contemplated, including those having fewer, additional, or different or combinations of components, steps, features, objects, benefits, and advantages, including those having different arrangements and / or orderings of components and / or steps.

[0110] The flowcharts and diagrams in the figures herein illustrate the architecture, functionality, and operation of possible implementations according to various embodiments of the present disclosure.

[0111] While the foregoing has been described in conjunction with exemplary embodiments, it is understood that the term "exemplary" means merely an example, not best or optimal. Except as immediately stated, nothing described or illustrated is intended to or should be construed as causing the general release of any element, step, feature, object, benefit, advantage, or equivalent, whether claimed or not.

[0112] The terms and expressions used herein will be understood to have the ordinary meanings ascribed to such terms and expressions with respect to their corresponding respective fields of study and research, unless a special meaning is otherwise stated herein. Relationship terms such as first and second, etc., may be used solely to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between the entities or actions. The terms "comprise," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements does not include only those elements, but may also include other elements not expressly listed or inherent in such process, method, article, or apparatus. An element preceded by "a" or "an" does not preclude, but is not limited to, the presence of additional identical elements in the process, method, article, or apparatus that includes that element.

[0113] The Abstract is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, it will be appreciated that the foregoing Detailed Description groups together various features in various embodiments for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments have more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter. [Explanation of symbols]

[0114] 105 Queue 110 users 115 Movies 150 groups 155 Matrix M 160 U 165 V 170 Ranking Calculation

Claims

1. 1. A computer-implemented method for training a machine learning model with shuffled gradient operations, comprising: uniformly randomly distributing data samples of training data; Calculating learning rates for unshuffled and shuffled schemes with randomly selected uniform sampling; computing the combined unshuffled and shuffled schemes on the training data using a stochastic gradient descent (SGD) algorithm; switching from the no-shuffle scheme to only performing the shuffle scheme based on one or more predetermined criteria; training the machine learning model with the training data based on the combined unshuffled scheme and the shuffled scheme; 20. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the data samples of the training data include big data.

3. 3. The computer-implemented method of claim 2, wherein the SGD algorithm used to perform the operations of the combined no-shuffle scheme and the shuffle scheme includes at least one of an Adagrad, Adam, RMSProp, or SGD Momentum algorithm as the no-shuffle scheme.

4. the one or more predetermined criteria include a machine learning rate; the switching to only performing the shuffle scheme occurs according to the learning rate. The computer-implemented method of claim 2.

5. the one or more predetermined criteria include a learning rate; the switching to only performing the shuffle scheme occurs according to the learning rate. The computer-implemented method of claim 2.

6. 3. The computer-implemented method of claim 2, wherein the predetermined criteria include a learning rate rule, and the computer-implemented method further comprises providing one or more learning rate rules that determine one or more of a predetermined theoretical complexity and practical performance of the SGD algorithm.

7. The learning rate rule is: A decreasing learning rate scheme and steps to obtain maximum convergence rate results in non-convex supervised learning models The computer-implemented method of claim 6, comprising:

8. The computer-implemented method of claim 2 , wherein the machine learning model is trained for a deep neural network (DNN) application.

9. The computer-implemented method of claim 2 , wherein the machine learning model is trained for one or more supervised learning operations.

10. 1. A computing device including a configurable machine learning framework for training machine learning models with shuffled gradient operations on big data, the machine learning framework comprising: a processor; a memory coupled to the processor, generating a machine learning framework including the operation of a combined no-shuffle scheme with randomly selected uniform sampling and a shuffle scheme using a stochastic gradient descent (SGD) algorithm, the machine learning framework being further configured to receive a predetermined criterion for switching from the no-shuffle scheme to performing only the shuffle scheme, and to train the machine learning model based on the operation of the combined no-shuffle scheme and the shuffle scheme. a memory storing instructions that cause the processor to perform actions including 1. A computing device comprising:

11. 11. The computing device of claim 10, further comprising receiving a user selection of the predetermined criteria for switching from the no-shuffle scheme to only the shuffle scheme.

12. 11. The computing device of claim 10, further comprising: configuring the machine learning framework to compute the combined shuffle scheme and the no-shuffle scheme from one or more SGD algorithms, including Adagrad, Adam, RMSProp, or SGD Momentum as the no-shuffle scheme.

13. 11. The computing device of claim 10, wherein the predetermined criteria for switching from the no-shuffle scheme to the shuffle-only scheme comprises an error rate.

14. 11. The computing device of claim 10, wherein the predetermined criteria for switching from the no-shuffle scheme to the only shuffle scheme comprises a learning rate.

15. 11. The computing device of claim 10, wherein the machine learning framework is configured to prompt at least one of an error rate or a learning rate as the predetermined criteria for switching from the no-shuffle scheme to performing only the shuffle scheme.

16. 1. A computing device configured for performing shuffled gradients for training machine learning models with big data using stochastic gradient descent (SGD), the computer-implemented method comprising: a processor; a memory coupled to the processor, uniformly randomly distributing data samples; Calculating learning rates for unshuffled and shuffled schemes with randomly selected uniform sampling; performing a combined unshuffled and shuffled scheme on the data samples; switching from the no-shuffle scheme to only the shuffle scheme based on predetermined criteria; training the machine learning model with the data samples based on performing the combined unshuffled and shuffled schemes; a memory storing instructions that cause the processor to perform actions including 1. A computing device comprising:

17. wherein the step of switching to performing only the shuffling scheme occurs according to a learning rate of a machine learning algorithm.

17. The computing device of claim 16.

18. 17. The computing device of claim 16, wherein the step of switching to performing only the shuffling scheme occurs according to a machine learning error rate.

19. 1. A computer-implemented method for determining a shuffled gradient method for training a machine learning model with big data using a stochastic gradient descent (SGD) method, comprising: uniformly randomly distributing data samples of training data; Calculating learning rates for unshuffled and shuffled schemes with randomly selected uniform sampling; Combining the no-shuffle and shuffle schemes into a framework; Implementing a decreasing learning rate scheme on randomly distributed samples in a non-convex supervised learning model to achieve the largest known convergence rate result; 20. A computer-implemented method comprising:

20. 20. The computer-implemented method of claim 19, wherein the machine learning model is trained for a deep neural network (DNN) application.

21. 20. The computer-implemented method of claim 19, wherein the machine learning model is trained for supervised learning operations.

22. 1. A non-transitory computer-readable storage medium tangibly embodying computer-readable program code having computer-readable instructions that, when executed, cause a computing device to perform a shuffled gradient method for training a machine learning model with big data using stochastic gradient descent (SGD), the shuffled gradient method comprising: uniformly randomly distributing data samples; Calculating learning rates for unshuffled and shuffled schemes with randomly selected uniform sampling; performing a combined unshuffled and shuffled scheme on the data samples; switching from the no-shuffle scheme to only performing the shuffle scheme based on one or more predetermined criteria; training the machine learning model with the data samples based on performing the combined no-shuffle scheme and the shuffle scheme; 1. A non-transitory computer-readable storage medium comprising:

23. 23. The non-transitory computer-readable storage medium of claim 22, wherein the predetermined criteria includes a learning rate rule, and the shuffled gradient method further includes providing one or more learning rate rules that determine one or more of a predetermined theoretical complexity and practical performance of an SGD algorithm.

24. 24. The non-transitory computer-readable storage medium of claim 23, wherein providing the learning rate rule comprises operating a decreasing learning rate scheme in a non-convex supervised learning model to achieve a maximum convergence rate result.

25. 23. The non-transitory computer-readable storage medium of claim 22, wherein the machine learning model is trained for one or more supervised learning operations.

Images