Boosting and matrix factorization

Abstract

The present disclosure provides methods, systems, and apparatuses including encoding on a computer storage medium a computer program for presenting a new machine learning model architecture. In some aspects, the method includes obtaining a training dataset having a plurality of training samples, the training dataset including feature variables and output variables. A first matrix is generated using the training dataset, the first matrix being a sparse representation of the training dataset. Generating the first matrix can include generating a categorical representation of numerical features and an encoded representation of categorical features. The method further includes generating a second matrix, a third matrix, and a fourth matrix. Each feature of the first matrix is then represented using a vector including a plurality of tunable parameters. The machine learning model can be learned by adjusting values of the tunable parameters using a combination of a loss function, the fourth matrix, and the first matrix.

Description

Background Technology

[0001] This manual covers data processing and machine learning models.

[0002] Machine learning is a form of artificial intelligence designed to teach computers how to learn and act without being explicitly programmed. More specifically, machine learning is a data analysis approach that involves building and tuning models, allowing computer-executable programs to "learn" from experience. Machine learning involves designing algorithms that tune their models to improve their predictive abilities. This is achieved by first training the machine learning model using historical data (training data) where the outcomes (labels) are known—a process known as supervised learning. During training, the computer can identify rules or relationships and learn the learning parameters of the machine learning model. Then, using new inputs, the machine learning model can generate predictions based on the identified rules or relationships. Summary of the Invention

[0003] Generally, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following operations: obtaining a training dataset comprising multiple training samples, wherein each training sample comprises feature variables and one or more output variables; using the training dataset, generating a first matrix, which is a sparse representation of the training dataset, wherein generating the first matrix comprises: generating a categorical representation of the feature variables based on each numerical feature variable among the feature variables; generating an encoded representation of each categorical feature variable among the feature variables by encoding each categorical feature variable; factorizing the matrix representation of the training dataset to generate one or more matrices comprising a second matrix; generating a third matrix using (i) the second matrix and (ii) a regularization term; generating a fourth matrix based on (i) one or more matrices or (ii) the third matrix; representing each feature of the first matrix using a vector comprising multiple adjustable parameters; and adjusting the values ​​of the adjustable parameters using a combination of (i) a loss function, (ii) the fourth matrix and (iii) the first matrix.

[0004] Other embodiments of this aspect include corresponding apparatus, systems, and computer programs configured to perform aspects of the method and encoded on a computer storage device. These and other embodiments may each optionally include one or more of the following features.

[0005] Methods may include a loss function that provides a result corresponding to a given result provided by a specific loss function of the following form:

[0006]

[0007] Where R is the residual, y i It is the output variable, c is a constant and E is the encoded representation.

[0008] The method may include generating a categorical representation of a feature variable based on each numerical feature variable, the numerical feature variable including: selecting a set of nodes; representing the numerical feature as: (i) a weighted sum of embeddings or (ii) a weighted average of embeddings; using interpolation techniques to generate corresponding weights for the embeddings; and using the corresponding weights to represent each numerical variable in a first matrix.

[0009] The method may include an interpolation technique for generating corresponding weights for an embedding, the embedding including spline interpolation.

[0010] The method may include generating a classification representation of a set of ordinal features included in the training samples, including: performing a discrete Fourier transform (DFT) or a discrete wavelet transform (DWT) on the set of ordinal features; and assigning the classification representation to the set of ordinal features, at least in part, based on the DFT or DWT transform matrix.

[0011] The method may include adjusting the values ​​of adjustable parameters, including iteratively generating sequence models to predict the residuals of the loss function until the residuals of the loss function can no longer be reduced, the model quality metric meets a quality threshold, or the model size has reached a maximum model size threshold.

[0012] The method may include further adjusting the values ​​of the adjustable parameters, and may further include generating pseudo residuals based on the derivative of the loss function.

[0013] The method may include training to predict the combined residuals of previously generated models in a model sequence of each subsequently generated model.

[0014] The method may further include adjusting the values ​​of the adjustable parameters, including at least in part based on the memory constraints of the device training or calling the model, iteratively adjusting the values ​​of the adjustable parameters until the model size reaches a model size threshold.

[0015] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. For example, the techniques discussed herein can generate trained machine learning models based on novel model architectures that require fewer training and computational resources to train and deliver high accuracy compared to other computationally expensive machine learning models that require longer training cycles. Generally, complex machine learning models can learn intricate relationships in training data, but this is less advantageous for devices with low computing power, such as mobile phones. However, this new architecture makes it feasible to learn or predict these intricate relationships using devices with low computing power, making it more efficient than other machine learning models. The latency of inference from complex models using other architectures also makes those other models less advantageous for real-time applications and critical systems in health and emergency situations, while the architecture discussed herein can be used for real-time applications and critical systems in health and emergency situations. In fact, models generated using the techniques discussed herein can be many orders of magnitude smaller and faster than typical neural network models.

[0016] Compared to other existing machine learning models, the machine learning model described in this specification also has a higher degree of interpretability, which further makes it suitable for use in areas such as health where model predictions must be meaningful to human operators, thus providing an improvement over other machine learning models. The machine learning model presented in this paper can also be used for a wide range of modeling problems, including binary and multi-task classification, regression, and ranking, making it suitable for a variety of modeling problems and thus making existing machine learning models more flexible than others that are limited by the types of modeling problems they can be applied to.

[0017] Details of one or more embodiments of the subject matter described in this specification are illustrated in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0018] Figure 1 This is a block diagram of an example training system for a machine learning model.

[0019] Figure 2 This is a flowchart of an example process for generating the first matrix.

[0020] Figure 3 This is a flowchart illustrating an example process of training model parameters for a machine learning model.

[0021] Figure 4 It is a block diagram of an example computer system that can be used to perform the described operations. Detailed Implementation

[0022] This document discloses methods, systems, apparatuses, and computer-readable media located on one or more computers at one or more locations, configured to generate and provide trained machine learning models in response to training a machine learning model on a training dataset.

[0023] As used in the specification and claims, unless the context clearly specifies otherwise, the following terms shall have the following meanings:

[0024] Characteristics: Characteristics are individual measurable properties or characteristics of an observed phenomenon.

[0025] Categorical features: Features are divided into groups. Typically, these features take a finite number of possible values ​​based on the group they are assigned to.

[0026] Numerical features: Features that can be expressed numerically, rather than described in natural language. Sometimes called quantitative features, numerical features are collected in numerical form. What distinguishes numerical features from other data types in numerical form is their ability to perform arithmetic operations on these numbers.

[0027] Ordinal features: These features are categorical data that have a set order or rank for them.

[0028] Splines: Functions composed of piecewise polynomials, each with a specific interval. The points where two piecewise polynomials intersect are called control points or nodes.

[0029] Figure 1 An example training system 100 for a machine learning model 102 is shown. The training system 100 is configured to generate and provide training for the machine learning model 102. The model 102 is configured to receive input and process the input based on the current values ​​of the set of machine learning model parameters to generate an output based on the input. In general, the model 102 can be configured to receive any kind of data input, including but not limited to image, video, audio, and text data, and generate any kind of score, prediction, classification, or regression output based on the input. The output data can be of the same or different type or form as the input data. Some applications of the model are described below; other applications include use in machine control systems.

[0030] For example, when model 102 is configured to process training examples as images or features already extracted from images, the output generated by model 102 for a given image can be a score for each object category in the set of object categories, where each score represents an estimated probability that the image depicts an object belonging to that category.

[0031] Regarding another example, if model 102 is configured to process training examples as Internet resources (e.g., web pages), documents, or parts of documents, or features extracted from Internet resources, documents, or parts of documents, then the output generated by model 102 for a given Internet resource, document, or part of a document can be a score for each topic in a set of topics, where each score represents an estimated probability that the Internet resource, document, or part of a document is relevant to that topic.

[0032] Regarding another example, if model 102 is configured to process training examples as features of the impression context of a particular digital component (e.g., audio, video, text, or other information, including information in the form of advertisements), then the output generated by model 102 may be a score representing the estimated probability that the particular digital component will be clicked.

[0033] Regarding another example, if model 102 is configured to process training examples of features as personalized recommendations for a user—for example, features characterizing the context of the recommendation, such as features characterizing the user's previous actions—then the output generated by model 102 could be a score for each recommendation in the recommendation set, where each score represents an estimated probability that the user will respond positively to the offered recommendation.

[0034] Regarding another example, if model 102 is configured to process training examples of text sequences in one language, the output generated by model 102 can be a score for each text sequence in a set of text sequences in another language, where each score represents an estimated probability that the text sequence in the other language is a correct translation of the input text into the other language.

[0035] Regarding another example, if model 102 is configured to process training examples as sequences representing spoken utterances, the output generated by model 102 can be a score for each text fragment in the set of text fragments, where each score represents an estimated probability that the text fragment is a correct transcribed record of the utterance.

[0036] The training dataset 104 includes multiple training examples. Each training example includes one or more features and a corresponding target output. In response to processing one or more features of a training example, the target output corresponding to the training example refers to the preferred or target output of model 102.

[0037] In some implementations, instead of specifying training inputs and corresponding target outputs, training examples specify a sequence of "experience tuples" that characterize the agent's interactions with the environment across multiple time steps. Each experience tuple includes: (i) an observation characterizing the state of the environment at the corresponding time step, (ii) an action performed by the agent at the corresponding time step, (iii) a subsequent observation characterizing the subsequent state of the environment at the next time step, and (iv) a reward received as a result of the agent performing an action at the time step.

[0038] System 100 can receive training dataset 104 in any manner. For example, system 100 can receive training dataset 104 as an upload from a remote user of the system via a data communication network, for example, using the application programming interface (API) provided by system 100.

[0039] System 100 may include a data analysis engine 110. In some implementations, the data analysis engine 110 may implement techniques and methods for creating or processing one or more features of training examples in training dataset 104 to create and / or select features to improve the predictive power of the learning algorithm. Generally, training data for machine learning models must be presented in a format that allows for model building. Whether it is structured (numerical / categorical / nominal) data or unstructured (text, audio, image, video) data, models typically accept a tabular format, where training examples form rows and features form columns. However, other formats are possible. For illustrative purposes, training dataset 104 may be defined as having a form {x i y i}, ...{x N y N N training examples of}, where x i Let y be the feature vector of the i-th training example, and y i This is the target output. Because the feature vector x i The dimension is fixed, and the training dataset can be represented using tables or matrices.

[0040] Embedding is a natural application for categorical features, but for numerical features, it typically requires converting the continuous values ​​of the numerical features into discrete points in the feature space. In some implementations, these discrete points are referred to as control points or nodes. In some implementations, the data analysis engine 110 uses cubic spline interpolation to process the value of each numerical feature in one or more features of the training dataset 104 into discrete points. Cubic spline interpolation can construct new points within the boundaries of a known set of points. These new points are the function values ​​of the interpolation function (called a spline), which itself consists of multiple cubic piecewise polynomials.

[0041] For illustrative purposes, consider training dataset 104, which includes a numerical feature x and n training examples of a target output y. In some implementations, data analysis engine 110 can use any number of nodes (for example, n+1 nodes) to quantify the valid feature value x, such that {(x...} i Y i ): i=1,...n}, where Y i It is x i The K-dimensional embedding representation, where x0 < x1 < ... < x n .

[0042] In some implementations, for the eigenvalue x k+1 +t values, where 0 ≤ t < x (k+2) -x k+1 The values ​​of numerical features can be represented by an embedding Y (i.e., a vector in the embedding space), which can be represented by {Y}. k Y k+1 Y k+2 Y k+3} and {w k w k+1 w k+2 w k+3 The weighted sum and determination of}. In some implementations, embedding Y can take the following form:

[0043] Y = w k *Y k +w k+1 *Y k+1 +w k+2 *Y k+2 +w k+3 *Y k+3 1

[0044] In some implementations, the weight {w k w k+1 w k+2 w k+3 The following equation can be used to calculate:

[0045]

[0046]

[0047]

[0048]

[0049] Among them, h 00 (t), h 10 (t), h 01 (t), h 11(t) represents the interval [x] . k+1 x k+2 The Hermite basis functions of the cubic Hermite spline interpolated between [ ] can be determined using the following equation:

[0050] Y = h 00 (t)*Y k+1 +h 10 (t)*m k+1 +h 01 (t)*Y k+2 +h 11 (t)*m k+2 6

[0051] h 00 (t)=2*t 3 -3*t 2 +1 7

[0052] h 10 (t)=t 3 -2*t 2 +t 8

[0053] h 01 (t)=-2*t 3 +3*t 2 9

[0054] h 11 (t)=t 3 -t 2 10

[0055]

[0056]

[0057] In some implementations, when the numerical feature x i The value is in the interval [x k+1 x k+2 Within this timeframe, the data analytics engine 110 can use weights w k w k+1 w k+2 w k+3 The embedding of numerical features is represented as individual features from x. i The weighted sum of the four embeddings of the four chosen values. These four values ​​can be x k x k+1 x k+2 and x k+3 These four points are examples of control points and nodes.

[0058] In some implementations, when the numerical feature x iWhen the value is within the interval [x0, x1], the data analysis engine 110 can use equations 13 and 14, in addition to the above equations 6 to 12, to calculate weights w0, w1, and w2 to identify the numerical feature x. i .

[0059]

[0060] m0=(Y1-Y0) / (x1-x0) 14

[0061] When using equations 13 and 14, equations 2, 3, and 4 can be taken in the following forms:

[0062]

[0063]

[0064]

[0065] In some implementations, if the numerical feature x i The value is in the interval [x n-1 x n Within this range, the data analysis engine 110 can use equations 18 and 19, in addition to the equations 6 to 12 mentioned above, to calculate the weight w. n-2 w n-1 and w n To identify numerical features x i .

[0066]

[0067]

[0068] When using equations 18 and 19, equations 2, 3, and 4 can be taken in the following forms:

[0069]

[0070]

[0071]

[0072] In some implementations, the data analytics engine 110 can further optimize (e.g., improve or otherwise adjust) the number and location of nodes. For example, suppose x i，j Let be the j-th numerical feature of the i-th training example. Further assume that the range of observed values ​​for the j-th feature is [min(x...]. i，j ), max(x i，j If x i，j Uniformly distributed in the range [min(x)]i，j ), max(x i，j Within the range, the analysis engine 110 can select nodes with equal distance and range. If x i，j If the values ​​are not evenly distributed within the range, then the data analysis engine 110 can base its analysis on the value x. i，j The density of nodes is selected. For example, the data analysis engine 110 can be based on x. i，j The distribution selects nodes based on quantiles, which allow any x i，j The probability of falling between adjacent node pairs is equal, thus providing sufficient support for learning the embedding.

[0073] In some implementations, the data analysis engine 110 may use the Hadamard transform (HT), a generalization of the Fourier transform (FT), to optimize node selection. In other implementations, in addition to HT, the data analysis engine 110 may also use the Haar wavelet transform (HWT). However, by understanding the operation of the machine learning model 102, appropriate methods and techniques for using HT and HWT will be explained. For illustrative purposes, the following sections of this paper will present the methods and techniques for training the machine learning model 102.

[0074] Figure 2 This is a flowchart of an example process 200 for generating the first matrix. To support the formulation of the training process, the first matrix is ​​referred to as A. For example, the operations of process 200 can be implemented by data analysis engine 110. The operations of process 200 can also be implemented as instructions stored on one or more computer-readable media, which may be non-transitory, and the execution of these instructions by one or more data processing devices can cause one or more data processing devices to perform the operations of process 200. The training dataset 104 is represented using a tabular format including rows and columns, where each row is a training example and each column is a feature of the training example. Of course, other suitable formats can be used.

[0075] In some implementations, the first matrix is ​​a sparse matrix generated by encoding the categorical features of the training dataset 104 using an encoding technique such as one-hot encoding. However, since the training dataset 104 may include other types of features (e.g., numerical and ordinal features), the data analysis engine can process the numerical and ordinal features and convert them into categorical features according to the method described above.

[0076] Data analysis engine 110 selects features from the training dataset (202). For example, data analysis engine 110 may iteratively select features from one or more features of training dataset 104 for processing. In other embodiments, data analysis engine 110 may simultaneously select multiple features from one or more features from training dataset 104 for processing, since the processing of each feature is independent of the processing of other features, thus consuming less computation time.

[0077] Data analysis engine 110 can determine whether the selected feature is a categorical feature (204). For example, data analysis engine 110 can implement methods known in the art to determine whether a feature selected from training dataset 104 is a categorical feature. If the feature is determined to be a categorical feature, data analysis engine 110 can proceed to step 208 of process 200. If the feature is determined not to be a categorical feature, data analysis engine 110 can use step 206 of process 200 to convert the feature into a categorical feature.

[0078] The data analysis engine 110 can transform the values ​​of non-categorical features into discrete points in the feature space (206). For example, after determining that a particular feature is not a categorical feature (i.e., the particular feature is a numerical feature), the data analysis engine 110 can use a cubic spline interpolation method (described above) to transform the value of the particular feature into a weighted average based on the nodes. For example, numerical feature x i k-dimensional embedding of Y can be used i express.

[0079] The data analysis engine 110 can generate encoded representations of features (208). For example, the data analysis engine 110 can use methods such as one-hot encoding to generate encoded representations of categorical features. Generally, one-hot encoding converts categorical features into vectors of 1s and 0s depending on the category of a particular feature. For example, if a particular feature identifying gender takes values ​​for male and female, the one-hot encoded representation could have two features such that if the particular feature is male, the first feature of the representation would have a value of 1 (and the second feature would have a value of 0). Correspondingly, if the particular feature is female, the first feature of the representation would have a value of 0 (and the second feature would have a value of 1).

[0080] Note that prior to step 208 of process 200, all features in the training dataset are categorical features, because features that are not categorical are processed in step 206 to generate categorical representations. It should also be noted that even though step 208 has been described as using one-hot encoding to generate coded representations of categorical features, any suitable encoding technique can be used for this purpose.

[0081] In some implementations, after generating the first matrix, the data analysis engine 110 can use singular value decomposition (SVD) to decompose the first matrix into one or more matrices. The SVD method can be represented by the following equation:

[0082] A=U*∑*V T twenty three

[0083] Where A is an n×m matrix such that n > m, U is an n×m orthogonal matrix, ∑ is an m×m diagonal matrix (also called the second matrix), and V is an m×m orthogonal matrix.

[0084] In some implementations, the data analysis engine 110 can calculate the third matrix ∑ -1 ∑ is the inverse matrix of the matrix ∑ using the regularization term λ. -1 The following equation can be used to calculate:

[0085]

[0086] Where Σ=diag(σ1,…σ r And σ1,…σ r It is a singular value.

[0087] In some implementations, Equation 24 can be used to calculate the inverse of the first matrix (also known as the fourth matrix) using Equation 25:

[0088]

[0089] In some embodiments, system 100 may further include a training engine 108, which may include one or more processors and is configured to train model 102 based on loss function 106 at each iteration. In some embodiments, training engine 108 may train model 102 by adjusting the values ​​of model parameters 103 according to their current values ​​to reduce the loss value generated by loss function 106.

[0090] In some implementations, the data analysis engine 110 can map each feature of the training dataset 104 to an embedding representation of dimension K, which is determined based on the unique value and / or level of the feature (also known as the cardinality of the feature). For example, if the training dataset 104 has N training examples and each training example has M features, the value of the j-th feature of the i-th training example can be represented using an embedding E of dimension K. i,j It means that, among them, E i,j ∈R K It should be noted that for categorical features x... j For each unique value of feature j, there exists |x| that results in feature j. j|Unique Embedding E j , where |x j | is the cardinality of the classification feature j.

[0091] In some implementations, during the training of the machine learning model 102, the machine learning model can make predictions according to the following equation:

[0092]

[0093] Where C can be a constant, and can be defined as:

[0094]

[0095] The training engine 108 can be trained by adjusting the trainable parameter c. k The values ​​and embeddings are used to train a machine learning model using K iterations. For example, in the k-th iteration of the training process, the training engine can compute each |x| for all features. j |Trainable parameters c of the embedding representation k And the kth value.

[0096] Generally, the residual of a machine learning model is the difference between the model's prediction and the target output. According to machine learning model 102, the residual of the k-th iteration of the training process can be calculated using equations 26 and 27. The residual can be expressed in the following form, where y i It is the target output of sample i.

[0097]

[0098] In some implementations, the target output of each subsequent training iteration can be the residual resulting from the previous iteration. For example, the target output y during the k-th iteration. i The residual R is calculated using Equation 28 in the (k-1)th training iteration. k-1，i Logically, this means that each training iteration attempts to learn the model parameters based on the residuals (also known as prediction errors) of previous training iterations. This training process can generate an overdetermined system with N equations, which can be expressed as:

[0099]

[0100] Where i = 1, 2, ... N.

[0101] In some implementations, an overdetermined system of N equations can be transformed into a logarithmic space, for example, by using c k Move to the right side of equation 29 and take the logarithm of both sides of the resulting equation, as shown in equation 30:

[0102]

[0103] Where, for any 1≤i≤N, R k-1，i -c k >0.

[0104] It should be noted that Where X is each |x| of all features concatenated with the columns corresponding to the first matrix. j |The k-th value of the embedding representation. Using Equation 30, the overdetermined system of equations N can be further expressed as:

[0105] A*X=B 31

[0106] Among them, B i =log(R) k-1，i -c k ).

[0107] It should be noted that Equation 31 can be analogous to the formula for a classic linear regression problem without an intercept. In some implementations, Equation 31 can be modified to [A 1]*[X b] by adding an extra column to the first matrix A. T =B. In some implementations, adding an extra column to the first matrix A can be achieved by adding a categorical feature with a single value to each training example in the training dataset 104.

[0108] In some implementations, the training engine 108 can compute X in equation 31 of the overdetermined system. This is because A*X = B and A = U*∑*V T The following equation can be derived:

[0109] U*∑*V T *X=B 32

[0110] This led to

[0111] V*∑ -1 *U T *U*∑*V T *X=V*∑ -1 *U T *B 33

[0112] Due to matrix U*U T =I, V*V T =I and ∑*∑ T =I, Equation 33 can be used to derive X as follows:

[0113] X=V*∑ -1 *U T *B=A -1 *B 34

[0114] Among them, A- 1 Calculate using equation 25, and B i =log log(R) k-1，i -c k ).

[0115] In some implementations, the training engine 108 may select c based on the following heuristics. k The value of .

[0116] If min(R) k-1，i )-c k >1 and log(max(R) k-1，i )-c k )>log(min(R k-1，i )-c k If ) > 0, then c k The value can be increased to decrease log(max(R)). k-1，i )-c k ) and max((log(R k-1，i -c k )) 2 ).

[0117] If max(R) k-1，i )-c k <1 and 0> log(max(R) k-1，i )-c k )>log(min(R k-1，i )-c k ), then c k The value can be reduced to decrease max((log(R)). k-1，i -c k )) 2 ).

[0118] when At that time, max((log(R) k-1，i -c k )) 2 The value of ) can be reduced to a minimum so that

[0119]

[0120] In some implementations, the training engine 108 can further select c k The optimal value of c is obtained by intuitively optimizing c. k The choice, |A*XB| 2 The value of can be minimized. This can be achieved using the function L(c k ) indicates that, among which:

[0121] L(c k ) = BT *(IA*(A T *A) -1 *A T )*B 35

[0122] By calculating C = IA * (A) T *A) -1 *A T Equation 35 can be taken in the following form:

[0123] L(c k ) = B T *C*B=∑ i ∑ j C i，j *log(R k-1，i -c k )*log(R k-1，j -c k 36

[0124] In some implementations, the training engine 108 can use gradient descent as a first-order iterative optimization algorithm to find the minimum L(c k ) of c k The value of . The gradient descent algorithm uses the function L(c k The first derivative of a function. The first derivative of a function takes the following form:

[0125]

[0126] In some implementations, the training engine 108 can use Newton's method or stochastic Newton's method as a second-order iterative optimization algorithm to find the minimum L(c k ) of c k The value of . Newton's method and stochastic Newton's method use the function L(c k The second derivative of a function is given by the following form:

[0127]

[0128] This optimization c k The techniques for determining the value of c may differ from those used in neural networks, for example, because here, each iteration only has a single unknown c. k This leads to a univariate optimization problem, whereas neural network optimization problems may require handling millions or billions of variables and finding local minima for hundreds or millions of variables. Therefore, it is easy to see that this optimization is less efficient than neural network optimization. k The technology provides a significant reduction in resources and training time.

[0129] In some implementations, the training engine 108 can directly use the stochastic gradient descent (SGD) algorithm to optimize the cost function. The cost function of the machine learning model can have the following form:

[0130]

[0131] In some implementations, the training engine 108 may use a coordinate descent algorithm to optimize the trainable parameters. In this implementation, the coordinate descent algorithm can compute a new value of C, where the new value C' can be computed using the following equation:

[0132]

[0133] In some implementations, the training engine 108 can compute embedded representations of the values ​​of specific features using an overdetermined system. For example, the overdetermined system A*X = B can be solved using QR decomposition into X. In this case, the solution X is a K-dimensional vector, and log(max(X)) = ... k′ Let ∈ be the new value of the k-th element in the embedding, and ∈ be a small positive number. This overdetermined system can be represented as follows:

[0134]

[0135]

[0136] Where, x i，q =v and p i ... = {0, 1, 2, ...}

[0137] In some implementations, the termination criterion for training the machine learning model 102 can be determined based on the residual after the k-th iteration of the training process. For example, if the iterative optimization method described above cannot further minimize the size of the residual, the training process can be terminated, and the current values ​​of the dimension K, constant C, and model parameters including all features and all embedded representations can be considered as the final values.

[0138] In some implementations, the termination criterion for the training process can be machine learning prediction accuracy, which can be measured using metrics such as precision, retrieval rate, F1 score, or PR AUC. For example, training engine 108 can terminate the training process when the accuracy of machine learning model 102 with the current model parameters reaches a predetermined threshold.

[0139] Returning to node selection using HT and HWT. For each dimension in the K dimensions, the data analysis engine 110 and the training engine 108 learn a function f′, which will fall within the range [min(x...]. i，j ), max(x i，jThe eigenvalues ​​x within )] i，j Mapping to floating-point numbers. Let's assume f represents the actual unknown smoothing function to be learned. For equally spaced nodes x0 < x1 < ... < x... n Where, x0 = min(x i，j And x n =max(x i，j The above method learns a cubic spline f′ that is sufficiently close to f. To evaluate the closeness, we evaluate any x i f and f′ at x0, where x0 < x i <x n Among them, the |f(x) of a well-trained model i )-f′(x i To represent a cubic spline with m+1 nodes, the model needs to learn m+1 parameters, i.e., {f′(x)} o ), f′(x1), ... f′(x) m If the selected value of m is significantly greater than the required value, then the model parameters {f′(x0), f′(x1), ... f′(x)} will be used. m There may be a lot of redundancy in the model parameters {f′(x0), f′(x1), ... f′(x)}. We might be able to compress the model parameters {f′(x0), f′(x1), ... f′(x)}. m This means reducing the model size without reducing the quality of the cubic spline fitness. If we define {f′(x0), f′(x1), ... f′(x)} as {f′(x0), f′(x1), ... f′(x)}, then we can achieve this. m If we consider a one-dimensional continuous signal sampled at m+1 points, we can convert it to the frequency domain using the Hadamard transform (HT), a generalization of the Fourier transform (FT), or the Haar wavelet transform (HWT), a possible wavelet transform. Continuous one-dimensional signals can generally be compressed better in the frequency domain.

[0140] In some implementations, the data analysis engine 110 can use the Hadamard transform to convert the function f to the frequency domain (denoted as F). The transformation matrix of the Hadamard transform can take the following form:

[0141]

[0142] Where i·j is the bitwise product of the binary representations of digits i and j. Since the transformation from f to F is equivalent to the projection vector {f′(x0), f′(x1), ... f′(x2)}, ..., f′(x3)}, ..., f′(x4)}, ..., f′(x5)}, ..., f′(x6)}, ..., f′(x7)}, ..., f′(x8)}, ..., f′(x9)} m The following relationship holds:

[0143] H n *{f′(x0), f′(x1),...f′(x m )} T={F′(x0), F′(x1),...F′(x m )} 43

[0144] Where m = 2 n .

[0145] In some implementations, training engine 108 can train machine learning model 102 in the frequency domain. For example, equations 6 to 12 and 44 can be used to conclude that each Y is all F′(x0), F′(x1), ... F′(x2) m A linear combination of ). Then, training engine 108 can construct the first matrix and directly use equations 31 to 38 to calculate F′(x0), F′(x1), ... F′(x2). m The value of ).

[0146]

[0147] When training the machine learning model 102 in the frequency domain, each element in the learned embedding is a coefficient of frequency in the frequency domain (i.e., the weights of a linear combination). For elements with very small values, the training engine 108 can set these element values ​​to 0 to reduce the number of model parameters, model capacity, and avoid overfitting. Compression in the frequency domain may be more effective.

[0148] In some implementations, the application of regularization, such as Equation 24, can reduce the element values ​​in the learned embeddings to 0. For elements with very small values, the training engine 108 can set these element values ​​to 0 to reduce the number of model parameters, model capacity, and avoid overfitting.

[0149] In some implementations, if the features in the training dataset 104 include ordinal features, the data analysis engine 110 can use the Hadmar transform method (as described above). In this implementation, the number of unique values ​​for the ordinal features can be increased to a certain extent. Ceiling(x) returns the smallest integer not less than x.

[0150] Generally, machine learning models can be used for a variety of tasks, such as regression, classification, and ranking. The following sections describe how the machine learning model 102 described above can be used for such tasks.

[0151] Multi-class classification

[0152] Assume there exists the form {x} i , l i}, ...{x N , l N The training dataset consists of N training examples of 102, where xi Let l be the feature vector of the i-th training example, and l i It is the corresponding target label, where l i It is an instance of all possible labels L, i.e., l i ∈L. Further assume that the number of features for each training example is M-1, i.e., x i，1 , ...x i，M-1 .

[0153] The multi-class classification problem is to find a classifier F to predict the class labels of training examples, i.e., F(x) i，1 , ...x i，M-1 )=l i The multi-class classification problem can be reformulated as a regression problem for each class, i.e., to find a classifier F' such that if x... i，M =l i Then F′(x) i，1 , ...x i，M ) = 1, otherwise for all x i，M ∈L, it is 0.

[0154] During training, the data analysis engine 110 can transform the N training examples of the training dataset 102 into N*L training examples, where the N training examples all have the target label 1, and the N*(L-1) training examples have the label 0. If the classification problem is caused by an imbalanced dataset, weights can be assigned to the training examples. For example, a hyperparameter δ > 0 can be defined, and the generated training examples and labels can be modified as follows. For all x i，M ∈L,

[0155] If x i，M =l i Then δ*(L-1)*F(x) i，1 , ...x i，M-1 x i，M )=δ*(L-1)

[0156] Otherwise F′(x) i，1 , ...x i，M-1 x i，M ) = 0 45

[0157] The weight of the training example with label 1 is δ*(L-1). It should be noted that choosing δ=1 will result in a perfectly balanced binary classification problem. It should also be noted that increasing the value of δ reduces the within-class variance of F's predictions.

[0158] Then, the optimal classifier F′ can be computed using Equation 26 and the corresponding solution described in this paper. It should be noted that during training, for each label l... i∈L will have a K-dimensional embedding. To predict the most likely label, Equation 26 can be reformulated as the dot product of two vectors of dimension K+1, where the label with the largest dot product is most likely the training example x. i，1 , ...x i，M-1 The tag.

[0159] The first of the two vectors (presented below) can be independent of the possible labels and is only related to the i-th training example x. i，1 , ...x i，M-1 The value of the feature is related:

[0160] {C, p1, ..., p K}in The second of the two vectors (presented below) can be the embedding of the label, and is independent of the i-th training example x. i，1 , ...x i，M-1 The value of the feature:

[0161] {1, q1, ..., q} K}in

[0162] Multi-task classification / regression

[0163] Suppose we have a training dataset 102 with N training examples, where each training example has a feature x. i，1 , ...x i，M-1 and tags Where {t1, ... t T} is a task.

[0164] During training, the data analysis engine 110 can define values ​​{t1, ... t}. T The virtual classification feature vector X m This transforms the multi-task problem into an equivalent single-task problem. The i-th training example can be further extended to T training examples as follows:

[0165]

[0166] Then, the training engine 108 can train the function F according to the above techniques and methods to learn the K-dimensional embedding for each task t. t It is similar to the multi-class classification problem (mentioned above), where each label has an embedding.

[0167] Similar to multi-class classification problems, the training engine 108 can compute vectors {p1, ... p1}. K},in, Independent of possible tasks, and only with the i-th training example xi，1 , ...x i，M-1 The values ​​of the features are related. Then, the prediction result for task t can be calculated using the following formula:

[0168] C+{p1,...p k}⊙embedding t 49

[0169] Sort

[0170] In general, the learning ranking problem can be formulated as an item-free set of contextual features {X1, ... X}. M} and the item-specific feature set {Y1, ... Y} N}. Typically, N=1, and Y1 is a classification feature. During training, a pairwise approach is used, and each training example includes (1) contextual features {X1, ... X1}. M The value of}, (2) the first term {y 1，1 , ...y 1，n The value of a specific characteristic of the term}, (3) the second term {y 2，1 , ...y 2，n The values ​​of the item-specific features and (4) indicate the order between the two items, i.e., if the first item should be ranked lower than the second item, L = -1, otherwise L = 1. When querying, each query includes (1) context features {x1, ... x M The value of} and (2) Z terms, whose term-specific characteristics have the value {y z，1 , ...y z，n}, where z∈[1,Z].

[0171] During training, the training engine 108 can define binary operators between two terms, the result of which is based on whether the left-hand side (LHS) of the operator should be ordered higher than the right-hand side (RHS) of the operator, whether true or false.

[0172] For {Y1, ... Y} N For each item-specific feature in the data analysis engine 110, two item-specific features can be defined to encode LHS Vs RHS, i.e., {LHS_Y1, ... LHS_Y}. N} and {RHS_Y1, ...RHS_Y N}, where Y i LHS_Y i and RHS_Y i Features have the same data type and their values ​​have the same semantics. The feature set in machine learning model 102 can be of size M+2*N in the form {X1, ... X...}. M, LHS_Y1, ... LHS_Y N ,RHS_Y1,...RHS-Y N}

[0173] For each training example {{x1, ... x...} M}, {y 1，1 , ...y 1，n}, {y 2，1 , ...y 2，n The training engine 108 can process four training examples. The training examples are as follows:

[0174] {x1, ... x} with label L m y 1，1 , ...y 1，n y 2，1 , ...y 2，n}

[0175] {x1, ... x} with label -L m y 2，1 , ...y 2，n y 1，1 , ...y 1，n}

[0176] {x1, ... x} with label 0 m y 1，1 , ...y 1，n y 1，1 , ...y 1，n}

[0177] {x1, ... x} with label 0 m y 2，1 , ...y 2，n y 2，1 , ...y 2，n 50

[0178] It should be noted that the first training example can be the original training example, and the second training example can be a negative example of the first training example, which has the observation that if the two items are swapped, the prediction of model 102 should be reversed. Training examples 3 and 4 are constraints indicating that two similar items should be ranked the same.

[0179] Suppose the values ​​of the context features in the query are {x1, ... x}. m The query requires sorting x items, i.e., {z1, ... z2}. x}, its specific characteristics have values Among them, z i ∈{z i , ...z xGiven two arbitrary terms z i z j ∈{z1,...z x The training engine 108 calculates the function F as follows: + and F - :

[0180]

[0181]

[0182] Then, the training engine 108 can be defined.

[0183] cap(x,minValue,maxValue)=min(max(x,minValue),maxValue)

[0184] This can be obtained

[0185]

[0186] Where -1≤F(z) i , z j )≤1. F(z) i , z j ) is z i The sorting is better than z j The reward.

[0187] Then, training engine 108 can find {z i ,....z x The arrangement of {z′} is the one that maximizes the overall reward. i ,...z′ x}, that is, ∑ 1＜i≤j≤x F(z′ i , z′ j ).

[0188] Figure 3 This is a flowchart of an example process 300 for training a machine learning model. The operations of process 300 can be, for example, performed by... Figure 1 The system 100 is implemented using a data analysis engine 110 and a training engine 108, or one or more data processing devices. The operation of process 300 can also be implemented as instructions stored on a computer-readable medium, wherein execution of the instructions can cause one or more data processing devices to perform the operation of process 300.

[0189] System 100 obtains a training dataset (302) comprising multiple training samples. For example, data analysis engine 110 may receive training dataset 104. Training dataset 104 comprises multiple training examples. Each training example includes one or more features and a corresponding target output. In response to processing one or more features of a training example, the target output corresponding to the training example refers to the preferred output of model 102. For example, training dataset 104 may be defined as having the form {x} i y i}, ...{x N y N N training examples of}, where x i Let y be the feature vector of the i-th training example, and y i This is the target output. Because the feature vector x i The dimension is fixed, and the training dataset can be represented using tables or matrices.

[0190] Data analysis engine 110 generates a first matrix (304) as a sparse representation of the training dataset. For example, data analysis engine 110 may iteratively select features from one or more features of the training dataset 104 for processing. The data analysis engine determines whether the selected features are categorical features. If a feature is determined to be non-categorical, data analysis engine 110 transforms the values ​​of the non-categorical feature into discrete points in the feature space. For example, after determining that a particular feature is not a categorical feature (i.e., the particular feature is a numerical feature), data analysis engine 110 may use a cubic spline interpolation method to transform the particular feature into discrete points. In some cases, Fourier transform (FFT) and / or wavelet transform can be used to train feature embeddings in the frequency domain to achieve better model compression, i.e., reducing the number of model parameters and preventing overfitting with regularization. For example, FFT (and / or wavelet) can be applied to training in the frequency domain, where the embeddings are coefficients (i.e., weights) of a linear combination of frequencies.

[0191] The data analysis engine 110 can generate encoded representations of features using methods such as one-hot encoding to generate encoded representations of categorical features. Generally, one-hot encoding converts categorical features into vectors of 1s and 0s based on the category of a particular feature. For example, if a specific feature identifying gender takes values ​​for male and female, the one-hot encoded representation will have two features such that if the specific feature is male, the first feature of the representation will have a value of 1 (and the second feature will have a value of 0). Correspondingly, if the specific feature is female, the first feature of the representation will have a value of 0 (and the second feature will have a value of 1). After encoding each feature of the training dataset 102, the result can be represented using a table or matrix called the first matrix.

[0192] The data analysis engine 110 can factorize the matrix representation of the dataset, referred to as the first matrix (306). For example, after generating the first matrix, the data analysis engine 110 uses singular value decomposition (SVD) to decompose the first matrix into one or more matrices. The SVD method can be expressed using the following equation: A = U * ∑ * V T , where A is an n×m matrix such that n>m, U is an n×m orthogonal matrix, ∑ is an m×m diagonal matrix (also called the second matrix), and V is an m×m orthogonal matrix.

[0193] The data analysis engine 110 can generate a third matrix (308) using the second matrix and a regularization term. For example, the data analysis engine 110 can compute the third matrix ∑ -1 ∑ is the inverse matrix of the matrix ∑ using the regularization term λ. -1 The following equation can be used to calculate:

[0194]

[0195] Among them, ∑=diag(σ1,...σ r ), and σ1,...σ r It is a singular value.

[0196] Data analysis engine 110 can generate a fourth matrix (310) based on one or more matrices and a third matrix. For example, equation 24 can be used to calculate the inverse of the first matrix (also called the fourth matrix) using equation 25:

[0197]

[0198] Data analysis engine 110 can represent each feature of the training dataset using a vector that includes a predetermined number of adjustable parameters (312). For example, data analysis engine 110 maps each feature of training dataset 104 to an embedding representation of dimension K, which is determined based on the unique value and / or level of the feature (also known as the cardinality of the feature). For example, if training dataset 104 has N training examples and each training example has M features, the value of the j-th feature of the i-th training example can be represented using an embedding E of dimension K. i，j It means that, among them, E i，j ∈R K It should be noted that for feature x... j The relatively small cardinality is beneficial for categorical features x. j For each unique value of feature j, there exists |x| that results in feature j. j |Unique Embedding E j , where |x j| represents the cardinality of the classification feature j. For features with relatively high cardinality, the data analysis engine 110 can divide the features with relatively high cardinality into features with relatively low cardinality. For example, feature x j It can be divided into x j1 x j2 , ...x jq In this case, feature x can use q unique embeddings E j express.

[0199] Training engine 108 can adjust the values ​​of adjustable model parameters of machine learning model 102 (314). In some implementations, adjustment may include iteratively training the sequence model to predict the residuals of the loss function until the loss can no longer be reduced, a metric of model quality meets a quality threshold, or the model size has reached a maximum model size threshold (e.g., based on the memory constraints of the device training or calling the model). For example, training engine 108 adjusts the trainable parameter c k The values ​​and embeddings are used to train a machine learning model using K iterations. Based on machine learning model 102, the residual of the k-th iteration of the training process can be calculated using equations 26 and 27. The residual can be represented in the following form:

[0200]

[0201] Among them, y i It is the target output of sample i.

[0202] In some implementations, each subsequent trained model can be trained to predict the combined residuals of previously generated models in the model sequence. In other words, the target output of each subsequent training iteration is the residual from the previous iteration. The training process can generate an overdetermined system of N equations, which can be converted to logarithmic space using the following equations:

[0203]

[0204] Where, for any R 1≤i≤N k-1，i -c k >0.

[0205] The overdetermined system of N equations can be further expressed as:

[0206] A*X=B

[0207] Among them, B i =log(R) k-1，i -c k ).

[0208] The training engine 108 can use the following equation to calculate X in the overdetermined system A*X=B:

[0209] X=V*∑ -1 *U T *B=A -1 *B 34

[0210] Among them, A -1 Calculate using equation 25, and B i =log log(R) k-1，i -c k ).

[0211] In some implementations, without applying Equation 34, the training engine 108 can compute X using an iterative method with selected regularization techniques, such as L1 regularization (aka Lasso regression) or L2 regularization (aka ridge regression). The training engine 108 can choose c based on the following heuristic or other suitable heuristics. k The value of c. The training engine 108 can be further optimized based on intuition. k The choice, that is, by choosing c k The optimal value can be minimized by |A*XB|. 2 The value of . This can be achieved using the function L(c k ) indicates that among them

[0212] L(c k ) = B T *(IA*(A T *A) -1 *A T )*B 35

[0213] By calculating C = IA * (A) T *A) -1 *A T Equation 35 can be in the form of:

[0214] L(c k ) = B T *C*B=∑ i ∑ j C i，j *log(R k-1，i -c k )*log(R k-1，j -c k 36

[0215] In some implementations, the training engine 108 can use gradient descent as a first-order iterative optimization algorithm to find the minimum L(c k ) of c k The training engine 108 can also use Newton's method, or stochastic Newton's method as a second-order iterative optimization algorithm, to find the value of L(c). k ) of c kThe training engine 108 can also directly use the stochastic gradient descent (SGD) algorithm to optimize the cost function. The training engine 108 can also use the coordinate descent algorithm to optimize the trainable parameters.

[0216] The training engine 108 can compute embedded representations of the values ​​of specific features using overdetermined systems. For example, the overdetermined system A*X = B can be solved by using QR decomposition to X.

[0217] In some implementations, the error function may differ from the mean squared error (MSE). In these implementations, in addition to using the residuals as training labels in each of the K steps, the training engine 108 can compute pseudo-residuals, i.e., representing A. i The derivative of the loss function of the true intermediate variable of the inner product of *X, where A i It is the row of the first matrix corresponding to the i-th training example. Let B i This is represented as the pseudo-residual computed during the i-th training iteration. The training engine 108 can solve the overdetermined system using any linear regression method described in the previous sections. A*X=B .

[0218] In some implementations, at each of the K steps in the iterative training process, the training engine 108 can explicitly represent the intercept of the linear regression result as...

[0219] In some implementations, the training engine 108 can explicitly define the learning rate, for example, γ, and change the objective function from... Modified to Where average(y) is the average value of the labels in the training dataset. In some implementations that support intercept and learning rate, the objective function can be...

[0220] By introducing a fictitious single-valued classification feature with a valid value, the above equation can be equated to: in, Furthermore, the k-th dimension in the embedding of the fictitious single-valued categorical feature is log(γ) + intercept. k .

[0221] Figure 4This is a block diagram of an example computer system 400 that can be used to perform the above operations. System 400 includes a processor 410, memory 420, storage device 430, and input / output device 440. Each of components 410, 420, 430, and 440 can be interconnected, for example, using system bus 450. Processor 410 is capable of processing instructions executed within system 400. In one embodiment, processor 410 is a single-threaded processor. In another embodiment, processor 410 is a multi-threaded processor. Processor 410 is capable of processing instructions stored in memory 420 or on storage device 430.

[0222] The memory 420 stores information within the system 400. In one embodiment, the memory 420 is a computer-readable medium. In one embodiment, the memory 420 is a volatile memory cell. In another embodiment, the memory 420 is a non-volatile memory cell.

[0223] Storage device 430 provides massive storage for system 400. In one embodiment, storage device 430 is a computer-readable medium. In various other embodiments, for example, storage device 430 may include a hard disk drive, an optical disk drive, a storage device shared by multiple computing devices (e.g., cloud storage devices) over a network, or some other high-capacity storage device.

[0224] Input / output device 440 provides input / output operations for system 400. In one embodiment, input / output device 440 may include one or more of the following: a network interface device such as an Ethernet card, a serial communication device such as an RS-232 port, and / or a wireless interface device such as an 802.11 card. In another embodiment, the input / output device may include a driver device configured to receive input data and send output data to other input / output devices, such as a keyboard, printer, and display device 370. However, other embodiments may also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0225] Although the example processing system already exists Figure 4 While described herein, the implementation of the subject matter and functional operations described herein may be implemented in other types of digital electronic circuit systems or in computer software, firmware, or hardware—including the structures disclosed herein and their structural equivalents—or in combinations of one or more of them.

[0226] Electronic documents (for brevity, we will simply call them documents) do not necessarily correspond to files. Documents can be stored in a section of a file that contains other documents, in a single file dedicated to the document being discussed, or in multiple coordinating files.

[0227] Embodiments of the subject matter and operations described in this specification may be implemented in digital electronic circuit systems or in computer software, firmware, or hardware—including the structures disclosed in this specification and their structural equivalents—or in combinations of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagating signal—e.g., a machine-generated electrical, optical, or electromagnetic signal generated for encoding information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or may be included therein. Furthermore, when the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in an artificially generated propagating signal. The computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0228] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0229] The term "data processing apparatus" includes various means, devices, and machines for processing data, such as programmable processors, computers, systems-on-a-chip (SoCs), or combinations of SoCs or the aforementioned components. Apparatus may include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. Apparatus and execution environments can implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0230] A computer program (also called a program, software, software application, script, or code) can be written in any programming language, including assembly or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but is not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document) or as a single file or multiple coordinating files specifically for the program in question (e.g., a file storing one or more modules, subroutines, or code sections). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a communication network.

[0231] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform actions by manipulating input data and generating outputs. The processes and logic flows can also be executed by a dedicated logic circuit system (e.g., a FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit)), and the apparatus can also be implemented as such a dedicated logic circuit system.

[0232] For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors. Generally, a processor receives instructions from read-only memory or random access memory, or both. The basic components of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to such mass storage devices, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by a dedicated logic circuit system or can be incorporated into that dedicated logic circuit system.

[0233] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound input, voice input, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents from the device used by the user, for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0234] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component as an application server, or a frontend component, such as a client computer having a graphical user interface or a web browser through which a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more such backend components, middleware components, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium—for example, a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0235] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a client device (e.g., to display data to a user interacting with the client device and to receive user input from that user). Data generated at the client device (e.g., the result of user interaction) may be received from the client device at the server.

[0236] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this specification within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while features may be described above as functioning in certain combinations and initially even being claimed in the same way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0237] Similarly, although the operations are depicted in a specific order in the diagrams, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or as performing all the operations shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0238] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, comprising: Obtain a training dataset that includes multiple training samples, where each training sample includes feature variables and one or more output variables; Generating a first matrix using the training dataset, wherein the first matrix is ​​a sparse representation of the training dataset, and generating the first matrix includes: Generate a categorical representation of the feature variables based on each numerical feature variable in the feature variables; An encoded representation of each categorical feature variable is generated by encoding each categorical feature variable. Factorize the matrix representation of the training dataset to generate one or more matrices including the second matrix; The third matrix is ​​generated using (i) the second matrix and (ii) the regularization term; A fourth matrix is ​​generated based on (i) one or more matrices and (ii) the third matrix; Each feature of the first matrix is ​​represented by a vector comprising multiple adjustable parameters; and The value of the adjustable parameter is adjusted using a combination of (i) the loss function, (ii) the fourth matrix, and (iii) the first matrix. The adjustment of the adjustable parameter includes iteratively generating a sequence model to predict the residuals of the loss function until the residuals of the loss function can no longer be reduced, the model quality metric meets a quality threshold, or the model size has reached a maximum model size threshold.

2. The computer-implemented method according to claim 1, wherein, The loss function is a loss function that provides a result corresponding to a given result provided by a specific loss function of the following form: Where R is the residual, y i The output variable is c, which is a constant, and E is the encoded representation.

3. The computer-implemented method according to claim 1, wherein, Generating a categorical representation of each numerical feature variable based on the following includes: Select the set of nodes; The numerical features are represented as: (i) a weighted sum of the embeddings, or (ii) a weighted average of the embeddings; Interpolation techniques are used to generate the corresponding weights for the embedding; and The corresponding weights are used to represent each numerical variable in the first matrix.

4. The computer-implemented method according to claim 3, wherein, The interpolation technique for generating the corresponding weights of the embedding includes spline interpolation.

5. The computer-implemented method according to claim 1, further comprising: Generating a classification representation of the ordinal feature set included in the training samples includes: Perform Discrete Fourier Transform (DFT) or Discrete Wavelet Transform (DWT) on the ordinal feature set; and The classification representation is assigned to the ordinal feature set, at least in part, based on the DFT or DWT transformation matrix.

6. The computer-implemented method according to claim 1, wherein, Adjusting the value of the adjustable parameter further includes generating pseudo residuals based on the derivative of the loss function.

7. The computer-implemented method according to claim 5, wherein, Each subsequently generated model is trained to predict the combined residual values ​​of the previously generated models in the model sequence.

8. The computer-implemented method according to claim 1, wherein, Adjusting the value of the adjustable parameter includes, at least in part, adjusting the value of the adjustable parameter based on the memory constraints of the device that trained or invoked the model, iteratively until the size of the model reaches a model size threshold.

9. A system comprising: Memory that stores instructions; as well as A processor communicatively coupled to the memory, wherein the instructions, when executed by the processor, cause the processor to perform operations, the operations including: Obtain a training dataset that includes multiple training samples, where each training sample includes feature variables and one or more output variables; Generating a first matrix using the training dataset, wherein the first matrix is ​​a sparse representation of the training dataset, and generating the first matrix includes: Generate a categorical representation of the feature variables based on each numerical feature variable in the feature variables; An encoded representation of each categorical feature variable is generated by encoding each categorical feature variable. Factorize the matrix representation of the training dataset to generate one or more matrices including the second matrix; The third matrix is ​​generated using (i) the second matrix and (ii) the regularization term; A fourth matrix is ​​generated based on (i) one or more matrices and (ii) the third matrix; Each feature of the first matrix is ​​represented by a vector comprising multiple adjustable parameters; and The value of the adjustable parameter is adjusted using a combination of (i) the loss function, (ii) the fourth matrix, and (iii) the first matrix. The adjustment of the adjustable parameter includes iteratively generating a sequence model to predict the residuals of the loss function until the residuals of the loss function can no longer be reduced, the model quality metric meets a quality threshold, or the model size has reached a maximum model size threshold.

10. The system according to claim 9, wherein, The loss function is a loss function that provides a result corresponding to a given result provided by a specific loss function of the following form: Where R is the residual, y i The output variable is c, which is a constant, and E is the encoded representation.

11. The system according to claim 9, wherein, The classification representation of the feature variable generated based on each numerical feature variable includes: Select the set of nodes; The numerical features are represented as: (i) a weighted sum of the embeddings, or (ii) a weighted average of the embeddings; Interpolation techniques are used to generate the corresponding weights for the embedding; and The corresponding weights are used to represent each numerical variable in the first matrix.

12. The system according to claim 11, wherein, The interpolation technique for generating the corresponding weights of the embedding includes spline interpolation.

13. The system according to claim 9, further comprising: Generating a classification representation of the ordinal feature set included in the training samples includes: Perform a Discrete Fourier Transform (DFT) or a Discrete Wavelet Transform (DWT) on the ordinal feature set; and The classification representation is assigned to the ordinal feature set, at least in part, based on the DFT or DWT transformation matrix.

14. The system according to claim 9, wherein, Adjusting the value of the adjustable parameter further includes generating pseudo residuals based on the derivative of the loss function.

15. The system according to claim 13, wherein, Each subsequently generated model is trained to predict the combined residual values ​​of the previously generated models in the model sequence.

16. The system according to claim 9, wherein, Adjusting the value of the adjustable parameter includes, at least in part, adjusting the value of the adjustable parameter based on the memory constraints of the device that trained or invoked the model, iteratively until the size of the model reaches a model size threshold.

17. A non-transitory computer-readable medium storing instructions that, when executed by one or more processing means, cause the one or more data processing means to perform operations, the operations including: Obtain a training dataset that includes multiple training samples, where each training sample includes feature variables and one or more output variables; Generating a first matrix using the training dataset, wherein the first matrix is ​​a sparse representation of the training dataset, and generating the first matrix includes: Generate a categorical representation of the feature variables based on each numerical feature variable in the feature variables; An encoded representation of each categorical feature variable is generated by encoding each categorical feature variable. Factorize the matrix representation of the training dataset to generate one or more matrices including the second matrix; The third matrix is ​​generated using (i) the second matrix and (ii) the regularization term; A fourth matrix is ​​generated based on (i) one or more matrices and (ii) the third matrix; Each feature of the first matrix is ​​represented by a vector comprising multiple adjustable parameters; and The value of the adjustable parameter is adjusted using a combination of (i) the loss function, (ii) the fourth matrix, and (iii) the first matrix. The adjustment of the adjustable parameter includes iteratively generating a sequence model to predict the residuals of the loss function until the residuals of the loss function can no longer be reduced, the model quality metric meets a quality threshold, or the model size has reached a maximum model size threshold.

18. The non-transitory computer-readable medium according to claim 17, wherein, The loss function is a loss function that provides a result corresponding to a given result provided by a specific loss function of the following form: Where R is the residual. y i The output variable is c, which is a constant, and E is the encoded representation.

Images