Machine learning using regular data

Abstract

The present disclosure relates to machine learning using periodic data. According to the present disclosure, a feature representation of an input data sample is obtained from a prediction model. A first Fourier coefficient of a first component in a Fourier expansion is determined by applying the feature representation to a first mapping model, and a second Fourier coefficient of a second component in the Fourier expansion is determined by applying the feature representation to a second mapping model. A Fourier expansion result is determined based on the first Fourier coefficient and the second Fourier coefficient in the Fourier expansion, and a prediction result of the input data sample is determined based on the Fourier expansion result.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority to U.S. patent application Ser. No. 17 / 666,076, entitled "MACHINE LEARNING WITH PERIODIC DATA," filed on February 7, 2022, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Periodic or cyclical data is frequently encountered in a wide range of machine learning scenarios. For example, in recommender systems, users may typically log into the application within a relatively fixed time frame each day (e.g., before bed or after work), which has been observed to result in strong periodic patterns in the recommendations made to users. In financial markets, asset prices may rise and fall periodically from year to year, a phenomenon commonly known as "seasonality." In search engines, hits for certain keywords may also show periodic patterns. Thus, how to exploit periodicity in training data to learn better predictive models is a key problem for these applications. [Brief description of the drawings]

[0003] The above and other objects, features, and advantages of exemplary embodiments disclosed herein will become more apparent through the following detailed description taken in conjunction with the accompanying drawings, in which several exemplary embodiments disclosed herein are illustrated by way of example and in a non-limiting manner, in which:

[0004] [Figure 1] 1 illustrates a block diagram of an environment in which embodiments of the present disclosure may be implemented.

[0005] [Diagram 2] FIG. 1 illustrates a block diagram of a machine learning system using Fourier learning, according to some example embodiments of the present disclosure.

[0006] [Diagram 3] FIG. 1 illustrates a block diagram of a machine learning system using Fourier learning, according to some other example embodiments of the present disclosure.

[0007] [Figure 4] FIG. 1 shows a block diagram of a machine learning system using Fourier learning, according to some further example embodiments of the present disclosure.

[0008] [Diagram 5] FIG. 1 shows a diagram of an example algorithm for Fourier learning using pseudo gradient descent, according to some embodiments of the present disclosure.

[0009] [Figure 6] 1 illustrates a flowchart of a process for Fourier learning according to some exemplary embodiments of the present disclosure.

[0010] [Figure 7] FIG. 1 shows a block diagram of an example computing system / device suitable for implementing example embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Next, the principles of the present disclosure will be described with reference to some embodiments. It should be understood that these embodiments are provided for illustrative purposes only, to assist those skilled in the art in understanding and implementing the present disclosure, and do not imply any limitations on the scope of the present disclosure. The disclosure described herein can be implemented in various ways other than those described below.

[0012] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0013] References in this disclosure to "one embodiment," "one embodiment," "exemplary embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but not all embodiments necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an exemplary embodiment, it is noted that it is within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0014] Terms such as "first" and "second" may be used herein to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0015] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. It will be further understood that the terms "comprise," "comprise," "have," "have," "include," and / or "comprise," as used herein, specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0016] As used herein, the term "model" refers to an association between inputs and outputs learned from training data, such that a corresponding output can be generated for a given input after training. The association can be represented by a function that processes the input to generate the output. The generation of the model can be based on machine learning techniques. Machine learning techniques can also be referred to as artificial intelligence (AI) techniques. In general, a machine learning model can be constructed that receives input information and makes a prediction based on the input information. Such a machine learning model can be referred to as a predictive model. For example, a classification model can predict a class of input information from a given set of classes, a recommendation model can predict a recommendation result for a user based on context information related to the user, and a model applied to a search engine can predict the probability of a hit for a particular keyword based on the user's actions. As used herein, a "model" can also be referred to as a "machine learning model", "learning model", "machine learning network", or "learning network", which are used interchangeably herein.

[0017] In general, machine learning may typically include three stages: a training stage, a validation stage, and an application stage (also called an inference stage). In the training stage, a large amount of training data may be used to iteratively train (or optimize) a given machine learning model until the model is able to obtain consistent inferences from the training data similar to those that human intelligence can make. During training, a set of parameter values ​​for the model are iteratively updated until a training goal is achieved. Through the training process, the machine learning model may be considered to be able to learn associations between inputs and outputs (also called input-output mappings) from the training data. In the validation stage, validation inputs are applied to the trained machine learning model to test whether the model can provide correct outputs to determine the model's performance. In the application stage, the resulting machine learning model may be used to process actual model inputs based on the set of parameter values ​​obtained from the training process to determine corresponding model outputs.

[0018] In contrast to batch learning methods, which generate an optimal machine learning model by learning the entire training data set at once, online machine learning is a method of machine learning in which training data becomes available sequentially and is used to update the optimal machine learning model for future data at each step. Example Environment

[0019] As mentioned above, it is expected that the periodicity in the training data can be exploited to learn a better predictive model. A predictive model is constructed and utilized according to machine learning techniques. The machine learning environment is described with reference to Figure 1.

[0020] 1 shows a block diagram of an environment 100 in which an embodiment of the present disclosure may be implemented. In the environment 100, it is expected to train and apply a machine learning model 105 for a predictive task. The machine learning model 105 may be any machine learning or deep learning architecture, for example, a neural network.

[0021] In a practical system, the machine learning model 105 may be configured to process input data samples and generate prediction results for the input data samples. The prediction task may be defined according to the practical application to which the machine learning model 105 is applied. As an example, in a recommendation system, the prediction task is to predict one or more items or objects that a user will be interested in and provide a recommendation to the user based on the prediction. In this example, the input data sample to the machine learning model 105 may include user information, contextual information related to the user, such as historical user interactions, and information related to the items to be recommended. The output from the machine learning model 105 is a prediction result indicating which items or types of items the user is interested in. As another example, in a financial application, the prediction task is to predict sales of a product in the future. In this example, the input data sample to the machine learning model 105 may include information related to further time, information related to the product and / or other related products, historical sales of the product and / or other related products, target geographical region and target users of the product, etc. It will be understood that only a limited number of examples are listed above and the machine learning model 105 may be configured to embody other prediction tasks.

[0022] The machine learning model 105 may be constructed as a function that processes input data and generates an output as a prediction result. The machine learning model 105 may be configured with a set of parameters whose values ​​are learned from training data through a training process. In FIG. 1, the model training system 110 is configured to embody a training process for training the machine learning model 105 based on a training data set 112. At an initial stage, the machine learning model 105 may be configured with initial parameter values. During the training process, the initial parameter values ​​of the machine learning model 105 may be iteratively updated until a learning goal is achieved.

[0023] The training data set 112 may include a number of input data samples provided to the machine learning model 105 and labeling information indicating corresponding ground truth labels of the input data samples. In some embodiments, an objective function is used to measure the error (or distance) between the output of the machine learning model 105 and the ground truth labels. Such an error may also be referred to as the machine learning loss, and the objective function may also be referred to as a loss function. The loss function is

number

[0024] After the training process, the trained machine learning model 105, configured with updated parameter values, may be provided to a model application system that applies real-world input data samples 122 to the machine learning model 105 and outputs prediction results for the input data samples 122.

[0025] In Fig. 1, the model training system 110 and the model application system 120 may be any system with computing capabilities. It should be understood that the components and arrangements within the environment shown in Fig. 1 are merely examples, and that a computing system suitable for implementing the exemplary implementations described in the subject matter described herein may include one or more different components, other components, and / or different arrangements. For example, what is shown as separate may be integrated into the same system or device. The embodiments of the present disclosure are not limited in this respect.

[0026] In some cases, the input data processed by the machine learning model may have a certain periodicity. Such data is referred to as periodic or cyclical data. For example, a user of an application may typically log into the application within a relatively fixed time frame each day (e.g., before going to bed or after work) and may exhibit the same interests during the same time frame on different days. Such periodic patterns may result in different predictive recommendations for the user. It is therefore expected that the machine learning model 105 may be trained to exploit the periodicity in the training data.

[0027] The problem of exploiting periodicity in training data to learn better predictive models can be formulated as follows.

number

number

number

number

number

[0028] X and Y are convex compact sets, and the loss function

number

number

number

[0029] The concept of periodicity plays an important role in equation (1). Specifically, periodicity determines the function f * t It is guaranteed that (x) is also a solution at t+nT (n is an integer greater than 0). This means that a predictive model trained at time t can provide useful information to improve the prediction accuracy at t+nT. Therefore, we are motivated to design a learning algorithm that can effectively utilize such useful information provided by the periodic nature of the data.

[0030] Surprisingly, existing optimization and machine learning methods provide little insight into how to exploit periodicity in the training data to efficiently solve equation (1) in big data environments, while industrial systems embody algorithms that simply underestimate periodicity in the training data.

[0031] When faced with machine learning of periodic data, one straightforward design to encode periodicity into the model structure is to simply include t as a model input and learn a function f(x,t). Unfortunately, this approach does not work as is. It has been shown that when a function f(x,t) is represented as a machine learning model, it is not possible to learn periodicity unless a special activation function is used. If f(x,t) belongs to a nonparametric family that encodes periodicity (such as Reproducing Kernel Hilbert Space (RKHS) with a periodic kernel or Sobolev-Hilbert Space with a periodic spline), the periodicity is automatically encoded across all input dimensions, but in equation (1), f(x,t) can be aperiodic in x.

[0032] An enhancement to this approach is to preprocess time t and learn a function denoted as f(x,mod(t,T)) that instead focuses on a single period of f(x,t). Preprocessing t mod(t,T) ensures periodicity at the inference stage, but still often requires laborious feature engineering, especially when x is high-dimensional and f(x,t) is of complex design.

[0033] An alternative approach to equation (1) is to simply train a predictive model every t. Since this is often not possible in practice, the time axis is often discretized, and the learner only needs to train a finite set of models at some discretized time points, resulting in a polytomous approach. In the case of machine learning systems, this finite set of models can share the "base" part of the neural network, differing only in the last few layers. The upside is that the time-dependent distribution D t If is piecewise constant in time, e.g.,

number

[0034] A further approach to training a predictive model using continuous data is to follow an online learning protocol where newly generated periodic data is applied to optimize the model. The performance of a learning algorithm is usually evaluated using the concept of dynamic regret, which measures the ability of the model to consistently and accurately predict the label of the latest batch of arriving data. Roughly speaking, where t takes on a discretized set of values, dynamic regret is the loss based on the trained model and f defined in equation (1): * t (x) is the cumulative sum of the difference between the optimal loss based on

number

[0035] In summary, the problem of exploiting periodic patterns in data distributions to train better models remains a largely unsolved problem in large-scale settings. Working Principles and Theoretical Analysis

[0036] According to an embodiment of the present disclosure, an improved approach to tackle machine learning problems with periodic data is provided. This approach proposes a new learning framework called Fourier Learning, which can be applied to train predictive models for use in a variety of applications where periodic data is generated.

[0037] Before describing the application of Fourier learning in predictive models, we theoretically analyze and prove how Fourier learning can solve problems that exploit periodicity in training data to learn better predictive models, e.g., the optimization problem of Equation (1).

[0038] In an embodiment of the present disclosure, the proposed Fourier learning can solve the set of optimization problems in Equation (1) as a single optimization problem in a function space that naturally contains time-periodic functions. In particular, the function space can be a tensor product of two Hilbert spaces, one containing model snapshots at fixed time points and the other containing time-periodic functions. As shown below, it can be seen that this leads to a partial Fourier expansion of these functions. In a convex analysis setting, it is also possible to learn the Fourier coefficients using streaming stochastic gradient descent (SGD). Theoretically, the proposed Fourier learning framework is supported from two different aspects: (i) from a modeling perspective, Fourier learning is naturally derived from a function optimization problem equivalent to the optimization problem in Equation (1) under a strongly convex and feasible setting. (ii) from an optimization perspective, it has been demonstrated that the coefficient functions updated in streaming SGD likely converge in the frequency domain. For practical applications, Fourier learning can be integrated into various predictive models to enable the predictive models to provide more accurate forecast results. By integrating with Fourier learning, one single model framework may be sufficient for forecasting periodic data.

[0039] The theoretical basis of the proposed Fourier learning is first introduced, which can be derived as a natural approach to function optimization problems. In an embodiment of the present disclosure, the set of learning problems in equation (1) is reformulated as a single learning problem in Hilbert space. In practice, this allows learning a unified model that takes both x and t as inputs. Specifically, the learning goal takes the form of the following equation (2), but in practice the expectation value can be replaced by the empirical mean value of the data set.

number

number

[0040] The key element in (2) is the design of the Hilbert space H in which f(x,t) is searched. In the problem of learning with periodic data, we focus specifically on functions in Hilbert spaces that are continuous, periodic in time, and have finite energy within a single period. We have discovered that the unifying objective of (2) is related to (1) via the following Lemma 1: Lemma 1. f in Eq.(1) * t Regarding (x),

number

[0041] In Lemma 1, T denotes the periodicity of x. The proof of Lemma 1 above is as follows: Starting from equation (2), for any f(x,t) ∈ H,

number

number

[0042] Lemma 1 states that (2) has a unique minimization function, and f * t It is implied that if (x) belongs to the Hilbert space H when treated as a function of both x and t, then the minimization function in (2) yields a solution to equation (1). Therefore, in a feasible setting, equation (2) serves as a surrogate for solving equation (1). The above proof shows that in a feasible setting and under the strict convexity used in Lemma 1, the desired set of solutions to equation (1) can be obtained by minimizing the surrogate loss specified in equation (2).

[0043] Another important element of (2) is the design of H. Here, we focus specifically on functions that are continuous, periodic in time, and have finite energy within a single period. Furthermore, functions in H need to degenerate to L2(X) as specified in equation (1) for a fixed t. Two important elements necessary to design such H are introduced:

[0044] Furthermore, defining functions on a circle is an important way to characterize periodic functions. Because these functions are defined for points on the circle, they take the angle information of the points as input, and so naturally have a period determined by the circumference. To facilitate optimization, we further define a Hilbert space structure on these functions based on the intuition of considering a circle as a line segment with its endpoints glued together, as follows:

number

[0045] Equation (5) indicates that the function f has finite energy.

number

number

number

[0046] L2(S T ) to a Hilbert space containing functions that depend on both x and t, requires the notion of a tensor product between Hilbert spaces, which is a direct extension of the notion of the Kronecker product between vectors in Euclidean space to function spaces.

[0047] Specifically,

number

number

number

number

number

number

number

[0048] L2(S T ) by its tensor product with L2(X),

number

number

number

number

number

number

number

[0049] The above lemma paves the way to a theoretically guaranteed algorithm for optimizing L(f) through a basis expansion of f in H, introduced below. Meanwhile, H is sufficiently general for learning purposes in the sense that the function f0(x,t) defined pointwise by the solution of equation (1) belongs to H under mild assumptions. Some definitions and assumptions are introduced below.

[0050] Definition 3 (Continuity under Total Variation). D t (x) to D t The conditional distribution of y given x under |t'-t|≦δ for any fixed t and any ε>0.

number

[0051] Assumption 4. Assumption: (i) X and Y are compact and convex sets. (ii) D in Definition 3. t (x) is continuous under the total variation of all x∈X. (iii) Loss function

number

number

[0052] Assumption 4 can be easily satisfied by a wide range of machine learning systems. For example, in deep neural networks (DNNs), the forced clipping of the final output usually results in limited output. The uniform and strong convexity of the loss function also holds for a wide range of l, such as the mean squared loss. With the above definitions and assumptions, we discover another lemma, Lemma 5.

[0053] Lemma 5. Under Assumption 4, f * t (x) is continuous at t for any given x∈X. Also,

number

[0054] Lemma 5 implies that under Assumption 4, the optimal solution f0(x,t) of equation (2) belongs to H. Combining Lemma 1 and Lemma 5, we see that satisfying Assumption 4 makes it possible to obtain a set of desirable solutions to equation (1) by solving equation (2). Fourier learning using periodic data

[0055] We then proceed to introduce Fourier learning, a learning framework that incorporates the periodicity of the data distribution into the model's structure via a partial Fourier expansion and learns the model by learning its Fourier coefficient functions. To do so, Lemma 2 is invoked, from a modeling perspective, where f(x,t)∈H can be expressed in terms of the following basis expansion:

number

number

[0056] Theorem 6 provides an explicit way to design periodic models and specifies how temporal features can be exploited. Note that it is perfectly possible to construct H with a weighted L2 space defined on a circle to guarantee periodicity. This allows us to deviate from trigonometric functions and potentially use other periodic functions to encode periodicity.

[0057] Using f(x,t) expressed by equation (10), the solution to the problem in equation (2) is a n (x) and b n(x), i.e., the Fourier coefficients of f(x,t) that are now independent of t and depend only on x. Furthermore, the sine and cosine components depend on t. Because equation (10) takes the form of a partial Fourier expansion of f(x,t), this training method can be called "Fourier training".

[0058] Fourier learning allows the designer to preserve the original model design, but at the same time mix in the expert advice of the last hidden layer in a time-dependent manner. This explicit role of t in the predictive model avoids the tedious feature engineering required when features t are implicitly added to the model in the form of f(x,t).

[0059] The goal now shifts to learning the coefficient functions in the frequency domain. For tractable learning, a cutoff frequency N / T can be introduced, so that we can instead express the truncated Fourier expansion of f(x,t) as:

number

[0060] The truncated Fourier expansion of equation (11) is an approximation of the Fourier expansion of equation (10). The approximation error of equation (11) for all f∈H can be determined as follows: N (f) can be expressed as:

number

[0061] In the Fourier expansion of the Fourier learning in equation (11), the Fourier coefficients a are used to generate the prediction results of the model f(x,t). n (x), b n (x) needs to be determined.n (x) and b n (x) can be thought of as a coefficient function that depends on x, which can be learned under different regimes. For example, they can be learned non-parametrically using function optimization algorithms.

[0062] In some embodiments, the Fourier coefficient a n (x) and b n When (x) has a parametric form, such as a neural network, stochastic gradient descent is known to converge to a stationary point at some rate under standard assumptions, which may be further explained below. In some embodiments, apart from the parametric framework above, Fourier learning also fits into a non-parametric regime, which may be further explained below. Machine learning system based on Fourier learning

[0063] In the following, we apply Fourier learning to large-scale machine learning scenarios by using neural networks. n (x) and b n We explain the parameterization of (x).

[0064] The above theoretical analysis shows that the model built based on Fourier learning can intuitively utilize the periodicity of the training data and can be expressed as a periodic function with the periodicity. Therefore, Fourier learning can be applied to machine learning-based prediction models. In the embodiment of the present disclosure, it is proposed to regard x as information related to an input data sample generated at a certain time t according to the Fourier expansion of f(x,t) in Equation (11). The input data sample can be a data sample of periodic data. The Fourier expansion result can be determined based on the Fourier expansion, and the prediction result of the input data sample is determined based on the Fourier expansion result.

[0065] 2, which illustrates a block diagram of a machine learning system 200 using Fourier learning in accordance with some example embodiments of the present disclosure. The machine learning system 200 may be embodied as a machine learning model 105 in the environment 100. As shown, the machine learning system 200 includes a predictive model 210 and a Fourier layer 220.

[0066] The predictive model 210 may be configured with any model architecture to embody a predictive task. In an embodiment of the present disclosure, the input data processed by the predictive model 210 is periodic data with a certain periodicity (represented by T). The input to the predictive model 210 is an input data sample generated at a certain time t. The Fourier layer 220 is introduced to allow for more accurate prediction results to be generated by taking into account the periodicity in the input data. In any case, it should be noted that the predictive model 210 may be constructed in any manner that may or may not exploit the periodicity of the input data, since the addition of the Fourier layer 220 can further exploit the periodicity.

[0067] The Fourier layer 220 is designed by the following intuition: if x is regarded as the output of the last hidden layer of the original predictive model, then equation (11) can be regarded as the output layer of a network with the architecture shown in Figure 2. Specifically, the Fourier layer 220 first divides x into a n (x) and b n (x), then multiply them element-wise by the basis vectors SIN and COS to obtain a (2N+1)-dimensional result. The results are summed to obtain a scalar output. In particular, for all n ≥ 1, n (x)=b n When (x) = 0, the final output is equal to b0(x), which itself can be interpreted as the output of the original model. This means that replacing the output layer of the original model with a Fourier layer 220 increases its capacity, avoiding the need for tedious feature engineering.

[0068] In particular, the Fourier layer 220 receives the feature representations of the input data samples extracted by the predictive model 210. The predictive model 210 may generally be considered to be composed of two parts, one to extract hidden features in the input data, and another to determine the model output based on the final hidden features. In some embodiments, the predictive model 210 may include multiple layers, including an input layer that receives the input data samples, one or more hidden layers that process the input data samples to generate feature representations that characterize the hidden features in the input data, and an output layer that generates the model output. The layers of the predictive model 210 are connected layer by layer, with the output from one layer being provided as input to the next layer.

[0069] In some embodiments, the feature representation extracted in the last hidden layer 212 of the predictive model 210 is provided as an input to the Fourier layer 220. This feature representation is represented by x. Typically, the input data sample may contain redundant information and may be of higher dimension. Through feature extraction in the predictive model, the feature representation may be able to characterize useful feature information in the input data sample with a relatively small dimension. The Fourier layer 220 may be able to further process the feature representation x to generate a prediction result for the input data sample.

[0070] Assume that x has dimension d1, and the predicted result of the input data sample has dimension d2. The dimensionality of the feature representation x and the dimensionality of the predicted result (d2) may depend on the configuration of the predictive model 210. In general, d1 may be greater than 1, and d2 may be equal to or greater than 1. For example, the predicted result may be a single dimensional output indicating, for example, the probability that the user is interested in a target item, or may be a multi-dimensional output indicating, for example, respective probabilities that the user is interested in multiple items.

[0071] Given the input (i.e., feature representation x) and output (i.e., prediction result) of the Fourier layer 220, the processing of the Fourier layer 220 can be viewed as mapping the input of dimension d1 to an output of dimension d2. The model structure of the Fourier layer 220 can be designed to embody such a mapping based on the Fourier expansion.

[0072] As shown, the Fourier layer 220 is a Fourier coefficient a n The mapping model 230 that generates (x) and the Fourier coefficients b n Following a truncated Fourier expansion of equation (11) having a predetermined number (N+1) terms, the mapping model 230 may be configured to convert the feature representation x of dimension d1 to an output of dimension N, and the mapping model 240 may be configured to convert the feature representation x of dimension d1 to an output of dimension (N+1). The N elements at the output of the mapping model 230 are represented by N Fourier coefficients

number

number

[0073] The mapping model 230 and the mapping model 240 may be constructed based on any machine learning architecture. In some embodiments, the mapping model 230 and the mapping model 240 may be constructed without an activation function. In general, the activation function (e.g., sigmoid function, tanh function, ReLU function) applied to the machine learning model may limit the amplitude of the model output to a certain range. In the Fourier expansion, there is no explicit limit on the amplitude of the Fourier coefficients, so the mapping model 230 and the mapping model 240 may be constructed without activation.

[0074] The Fourier expansion generally includes a sine function-based component and a cosine function-based component. Thus, as shown in FIG. 2, the Fourier layer 220 further includes a sine function unit 232 for determining a value of the sine component in the Fourier expansion, and a cosine function unit 242 for determining a value of the cosine component in the Fourier expansion. As shown in Equation (11), the sine component is based on a sine function that is a periodic function with periodicity T and dependent on time t, and the cosine component is based on a cosine function that is a periodic function with periodicity T and dependent on time t.

[0075] The sine function unit 232 converts the set of sine component values ​​into a column vector

number

number

[0076] N sine component values

number

number

[0077] In the Fourier layer 220, in response to the input data samples generated at each time point, the Fourier coefficients

number

number

number

[0078] The N sine component values ​​and the N Fourier coefficients may be provided to a multiplier 234. The multiplier 234 is configured to perform element-wise multiplications on the N sine component values ​​and the N Fourier coefficients to generate N products. The (N+1) cosine component values ​​and the (N+1) Fourier coefficients are provided to a multiplier 244. The multiplier 244 is configured to perform element-wise multiplications on the (N+1) cosine component values ​​and the (N+1) Fourier coefficients to generate (N+1) products. The (2N+1) products correspond to individual terms involved in the Fourier expansion.

[0079] To obtain a predicted result for dimension d2, the N products from multiplier 234 may be input to mapping model 236, and the (N+1) products from multiplier 244 may be input to mapping model 246. The mapping model 236 may be configured to convert the N products from multiplier 234 into a first intermediate unfolded result for dimension d2, and the mapping model 246 may be configured to convert the (N+1) products from multiplier 244 into a second intermediate unfolded result for dimension d2. The first and second intermediate unfolded results may be provided to an aggregator 250 configured to perform element-wise summation on the first and second intermediate unfolded results to provide a Fourier unfolded result, which may be determined as a predicted result for the input data sample for dimension d2.

[0080] In some embodiments, if the predicted result is a one-dimensional output, the mapping models 236 and 246 may be omitted from the Fourier layer 220. In this case, the products from the multipliers 234 and 244 are summed to provide a Fourier expansion result that may be determined as the predicted result.

[0081] In some embodiments, mapping model 230 and mapping model 240 may be constructed as multi-layer perceptron (MLP) models. In some embodiments, mapping model 236 and mapping model 246 may be constructed as MLP models. Thus, Fourier layer 220 may be considered as a Fourier MLP (F-MLP) layer.

[0082] Parameter values ​​of the mapping models 220, 240, 236, 246 in the Fourier layer 220 may be determined through a training process. In some embodiments, these mapping models may be trained with the predictive model 210. The training data may include input data samples to the predictive model 210 and labeling information indicating corresponding ground truth labels for the input data samples. In some embodiments, the mapping models in the Fourier layer may be trained in an end-to-end manner with the predictive model 210. In some embodiments, the predictive model 210 may be trained first and then retrained together with the mapping models in the Fourier layer.

[0083] In some embodiments, the Fourier layer 220 may be configured as follows for an F-MLP with input dimension d and output dimension d:

number

number

number

number

number

number

number

number

[0084] It should be noted that there are several approaches that propose combining Fourier analysis with deep learning systems. However, most of these approaches aim to learn the inherent high frequency components in the distribution of the input data itself, rather than focusing on the periodicity of the distribution in time. In particular, the inventors have observed that the realization of the Fourier layer of the present disclosure into existing designs of predictive models fundamentally changes the physical meaning of each processing unit in the model: in the usual model, each processing unit is an expert that changes its decision over time. Under the Fourier layer of the present disclosure, each processing unit holds an expert frequency component that determines how significantly it changes its decision over time at a given frequency. While the former requires designing an online learning algorithm that tracks the constantly changing optimum for each expert, the latter allows for future optimum values ​​to be predicted using trigonometric interpolation. This provides an advantage over online learning approaches.

[0085] In the exemplary embodiment of Figure 2, the Fourier layer 220 is introduced as an output layer of the predictive model 210, and thus its output is determined as a predicted result of the input data sample. In some embodiments, the Fourier layer 220 may operate together with the complete predictive model 210 (including its own output layer), and the output from the Fourier layer 220 and the output from the predictive model 210 are aggregated to generate a final predicted result. Figure 3 illustrates a machine learning system 200 according to such an embodiment.

[0086] As shown in Fig. 3, the predictive model 210 includes, among other layers, an output layer 312 that receives the feature representation x from the last hidden layer 212. The output layer 312 of the predictive model 210 may process the feature representation x and generate intermediate prediction results. The processing in the output layer 312 may depend on the configuration of the predictive model 210, which may vary in different prediction tasks. The Fourier layer 220 may also receive the feature representation x from the last hidden layer 212 and generate intermediate prediction results based on the Fourier expansion results, as described according to the embodiment with respect to Fig. 2.

[0087] The machine learning system 200 may further include an aggregator 330 configured to mix two intermediate prediction results from the predictive model 210 and the Fourier layer 220. For example, the aggregator 330 may determine a weighted sum of the two intermediate prediction results. The aggregation of the intermediate prediction results may be expressed as:

number

[0088] In some embodiments, the predictive model 210 may have a complex structure, e.g., may include multiple sub-models with different model structures. In this case, the input to the Fourier layer 220 may be carefully designed. Figure 3 illustrates a machine learning system 200 according to such an embodiment.

[0089] 4, predictive model 210 may include multiple submodels (e.g., K submodels), such as submodel 410-1, ..., submodel 410-K (collectively or individually referred to as submodels 410 for purposes of discussion), where K is an integer greater than 1. The outputs of the submodels may be summed at an output layer 312 in predictive model 210 to provide an output for the model. In this case, output layer 312 may include an aggregator that performs a summation on each output from the K submodels 410. Thus, predictive model 210 may include:

number

[0090] With the structure of the predictive model 210 shown in FIG. 4, the sub-model 410 may extract feature representations from the input data samples in the last hidden layer and determine its own output in its output layer based on the feature representations. The feature representations from the sub-models 410 may be aggregated to generate a feature representation x that is input to the Fourier layer 220. In some embodiments, the feature representations from the sub-models 410 may be of different dimensions. In the embodiment of FIG. 4, to aggregate the feature representations from the sub-models 410, the machine learning model 200 may further include a dimension alignment layer 420 that converts each feature representation having different dimensions from the sub-models 410 into a feature representation having the same dimensions. In some embodiments, the dimension alignment layer 420 may include multiple MLPs each configured to convert a feature representation from one of the sub-models 410 into a feature representation having the same dimensions.

[0091] Because Fourier layer 220 performs a linear transformation on its input, it may add feature representations with the same dimensions generated from dimensional alignment layer 420 to obtain feature representation x to input to Fourier layer 220. In this case, dimensional alignment layer 420 may transform feature representations with different dimensions from sub-model 410 into a feature representation with a dimension of d1.

[0092] In Figure 4, the output from the Fourier layer 220 is shown aggregated with the output from the predictive model 210 by an aggregator 330, similar to Figure 3. It will be appreciated that in other embodiments, processing on the feature representation may be integrated into the system 200 shown in Figure 2. Training a Machine Learning System

[0093] The training of the Fourier layer 220 is performed in conjunction with the prediction model according to the procedure of streaming SGD. This procedure actually involves the sample data (x,y,t) ∼D tThis differs from standard SGD, which requires that (x,y) p(t). However, sampling from p(t) is difficult for many online applications due to real-time update requirements where data arrive sequentially. Here we show that streaming SGD can be used to circumvent this problem while maintaining good practical performance and convergence guarantees.

[0094] The training procedure is as follows: n (x) and b n (x) is a n (x;θ n ) and b n (x;ρ n ) can be parameterized by θ n and ρ n are the neural network parameters. In the case of periodic data, the τth mini-batch of data is collected in the kth cycle and the model can be updated with the following update rule:

number

number

number

number

number

[0095] The convergence property is discussed when training a machine learning system based on Fourier learning using streaming SGD. Truncating f N (x,t) is used, and it is recalled that the problem in equation (2) reduces to finding the optimal value in the frequency domain given by the following equation (17):

Number

Number

[0096] Below, first, the gradient norm convergence results of streaming SGD under a general non-convex setting are shown, and then the global convergence results under the assumption of strong convexity are introduced. Before that, some additional assumptions are introduced.

[0097] Assumption 7. Hypothesis: (i) For all n, k, τ, the second moment of the update direction is bounded: for some

Number

Number

Number

[0098] Assumption 7 assumes bounded second moments in the update direction and Lipschitz property of the gradient, which are typically required in the convergence analysis of SGD-type algorithms. The following results show that streaming SGD with an appropriate learning rate achieves convergence under both non-convex and strongly convex settings.

[0099] Theorem 8 (Convergence of streaming SGD). Suppose that Assumptions 4 and (i) and (ii) of Assumptions 7 hold, then for all

number

number

number

[0100] In simple terms, the learning framework is

number

number

[0101] As mentioned above, apart from the above parametric framework, in some embodiments the proposed Fourier learning also fits into the non-parametric regime, where a n and b nis updated directly:

number

[0102] Since L2 function gradients often involve Dirac delta functions, which leads to discontinuous updates, we instead replace the function gradients with their kernel embeddings. Specifically,

number

number

number

number

number

number

number

[0103] 6 illustrates a flowchart of a process 600 for Fourier learning according to some example embodiments of the present disclosure. The process 600 may be embodied in the machine learning system 200 or by a model application system 120 that can apply input data to the machine learning system 200 to perform a corresponding predictive task. For purposes of discussion, the process 600 will be discussed with reference to FIG.

[0104] At block 610, the model application system 120 obtains feature representations of input data samples from a predictive model configured to process input data with a periodicity, the input data samples being samples of the input data generated at times within the period.

[0105] In block 620, the model application system 120 applies the feature representation to a first mapping model to determine a first Fourier coefficient of a first component in a Fourier expansion, the Fourier expansion being periodic and dependent on the time instant and the feature representation. In block 620, the model application system 120 applies the feature representation to a second mapping model to determine a second Fourier coefficient of a second component in a Fourier expansion.

[0106] At block 640, the model application system 120 determines a Fourier expansion result based on the first Fourier coefficient and the second Fourier coefficient in the Fourier expansion. At block 650, the model application system 120 determines a predicted result for the input data sample based on the Fourier expansion result.

[0107] In some embodiments, the Fourier expansion comprises a truncated Fourier expansion having a predetermined number of terms, and the number of the first Fourier coefficients and the number of the second Fourier coefficients are based on the predetermined number.

[0108] In some embodiments, the first component depends on the time point and is based on a sine function having the periodicity, and the second component is based on a cosine function having the periodicity.

[0109] In some embodiments, to determine the Fourier expansion result, the model application system 120 determines a set of first component values ​​for the first component by shifting the frequency of a sine function a predetermined number of times, and determines a set of second component values ​​for the second component by shifting the frequency of a cosine function a predetermined number of times. The model application system 120 determines the Fourier expansion result by multiplying the first Fourier coefficients by the first component values, respectively, and multiplying the second Fourier coefficients by the second component values.

[0110] In some embodiments, to determine the Fourier expansion result, the model application system 120 calculates a first product by multiplying a first Fourier coefficient by a first component value and calculates a second product by multiplying a second Fourier coefficient by a second component value. The model application system 120 maps the first product to a first intermediate unfolding result using a third mapping model and maps the second product to a second intermediate unfolding result using a fourth mapping model. The model application system 120 determines the Fourier expansion result by aggregating the first intermediate unfolding result and the second intermediate unfolding result.

[0111] In some embodiments, to determine a prediction result, the model application system 120 determines a first intermediate prediction result from the Fourier expansion result, and obtains a second intermediate prediction result generated from an output layer of a predictive model based on the feature representation. The model application system 120 determines a prediction result by aggregating the first intermediate prediction result and the second intermediate prediction result.

[0112] In some embodiments, the predictive model includes multiple sub-models configured to extract multiple feature representations from the input data samples, and in some embodiments, the model application system 120 obtains multiple feature representations from the multiple sub-models and generates a feature representation by aggregating the multiple feature representations.

[0113] In some embodiments, the first mapping model and the second mapping model are constructed without an activation function. In some embodiments, the third mapping model and the fourth mapping model are constructed without an activation function. In some embodiments, the mapping models are trained jointly with the predictive model. Exemplary Systems / Devices

[0114] 7 shows a block diagram of an exemplary computing system / device 700 suitable for implementing an exemplary embodiment of the present disclosure. The model application system 120 and / or the model training system 110 may be embodied as or included in the system / device 700. The system / device 700 may be a general-purpose computer or computer system, a physical computing system / device, or a portable electronic device, or may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. The system / device 700 may be used to implement the process 600 of FIG. 6.

[0115] As shown in the figure, the system / device 700 includes a processor 701 capable of executing various processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 to a random access memory (RAM) 703. The RAM 703 also stores data and the like required for the processor 701 to execute various processes as necessary. The processor 701, the ROM 702, and the RAM 703 are connected to one another via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0116] The processor 701 may be of any type suitable for a local technology network and may include, as non-limiting examples, one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), a graphic processing unit (GPU), a co-processor, and a processor based on a multi-core processor architecture. The system / device 700 may have multiple processors, such as application specific integrated circuit chips slaved to a clock that synchronizes a main processor.

[0117] The multiple components in the system / device 700 are connected to an I / O interface 705 including input units 707 such as a keyboard, mouse, etc., output units 707 including displays such as a cathode ray tube (CRT), liquid crystal display (LCD), etc., loudspeakers, etc., a storage unit 708 such as a disk, optical disk, etc., and a communication unit 709 such as a network card, modem, wireless transceiver, etc. The communication unit 709 enables the system / device 700 to exchange information / data with other devices over a communication network such as the Internet, various telecommunication networks, etc.

[0118] The above-mentioned methods and processes, such as process 600, may also be executed by the processor 701. In some embodiments, the process 600 may be embodied as a computer software program or computer program product tangibly contained in a computer-readable medium, such as the storage unit 708. In some embodiments, the computer program may be partially or completely loaded and / or embodied in the system / device 700 via the ROM 702 and / or the communication unit 709. The computer program includes computer executable instructions that are executed by the associated processor 701. When the computer program is loaded into the RAM 703 and executed by the processor 701, it may embody one or more operations of the above-mentioned process 600. Alternatively, the processor 701 may be configured via any other suitable manner (e.g., by firmware) to execute the process 600 in other embodiments.

[0119] In some exemplary embodiments of the present disclosure, a computer program product is provided that includes instructions that, when executed by a processor of an apparatus, cause said apparatus to perform the steps of any one of the methods described above.

[0120] In some exemplary embodiments of the present disclosure, a computer-readable medium is provided that includes program instructions for causing an apparatus to perform the steps of any one of the methods described above. In some embodiments, the computer-readable medium may be a non-transitory computer-readable medium.

[0121] In an eighth aspect, an exemplary embodiment of the present disclosure provides a computer-readable medium comprising program instructions for causing an apparatus to perform at least the method of the second aspect described above. In some embodiments, the computer-readable medium may be a non-transitory computer-readable medium.

[0122] In general, various exemplary embodiments of the present disclosure may be embodied in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be embodied in firmware or software that may be executed by a controller, microprocessor, or other computing device. Although various aspects of the exemplary embodiments of the present disclosure have been illustrated and described using block diagrams, flowcharts, or other graphical representations, it will be understood that the blocks, apparatus, systems, techniques, or methods described herein may be embodied in, by way of non-limiting examples, hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or any combination thereof.

[0123] The present disclosure also provides at least one computer program product tangibly stored in a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in a program module, that execute on a target real or virtual processor device to perform the above-described method / process. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. The computer-executable instructions of the program modules may be executed in local or distributed devices. In distributed devices, the program modules may be located in both local and remote storage media.

[0124] The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium may include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] Computer program code for carrying out the methods disclosed herein may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus such that, when executed by the processor or controller, the program code embodies the functions / operations specified in the flowcharts and / or block diagrams. The program code may be executed entirely on the computer, partially on the computer, as a stand-alone software package, partially on the computer and partially on a remote computer, or entirely on a remote computer or server. The program code may be distributed on specially programmed devices, generally referred to herein as "modules." The software component parts of a module may be written in any computer language and may be part of a monolithic code base, or may be developed in more separate code parts as is common in object-oriented computer languages. Furthermore, modules may be distributed across multiple computer platforms, servers, terminals, mobile devices, etc. A given module may be embodied such that the described functions are executed by separate processors and / or computing hardware platforms.

[0126] Although operations are shown in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above description, these should not be construed as limitations on the scope of the disclosure, but rather as descriptions of features specific to certain embodiments. Certain features that are described in the context of separate embodiments may also be embodied in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be embodied in multiple embodiments separately or in any suitable subcombination.

[0127] Although the present disclosure has been described in language specific to structural features and / or methodological acts, it is to be understood that the present disclosure, as defined by the appended claims, is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. 1. A method comprising: obtaining feature representations of input data samples from a predictive model, the predictive model being configured to process input data with a periodicity, the input data samples being samples of the input data generated at a point in time within the period; determining a first Fourier coefficient of a first component in a Fourier expansion by applying the feature representation to a first mapping model, the Fourier expansion being dependent on the time instant and the feature representation, the Fourier expansion having the periodicity; determining second Fourier coefficients of a second component in the Fourier expansion by applying the feature representation to a second mapping model; determining a Fourier expansion result based on the first Fourier coefficient and the second Fourier coefficient in the Fourier expansion; determining a predicted outcome for the input data samples based on the Fourier expansion results.

2. The method of claim 1 , wherein the Fourier expansion comprises a truncated Fourier expansion having a predetermined number of terms, and the number of the first Fourier coefficients and the number of the second Fourier coefficients are based on the predetermined number.

3. The first component is based on a sine function having the periodicity depending on the time point, and the second component is based on a cosine function having the periodicity, and the step of determining the Fourier expansion result includes: determining a set of first component values ​​for the first component by shifting the frequency of the sine function the predetermined number of times; determining a set of second component values ​​for the second component by shifting the frequency of the cosine function the predetermined number of times; and determining the Fourier expansion result by multiplying the first Fourier coefficients by the first component values, respectively, and multiplying the second Fourier coefficients by the second component values.

4. determining the Fourier expansion result by multiplying the first Fourier coefficients by the first component values ​​and multiplying the second Fourier coefficients by the second component values, calculating a first product by multiplying the first Fourier coefficient by the first component value; calculating a second product by multiplying the second Fourier coefficient by the second component value; mapping the first product to a first intermediate unfolded result using a third mapping model and mapping the second product to a second intermediate unfolded result using a fourth mapping model; The method of claim 3 , further comprising determining the Fourier expansion result by aggregating the first intermediate expansion result and the second intermediate expansion result.

5. determining a predicted result for the input data sample based on the Fourier expansion result, determining a first intermediate prediction result from the Fourier expansion result; obtaining a second intermediate prediction result generated from an output layer of the prediction model based on the feature representation; The method according to any one of claims 1 to 4, further comprising: determining the prediction result by aggregating the first intermediate prediction result and the second intermediate prediction result.

6. The predictive model includes a plurality of sub-models configured to extract a plurality of feature representations from the input data samples, and obtaining the feature representations comprises: obtaining the plurality of feature representations from the plurality of sub-models; The method of any one of claims 1 to 5, further comprising: generating the feature representation by aggregating the plurality of feature representations.

7. The method according to any one of claims 1 to 6, wherein the first mapping model and the second mapping model are constructed without an activation function.

8. At least one processor; at least one memory, The memory is communicatively coupled to the at least one processor, and when executed by the at least one processor, causes the at least one processor to: obtaining feature representations of input data samples from a predictive model, the predictive model being configured to process input data with a periodicity, the input data samples being samples of the input data generated at a point in time within the period; determining a first Fourier coefficient of a first component in a Fourier expansion by applying the feature representation to a first mapping model, the Fourier expansion being dependent on the time instant and the feature representation, the Fourier expansion having the periodicity; determining second Fourier coefficients of a second component in the Fourier expansion by applying the feature representation to a second mapping model; determining a Fourier expansion result based on the first Fourier coefficient and the second Fourier coefficient in the Fourier expansion; determining a predicted outcome for the input data samples based on the Fourier expansion results.

9. 9. The system of claim 8, wherein the Fourier expansion includes a truncated Fourier expansion having a predetermined number of terms, and the number of the first Fourier coefficients and the number of the second Fourier coefficients are based on the predetermined number.

10. The first component is dependent on the time point and is based on a sine function having the periodicity, and the second component is based on a cosine function having the periodicity, and determining the Fourier expansion result includes: determining a set of first component values ​​for the first component by shifting the frequency of the sine function the predetermined number of times; determining a set of second component values ​​for the second component by shifting the frequency of the cosine function the predetermined number of times; and determining the Fourier expansion result by multiplying the first Fourier coefficients by the first component values, respectively, and multiplying the second Fourier coefficients by the second component values.

11. determining the Fourier expansion result by multiplying the first Fourier coefficients by the first component values, respectively, and multiplying the second Fourier coefficients by the second component values, calculating a first product by multiplying the first Fourier coefficient by the first component value; calculating a second product by multiplying the second Fourier coefficient by the second component value; mapping the first product to a first intermediate unfolded result using a third mapping model and mapping the second product to a second intermediate unfolded result using a fourth mapping model; and determining the Fourier expansion result by aggregating the first intermediate expansion result and the second intermediate expansion result.

12. Determining a predicted result for the input data samples based on the Fourier expansion result includes: determining a first intermediate prediction result from the Fourier expansion result; obtaining a second intermediate prediction result generated from an output layer of the prediction model based on the feature representation; and determining the prediction result by aggregating the first intermediate prediction result and the second intermediate prediction result.

13. The predictive model includes a plurality of sub-models configured to extract a plurality of feature representations from the input data samples, and obtaining the feature representations comprises: obtaining the plurality of feature representations from the plurality of sub-models; and generating the feature representation by aggregating the plurality of feature representations.

14. The system of any one of claims 8 to 13, wherein the first mapping model and the second mapping model are constructed without an activation function.

15. A non-transitory computer-readable storage medium storing computer-readable instructions, comprising: The computer readable instructions, when executed by a computing device, cause the computing device to: obtaining feature representations of input data samples from a predictive model, the predictive model being configured to process input data with a periodicity, the input data samples being samples of the input data generated at a point in time within the period; determining a first Fourier coefficient of a first component in a Fourier expansion by applying the feature representation to a first mapping model, the Fourier expansion being dependent on the time instant and the feature representation, the Fourier expansion having the periodicity; determining second Fourier coefficients of a second component in the Fourier expansion by applying the feature representation to a second mapping model; determining a Fourier expansion result based on the first Fourier coefficient and the second Fourier coefficient in the Fourier expansion; and determining a predicted outcome for the input data samples based on the Fourier expansion results.

16. 16. The non-transitory computer-readable storage medium of claim 15, wherein the Fourier expansion includes a truncated Fourier expansion having a predetermined number of terms, and a number of the first Fourier coefficients and a number of the second Fourier coefficients are based on the predetermined number.

17. The first component is dependent on the time point and is based on a sine function having the periodicity, and the second component is based on a cosine function having the periodicity, and determining the Fourier expansion result includes: determining a set of first component values ​​for the first component by shifting the frequency of the sine function the predetermined number of times; determining a set of second component values ​​for the second component by shifting the frequency of the cosine function the predetermined number of times; and determining the Fourier expansion result by multiplying the first Fourier coefficients by the first component values, respectively, and multiplying the second Fourier coefficients by the second component values.

18. determining the Fourier expansion result by multiplying the first Fourier coefficients by the first component values, respectively, and multiplying the second Fourier coefficients by the second component values, calculating a first product by multiplying the first Fourier coefficient by the first component value; calculating a second product by multiplying the second Fourier coefficient by the second component value; mapping the first product to a first intermediate unfolded result using a third mapping model and mapping the second product to a second intermediate unfolded result using a fourth mapping model; and determining the Fourier expansion result by aggregating the first intermediate expansion result and the second intermediate expansion result.

19. Determining a predicted result for the input data samples based on the Fourier expansion result includes: determining a first intermediate prediction result from the Fourier expansion result; obtaining a second intermediate prediction result generated from an output layer of the prediction model based on the feature representation; and determining the prediction result by aggregating the first intermediate prediction result and the second intermediate prediction result.

20. The predictive model includes a plurality of sub-models configured to extract a plurality of feature representations from the input data samples, and obtaining the feature representations comprises: obtaining the plurality of feature representations from the plurality of sub-models; and generating the feature representation by aggregating the plurality of feature representations.

Images