Machine learning using periodic data

Abstract

The present disclosure relates to machine learning using periodic data. According to the present disclosure, a feature representation of an input data sample is obtained from a prediction model. A first Fourier coefficient of a first component in a Fourier expansion is determined by applying the feature representation to a first mapping model, and a second Fourier coefficient of a second component in the Fourier expansion is determined by applying the feature representation to a second mapping model. A Fourier expansion result is determined based on the first Fourier coefficient and the second Fourier coefficient in the Fourier expansion, and a prediction result of the input data sample is determined based on the Fourier expansion result.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims the benefit of priority of U.S. Patent Application No. 17 / 666,076, entitled "MACHINE LEARNING WITH PERIODIC DATA," filed on February 7, 2022, the entire content of which is incorporated herein by reference.

Background Art

[0002] Periodic or cyclic data is frequently encountered in a wide range of machine - learning scenarios. For example, in a recommender system, users can usually log in to an application within a relatively fixed time frame (e.g., before bedtime or after work) every day, and as a result, it has been observed that there is a strong periodic pattern in the recommendations to users. In the financial market, asset prices can fluctuate regularly every year, which is a generally known phenomenon as "seasonality." In a search engine, periodic patterns may also be displayed due to hits on certain keywords. Therefore, how to utilize the periodicity in training data to learn better prediction models is an important issue for these applications.

Brief Description of the Drawings

[0003] The above and other objects, features, and advantages of the exemplary embodiments disclosed herein will become more understandable through the following detailed description with reference to the accompanying drawings. In the drawings, some exemplary embodiments disclosed herein are shown by way of example and in a non - limiting manner.

[0004] [Figure 1] A block diagram of an environment in which embodiments of the present disclosure can be implemented is shown.

[0005] [Figure 2] A block diagram of a machine - learning system using Fourier learning according to some exemplary embodiments of the present disclosure is shown.

[0006] [Figure 3] A block diagram of a machine learning system using Fourier learning, according to some other exemplary embodiments of the present disclosure, is shown.

[0007] [Figure 4] A block diagram of a machine learning system using Fourier learning, according to some further exemplary embodiments of the present disclosure, is shown.

[0008] [Figure 5] The diagrams show exemplary algorithms for Fourier learning using pseudo-gradient descent according to several embodiments of this disclosure.

[0009] [Figure 6] A flowchart of the Fourier learning process according to some exemplary embodiments of this disclosure is shown.

[0010] [Figure 7] A block diagram of an exemplary computing system / device suitable for embodying an exemplary embodiment of the present disclosure is shown. [Modes for carrying out the invention]

[0011] Next, the principles of this disclosure will be described with reference to several embodiments. These embodiments are provided for illustrative purposes only and are intended to help those skilled in the art understand and implement this disclosure; they should not be interpreted as implying any limitation on the scope of this disclosure. The disclosures described herein can be implemented in a variety of ways other than those described below.

[0012] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those generally understood by those skilled in the art to which this disclosure belongs.

[0013] References in this disclosure such as “one embodiment,” “one example,” and “exemplary embodiment” indicate that the described embodiments may include certain features, structures, or characteristics, but not all embodiments necessarily include those features, structures, or characteristics. Furthermore, such phrases do not necessarily refer to the same embodiment. In addition, if certain features, structures, or characteristics are described in relation to an exemplary embodiment, it should be noted that, whether explicitly stated or not, the influence of such features, structures, or characteristics in relation to other embodiments is within the knowledge of those skilled in the art.

[0014] Terms such as “first” and “second” may be used herein to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of the exemplary embodiments, the first element may be called the second element, and similarly, the second element may be called the first element. The term “and / or” as used herein includes any combination of one or more of the enumerated terms.

[0015] The terms used herein are for the sole purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” also include the plural unless the context clearly indicates otherwise. It will be further understood that the terms “include,” “equip,” “have,” “possess,” “include,” and / or “incorporate,” as used herein, identify the presence of the described features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0016] As used herein, the term “model” refers to an association between inputs and outputs learned from training data, and therefore, a corresponding output can be generated for a given input after training. This association can be represented by a function that processes the input and produces an output. Model generation may be based on machine learning techniques, also known as artificial intelligence (AI) techniques. Generally, machine learning models can be constructed that receive input information and make predictions based on that input information. Such machine learning models may be called predictive models. For example, a classification model may predict the class of input information from a given set of classes; a recommendation model may predict recommendation results for a user based on contextual information relevant to the user; and a model applied to a search engine may predict the probability of a particular keyword being hit based on user behavior. As used herein, “model” may also be referred to as a “machine learning model,” “learning model,” “machine learning network,” or “learning network,” but these terms are used interchangeably herein.

[0017] Generally, machine learning can involve three stages: the training stage, the validation stage, and the application stage (also called the inference stage). In the training stage, a given machine learning model can be iteratively trained (or optimized) using a large amount of training data until the model can derive consistent inferences from the training data similar to those that human intelligence can perform. During training, the set of parameter values ​​for the model is iteratively updated until the training goal is achieved. Throughout the training process, the machine learning model can be considered to have learned the relationships between inputs and outputs (also called input-output mappings) from the training data. In the validation stage, validation inputs are applied to the trained machine learning model to test whether the model can provide the correct output and determine the model's performance. In the application stage, the resulting machine learning model can be used to process real-world model inputs based on the set of parameter values ​​obtained from the training process and determine the corresponding model outputs.

[0018] In contrast to batch learning methods, which generate an optimal machine learning model by training on the entire set of training data at once, online machine learning is a machine learning method in which training data becomes available sequentially, and the machine learning model is updated at each step to be optimal for future data. Exemplary environment

[0019] As mentioned earlier, it is expected that a better predictive model can be learned by utilizing the periodicity within the training data. The predictive model is constructed and used according to machine learning techniques. Refer to Figure 1 to explain the machine learning environment.

[0020] Figure 1 shows a block diagram of an environment 100 in which embodiments of the present disclosure may be realized. In environment 100, it is expected that a machine learning model 105 for a prediction task will be trained and applied. The machine learning model 105 may be any machine learning or deep learning architecture, such as a neural network.

[0021] In a real system, the machine learning model 105 may be configured to process input data samples and generate prediction results for said input data samples. The prediction task may be defined according to the actual application to which the machine learning model 105 is applied. For example, in a recommendation system, the prediction task is to predict one or more items or objects that a user may be interested in and provide recommendations to the user based on that prediction. In this example, the input data sample to the machine learning model 105 may include user-related contextual information such as user information and historical user interactions, and information related to the items to be recommended. The output from the machine learning model 105 is a prediction result showing which items or types of items the user may be interested in. In another example, in a financial application, the prediction task is to predict future sales of a product. In this example, the input data sample to the machine learning model 105 may include information related to further time, the product and / or other related products, historical sales of the product and / or other related products, target geographical regions and target users of the product, etc. Only a limited number of examples are listed above, and it will be understood that the machine learning model 105 can be configured to embody other prediction tasks.

[0022] The machine learning model 105 can be constructed as a function that processes input data and generates an output as a prediction result. The machine learning model 105 can consist of a set of parameters whose values ​​are learned from training data throughout the training process. In Figure 1, the model training system 110 is configured to embody a training process that trains the machine learning model 105 based on the training data set 112. Initially, the machine learning model 105 may have initial parameter values ​​configured. During the training process, the initial parameter values ​​of the machine learning model 105 can be iteratively updated until the learning objective is achieved.

[0023] The training data set 112 may include a number of input data samples provided to the machine learning model 105 and labeling information indicating the corresponding ground truth labels for the input data samples. In some embodiments, an objective function is used to measure the error (or distance) between the output of the machine learning model 105 and the ground truth labels. Such an error is also called the machine learning loss, and the objective function may also be called the loss function. The loss function is

number

[0024] After the training process, the trained machine learning model 105, configured with updated parameter values, can be provided to a model application system that applies real-world input data samples 122 to the machine learning model 105 and outputs prediction results for the input data samples 122.

[0025] In Figure 1, the model training system 110 and the model application system 120 may be any system with computing capabilities. The components and arrangement in the environment shown in Figure 1 are merely examples, and it should be understood that a computing system suitable for embodying the exemplary implementations described in the subject matter herein may include one or more different components, other components, and / or different arrangements. For example, components shown separately may be integrated into the same system or device. Embodiments of this disclosure are not limited thereto.

[0026] In some cases, the input data processed by a machine learning model may have a certain periodicity. Such data is called periodic or cyclical data. For example, an application user may typically log in to the application within a relatively fixed time frame each day (e.g., before going to bed or after work) and may show the same interests at the same time frame on different days. Such cyclical patterns can lead to different predictive recommendations for the user. Therefore, it is expected that the machine learning model 105 can be trained to take advantage of the periodicity in the training data.

[0027] The problem of utilizing periodicity in training data to learn a better predictive model can be framed as follows:

number

number

number

number

number

[0028] X and Y are convex compact sets, and the loss function

number

number

number

[0029] The concept of periodicity plays an important role in equation (1). Specifically, due to periodicity, the function f with respect to time t * t It is guaranteed that (x) is also a solution at t+nT (where n is an integer greater than 0). This means that a predictive model learned at time t can provide useful information to improve the predictive accuracy at t+nT. Therefore, the inventors are motivated to design a learning algorithm that can effectively utilize such useful information provided by the periodic nature of the data.

[0030] Surprisingly, existing optimization and machine learning techniques provide little insight into how to efficiently solve Equation (1) in a big data environment by leveraging periodicity in the training data, while industrial systems embody algorithms that simply undervalue the periodicity in the training data.

[0031] When facing machine learning with periodic data, one straightforward design for encoding periodicity into the model structure is to simply include t as a model input and learn a function f(x,t). Unfortunately, this approach does not work as is. When the function f(x,t) is represented as a machine learning model, it has been shown that periodicity cannot be learned without using special activation functions. If f(x,t) belongs to a non-parametric family that encodes periodicity (such as a reproducing kernel Hilbert space (RKHS) using a periodic kernel or a Sobolev Hilbert space using periodic splines), periodicity is automatically encoded across all input dimensions, but in Equation (1), f(x,t) can be non-periodic in x.

[0032] An enhanced version of this approach is to preprocess time t and instead learn a function represented as f(x,mod(t,T)) that focuses on a single period of f(x,t). Preprocessing t to mod(t,T) guarantees periodicity at the inference stage, but it is often still necessary to perform laborious feature engineering, especially when x is high-dimensional and f(x,t) has a complex design.

[0033] Another approach to Equation (1) is to simply learn a prediction model for each t. Since this is often practically impossible, the time axis is often discretized, and the learner only needs to learn a finite set of models at several discrete time points, resulting in a multivariate approach. In the case of a machine learning system, this finite set of models can share the "base" part of the neural network, with only the last few layers being different. Advantageously, when the time-dependent distribution D t is stationary over time intervals, for example,

number

[0034] A further approach to training predictive models using continuous data is to follow an online learning protocol to which newly generated periodic data is applied to optimize the model. The performance of a learning algorithm is typically evaluated using the concept of dynamic regret, which measures the model's ability to consistently and accurately predict the labels of the latest batch of incoming data. Roughly speaking, if t takes the form of a set of discretized values, dynamic regret is the loss based on the learned model and f defined by equation (1). * t This is the cumulative sum of the differences between (x) and the optimal loss.

number

[0035] In summary, the problem of using periodic patterns in data distributions to train better models remains largely unsolved in large-scale settings. Working principles and theoretical analysis

[0036] Embodiments of this disclosure provide an improved approach to addressing machine learning challenges using periodic data. This approach proposes a novel learning framework called Fourier learning, which can be applied to training predictive models used in a variety of applications where periodic data is generated.

[0037] Before explaining the application of Fourier learning within predictive models, we will theoretically analyze and prove how Fourier learning can solve problems that utilize periodicity in training data to train better predictive models, such as the optimization problem of equation (1).

[0038] In embodiments of this disclosure, the proposed Fourier learning can solve the set of optimization problems in equation (1) as a single optimization problem in a function space that naturally contains time-periodic functions. In particular, the function space may be the tensor product of two Hilbert spaces, one containing a model snapshot at a fixed time point and the other containing a time-periodic function. As shown below, this leads to a partial Fourier expansion of these functions. In a convex analysis setting, it is also possible to learn the Fourier coefficients using streaming stochastic gradient descent (SGD). Theoretically, the proposed Fourier learning framework is supported in two different aspects: (i) from a modeling perspective, Fourier learning is naturally derived from a function optimization problem equivalent to the optimization problem in equation (1) under a setting that is strongly convex and achievable; (ii) from an optimization perspective, it has been demonstrated that the coefficient function updated by streaming SGD probably converges in the frequency domain. For practical applications, Fourier learning can be integrated into various predictive models to enable the predictive models to provide more accurate prediction results. By integrating with Fourier learning, a single model framework may suffice for predicting periodic data.

[0039] The theoretical foundation of the proposed Fourier learning is first introduced, which can be derived as a natural approach to function optimization problems. In embodiments of this disclosure, the set of learning problems in equation (1) is reformulated as a single learning problem in a Hilbert space. In practice, this makes it possible to learn a unified model that accepts both x and t as inputs. Specifically, the learning goal takes the form of equation (2), but in practice the expected value can be replaced with the empirical mean of the data set.

number

number

[0040] A key element in equation (2) is the design of the Hilbert space H from which f(x,t) is retrieved. In learning problems using periodic data, we particularly focus on functions in Hilbert spaces that are continuous, time-periodic, and have a finite energy within a single period. The inventors have discovered that the unification objective of equation (2) relates to equation (1) via the following Lemma Theorem 1. Lemma 1. f in equation (1) * t Regarding (x),

number

[0041] In Lemma 1, T represents the periodicity of x. The proof of Lemma 1 is as follows: Starting from equation (2), for any f(x,t)∈H,

number

number

[0042] Lemma 1 states that (2) has a unique minimization function, and f * t If (x) belongs to a Hilbert space H when treated as a function of both x and t, then the minimization function of (2) means that it yields a solution to equation (1). Thus, under feasible settings, equation (2) serves as a proxy for solving equation (1). As shown in the proof above, under feasible settings and strictly convexity used in Lemma 1, it can be shown that by minimizing the proxy loss specified in equation (2), a desired set of solutions to equation (1) can be obtained.

[0043] Another important element of (2) is the design of H. Here we focus particularly on a function that is continuous, time-periodic, and has a finite energy within a single period. Furthermore, the function in H must degenerate to L2(X) as specified by equation (1) for each fixed t. Two important elements are introduced that are necessary to design such H.

[0044] Furthermore, defining functions on a circle is an important way to characterize periodic functions. Since these functions are defined for points on the circle, they take angular information of the points as input, and therefore naturally have a period determined by the circumference. To facilitate optimization, we further define Hilbert space structures on these functions, based on the intuition that the circle is a line segment with its endpoints joined together, as follows:

number

[0045] Equation (5) shows that the function f has a finite energy.

number

number

number

[0046] L2(S T To further enhance this to a Hilbert space containing functions that depend on both x and t, the concept of the tensor product between Hilbert spaces is needed, which is a direct extension of the Kronecker product concept between vectors in Euclidean space to a function space.

[0047] Specifically, each

number

number

number

number

number

number

number

[0048] L2(S T To enhance ) by the tensor product with L2(X),

number

number

number

number

number

number

number

[0049] The above lemma opens the way to a theoretically guaranteed algorithm that optimizes L(f) through a basis expansion of f in H, which will be introduced below. On the other hand, H is general enough for learning purposes in the sense that the function f0(x,t) defined point by the solution of equation (1) belongs to H under mild assumptions. Some definitions and assumptions are introduced below.

[0050] Definition 3 (Continuity under overall fluctuation). D t (x) D t Let this be the conditional distribution of y given x under the following conditions: For any fixed t and any ε>0, where |t'-t|≦δ

number

[0051] Assumption 4. Assumption: (i) X and Y are compact and convex sets. (ii) D according to Definition 3. t (x) is continuous under all total variation of x ∈ X. (iii) Loss function

number

number

[0052] Assumption 4 can be easily satisfied by a wide range of machine learning systems. For example, in deep neural networks (DNNs), the output is usually limited when clipping of the final output is forced. The uniform and strongly convex nature of the loss function also applies to a wide range of l, such as mean squares loss. With the above definitions and assumptions, the inventors have discovered another lemma, Lemma 5.

[0053] Lemma 5. Under assumption 4, f * t (x) is continuous at t for any given x ∈ X. Also,

number

[0054] Lemma 5 means that, under assumption 4, the optimal solution f0(x,t) of equation (2) belongs to H. Combining Lemma 1 and Lemma 5, we can see that satisfying assumption 4 means that by solving equation (2), we can obtain the set of desirable solutions to equation (1). Fourier learning using periodic data

[0055] Next, we will introduce Fourier learning, a learning framework that incorporates the periodicity of the data distribution into the model structure via a partial Fourier expansion and learns the model by learning its Fourier coefficient function. To do this, Lemma 2 is invoked from a modeling perspective, and f(x,t)∈H can be expressed by the following basis expansion:

number

number

[0056] Theorem 6 provides an explicit method for designing periodic models and specifies how time features can be utilized. It should be noted that, to guarantee periodicity, it is entirely possible to construct H using a weighted L2 space defined on a circle. This allows for a departure from trigonometric functions and potentially the use of other periodic functions to encode periodicity.

[0057] Using f(x,t) expressed by equation (10), the solution to problem (2) is a n (x) and b n(x), that is, the Fourier coefficients of f(x,t) that are currently independent of t and depend only on x are learned. Furthermore, the sine and cosine components depend on t. Since equation (10) takes the form of a partial Fourier expansion of f(x,t), this learning method can be called "Fourier learning".

[0058] Fourier learning allows designers to retain their original model design while simultaneously mixing in expert advice from the final hidden layer in a time-dependent manner. This explicit role of t in the predictive model avoids the cumbersome feature engineering that would be required when features t are implicitly added to the model in the form f(x,t).

[0059] The goal then shifts to learning the coefficient function in the frequency domain. For easier learning, a cutoff frequency N / T can be introduced, so instead, the truncated Fourier expansion of f(x,t) can be expressed as follows:

number

[0060] The truncated Fourier expansion of equation (11) is an approximation of the Fourier expansion of equation (10). The approximation error for all f∈H in equation (11) can be determined as follows: E N It can be expressed as (f):

number

[0061] In the Fourier expansion of equation (11) in Fourier learning, the Fourier coefficient a is used to generate the prediction result of the model f(x,t). n (x), b n We need to determine (x). Fourier coefficient an (x) and b n (x) can be thought of as a coefficient function that depends on x and can be learned under various regimes. For example, these can be learned nonparametrically using function optimization algorithms.

[0062] In some embodiments, the Fourier coefficient a n (x) and b n When (x) has a parametric form such as a neural network, it is known that stochastic gradient descent converges to a stationary point at a certain rate under standard assumptions, which can be described in detail below. In some embodiments, Fourier learning also fits nonparametric regimes, separate from the parametric framework described above, which can be described in detail below. Machine learning system based on Fourier learning

[0063] Below, we will discuss how to apply Fourier learning to large-scale machine learning scenarios using neural networks. n (x) and b n Let's explain the parameterization of (x).

[0064] The theoretical analysis described above demonstrates that models built on Fourier learning can intuitively utilize the periodicity of training data and express it as a periodic function possessing that periodicity. Therefore, Fourier learning can be adapted to machine learning-based predictive models. In embodiments of this disclosure, it is proposed that x be considered as information related to an input data sample generated at a certain time t, according to the Fourier expansion of f(x,t) in equation (11). The input data sample may be a data sample of periodic data. The Fourier expansion result can be determined based on the Fourier expansion, and the prediction result of the input data sample is determined based on the Fourier expansion result.

[0065] Referring now to Figure 2, Figure 2 shows a block diagram of a machine learning system 200 using Fourier learning according to some exemplary embodiments of the present disclosure. The machine learning system 200 may be embodied as a machine learning model 105 in an environment 100. As shown, the machine learning system 200 includes a predictive model 210 and a Fourier layer 220.

[0066] The prediction model 210 may be configured with any model architecture to embody the prediction task. In embodiments of this disclosure, the input data processed by the prediction model 210 is periodic data having a certain periodicity (represented by T). The input to the prediction model 210 is a sample of input data generated at a certain time t. A Fourier layer 220 is introduced to enable the generation of more accurate prediction results by taking into account the periodicity in the input data. In any case, it should be noted that the prediction model 210 can be constructed in any way that takes advantage of the periodicity of the input data, or does not, since the periodicity can be further utilized by adding the Fourier layer 220.

[0067] Fourier layer 220 is designed by the following intuition: If x is considered the output of the last hidden layer of the original prediction model, then equation (11) can be considered the output layer of a network having the architecture shown in Figure 2. Specifically, Fourier layer 220 first takes x as a n (x) and b n We convert them to (x), and then multiply them element by element with the basis vectors SIN and COS to obtain a (2N+1)-dimensional result. These results are added together to obtain a scalar output. In particular, for all n≧1 a n (x=b n When (x)=0, the final output is equal to b0(x), which can be interpreted as the output of the original model. This means that the output layer of the original model can be increased by replacing it with a 220 Fourier layer, thus avoiding the need for tedious feature engineering.

[0068] In particular, the Fourier layer 220 receives feature representations of input data samples extracted by the prediction model 210. The prediction model 210 can generally be thought of as consisting of two parts: one that extracts hidden features in the input data, and the other that determines the model output based on the final hidden features. In some embodiments, the prediction model 210 may include multiple layers, each including an input layer that receives input data samples, one or more hidden layers that process the input data samples to generate feature representations characterizing the hidden features in the input data, and an output layer that generates the model output. The layers of the prediction model 210 are connected layer by layer, with the output from one layer being provided as input to the next layer.

[0069] In some embodiments, the feature representation extracted in the last hidden layer 212 of the predictive model 210 is provided to the Fourier layer 220 as input. This feature representation is denoted by x. Typically, the input data samples may contain redundant information and may be of higher dimension. Through feature extraction in the predictive model, the feature representation may be able to characterize useful feature information within the input data samples of a relatively low dimension. The Fourier layer 220 may be able to further process the feature representation x to generate a prediction result for the input data samples.

[0070] Assume that x has dimension d1 and the prediction result of the input data sample has dimension d2. The dimension of the feature representation x and the dimension of the prediction result (d2) may depend on the configuration of the prediction model 210. In general, d1 may be greater than 1 and d2 may be greater than or equal to 1. For example, the prediction result may be a one-dimensional output showing, for example, the probability that a user is interested in a target item, or it may be a multi-dimensional output showing, for example, the probability that a user is interested in each of several items.

[0071] Given the input (i.e., feature representation x) and output (i.e., prediction result) of the Fourier layer 220, the operation of the Fourier layer 220 can be considered as mapping the input of dimension d1 to the output of dimension d2. The model structure of the Fourier layer 220 can be designed to embody such a mapping based on the Fourier expansion.

[0072] As shown in the figure, the Fourier layer 220 is the Fourier coefficient a in the Fourier expansion. n The mapping model 230 that generates (x), and the Fourier coefficients b in the Fourier expansion. n The mapping model 240 generates (x). Following a truncated Fourier expansion of equation (11) having a predetermined number of (N+1) terms, the mapping model 230 may be configured to transform a feature representation x of dimension d1 into an output of dimension N, and the mapping model 240 may be configured to transform a feature representation x of dimension d1 into an output of dimension (N+1). The N elements in the output of the mapping model 230 are the N Fourier coefficients.

number

number

[0073] Mapping models 230 and 240 can be constructed based on any machine learning architecture. In some embodiments, mapping models 230 and 240 can be constructed without activation functions. Generally, activation functions applied to machine learning models (e.g., sigmoid function, tanh function, ReLU function) can restrict the amplitude of the model output to a certain range. Since there is no explicit restriction on the amplitude of the Fourier coefficients in Fourier expansion, mapping models 230 and 240 can be constructed without activation.

[0074] A Fourier expansion generally includes a sine function-based component and a cosine function-based component. Therefore, as shown in Figure 2, the Fourier layer 220 further includes a sine function unit 232 that determines the value of the sine component in the Fourier expansion, and a cosine function unit 242 that determines the value of the cosine component in the Fourier expansion. As shown in equation (11), the sine component is based on a sine function that depends on time t, which is a periodic function with periodicity T, and the cosine component is based on a cosine function that depends on time t, which is a periodic function with periodicity T.

[0075] The sine function unit 232 stores the set of sine component values ​​in a column vector.

number

number

[0076] N sign component values

number

number

[0077] In Fourier layer 220, the Fourier coefficients are generated in response to the input data samples generated at each time point.

number

number

number

[0078] N sine component values ​​and N Fourier coefficients can be provided to multiplier 234. Multiplier 234 is configured to perform element-wise multiplication on the N sine component values ​​and N Fourier coefficients to produce N products. (N+1) cosine component values ​​and (N+1) Fourier coefficients can be provided to multiplier 244. Multiplier 244 is configured to perform element-wise multiplication on the (N+1) cosine component values ​​and (N+1) Fourier coefficients to produce (N+1) products. The (2N+1) products correspond to the individual terms involved in the Fourier expansion.

[0079] To obtain a prediction result for dimension d2, N products from multiplier 234 may be input to mapping model 236, and (N+1) products from multiplier 244 may be input to mapping model 246. Mapping model 236 may be configured to convert the N products from multiplier 234 into a first intermediate expansion result for dimension d2, and mapping model 246 may be configured to convert the (N+1) products from multiplier 244 into a second intermediate expansion result for dimension d2. The first and second intermediate expansion results may be provided to an aggregator 250 configured to perform an element-wise sum on the first and second intermediate expansion results to provide a Fourier expansion result which can be determined as a prediction result for input data samples of dimension d2.

[0080] In some embodiments, if the prediction result is a one-dimensional output, mapping models 236 and 246 may be omitted from the Fourier layer 220. In this case, the products from multipliers 234 and 244 are summed to provide a Fourier expansion result that can be determined as the prediction result.

[0081] In some embodiments, mapping models 230 and 240 can be constructed as multilayer perceptron (MLP) models. In some embodiments, mapping models 236 and 246 can be constructed as MLP models. Thus, the Fourier layer 220 can be considered a Fourier MLP (F-MLP) layer.

[0082] The parameter values ​​of the mapping models 220, 240, 236, and 246 in the Fourier layer 220 can be determined through the training process. In some embodiments, these mapping models can be trained using a predictive model 210. The training data may include input data samples to the predictive model 210 and labeling information indicating the corresponding ground truth labels for the input data samples. In some embodiments, the mapping models in the Fourier layer are obtained by training end-to-end using the predictive model 210. In some embodiments, the predictive model 210 can be trained first and then retrained together with the mapping models in the Fourier layer.

[0083] In some embodiments, the Fourier layer 220 is added to an F-MLP having input dimension d1 and output dimension d2.

number

number

number

number

number

number

number

number

[0084] It should be noted that there are several approaches that propose combining Fourier analysis with deep learning systems. However, most of these approaches aim to learn intrinsic high-frequency components within the distribution of the input data itself, rather than focusing on the periodicity of the distribution over time. In particular, the inventors have observed that the embodiment of the Fourier layer of this disclosure into existing designs of predictive models fundamentally alters the physical meaning of each processing unit in the model: in a typical model, each processing unit is an expert who modifies decisions over time. Under the Fourier layer of this disclosure, each processing unit holds the frequency components of an expert who determines how significantly they modify their decisions over time at a given frequency. The former requires designing an online learning algorithm that tracks the constantly changing optimal value for each expert, whereas the latter allows for the prediction of future optimal values ​​using trigonometric interpolation. This provides advantages over the online learning approach.

[0085] In the exemplary embodiment shown in Figure 2, the Fourier layer 220 is introduced as the output layer of the prediction model 210, and its output is therefore determined as the prediction result for the input data samples. In some embodiments, the Fourier layer 220 may operate together with the complete prediction model 210 (including its own output layer), and the output from the Fourier layer 220 and the output from the prediction model 210 are aggregated to produce the final prediction result. Figure 3 shows a machine learning system 200 according to such an embodiment.

[0086] As shown in Figure 3, the prediction model 210 includes an output layer 312 that receives a feature representation x from the last hidden layer 212, among other layers. The output layer 312 of the prediction model 210 may process the feature representation x and generate an intermediate prediction result. The processing in the output layer 312 may depend on the configuration of the prediction model 210, which may be modified for different prediction tasks. The Fourier layer 220 may also receive a feature representation x from the last hidden layer 212 and generate an intermediate prediction result based on the Fourier expansion result, as described according to the embodiment relating to Figure 2.

[0087] The machine learning system 200 may further include an aggregator 330 configured to mix two intermediate prediction results from the prediction model 210 and the Fourier layer 220. For example, the aggregator 330 may determine a weighted sum of the two intermediate prediction results. The aggregation of the intermediate prediction results may be expressed as follows:

number

[0088] In some embodiments, the predictive model 210 may have a complex structure, for example, including multiple submodels with different model structures. In this case, the input to the Fourier layer 220 may be carefully designed. Figure 3 shows a machine learning system 200 according to such an embodiment.

[0089] As shown in Figure 4, the prediction model 210 may include multiple submodels (e.g., K submodels), such as submodel 410-1, ..., submodel 410-K (referred to collectively or individually as submodel 410 for the purposes of discussion), where K is an integer greater than 1. The outputs of the submodels may be added together in the output layer 312 of the prediction model 210 to provide the output of the model. In this case, the output layer 312 may include an aggregator that performs a sum on each of the K submodels 410. Thus, the prediction model 210 is,

number

[0090] The structure of the predictive model 210 shown in Figure 4 allows a submodel 410 to extract feature representations from input data samples in the final hidden layer and determine its own output in its output layer based on these feature representations. Feature representations from the submodel 410 can be aggregated to generate feature representation x, which is input to the Fourier layer 220. In some embodiments, the feature representations from the submodel 410 may be of different dimensions. In the embodiment of Figure 4, to aggregate the feature representations from the submodel 410, the machine learning model 200 may further include a dimensionality-aligned layer 420 that transforms each feature representation from the submodel 410 having different dimensions into a feature representation having the same dimension. In some embodiments, the dimensionality-aligned layer 420 may include a plurality of MLPs, each configured to transform a feature representation from one of the submodels 410 into a feature representation having the same dimension.

[0091] Since the Fourier layer 220 performs a linear transformation on its input, it can obtain a feature representation x to input to the Fourier layer 220 by adding feature representations of the same dimension generated from the dimensional sorting layer 420. In this case, the dimensional sorting layer 420 can transform feature representations of different dimensions from the submodel 410 into feature representations of dimension d1.

[0092] Figure 4 shows that the output from the Fourier layer 220 is aggregated with the output from the predictive model 210 by the aggregator 330, similar to Figure 3. In other embodiments, it will be understood that processing of feature representations can be integrated into the system 200 shown in Figure 2. Training a machine learning system

[0093] The Fourier layer 220 is trained in conjunction with the predictive model following the streaming SGD procedure. This procedure is actually performed using sample data (x,y,t) ~ D tThis differs from standard SGD, which requires (x,y)p(t). However, sampling from p(t) is difficult in many online applications due to the real-time update requirement where data arrives sequentially. Here, we demonstrate that using streaming SGD can circumvent this problem while maintaining excellent practical performance and convergence guarantees.

[0094] The training procedure is as follows: a n (x) and b n (x) is a n (x;θ n ) and b n (x;ρ n It can be parameterized by θ. n and ρ n are the neural network parameters. For periodic data, the τ-th mini-batch of data is collected in the k-th cycle, and the model can be updated according to the following update rule:

number

number

number

number

number

[0095] The convergence properties are discussed when training a machine learning system based on Fourier learning using streaming SGD. Truncating f N (x,t) is used, and it is recalled that the problem in equation (2) reduces to finding the optimum value in the frequency domain given by equation (17) below:

Number

Number

[0096] Below, first, the gradient norm convergence results of streaming SGD under a general non-convex setting are shown, and then the global convergence results under the assumption of strong convexity are introduced. Before that, some additional assumptions are introduced.

[0097] Assumption 7. Hypothesis: (i) For all n, k, τ, the second moment of the update direction is bounded: for some

Number

Number

Number

[0098] [[ID=**57**]] Assumption 7 assumes a bounded second moment in the update direction and the Lipschitz property of the gradient, which are typically required in convergence analysis of SGD-type algorithms. The following results show that streaming SGD with an appropriate learning rate achieves convergence under both non-convex and strongly convex settings.

[0099] Theorem 8 (Convergence of Streaming SGD). Assume that (i) and (ii) of Assumption 4 and Assumption 7 hold, all

number

number

number

[0100] Simply put, the learning framework is, under general non-convex settings,

number

number

[0101] As described above, apart from the parametric framework described above, in some embodiments, the proposed Fourier learning also fits nonparametric regimes, where a n and b nThis will be updated directly:

number

[0102] The L2 function gradient often contains the Dirac delta function, which causes discontinuous updates; therefore, the function gradient is replaced with its kernel embedding instead. Specifically,

number

number

number

number

number

number

number

[0103] Figure 6 shows a flowchart of a process 600 for Fourier learning according to some exemplary embodiments of the present disclosure. The process 600 may be embodied in a machine learning system 200, or in a model application system 120 that can apply input data to the machine learning system 200 to perform a corresponding prediction task. For the purposes of discussion, the process 600 will be discussed with reference to Figure 1.

[0104] In block 610, the model application system 120 obtains feature representations of input data samples from a predictive model. The predictive model is configured to process the input data periodically. The input data samples are samples of input data generated at points in time within the period.

[0105] In block 620, the model application system 120 determines the first Fourier coefficient of the first component in the Fourier expansion by applying the feature representation to a first mapping model. The Fourier expansion is periodic and depends on the time and the feature representation. In block 620, the model application system 120 determines the second Fourier coefficient of the second component in the Fourier expansion by applying the feature representation to a second mapping model.

[0106] In block 640, the model application system 120 determines the Fourier expansion result based on the first and second Fourier coefficients in the Fourier expansion. In block 650, the model application system 120 determines the prediction result of the input data sample based on the Fourier expansion result.

[0107] In some embodiments, the Fourier expansion includes a truncated Fourier expansion having a predetermined number of terms, where the number of the first Fourier coefficients and the number of the second Fourier coefficients are based on the predetermined number.

[0108] In some embodiments, the first component is based on a time-dependent, periodic sine function, and the second component is based on a periodic cosine function.

[0109] In some embodiments, to determine the Fourier expansion result, the model application system 120 determines a set of first component values ​​for the first component by shifting the frequency of the sine function a predetermined number of times, and determines a set of second component values ​​for the second component by shifting the frequency of the cosine function a predetermined number of times. The model application system 120 determines the Fourier expansion result by multiplying the first Fourier coefficients by the first component values ​​and the second Fourier coefficients by the second component values.

[0110] In some embodiments, to determine the Fourier expansion result, the model application system 120 calculates a first product by multiplying the first Fourier coefficients by the first component values, and a second product by multiplying the second Fourier coefficients by the second component values. The model application system 120 maps the first product to a first intermediate expansion result using a third mapping model, and maps the second product to a second intermediate expansion result using a fourth mapping model. The model application system 120 determines the Fourier expansion result by aggregating the first and second intermediate expansion results.

[0111] In some embodiments, to determine the prediction result, the model application system 120 determines a first intermediate prediction result from the Fourier augmentation result and obtains a second intermediate prediction result generated from the output layer of the prediction model based on the feature representation. The model application system 120 determines the prediction result by aggregating the first intermediate prediction result and the second intermediate prediction result.

[0112] In some embodiments, the predictive model includes multiple submodels configured to extract multiple feature representations from input data samples. In some embodiments, the model application system 120 obtains multiple feature representations from the multiple submodels and generates a feature representation by aggregating the multiple feature representations.

[0113] In some embodiments, the first and second mapping models are constructed without activation functions. In some embodiments, the third and fourth mapping models are constructed without activation functions. In some embodiments, the mapping models are trained in conjunction with the prediction model. Exemplary system / device

[0114] Figure 7 shows a block diagram of an exemplary computing system / device 700 suitable for embodying exemplary embodiments of the present disclosure. The model application system 120 and / or model training system 110 may be embodied as system / device 700 or may be included in system / device 700. System / device 700 may be a general-purpose computer or computer system, a physical computing system / device, or a portable electronic device, or may run in a distributed cloud computing environment where tasks are performed by remote processing devices linked over a communication network. System / device 700 may be used to embody process 600 in Figure 6.

[0115] As shown in the figure, the system / device 700 includes a processor 701 that can perform various operations according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 also stores data necessary for the processor 701 to perform various operations as needed. The processor 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0116] The processor 701 can be any type suitable for a local technology network and may, in non-limiting examples, include one or more of the following: general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), graphics processing units (GPUs), coprocessors, and processors based on multi-core processor architectures. The system / device 700 may have multiple processors, such as application-specific integrated circuit chips that are slaved to a clock that synchronizes the main processor.

[0117] Multiple components within the system / device 700 are connected to input units 707 such as keyboards and mice, output units 707 including displays such as cathode ray tubes (CRTs) and liquid crystal displays (LCDs), I / O interfaces 705 including loudspeakers, storage units 708 such as disks and optical disks, and communication units 709 such as network cards, modems, and wireless transceivers. The communication units 709 enable the system / device 700 to exchange information / data with other devices via communication networks such as the Internet and various telecommunications networks.

[0118] The above-described methods and processes, such as process 600, can also be executed by a processor 701. In some embodiments, process 600 can be embodied as a computer software program or a computer program product tangibly included in a computer-readable medium, such as storage unit 708. In some embodiments, the computer program can be partially or fully loaded and / or embodied in system / device 700 via ROM 702 and / or communication unit 709. The computer program includes computer-executable instructions executed by an associated processor 701. When the computer program is loaded into RAM 703 and executed by processor 701, one or more operations of the above-described process 600 can be embodied. As an alternative, in other embodiments, processor 701 can be configured via any other suitable means (e.g., by firmware) to execute process 600.

[0119] In some exemplary embodiments of the present disclosure, there is provided a computer program product including instructions that, when executed by a processor of a device, cause the device to execute any one of the steps of the above-described methods.

[0120] In some exemplary embodiments of the present disclosure, there is provided a computer-readable medium including program instructions for causing a device to execute any one of the steps of the above-described methods. In some embodiments, the computer-readable medium can be a non-transitory computer-readable medium.

[0121] In an eighth aspect, an exemplary embodiment of the present disclosure provides a computer-readable medium including program instructions for causing a device to execute at least the method of the second aspect described above. In some embodiments, the computer-readable medium can be a non-transitory computer-readable medium.

[0122] In general, the various exemplary embodiments of the present disclosure may be embodied in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be embodied in firmware or software that may be executed by a controller, microprocessor, or other computing device. Although the various aspects of the exemplary embodiments of the present disclosure have been illustrated and described using block diagrams, flowcharts, or other graphical representations, the blocks, devices, systems, techniques, or methods described herein may be embodied, by way of non-limiting example, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0123] The present disclosure also provides at least one computer program product tangibly stored in a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules that are executed on a device on a target physical processor or virtual processor to perform the methods / processes described above. In general, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or embody particular abstract types. The functions of program modules may be combined or divided among program modules as needed in various embodiments. The computer-executable instructions of program modules may be executed within a local or distributed device. In a distributed device, program modules may be located in both local and remote storage media.

[0124] Computer-readable media may be computer-readable signal media or computer-readable storage media. Computer-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More specific examples of computer-readable storage media may include electrical connections having one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0125] Computer program code for performing the methods disclosed herein may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable data processing device so that, when executed by the processor or controller, the program code embodies the functions / operations specified in the flowcharts and / or block diagrams. The program code may run entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server. The program code may be distributed on specially programmed devices, commonly referred to herein as “modules.” The software component portions of a module may be written in any computer language, may be part of a monolithic codebase, or may be developed in more isolated code portions, as is common in object-oriented computer languages. Furthermore, modules may be distributed across multiple computer platforms, servers, terminals, mobile devices, and the like. A given module may be embodied so that the described functions are performed by separate processors and / or computing hardware platforms.

[0126] While the operations are shown in a specific order, this should not be understood as requiring that such operations be performed in a specific or sequential order, or that all illustrated operations be performed, in order to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, while some specific embodiment details are included in the above description, these should not be interpreted as limitations on the scope of this disclosure, but rather as descriptions of features specific to a particular embodiment. Certain features described in the context of a separate embodiment may be embodied in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be embodied individually or in any suitable subcombination in multiple embodiments.

[0127] Although this disclosure is described in language specific to structural features and / or methodological actions, it should be understood that the disclosure as defined in the attached claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms that embody the claims.

Claims

1. A method performed by an electronic device, A step of obtaining a feature representation of an input data sample from a predictive model, wherein the predictive model is configured to process periodic input data, and the input data sample is a sample of the input data generated at some point in time within the period. A step of determining the first Fourier coefficient of the first component in a Fourier expansion by applying the feature representation to a first mapping model, wherein the Fourier expansion depends on the time and the feature representation, the Fourier expansion has the periodicity, and the first component is based on a sine function that depends on the time and has the periodicity; A step of determining the second Fourier coefficient of the second component in the Fourier expansion by applying the feature representation to a second mapping model, wherein the second component is based on the periodic cosine function, A step of determining the Fourier expansion result based on the first Fourier coefficient and the second Fourier coefficient in the Fourier expansion, A method comprising the step of determining a prediction result of the input data sample based on the Fourier expansion result.

2. The method according to claim 1, wherein the Fourier expansion includes a truncated Fourier expansion having a predetermined number of terms, and the number of the first Fourier coefficients and the number of the second Fourier coefficients are based on the predetermined number.

3. The step of determining the Fourier expansion result is: The steps include determining a set of first component values ​​for the first component by shifting the frequency of the sine function a predetermined number of times, The steps include determining the set of second component values ​​for the second component by shifting the frequency of the cosine function a predetermined number of times, The method according to claim 2, further comprising the step of determining the Fourier expansion result by multiplying the first Fourier coefficients by the first component values ​​and multiplying the second Fourier coefficients by the second component values.

4. The step of determining the Fourier expansion result by multiplying the first Fourier coefficient by the first component value and multiplying the second Fourier coefficient by the second component value is as follows: The first step is to calculate the first product by multiplying the first Fourier coefficient by the first component value, The steps include: calculating the second product by multiplying the second Fourier coefficient by the second component value; The steps include mapping the first product to a first intermediate expansion result using a third mapping model, and mapping the second product to a second intermediate expansion result using a fourth mapping model, The method according to claim 3, further comprising the step of determining the Fourier expansion result by aggregating the first intermediate expansion result and the second intermediate expansion result.

5. The step of determining the prediction result of the input data sample based on the Fourier expansion result is: The steps include determining a first intermediate prediction result from the Fourier expansion result, The steps include obtaining a second intermediate prediction result generated from the output layer of the prediction model based on the aforementioned feature representation, The method according to claim 1, further comprising the step of determining the prediction result by aggregating the first intermediate prediction result and the second intermediate prediction result.

6. The prediction model includes a plurality of submodels configured to extract a plurality of feature representations from the input data samples, and the step of obtaining the feature representations is: A step of obtaining the plurality of feature representations from the plurality of submodels, The method according to claim 1, further comprising the step of generating the feature expression by aggregating the plurality of feature expressions.

7. The method according to claim 1, wherein the first mapping model and the second mapping model are constructed without an activation function.

8. At least one processor, A system including at least one memory, The memory is communicatively coupled to the at least one processor, and when executed by the at least one processor, it is sent to the at least one processor. A step of obtaining a feature representation of an input data sample from a predictive model, wherein the predictive model is configured to process periodic input data, and the input data sample is a sample of the input data generated at some point in time within the period. A step of determining the first Fourier coefficient of the first component in the Fourier expansion by applying the feature representation to a first mapping model, wherein the Fourier expansion depends on the time and the feature representation, the Fourier expansion has the periodicity, and the first component is based on a sine function that depends on the time and has the periodicity; A step of determining the second Fourier coefficient of the second component in the Fourier expansion by applying the feature representation to a second mapping model, wherein the second component is based on the periodic cosine function, A step of determining the Fourier expansion result based on the first and second Fourier coefficients in the Fourier expansion, A system including a computer-readable instruction causing an operation to be performed, which includes the step of determining a prediction result for the input data sample based on the Fourier expansion result.

9. The system according to claim 8, wherein the Fourier expansion includes a truncated Fourier expansion having a predetermined number of terms, and the number of the first Fourier coefficients and the number of the second Fourier coefficients are based on the predetermined number.

10. Determining the Fourier expansion result is The steps include determining a set of first component values ​​for the first component by shifting the frequency of the sine function a predetermined number of times, The steps include determining the set of second component values ​​for the second component by shifting the frequency of the cosine function a predetermined number of times, The system according to claim 9, comprising the step of determining the Fourier expansion result by multiplying the first Fourier coefficients by the first component values ​​and multiplying the second Fourier coefficients by the second component values.

11. Determining the Fourier expansion result by multiplying the first Fourier coefficient by the first component value and the second Fourier coefficient by the second component value is: The first step is to calculate the first product by multiplying the first Fourier coefficient by the first component value, The steps include: calculating the second product by multiplying the second Fourier coefficient by the second component value; The steps include mapping the first product to a first intermediate expansion result using a third mapping model, and mapping the second product to a second intermediate expansion result using a fourth mapping model, The system according to claim 10, further comprising the step of determining the Fourier expansion result by aggregating the first intermediate expansion result and the second intermediate expansion result.

12. Determining the prediction result of the input data sample based on the Fourier expansion result is: The steps include determining a first intermediate prediction result from the Fourier expansion result, The steps include obtaining a second intermediate prediction result generated from the output layer of the prediction model based on the aforementioned feature representation, The system according to claim 8, further comprising the step of determining the prediction result by aggregating the first intermediate prediction result and the second intermediate prediction result.

13. The prediction model includes a plurality of submodels configured to extract a plurality of feature representations from the input data samples, and obtaining the feature representations is A step of obtaining the plurality of feature representations from the plurality of submodels, The system according to claim 8, further comprising the step of generating the feature representation by aggregating the plurality of feature representations.

14. The system according to claim 8, wherein the first mapping model and the second mapping model are constructed without activation functions.

15. A non-temporary computer-readable storage medium for storing computer-readable instructions, When the computer-readable instruction is executed by the computing device, the computing device will: A step of obtaining a feature representation of an input data sample from a predictive model, wherein the predictive model is configured to process periodic input data, and the input data sample is a sample of the input data generated at some point in time within the period. A step of determining the first Fourier coefficient of the first component in the Fourier expansion by applying the feature representation to a first mapping model, wherein the Fourier expansion depends on the time and the feature representation, the Fourier expansion has the periodicity, and the first component is based on a sine function that depends on the time and has the periodicity; A step of determining the second Fourier coefficient of the second component in the Fourier expansion by applying the feature representation to a second mapping model, wherein the second component is based on the periodic cosine function, A step of determining the Fourier expansion result based on the first and second Fourier coefficients in the Fourier expansion, A non-temporary computer-readable storage medium that performs an operation including the step of determining a prediction result for the input data sample based on the Fourier expansion result.

16. The non-temporary computer-readable storage medium according to claim 15, wherein the Fourier expansion includes a truncated Fourier expansion having a predetermined number of terms, and the number of the first Fourier coefficients and the number of the second Fourier coefficients are based on the predetermined number.

17. Determining the Fourier expansion result is The steps include determining a set of first component values ​​for the first component by shifting the frequency of the sine function a predetermined number of times, The steps include determining the set of second component values ​​for the second component by shifting the frequency of the cosine function a predetermined number of times, A non-temporary computer-readable storage medium according to claim 16, comprising the step of determining the Fourier expansion result by multiplying the first Fourier coefficient by the first component value and multiplying the second Fourier coefficient by the second component value.

18. Determining the Fourier expansion result by multiplying the first Fourier coefficient by the first component value and the second Fourier coefficient by the second component value is: The first step is to calculate the first product by multiplying the first Fourier coefficient by the first component value, The steps include: calculating the second product by multiplying the second Fourier coefficient by the second component value; The steps include mapping the first product to a first intermediate expansion result using a third mapping model, and mapping the second product to a second intermediate expansion result using a fourth mapping model, A non-temporary computer-readable storage medium according to claim 17, comprising the step of determining the Fourier expansion result by aggregating the first intermediate expansion result and the second intermediate expansion result.

19. Determining the prediction result of the input data sample based on the Fourier expansion result is: The steps include determining a first intermediate prediction result from the Fourier expansion result, The steps include obtaining a second intermediate prediction result generated from the output layer of the prediction model based on the aforementioned feature representation, A non-temporary computer-readable storage medium according to claim 15, comprising the step of determining the prediction result by aggregating the first intermediate prediction result and the second intermediate prediction result.

20. The prediction model includes a plurality of submodels configured to extract a plurality of feature representations from the input data samples, and obtaining the feature representations is A step of obtaining the plurality of feature representations from the plurality of submodels, A non-temporary computer-readable storage medium according to claim 15, comprising the step of generating the feature representation by aggregating the plurality of feature representations.

Images