Feature Token conversion method and system based on Kolmogorov-Arnold network

Abstract

The invention discloses a feature Token conversion method and system based on a Kolmogorov-Arnold network, and the method comprises the steps: carrying out the nonlinear transformation of an input feature through a Gaussian process of a cubic B-spline function or a Materon kernel, and generating an embedded representation through the combination of basic transformation. Compared with an original FT-Transform, the FT-Transform in which the feature token conversion method of the Kolmogorov-Arnold network based on the cubic B-spline function is embedded has the advantages that the training speed is increased, and the training time is prolonged by about 12.44% under an early stop mechanism with the path being 16. Compared with the prior art, the feature token conversion method of the Kolmogorov-Arnold network using the cubic B-spline function is further optimized through the Mateur Gaussian kernel, and compared with the FT-Transform using the cubic B-spline function, the forward reasoning speed is increased by about 70.82%, and the back propagation speed is increased by about 36.25%. The end-side system based on the feature token conversion method of the Kolmogorov-Arnold network improves the back propagation speed, the training speed and the reasoning speed, significantly reduces the training time and energy consumption of end-side equipment, and provides powerful support for efficient training and reasoning in a resource-constrained environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a feature Token conversion method and system based on a Kolmogorov-Arnold network. BACKGROUND

[0002] End-side training refers to the process of training artificial intelligence models directly on terminal devices such as smartphones, Internet of Things devices, etc. Unlike traditional cloud training modes, end-side training moves data processing and model training tasks to local devices, with the following advantages:

[0003] Privacy protection: User data does not need to be uploaded to the cloud, effectively protecting user privacy.

[0004] Low latency: Local processing reduces data transmission time and improves response speed.

[0005] Offline function: The device can update and train the model without network connection.

[0006] However, the computing and storage resources of end-side devices are usually limited, which poses challenges to the complexity and training efficiency of the model. Therefore, model compression, quantization and other technologies need to be used to ensure that the model adapts to the resource limitations of the end-side environment while ensuring performance.

[0007] FT-Transformer (Feature Token Transformer) is an architecture that inputs structured data (such as table data) into a Transformer model. In traditional Transformers, input data is usually in sequence form, such as text or time series. However, for structured data, directly inputting it into the Transformer may not work well, because the feature dimension and representation of these data are different from sequence data. To solve this problem, researchers have proposed the FT-Transformer architecture, which "tokenizes" the features of structured data to make them suitable for the processing of Transformers.

[0008] In FT-Transformer, the Feature Tokenizer component plays a crucial role. The main function of this component is to convert the input structured data into a sequence of tokens suitable for Transformer processing, which includes handling numerical features and categorical features, as well as adding a special [CLS] token at the beginning of the feature sequence. The Transformer model then processes these tokens, and finally uses the representation of the [CLS] token for prediction. Specifically, the Feature Tokenizer component performs the following operations:

[0009] Numerical Feature Processing: For numerical features, the Feature Token component usually applies a linear transformation or other non-linear mapping to convert them into embedding vectors. These embedding vectors are input to the Transformer as tokens.

[0010] Categorical Feature Processing: For categorical features, the component uses embedding lookup to map each category to an embedding vector, which is also input to the Transformer as tokens.

[0011] Through the above processing, the Feature Token component converts the features of structured data into tokens, making them available for effective processing by the Transformer model. This approach takes full advantage of the Transformer's ability to capture complex dependencies between features, improving the model's performance in processing structured data.

[0012] However, the expressiveness of the Feature Tokenizer method is limited, and it may not be able to fully capture the complex nonlinear relationships between features, resulting in poor performance of the model in processing complex data. Specifically:

[0013] 1. Limitations of simple linear transformation: Existing Feature Tokenizer methods usually use simple linear transformations (such as fully connected layers) to map input features to embedding space. The expressiveness of this method is limited, and it may not be able to fully capture the complex nonlinear relationships between features, resulting in poor performance of the model in processing complex data.

[0014] 2. Lack of non-linear modeling ability: Due to the lack of non-linear activation functions, existing linear transformations may not be able to effectively fit complex feature interaction patterns, limiting the performance of the model in tasks that require high expressiveness.

[0015] 3. Insufficient ability to fit complex patterns: When dealing with data with complex patterns and high-order feature interactions, simple linear transformations may fail to capture key feature relationships, affecting the model's prediction accuracy.

[0016] The three drawbacks mentioned above all lead to longer training times for the model to learn these complex patterns. On edge devices, in resource-constrained environments, how to perform feature processing and model training quickly and efficiently has become a pressing issue. Existing feature tokenization methods suffer from inefficiencies and limited feature representation capabilities when processing complex data, resulting in inefficient training.

[0017] Therefore, it is necessary to design an efficient feature token conversion method. Summary of the Invention

[0018] This invention aims to overcome the shortcomings of existing technologies and provide a feature token transformation method based on the Kolmogorov-Arnold network and its efficient edge training system to improve the efficiency and performance of feature processing. Specifically, the efficient feature token transformation method based on the Kolmogorov-Arnold network is used to replace the feature tokenizer structure in the traditional FT-Transformer.

[0019] This invention is specifically achieved through the following technical solution:

[0020] A feature token transformation method based on the Kolmogorov-Arnold network, characterized in that the method includes:

[0021] Input processing: Obtain an input tensor with a batch size of B and a number of features of N. Where B is the batch size and N is the number of features;

[0022] Mesh initialization: Set the mesh range [a,b] to [-1,1], the mesh size to G, calculate the step size h = (ba) / G, and generate a set of mesh points gridd that extends to [-1-3h,1+3h].

[0023] grid = {t -3 ,t -2 ,t -1 ,t0,…,t G ,t G+1 ,t G+2 ,t G+3};

[0024] Cubic B-spline basis function calculation: For input x, calculate the cubic B-spline basis function B according to the recursive formula. i,3 (x), where the basis function is in [t] when k = 3.i ,t i+4 [Non-zero]

[0025] Spline transformation: Calculate the spline basis matrix Combine spline weights The spline output, spline_output∈R, is obtained by Einstein summation. B×N×D ;

[0026] Basic transformation: Apply the SiLU activation function f(x) = x·σ(x) to the input x, combined with the basic weights W. b ∈R N×D The base output ∈ R is obtained. B×N×D ;

[0027] Final output: The spline output is superimposed with the base output to obtain the feature token representation. Where D is the embedding dimension, i.e.

[0028] y b,n,d =f(x) b,n )·W b,n,d +∑ i B i,3 (x b,n )·W s,n,d,i .

[0029] As a further improvement of the present invention, the recursive formula of the cubic B-spline basis function is as follows:

[0030] 0th order basis functions:

[0031]

[0032] k-th order basis functions, k = 1, 2, 3:

[0033]

[0034] As a further improvement of the present invention, the Einstein summation expression for the spline transformation is:

[0035] spline_output b,n,d =∑ i spline_bases b,n,i ·W s,n,di .

[0036] A feature token transformation method based on the Kolmogorov-Arnold network, characterized in that the method includes:

[0037] Input processing: Obtain an input tensor with a batch size of B and a number of features of N. Where B is the batch size and N is the number of features;

[0038] Matérn kernel definition: Based on the smoothing parameter v and the length scale l, the distance d = |x1-x2| between inputs x1 and x2 is defined, and the root term... Calculate the Matérn kernel k(x1,x2), where:

[0039] When v = 0.5:

[0040]

[0041] When v = 1.5:

[0042]

[0043] When v = 2.5:

[0044]

[0045] Gaussian process transformation: setting the mesh Where G is the number of grid points, ranging from [grid_min, grid_max]. For each input dimension i, calculate the Matérn kernel value of x[:,i] of input dimension B×1 and the grid points:

[0046]

[0047] Finally obtained

[0048] GP output calculation: combining GP weights Obtained through Einstein's summation method

[0049] Basic transformation: Apply the SiLU activation function f(x) = SiLU(x) - x·σ(x) to the input x, combined with the basic weights W. b ∈R N×D The base output ∈ R is obtained. B×N×D ;

[0050] Final output: The GP output is superimposed with the basic output to obtain the feature token representation. Where D is the embedding dimension, i.e.

[0051] y b,i,d =f(x) b,i )·W b,i,d +∑ k k(x b,i ,grid k )·W gp,i,n,d .

[0052] As a further improvement of the present invention, the Einstein summation expression for the Gaussian process transform is:

[0053] gp_output h,i,d =∑ n gp_features b,j,n ·W gp,i,n,d .

[0054] A feature token conversion system based on the Kolmogorov-Arnold network, characterized in that the system comprises:

[0055] Data acquisition module: used to acquire sensor data, including two-dimensional image data (H×W×C) and one-dimensional time series data;

[0056] Data preprocessing module: performs flattening, denoising, normalization, missing value imputation, and dimensionality checking on two-dimensional image data, and performs denoising, normalization, missing value imputation, and dimensionality checking on one-dimensional time series data;

[0057] KAFT / MKAFT module: Based on the above transformation method, it performs feature token transformation on the preprocessed data to generate feature representation.

[0058] Transformer module: contains a multi-layer self-attention mechanism and a feedforward network, which refines the feature representation T layer by layer;

[0059] Prediction module: Based on the feature representation output by Transformer, performs classification or regression tasks and outputs the result y.

[0060] The beneficial effects of this invention are as follows: Compared to the original FT-Transformer, the FT-Transformer embedding the feature token transformation method of the Kolmogorov-Arnold network based on cubic B-spline functions of this invention improves training speed and training time by approximately 12.44% under an early stopping mechanism with a patience of 16. The Matérn Gaussian kernel further optimizes the feature token transformation method of the Kolmogorov-Arnold network using cubic B-spline functions. Compared to the FT-Transformer using cubic B-spline functions, the Kolmogorov-Arnold feature token transformation method using Matérn Gaussian kernels improves forward inference speed by approximately 70.82% and backpropagation speed by approximately 36.25%. Furthermore, the edge system based on the feature token transformation method of the Kolmogorov-Arnold network of this invention improves backpropagation speed, training speed, and inference speed, significantly reducing training time and energy consumption of edge devices, providing strong support for efficient training and inference in resource-constrained environments. Attached Figure Description

[0061] Figure 1 This is a flowchart of the feature token transformation method for Kolmogorov-Arnold networks based on cubic B-spline functions according to the present invention.

[0062] Figure 2 This is a flowchart of the feature token conversion method for the Kolmogorov-Arnold network based on the Matérn kernel of the present invention;

[0063] Figure 3 A diagram of the edge system architecture for applying the feature token conversion method based on the Kolmogorov-Arnold network of this invention;

[0064] Figure 4 This is a schematic diagram of the theoretical structure of the Kolmogorov-Arnold network (KAN). Detailed Implementation

[0065] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0066] The feature token transformation method based on the Kolmogorov-Arnold network of the present invention includes a feature token transformation method based on cubic B-spline function (KAFT) and a feature token transformation method based on Matérn kernel (MKAFT).

[0067] The feature token transformation method for the Kolmogorov-Arnold network based on cubic B-spline functions is attached. Figure 1 As shown, the core of this method lies in using a cubic B-spline function (spline_order=3) to perform a nonlinear transformation on the input features and combining it with the basic transformation to generate an embedded representation.

[0068] 1. Input and Output Definitions

[0069] enter: Where B is the batch size and N is the number of features.

[0070] Output: Where D is the embedding dimension.

[0071] 2. Mesh initialization

[0072] Grid range: [a,b] = [-1,1], grid size is G.

[0073] Step size: h = (ba) / G.

[0074] Grid point: t i =a+i·h, extended to [-1-3h,1+3h] to support the interval of cubic B-spline.

[0075] Mathematical representation: grid = {t} -3 ,t -2 ,t -1 ,t0,…,t G ,t G+1 ,t G+2 ,t G+3}

[0076] 3. Calculation of cubic B-spline basis functions

[0077] B-spline basis functions are defined by recursive formulas. For input x and grid:

[0078] 0th order basis functions:

[0079]

[0080] k-th order basis functions (k = 1, 2, 3):

[0081]

[0082] For a cubic B-spline (k=3), each basis function B i,3 (x) in [t i ,t i+4 It is non-zero on the upper bound, smooth, and has a continuous second derivative.

[0083] 4. Spline Transformation

[0084] spline basis:

[0085] Spline weights:

[0086] Spline output:

[0087]

[0088] Using Einstein's summation method, it can be expressed as:

[0089]

[0090] 5. Basic Transformation

[0091] Activation function:

[0092] Basic weights:

[0093] 6. Final Output

[0094]

[0095] The characteristic token conversion method of the Kolmogorov-Arnold network based on the Matérn kernel is attached. Figure 2 As shown, the core of this method lies in using the Gaussian Process (GP) of the Matérn kernel to perform a nonlinear transformation on the input features, and combining it with the basic transformation to generate an embedded representation.

[0096] 1. Input and Output Definitions

[0097] ·enter: Where B is the batch size and N is the number of features.

[0098] Output: Where D is the embedding dimension.

[0099] 2. Definition of Matérn kernel

[0100] The Matérn kernel is a commonly used covariance function for Gaussian processes. Its form is controlled by the smoothing parameter v and the length scale l. For inputs x1, x2, the mathematical expression of the Matérn kernel is:

[0101] Distance: d = |x1 - x2|

[0102] Root term:

[0103] The specific form of the Matérn core:

[0104] When v = 0.5:

[0105]

[0106] When v = 1.5:

[0107]

[0108] When v = 2.5:

[0109]

[0110] 3. Gaussian Process (GP) Transformation

[0111] Grid: Where G is the number of grid points, ranging from [grid_min, grid_max].

[0112] For each input dimension i, calculate the Matérn kernel value of the input x[:,i] (dimension i×1) and the grid points:

[0113]

[0114] Finally obtained

[0115] 4. GP Output Calculation

[0116] GP weights:

[0117] GP output:

[0118] gp_output = gp_features·W gp

[0119] Using Einstein's summation and expressing it as:

[0120]

[0121] Where b represents the b-th vector (there are B vectors in total); i represents the i-th element of the vector (a vector has N elements); d represents the d-th element of a new vector formed by "upgrading" the i-th element of the b-th vector to a new dimension; the resulting dimension is...

[0122] 5. Basic Transformation

[0123] Activation function:

[0124] Basic weights:

[0125] Basic output:

[0126] In the specific calculation, f(x) has a dimension of B×N, which is expanded to B×N×1 using unsqueeze(-1), while W... b Expanding to 1×N×D using unsqueeze(0), and then multiplying the matrix, we get B×N×D.

[0127] 6. Final Output

[0128] y = base_output + gp_output

[0129] That is: y b,i,d =f(x) b,i )·W b,i,d +∑ k k(x b,i ,grid k )·W gp,i,n,d .

[0130] Figure 3 The overall architecture of the edge system based on the feature token transformation method of the Kolmogorov-Arnold network is demonstrated, covering the complete process from data acquisition to final prediction. Figure 3 In this process, the input data X (with dimensions B×N, where B is the batch size and N is the number of features) is first acquired by sensors on the edge device. Specifically, this includes acquiring two-dimensional image data (e.g., H×W×C, where H and W are the image height and width, respectively, and C is the number of channels) through a camera, and acquiring one-dimensional time series data through temperature, pressure, or acceleration sensors.

[0131] The data is then sent to the data preprocessing module. The preprocessing process varies depending on the data type: For two-dimensional image data acquired by a camera, flattening is required to flatten the H×W×C image into a one-dimensional vector (with dimensions of H×W×C to fit the model input). Then, denoising (removing Gaussian noise or salt-and-pepper noise from the image), normalization (scaling pixel values ​​to [0,1] or standardizing them to zero mean and unit variance), missing value imputation (handling missing pixels using mean or interpolation methods), and dimension checking are performed. For one-dimensional data acquired by temperature, pressure, or acceleration sensors, flattening is not required. Instead, denoising (removing sensor noise, such as high-frequency interference), normalization (scaling data to [0,1] or standardizing), missing value imputation (handling missing data points using mean or interpolation methods), and dimension checking are performed directly to ensure that the data format is consistent with the model input requirements.

[0132] The preprocessed data is fed into the KAFT / MKAFT module, where KAFT specifically refers to the Kolmogorov-Arnold feature token transformation method based on cubic B-spline functions, and MKAFT specifically refers to the Kolmogorov-Arnold feature token transformation method based on Matérn kernels. A smooth nonlinear feature transformation is achieved through cubic B-spline curves or Gaussian processes using Matérn kernels, generating an initial feature representation T. Embedding categorical features transforms this into T0, which is then fed into a multi-layer Transformer model. Each layer contains a self-attention mechanism and a feedforward network to progressively refine the feature representation. Finally, the prediction module outputs the result y, which can be used for classification or regression tasks.

[0133] Figure 4 The theoretical structure of the Kolmogorov-Arnold network (KAN) is presented. It approximates arbitrary functions through a combination of hierarchical basis functions (circular nodes represent intermediate features, and square nodes represent basis functions, such as the Matérn kernel or B-spline functions), demonstrating the powerful function approximation ability of the KAN method, especially its ability to capture complex patterns through nonlinear transformations in feature token transitions. The upper right half shows the posterior distribution characteristics of the Matérn kernel, with the horizontal axis representing the input value X and the vertical axis representing the posterior distribution value. The figure includes multiple posterior sample curves, the mean curve (thick black line), and the standard deviation confidence interval (gray shaded area), reflecting the smoothness and uncertainty estimation ability of the Matérn kernel in nonlinear modeling. The lower right half shows the effect of the cubic B-spline function approximating a noisy sine function, with the horizontal axis representing the input value and the vertical axis representing the function value. The figure includes the original sine function (smooth curve), the noisy sine function (scattered data), and the approximation result of the cubic B-spline function (smooth approximation curve), demonstrating the superiority of the B-spline function in terms of smoothness and noise robustness.

[0134] The system combines structured data (tabular input) with differentiated data preprocessing (two-dimensional data needs to be flattened), cubic B-spline function and Matérn kernel Gaussian process nonlinear transformation, and the powerful feature extraction capability of Transformer to achieve efficient feature token conversion. The system accelerates training and inference speed by fitting the network with the above powerful functions, so as to adapt to resource-constrained edge devices.

[0135] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention. For those skilled in the art, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered within the protection scope of the present invention.

Claims

1. A feature token conversion method based on the Kolmogorov-Arnold network, characterized in that, The method includes: Input processing: Obtain an input tensor with a batch size of B and a number of features of N. Where B is the batch size and N is the number of features; Mesh initialization: Set the mesh range [a,b] to [-1,1], the mesh size to G, calculate the step size h = (ba) / G, and generate a set of mesh points gridd that extends to [-1-3h,1+3h]. grid={t -3 ,t -2 ,t -1 ,t0,…,t G ,t G+1 ,t G+2 ,t G+3 }; Cubic B-spline basis function calculation: For input x, calculate the cubic B-spline basis function B according to the recursive formula. i,3 (x), where the basis function is in [t] when k = 3. i ,t i+4 [Non-zero] Spline transformation: Calculate the spline basis matrix Combine spline weights The spline output, spline_output∈R, is obtained by Einstein summation. B×N×D ; Basic transformation: Apply the SiLU activation function f(x) = x·σ(x) to the input x, combined with the basic weights W. b ∈R N×D The base output ∈ R is obtained. B×N×D ; Final output: The spline output is superimposed with the base output to obtain the feature token representation. Where D is the embedding dimension, i.e. y b,n,d =f(x b,n )·W b,n,d +∑ i B i,3 (x b,n )·W s,n,d,i 。 2. The method according to claim 1, wherein, The recursive formula for the cubic B-spline basis function is: 0th order basis functions: k-th order basis functions, k = 1, 2, 3:

3. The method according to claim 1, wherein, The Einstein summation expression for the spline transformation is: spline_output b,n,d =∑ i spline_bases b,n,i ·W s,n,di 。 4. A feature token conversion method based on the Kolmogorov-Arnold network, characterized in that, The method includes: Input processing: Obtain an input tensor with a batch size of B and a number of features of N. Where B is the batch size and N is the number of features; Matérn kernel definition: Based on the smoothing parameter v and the length scale l, the distance d = |x1-x2| between inputs x1 and x2 is defined, and the root term... Calculate the Matérn kernel k(x1,x2), where: When v = 0.5: When v = 1.5: When v = 2.5: Gaussian process transformation: setting the mesh Where G is the number of grid points, ranging from [grid_min, grid_max]. For each input dimension i, calculate the Matérn kernel value of x[:,i] of input dimension B×1 and the grid points: Finally obtained GP output calculation: combining GP weights Obtained through Einstein's summation method Basic transformation: Apply the SiLU activation function f(x) = SiLU(x) = x·σ(x) to the input x, combined with the basic weights W. b ∈R N ×D The base output ∈ R is obtained. B×N×D ; Final output: The GP output is superimposed with the basic output to obtain the feature token representation. Where D is the embedding dimension, i.e. y b,i,d =f(x b,i )·W b,i,d +∑ k k(x b,i ,grid k )·W gp,i,n,d 。 5. The method according to claim 4, wherein, The Einstein summation expression for the Gaussian process transform is: gp_output h,i,d =∑ n g p_features b,j,n ·W gp,i,n,d 。 6. A feature token conversion system based on the Kolmogorov-Arnold network, characterized in that, The system includes: Data acquisition module: used to acquire sensor data, including two-dimensional image data (H×W×C) and one-dimensional time series data; Data preprocessing module: performs flattening, denoising, normalization, missing value imputation, and dimensionality checking on two-dimensional image data, and performs denoising, normalization, missing value imputation, and dimensionality checking on one-dimensional time series data; KAFT / MKAFT module: The method according to any one of claims 1-3 or the method according to claim 4 or 5 performs feature token transformation on the preprocessed data to generate a feature representation. Transformer module: contains a multi-layer self-attention mechanism and a feedforward network, which refines the feature representation T layer by layer; Prediction module: Based on the feature representation output by Transformer, performs classification or regression tasks and outputs the result y.

Images