A data restoration method and system based on tensor decomposition

Abstract

The present invention relates to a data restoration method and system based on tensor decomposition, and the method comprises: S1: obtaining the original traffic data of a content distribution network, and constructing a third-order tensor based on the domain name, time and server IP address in the original traffic data, wherein N, P and Q are the dimensions of the domain name, time and CDN server IP address respectively; S2: constructing a two-dimensional matrix based on the time and traffic of the original traffic data S3: using the gradient descent method to solve X and Y, obtaining a core third-order tensor A, a two-dimensional matrix R, a two-dimensional matrix S and a two-dimensional matrix T, restoring X according to A, R, S and T, filling in missing values, and obtaining a restored third-order tensor X'. The method provided by the present invention uses a tensor decomposition algorithm to establish a high-order tensor model, and combines historical data to form a two-dimensional matrix to fill in sparse data, so as to improve data utilization efficiency.

Description

Technical Field

[0001] The present invention relates to the field of noise data, and in particular to a data restoration method and system based on tensor decomposition. Background Art

[0002] As the complexity and scale of modern systems increase, the data generated by the system may be missing due to internal or external reasons. For example, the currently widely deployed content distribution network system CDN (Content Delivery Network) introduces domain names with low access traffic, which usually have low access in the early morning and other time periods, resulting in discontinuity in the time series of indicators such as traffic or latency, making it difficult to conduct regular monitoring work.

[0003] Traditional sparse data processing methods include filling missing values ​​with 0, or taking the average of the data corresponding to the two timestamps before and after, so as to keep the time series of a certain indicator continuous. However, in the content distribution network system, domain names with less visits generate less data or are missing in time periods such as early morning. When they are filled with 0 values, it will cause false alarms in the monitoring system. When they are filled with the average of the two timestamps before and after, it will also cause erroneous data in the larger time series, leading to false alarms. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a data restoration method and system based on tensor decomposition.

[0005] The technical solution of the present invention is: a data restoration method based on tensor decomposition, comprising:

[0006] Step S1: Obtain the original traffic data of the content distribution network, and construct a third-order tensor based on the domain name, time and server IP address in the original traffic data Among them, N, P, and Q are the dimensions of domain name, time, and CDN server IP address respectively;

[0007] Step S2: Construct a two-dimensional matrix based on the time and flow of the original flow data

[0008] Step S3: Use the gradient descent method to solve X and Y to obtain the core third-order tensor A, two-dimensional matrix R, two-dimensional matrix S and two-dimensional matrix T. According to A, R, S and T, restore X and fill in the missing values ​​to obtain the restored third-order tensor X'.

[0009] Compared with the prior art, the present invention has the following advantages:

[0010] The present invention discloses a data restoration method based on tensor decomposition. Sparse data is modeled using a high-order tensor model, and is decomposed using a two-dimensional matrix formed by historical data. Since the historical data is relatively complete, the core tensor and two-dimensional matrix finally obtained are more accurate, which has obvious advantages over traditional methods of filling in zeros or filling in the mean of adjacent timestamps. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of a data restoration method based on tensor decomposition in an embodiment of the present invention;

[0012] Figure 2 Schematic diagram of a third-order tensor model in an embodiment of the present invention;

[0013] Figure 3 This is a two-dimensional matrix model in an embodiment of the present invention;

[0014] Figure 4 It is a structural block diagram of a data restoration system based on tensor decomposition in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention provides a data restoration method based on tensor decomposition, which uses a tensor decomposition algorithm to establish a high-order tensor model, and combines historical data to form a two-dimensional matrix to fill in sparse data to improve data utilization efficiency.

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0017] Embodiment 1

[0018] like Figure 1 As shown, a data restoration method based on tensor decomposition provided by an embodiment of the present invention includes the following steps:

[0019] Step S1: Obtain the original traffic data of the content distribution network, and construct a third-order tensor based on the domain name, time and server IP address in the original traffic data Among them, N, P, and Q are the dimensions of domain name, time, and CDN server IP address respectively;

[0020] Step S2: Construct a two-dimensional matrix based on the time and flow of the original traffic data

[0021] Step S3: Use the gradient descent method to solve X and Y to obtain the core third-order tensor A, two-dimensional matrix R, two-dimensional matrix S and two-dimensional matrix T. According to A, R, S and T, restore X and fill in the missing values ​​to obtain the restored third-order tensor X'.

[0022] In one embodiment, the above step S1: obtains the original traffic data of the content distribution network, and constructs a third-order tensor based on the domain name, time and server IP address in the original traffic data Among them, N, P, and Q are the dimensions of domain name, time, and CDN server IP address, respectively, including:

[0023] Read the original traffic data of the content distribution network CDN. In the embodiment of the present invention, the three dimensions of the original traffic data are used but not limited to: domain name, time, and server IP for data modeling. In actual applications, corresponding dimensions can be added or deleted. A third-order tensor model is established based on the above three dimensions, such as Figure 2 The third-order tensor shown Where N, P, and Q represent Figure 2 The dimensions of domain name, time, and CDN server IP address, Figure 2 Each point on the cube represents the traffic volume generated by a specific domain name at a CDN server IP address at a specific time. It can be seen that the traffic value is likely to be discontinuous on the time axis. The time axis represents different granularities of 24 hours a day, such as every 5 minutes.

[0024] In one embodiment, the above step S2: constructs a two-dimensional matrix based on the time and flow of the original flow data Specifically include:

[0025] Based on the historical data of the original traffic data, such as the logs of the past month, a two-dimensional matrix of time and traffic is summarized, such as Figure 3 The Y shown, Since the historical time period is long (logs of a month or longer range can be taken), the two-dimensional matrix Y can ensure that most values ​​are continuous. Supplementing the tensor X with it can well fill the missing values ​​of X. The embodiment of the present invention uses but is not limited to the use of time granularity. In actual applications, dimensions such as domain name and CDN server IP can be added to assist in decomposing high-order tensors.

[0026] In one embodiment, the above step S3: using the gradient descent method to solve X and Y, obtaining a core third-order tensor A, a two-dimensional matrix R, a two-dimensional matrix S and a two-dimensional matrix T, restoring X according to A, R, S and T, filling in missing values, and obtaining a restored third-order tensor X', specifically includes:

[0027] Step S31: Use Tucker decomposition method to model X and obtain the objective function, as shown in formula (1):

[0028]

[0029] Among them, ||.|| 2 represents the second norm of the tensor, Represents the penalty term to prevent overfitting; A is the core third-order tensor, w R 、w S 、w T are the dimensions of the two-dimensional matrices R, S, and T respectively, and F is a two-dimensional auxiliary matrix; λ1 and λ2 represent the parameters controlling the contribution of the normalization factor; the symbol × L Represents the product of tensor and matrix, L represents the direction;

[0030] Step S32: Initialize A, S, R, T, assign random values, and calculate the loss function value L t Initialize, set the loss function to mean square error function, set the gradient descent step size ρ, and set the termination condition ε;

[0031] Step S33: Determine L t+1 -L t <ε is true; if true, go to step S37; otherwise, go to step S34;

[0032] Step S34: For each non-zero element q in the third-order tensor X, calculate according to formulas (2) to (7) in sequence:

[0033] Z ijk =A× R R i* × S S j* × T T k* (2)

[0034] R i* =R i* -ρλ2R i* -ρ(Z ijk -q)×A× S S j* × T T k* (3)

[0035] S j* =S j* -ρλ2S j* -ρ(Z ijk -q)×A× R R i* × T T k* (4)

[0036] T k* =T k* -ρλ2T k*-ρ(Z ijk -q)×A× R R i* × S S j* -ρλ1(T k* ×FY k* )×F (5)

[0037] A=A-ρλ1A-ρ(Z ijk -q)×R i* ×S j* ×T k* (6)

[0038] F=F-ρλ2F-ρλ1(T k* ×FY k* )×T k* (7);

[0039] Step S35: Update the loss function value for the third-order tensors X and X' Where X' = A × R R× S S× T T,y i represents the elements in X, represents the corresponding element in X', and m is the number of non-zero elements in X;

[0040] Step S36: Go to step S33;

[0041] Step S37: output A, R, S, T;

[0042] Step S38: Fill in the missing values ​​of X according to A, R, S, and T to obtain the restored third-order tensor X'=A× R R× S S× T T.

[0043] The present invention discloses a data restoration method based on tensor decomposition. A high-order tensor model is used to model sparse data, and a two-dimensional matrix formed by historical data is used to decompose the sparse data. Since the historical data is relatively complete, the core tensor and two-dimensional matrix finally obtained are more accurate, which has obvious advantages over the traditional method of filling with zeros or filling the mean of adjacent timestamps.

[0044] Embodiment 2

[0045] like Figure 4 As shown, an embodiment of the present invention provides a data restoration system based on tensor decomposition, including the following modules:

[0046] The third-order tensor building module 41 is used to obtain the original traffic data of the content distribution network and build a third-order tensor based on the domain name, time and server IP address in the original traffic data. Among them, N, P, and Q are the dimensions of domain name, time, and CDN server IP address respectively;

[0047] The two-dimensional matrix construction module 42 is used to construct a two-dimensional matrix based on the time and flow of the original flow data.

[0048] The third-order tensor restoration module 43 is used to solve X and Y using the gradient descent method to obtain a core third-order tensor A, a two-dimensional matrix R, a two-dimensional matrix S and a two-dimensional matrix T. X is restored according to A, R, S and T, missing values ​​are filled, and the restored third-order tensor X' is obtained.

[0049] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included within the scope of the present invention.

Claims

1. A data restoration method based on tensor decomposition, characterized in that: include: Step S1: Obtain the original traffic data of the content distribution network, and construct a third-order tensor based on the domain name, time and server IP address in the original traffic data , where N, P, and Q are the dimensions of domain name, time, and CDN server IP address, respectively; Step S2: Construct a two-dimensional matrix based on the time and flow of the original flow data ; Step S3: Use the gradient descent method to solve X and Y to obtain the core third-order tensor A, two-dimensional matrix R, two-dimensional matrix S and two-dimensional matrix T. According to A, R, S and T, restore X and fill in the missing values ​​to obtain the restored third-order tensor. , comprising the following steps: Step S31: Use Tucker decomposition method to model X and obtain the objective function, as shown in formula (1): (1) in, represents the second norm of the tensor, Represents the penalty term to prevent overfitting; A is the core third-order tensor, , , , , , , , are the dimensions of the two-dimensional matrices R, S, and T respectively, and F is the two-dimensional auxiliary matrix; represents the parameter controlling the contribution of the normalization factor; the symbol Represents the product of tensor and matrix, and L represents the direction.

2. The data restoration method based on tensor decomposition according to claim 1, characterized in that: Step S3: Using the gradient descent method to solve X and Y, obtain the core third-order tensor A, the two-dimensional matrix R, the two-dimensional matrix S and the two-dimensional matrix T, restore X according to A, R, S and T, fill in the missing values, and obtain the restored third-order tensor , further comprising the steps of: Step S32: Initialize A, S, R, T, assign random values, and calculate the loss function value L t Initialize, the loss function is the mean square error function, set the gradient descent step size , set the termination condition ; Step S33: Determine L t+1 -L t < Is it established? If established, go to step S37; Otherwise, go to step S34; Step S34: For each non-zero element q in the third-order tensor X, calculate according to formulas (2) to (7) in sequence: (2) (3) (4) (5) (6) (7)； Step S35: For the third-order tensors X and , update the loss function value ,in ,y i represents the elements in X, express The corresponding element in , m is the number of non-zero elements in X; Step S36: Go to step S33; Step S37: output A, R, S, T; Step S38: Fill in the missing values ​​of X according to A, R, S, and T to obtain the restored third-order tensor .

3. A data restoration system based on tensor decomposition, characterized in that: Includes the following modules: Construct a third-order tensor module to obtain the original traffic data of the content distribution network, and construct a third-order tensor based on the domain name, time and server IP address in the original traffic data , where N, P, and Q are the dimensions of domain name, time, and CDN server IP address, respectively; A two-dimensional matrix construction module is used to construct a two-dimensional matrix based on the time and flow of the original flow data. ; The third-order tensor restoration module is used to solve X and Y using the gradient descent method to obtain the core third-order tensor A, two-dimensional matrix R, two-dimensional matrix S and two-dimensional matrix T. According to A, R, S and T, X is restored to fill in the missing values ​​and the restored third-order tensor is obtained. , comprising the following steps: Step S31: Use Tucker decomposition method to model X and obtain the objective function, as shown in formula (1): (1) in, represents the second norm of the tensor, Represents the penalty term to prevent overfitting; A is the core third-order tensor, , , , , , , , are the dimensions of the two-dimensional matrices R, S, and T respectively, and F is the two-dimensional auxiliary matrix; represents the parameter controlling the contribution of the normalization factor; the symbol Represents the product of tensor and matrix, and L represents the direction.

Images