Sampling and polynomial-based learnable property estimation method on data stream

Abstract

The invention provides a method for estimating learnable properties on a data stream based on sampling and polynomial. Comprising the steps of S1, extracting a sample from a data stream; s2, generating training data; s3, according to different data property estimation problems, designing different learnable neural networks, and solving the data property estimation problems, S4, inputting the random data distribution group generated in the step S2 into the step S3 to serve as training data for training, and taking the loss function constructed in the step S3 as a training termination condition; s5, estimating a data property problem by combining the final model obtained in the step S4 with the data information features extracted in the step S1; after estimation is carried out through the neural network, obtained data property estimation is used for answering a data property estimation problem. Through the above means, the method is applied to a data stream environment, adapts to different data distributions, learns the estimator independently, and improves the effect of estimating the property of the estimated data.

Description

Technical Field

[0001] The present disclosure relates to the field of databases, and more particularly, to a method for estimating learnable properties based on sampling and polynomials on a data stream. Background Art

[0002] With the advent of the big data era, the scale of data has grown dramatically. Data usually appears in large quantities at high speed. It is necessary to return some estimates of data properties in a short time. Brief data properties can intuitively reflect the distribution of data, applicable data algorithms, and detect whether the data has abnormalities and whether it can meet the previous data assumptions. For high-speed continuous data, the continuity of data is used to construct a data curve fitting graph to obtain continuous data with certain rules, thereby answering the above questions. However, the data properties of discrete data are also particularly important, but there is a lack of universal algorithms to analyze the data properties of these discrete data. To return the data properties of discrete data, a complete data frequency dictionary is usually required for calculation. To obtain the complete data frequency on the data stream, it is necessary to save and merge all the accessed data and then count them. When the data elements are different, saving all the data is difficult to meet the space requirements, and it is inefficient and difficult to meet real-time requirements. How to efficiently obtain the data properties of discrete data has become a challenge.

[0003] In databases, estimating the properties of discretely distributed data is one of the core issues in this field. The properties of discretely distributed data include the number of different elements of the data, the entropy of the data, the p-th moment of the data, and so on. Moreover, the properties of discrete data distribution can be applied to various aspects of analyzing data problems, such as optimizing query efficiency and optimizing compression algorithms. This problem is particularly important in data analysis on stream data. Sampling algorithms are usually used to estimate these data properties without recording all elements. By uniformly sampling the input stream data and retaining a portion of the data information, the overall properties of the data are estimated. Traditional sampling estimation methods target a specific data property and analyze its inherent distribution properties to obtain a fixed sampling estimator to deal with specific data property estimation problems. For example, there are a series of statistics for different numbers of data items, which can estimate the number of different data items through sampling frequency. But the current problem is that these methods are independent of each other and lack a unified way to solve different data property estimation problems.

[0004] In data streams, data distribution is not static. Since data distribution changes over time, fixed estimators designed for specific data distributions cannot cope well with estimation problems under different data distributions. In the face of changes in data distribution, a more reasonable solution is to use machine learning methods to learn different estimators for different data distributions and adjust the estimators according to changes in data distribution.

[0005] Therefore, it is necessary to design a uniform estimator for the properties of sampled data under data streams that can change according to the data changes. In addition, using machine learning methods to improve the estimation effect of the estimator at a fixed sampling rate is also an important part of designing the estimator. Summary of the invention

[0006] The purpose of the embodiments of the present disclosure is to provide a method for estimating learnable properties based on sampling and polynomials on data streams. The main research goal of the present invention is to realize the automatic generation of code summary documents with project-level granularity through the organic combination of hierarchical-based models and program analysis technology. The specific object of the study is English documents, and the programming language targeted is Java. The summary document is mainly reflected in a short natural language summary, which does not involve complex information such as directory structure, instructions for use, etc. Based on this model, an open source code repository summary automatic generation tool is designed and implemented on the Eclipse IDE in the form of an Eclipse plug-in.

[0007] In a general aspect, a method for estimating learnable properties based on sampling and polynomials on a data stream is provided, comprising steps S1-S5:

[0008] Step S1, extract samples from the data stream; the data stream scenario is limited to discrete stream data, and the data volume is large, which cannot be fully maintained in the memory, but the discrete distribution properties of the data need to be analyzed. For example, analyze whether the log data is abnormal. At this time, the log data is continuously input, but it is difficult to fully maintain it in the memory. It can only be calculated by sampling whether the log data distribution has changed.

[0009] Step S2, generating training data;

[0010] The specific steps include S2.1-S2.3:

[0011] S2.1 From the uniform distribution U(0, B), sample a positive integer r, N = 10 r As the total number of training sets;

[0012] S2.2 From the uniform distribution U(0, B), sample a positive integer s, let M = 10 s ;

[0013] S2.3 Let F i is the number of elements that appear i times in the population, and let SF i =∑ i F i , in the restriction Next, randomly select SF i , and reversely calculate the distribution of the constructed data population F = {F1, F2, ...,}, F i =SF i -SF i+1 ;

[0014] Repeat S2.1-3 in batches to generate sampled frequency data, which is used as training data and input into the downstream model;

[0015] Step S3, design different learnable neural networks according to different data property estimation problems to solve data property estimation problems.

[0016] Using data property function induction, the data property estimates of these discrete data distributions are summarized into the following calculation objectives:

[0017]

[0018] in, The probability of a data item being sampled j times in the corresponding population is Then it corresponds to the property estimation function acting on probability; the data property estimator is set as: Among them, a i is the linear coefficient, f i Represents the number of elements that appear i times in the sample, This corresponds to the frequency of the element that appears i times in a sample of size n; the first half of the equation Approximately, the error is:

[0019]

[0020] Among them, Poly(N,n,j,t) is a polynomial about j, N, n, and t. For different property estimation problems, it can be input into the above formula to obtain different approximate targets. The error ε is combined with the absolute error between the estimated value of the estimator and the true value as the final loss function.

[0021]

[0022] Among them, w j , a t They are all two-layer MLPs, or they can be more complex and have stronger expressive network structures, and the network can be updated;

[0023] Step S4, inputting the random data distribution group generated in step S2 into step S3 as training data, using the loss function constructed in S3 as the termination condition for training. When the loss function value is updated to be small, corresponding to the actual operation, the function value curve of the loss function is drawn, and the function value curve decreases rapidly, and then the rate of decrease decays until it becomes flat. At this time, the update is stopped, and the model is retained in the data stream cache.

[0024] Detect whether the data has undergone large-scale changes. The specific criteria are whether the scale of the input data has changed, whether the sampling rate range has changed, whether its numerical domain has changed, and whether its data skewness has changed. When the data has undergone large-scale migration or change, re-run steps S2 and S3 to obtain a new model.

[0025] Step S5, the final model obtained in step S4 is combined with the data information features extracted in step S1 to estimate the data property problem. After the estimation is performed through the neural network, the data property estimation obtained can answer the data property estimation problem.

[0026] The specific method of extracting samples from the data stream is as follows: selecting a sampling rate between 0.01 and 0.001, assuming that the total size of the data stream is N, maintaining a water pond sampling pool n in the machine, performing frequency statistics on the data obtained from the data, and finally maintaining a set of data frequencies in the machine memory; when it is necessary to estimate the properties of the data, the frequency dictionary {e1:x1,e2:x2,…,e D :x D}, as the input of the model; the frequency dictionary of its data frequency is counted as: {f1,f2,…,f i ,…}, where f i In the sample, there are a total of f i An element appears i times.

[0027] The innovation of the embodiment of the present invention lies in:

[0028] 1. The present invention can utilize polynomial estimation to cope with estimation of different data properties based on sampling without the need to design an algorithm for each problem separately.

[0029] 2. The present invention can be applied in a data stream environment and can adapt to different data distributions and learn estimators independently.

[0030] 3. The present invention improves the effect of estimating data properties in a data stream environment and provides theoretically optimal sampling complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0032] Figure 1 It is a schematic diagram showing the architecture of a method for estimating learnable properties based on sampling and polynomials on a data stream according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0034] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0035] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0036] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0037] In the specification, when an element (such as a layer, a region or a substrate) is described as being “on”, “connected to” or “coupled to” another element, the element may be directly “on”, “connected to” or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to” or “directly coupled to” another element, there may be no other elements present therebetween.

[0038] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0039] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0040] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0041] Figure 1 is a schematic diagram showing a sampling and polynomial-based learnable property estimation method on a data stream according to an embodiment of the present disclosure.

[0042] The present invention is described in detail below with reference to the accompanying drawings and embodiments, but it should be understood that the protection scope of the present invention is not limited by the specific implementation methods.

[0043] like Figure 1 As shown, the present invention is based on sampling, and a flowchart of the method for estimating different elements in a distributed environment. The specific execution scenarios are: the stream data environment is such as detecting whether the data of the operation and maintenance log input in the form of a data stream is abnormal, or extracting its discrete distribution data features; the distributed environment is such as large-scale log historical data stored on the disk. When it is impossible to scan the data packets on each disk on a large scale, this method can only be used by sampling and analyzing the disk data and calculating the discrete data features. It includes steps S1-S5.

[0044] Step S1, extract samples from the data stream, usually the sampling rate is between 0.01 and 0.001. In actual operation, assuming that the total size of the data stream is N, a pond sampling pool n can be maintained in the machine, and the frequency statistics of the data obtained are performed, and finally a set of data frequencies are maintained in the machine memory. When it is necessary to estimate the properties of the data, the frequency dictionary {e1:x1,e2:x2,…,e D :x D}, as the input of the model. The frequency dictionary of its data frequency can be calculated as: {f1,f2,…,f i ,…}, where f i In the sample, there are a total of f i The element appears i times. By sampling the frequency of the data, the overall nature of the data can be estimated in the following steps.

[0045] Step S2, generate training data. Training data can be generated according to different research questions and different data distributions. The most direct method is to input historical data as empirical data. In order to ensure the learning effect of property estimation, specific training data can also be designed according to theoretical settings to avoid extreme problems caused by overfitting. The data generated in this way will cover more extreme situations as training. The specific steps are as follows:

[0046] S2.1 Sampling a positive integer r from the uniform distribution U(0,B), N=10 r As the total number of training sets.

[0047] S2.2 Sampling a positive integer s from the uniform distribution U(0,B), let M = 10 s .

[0048] S2.3 Let F i is the number of elements that appear i times in the population. And let SF i =∑ i F i , in the restriction Next, randomly select SF i , and reversely calculate the distribution of the constructed data population F = {F1, F2, ...,}, F i =SF i -SF i+1 .

[0049] Repeating S2.1-3 in batches can obtain a large amount of training data. These constructed data distributions can quickly obtain the constructed overall data property information, such as the number of different elements, entropy, etc. By sampling the distribution of the constructed data population F = {F1, F2, ...,} and the binomial distribution, the sampled frequency data can be quickly generated as training data and input into the downstream model.

[0050] Step S3, according to different data property estimation problems, different learnable neural networks are designed to solve data property estimation problems. For different data property estimation problems, data property function induction can be used to summarize the data property estimation of these discrete data distributions into the following calculation objectives:

[0051]

[0052] in, The probability of a data item being sampled j times in the corresponding population is This corresponds to a property estimation function acting on probability. For example, when faced with the estimation of the number of different data items, this function corresponds to When faced with the entropy estimation problem, this function corresponds to In general, the problem of estimating data properties is essentially to approximate different The theory behind this is polynomial. We found that if the data property estimator is set to: When , we can get the relationship between polynomial approximation and property estimation, where a i is the linear coefficient, f i Represents the number of elements that appear i times in the sample, The corresponding frequency of the element that appears i times in a sample of size n is used to estimate the probability of being sampled. The second half of the equation represents the estimate of the sampled element for the real problem. This estimate is unbiased and has theoretical guarantees. For the first half of the equation It can be approximated by a polynomial with an error of:

[0053]

[0054] Where Poly(N,n,j,t) is a polynomial about j, N, n, and t. Different property estimation problems can be input into the above formula to obtain different approximate targets. The parameterization of can learn different data distributions. t Transformed into a multi-layer MLP neural network, the neural network can be trained through back propagation according to different frequency data inputs to obtain the optimal polynomial approximation. In addition to the approximation of polynomial theory, the estimated value of the linear estimator should also conform to the estimated value of the target property. Therefore, when designing the loss function, it is necessary to combine the error ε with the absolute error between the estimated value of the estimator and the true value as the final loss function

[0055]

[0056] Among them, w j , a t They are all two-layer MLPs, but they can also be more complex network structures with stronger expressive capabilities, and the network can be updated.

[0057] Step S4, input the random data distribution group generated in step S2 into step S3 as training data. The loss function constructed in S3 is used as the termination condition for training. When the loss function value is updated to a small value, the function value curve of the loss function is drawn in actual operation. The function value curve will drop rapidly, and then the drop rate will decay until it is flat. At this time, the update is stopped and the model is retained in the data stream cache.

[0058] Detect whether the data has undergone large-scale changes. The specific criteria are whether the scale of the input data has changed, whether the sampling rate range has changed, whether its numerical domain has changed, and whether its data skewness has changed. When the data has undergone large-scale migration or change, re-run steps S2 and S3 to obtain a new model.

[0059] Step S5, the final model obtained in step S4 is combined with the data information features extracted in step S1 to estimate the data property problem. The specific operation is to divide the input frequency by the predefined frequency division domain value L using the data information features obtained in S1, and input the low-frequency data features into the saved model obtained in S4. After the estimation is performed through the neural network, the data property estimation obtained can answer the data property estimation problem.

[0060] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for estimating learnable properties based on sampling and polynomials on data streams, characterized in that: The method comprises steps S1-S5: Step S1, extracting samples from the data stream of the analysis log data; Step S2, generating training data; The specific steps include S2.1-S2.3: S2.1 Sampling a positive integer r from the uniform distribution U(0,B), N=10 r As the total number of training sets; S2.2 Sampling a positive integer s from the uniform distribution U(0,B), let M = 10 s ; S2.3 Let F i is the number of elements that appear i times in the population, and let SF i =∑ i F i , in the restriction Next, randomly select SF i , and reversely calculate the distribution of the constructed data population F = {F1, F2, ...,}, F i =SF i -SF i+1 ; Repeat S2.1-3 in batches to generate sampled frequency data, which is used as training data and input into the downstream model; Step S3, design different learnable neural networks according to different data property estimation problems to solve data property estimation problems. Using data property function induction, the data property estimates of these discrete data distributions are summarized into the following calculation objectives: in, The probability of a data item being sampled j times in the corresponding population is Then it corresponds to the property estimation function acting on probability; the data property estimator is set as: Among them, a i is the linear coefficient, f i Represents the number of elements that appear i times in the sample, This corresponds to the frequency of the element that appears i times in a sample of size n; the first half of the equation Approximately, the error is: Among them, Poly(N,n,j,t) is a polynomial about j, N, n, and t. For different property estimation problems, it can be input into the above formula to obtain different approximate targets. The error ε is combined with the absolute error between the estimated value of the estimator and the true value as the final loss function: Among them, w j , a t They are all two-layer MLPs, or they can be more complex and have stronger expressive network structures, and the network can be updated; Step S4, input the random data distribution group generated in step S2 into step S3 as training data for training, and use the loss function constructed in S3 as the termination condition for training. When the loss function value is updated to a small value, corresponding to the actual operation, the function value curve of the loss function is drawn, and the function value curve will drop rapidly, and then the drop rate will decay until it is flat. At this time, the update is stopped, and the model is retained in the data stream cache; when the data is migrated or changed on a large scale, steps S2 and S3 are re-run to obtain a new model; Step S5, the final model obtained in step S4 is combined with the data information features extracted in step S1 to estimate the data property problem; the data property estimation obtained after estimation through the neural network is used to answer the data property estimation problem.

2. A method for estimating learnable properties based on sampling and polynomials on a data stream as claimed in claim 1, characterized in that: The specific method of extracting samples from the data stream is as follows: selecting a sampling rate between 0.01 and 0.001, assuming that the total size of the data stream is N, maintaining a pond sampling pool n in the machine, performing frequency statistics on the data obtained from the data, and finally maintaining a set of data frequencies in the machine memory; when it is necessary to estimate the properties of the data, the frequency dictionary {e1:x1,e2:x2,…,e D :x D }, as the input of the model; the frequency dictionary of its data frequency is counted as: {f1,f2,…,f i ,…}, where f i In the sample, there are a total of f i An element appears i times.

Images