Method for evaluating large-scale streaming data sampling based on dynamic drilling

Abstract

One technical scheme of the present application is to provide a large-scale stream data sampling method based on dynamic drilling. Another technical scheme of the present application is to provide a large-scale stream data evaluation method based on dynamic drilling, characterized in that the sample set is obtained from the original stream data set by using the above-mentioned large-scale stream data sampling method, and the value characteristics of the original data set are evaluated based on the sample set. Based on the idea of mineral drilling exploration, the present application proposes a dynamic drilling sampling method, which takes the well as the analysis unit and dynamically changes the size and position of the well to accurately locate the position and range of discrete data. Further, a new stream data value evaluation model is proposed, which evaluates the original stream data set from the sample set obtained based on the dynamic drilling sampling method from three dimensions of dispersion, concentration and integrity, which has important research significance for big data value evaluation.

Description

Technical Field

[0001] This invention relates to a large-scale streaming data sampling method based on dynamic drilling, belonging to the field of information technology. Background Technology

[0002] In the era of big data, the valuation of data is one of the core requirements. The marketization of data elements has become an indispensable part of building a digital China, and the era of data assets has arrived. Currently, the actions of various stakeholders are clearing obstacles for the process of data assetization.

[0003] With the development of the Industrial Internet and the rapid advancement of big data information technology, data has become a commodity that major manufacturers are vying to acquire. The volume of data is also growing rapidly and dynamically, with fields such as cybersecurity, daily transactions, social media, and transportation generating a continuous stream of data. Sampling is an indispensable method in data mining, with wide applications in fraud detection, data mining, and transportation, among many other practical applications. Using sampling techniques to extract sample sets that retain the original data characteristics from large datasets enables the assessment and prediction of data quality and value, while reducing computational costs and storage resources.

[0004] Currently, sampling methods for streaming data are mainly divided into three categories. The first category is unbiased sampling: stratified sampling, random sampling, reservoir sampling, etc. Unbiased sampling is random, and the sampled streaming data will lose some key information, ultimately leading to inaccurate value assessment of the streaming data. The second category is biased sampling: probability density sampling, which solves the data loss problem of unbiased sampling. Biased sampling can well preserve a large amount of discrete data in streaming data, but it amplifies the influence of discrete data in the sample set. The third category is mixed sampling. Mixed sampling can obtain a good sample of the overall dataset, but the sampling accuracy for discrete data is low. Furthermore, in anti-fraud financial risk control systems, anomalous data plays a very important role, containing a large amount of valuable information. For example, anomalies in financial transaction data may indicate potential fraud or money laundering activities; therefore, anomalous data needs to be retained in the sample.

[0005] At present, the value assessment of big data is mainly reflected in the field of economics, while the value assessment of streaming data still needs further research.

[0006] In summary, the inconsistency in the objectives of unbiased sampling, biased sampling, and mixed sampling makes it difficult to comprehensively and accurately assess the value of streaming data. Therefore, although many sampling methods exist to preserve discrete data in streaming data, several challenges remain. Summary of the Invention

[0007] The technical problem to be solved by this invention is: to accurately predict the location and range of discrete data in streaming data; and to make efficient and accurate evaluations of the value characteristics of streaming data.

[0008] To address the aforementioned technical problems, one technical solution of the present invention is to provide a large-scale flow data sampling method based on dynamic drilling, characterized in that it is used to sample flow data S, wherein the flow data S is represented as: S={(id i time i value i |1≤i≤N and i∈N + In the formula, id i For the order in which the i-th stream of data arrives, time i Let `value` be the arrival time of the i-th stream data. i For the i-th stream data value, the large-scale stream data sampling method includes the following steps:

[0009] Step 1: Using wells as the analysis unit, determine the location and range of discrete data in the flow data, where the i-th well is denoted as W. i The size of the i-th well is denoted as WS. i Let WN be the number of sampling wells in the original stream dataset. i Internal data is represented as:

[0010] W i ={(id j time j value j )|1≤j≤WS i and j∈N + (1≤i≤WN)

[0011] Step 2: Calculate the well interval, where the i-th well interval is denoted as WI. i The size of the i-th well interval is denoted as WIS. i Then the i-th well interval WI i Represented as:

[0012] WI i ={(id j time j value j )|id wi_max +1≤id j ≤id wi+1_min -1}

[0013] In the formula, id wi_max Let id be the largest id among all flow data in the i-th well. wi+1_min The smallest ID among all flow data in the (i+1)th well;

[0014] Step 3: Use the skewness coefficient SK to determine the dispersion of the well. If the distribution of the flow data in the well is symmetrical, the skewness coefficient SK is equal to 0. If the skewness coefficient SK is obviously not equal to 0, it indicates that the distribution of the flow data in the well is asymmetrical. A positive skewness coefficient SK indicates a right-skewed distribution, and a negative skewness coefficient SK indicates a left-skewed distribution.

[0015] If the distribution of the well flow data in the i-th well is asymmetric, and the skewness coefficient of the i-th well is denoted as SK i Then we have: If SK i If ∈[-0.5,0.5], then the dispersion of the i-th well is relatively small; if SK i If ∈(-∞, -1) or (1, +∞), then the i-th well has a large degree of dispersion, which is called a highly skewed distribution; if SK i If the distribution is ∈(-1,-0.5) or (0.5,1), then it is considered to be a moderately skewed distribution.

[0016] Step 4: Dynamically adjust the sampling rate and well interval using the skewness coefficient. Assume the current sample is the i-th well, and the initial sampling rate is p. init The adjusted sampling rate p is then expressed as:

[0017]

[0018] Adjusted size of the i-th well interval WIS i It is expressed as follows:

[0019]

[0020] In the formula, WIS init This represents the initial well spacing size for the i-th well;

[0021] Step 5: Dynamically adjust the well size using an algorithm combining Pearson correlation coefficient and coefficient of variation:

[0022] Representative wells are recorded in a well set. Then, a sliding window is used to receive new flow data. The well set is traversed, and the size of the sliding window is set to the size of different wells in the well set. The size of the well is determined by calculating the Pearson correlation coefficient and the coefficient of variation between the wells in the well set and the sliding window.

[0023] Step 6: Dynamic drilling sampling:

[0024] Within the well, unbiased sampling within the class and biased sampling between the classes are performed. Among them, the sampling rate of the minority class is dynamically increased according to the magnitude of the skewness coefficient between classes. Within the well interval, equidistant sampling is adopted to reduce the access rate of the sampled stream data. According to the algorithm for dynamically adjusting the size of the well as described in step 5, the range of discrete data is accurately located. Then, the size of the well interval is dynamically adjusted according to the skewness coefficient to dynamically locate the position of the discrete data.

[0025] Preferably, in step 3, the i-th in-well skewness coefficient SK i Represented as:

[0026]

[0027] In the formula, This represents the mean of the data in the i-th well.

[0028] Preferably, in step 5, the Pearson correlation coefficient ρ(X,Y) of the two sets of variables X and Y is expressed as:

[0029]

[0030] In the formula, σ X σ Y Let Y and Y' be the standard deviations of the two variables Y and Y' respectively, and let cov(X,Y) be the Pearson correlation coefficient before adjustment, expressed by the following formula:

[0031]

[0032] In the formula, n represents the amount of data in variables X and Y, and x i Let y represent the i-th variable in the current set of variables X. i This represents the i-th variable in the current set of variables Y. This represents the mean of the current set of variables X. This represents the mean of the current set of variables Y.

[0033] Preferably, in step 5, the coefficient of variation CV is expressed as:

[0034]

[0035] In the formula, σ is the standard deviation of the current set of variables, and μ is the mean of the current set of variables.

[0036] Another technical solution of the present invention is to provide a method for evaluating large-scale flow data based on dynamic drilling. The method is characterized by obtaining a sample set from the original flow dataset using the aforementioned large-scale flow data sampling method. Based on the sample set, the value characteristics of the original dataset are evaluated using discrete mean accuracy, discrete coefficient of variation accuracy, discrete sampling accuracy, lumped mean accuracy, lumped coefficient of variation accuracy, overall mean accuracy, overall coefficient of variation accuracy, and the JSD index.

[0037] Discrete mean accuracy is the accuracy of estimating the mean of the discrete dataset attributes of the original streaming dataset using the mean of the discrete dataset attributes of the sample set.

[0038] Discrete Coefficient of Variation Accuracy: This is the accuracy of estimating the coefficient of variation of the discrete dataset attributes (values) of the original streaming dataset using the coefficient of variation of the discrete dataset attributes (values) of the sample set.

[0039] Discrete sampling accuracy is the ratio of the number of intersections between the discrete dataset of the sample set and the discrete dataset of the original streaming dataset to the length of the discrete dataset of the sample set.

[0040] Mass mean accuracy is the accuracy of estimating the mean of the mass attribute values ​​of the original streaming dataset using the mean of the mass dataset attribute values ​​of the sample set.

[0041] The accuracy of the lumped set coefficient of variation is the accuracy of estimating the lumped set attribute value of the original streaming dataset using the lumped set coefficient of variation of the lumped set attribute value.

[0042] Overall mean accuracy: This is the accuracy of estimating the mean of the attribute values ​​in the original streaming dataset using the mean of the attribute values ​​in the sample set.

[0043] Overall coefficient of variation accuracy is the accuracy of estimating the coefficient of variation of the original stream dataset's attribute values ​​using the coefficient of variation of the attribute values ​​in the sample set.

[0044] The JSD metric measures the distance between two probability distributions by calculating their average distribution and their KL divergence. A smaller JSD metric indicates a greater similarity between the sample set and the original streaming data distribution, while a larger JSD metric indicates a greater difference between the sample set and the original streaming dataset distribution. The JSD metric is calculated using the following formula:

[0045]

[0046] In the formula, P(x) is the probability distribution of the sample set, Q(X) is the probability distribution of the original stream dataset, and D(P||Q) represents the KL divergence between P(x) and Q(X). Then we have...

[0047] Current sampling methods for real-time and rapidly changing streaming data tend to lose a large amount of value and information from discrete data, making it difficult to efficiently and accurately evaluate the value characteristics of streaming data. Based on the concept of mineral drilling exploration, this invention proposes a dynamic drilling sampling method. This method uses the well as the analysis unit, dynamically changing the size and location of the well to accurately locate the position and range of discrete data. Furthermore, a novel streaming data value assessment model is proposed. This model evaluates the original streaming dataset from three dimensions—discrete, centralized, and overall—based on the sample set obtained by the dynamic drilling sampling method, which has significant research implications for big data value assessment. Attached Figure Description

[0048] Figure 1 It is a large-scale flow data sampling framework diagram for dynamic drilling;

[0049] Figure 2 This is a diagram of a streaming data value assessment model;

[0050] Figure 3 This is a diagram illustrating streaming data;

[0051] Figures 4(a) and 4(b) are classification diagrams of peaks and troughs in streaming data. Detailed Implementation

[0052] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0053] This invention relates to a large-scale streaming data sampling method based on dynamic drilling, and also to an evaluation model establishment method for assessing the value of streaming data obtained using the large-scale streaming data sampling method.

[0054] The general steps of a large-scale streaming data sampling method based on dynamic drilling disclosed in this invention are as follows: First, set the initial wells and initial well spacing, and perform k-means clustering on the data in the wells; then, use the skewness coefficient to determine the dispersion of the data in the wells, the sampling rate of each class, and the well spacing, performing unbiased sampling within each well and biased sampling between wells, and equal-distance sampling at the well spacing; finally, determine the size of the next well using the correlation coefficient and the coefficient of variation. The specific sampling method framework is as follows: Figure 1 As shown, it includes the following steps:

[0055] The research object of this invention is real-time and high-speed changing discrete streaming data. The size of the discrete streaming data is denoted as N, and the size increases infinitely with time.

[0056] S101: Represent the stream data S as:

[0057] S={(id i ,time i ,value i )|1≤i≤Nandi∈N +

[0058] In the formula, id i For the order in which the i-th stream of data arrives, time i Let `value` be the arrival time of the i-th stream data. i Let be the value of the i-th stream data. The stream data distribution is as follows: Figure 3 As shown.

[0059] S102: Well:

[0060] Since the location and range of discrete data in streaming data are uncertain, the concept of a well is introduced based on the drilling sampling approach. By using the well as the analysis unit, the location and range of discrete data in the streaming data are determined. The i-th well is denoted as W. i The size of the i-th well is denoted as WS. i Let WN be the number of sampling wells in the original dataset. i Internal data is represented as:

[0061] W i ={(id j time j value j )|1≤j≤WS i and j∈N + (1≤i≤WN)

[0062] S103: Well Interval:

[0063] To reduce the access rate of streaming data sampling, well intervals are set between wells. The i-th well interval is denoted as WI. i The size of the i-th well interval is denoted as WIS. i Then the i-th well interval WI i Represented as:

[0064] WI i ={(id j time j value j )|id wi_max +1≤id j ≤id wi+1_min -1}

[0065] In the formula, id wi_maxLet id be the largest id among all flow data in the i-th well. wi+1_min It is the smallest ID among all the flow data in the (i+1)th well.

[0066] S104: Determine the dispersion of the well:

[0067] To determine the location of discrete data, this invention introduces the concept of "skewness coefficient" from statistics, denoted as SK. The skewness coefficient SK of the i-th well is then... i It is expressed as follows:

[0068]

[0069] In the formula, This represents the mean of the data in the i-th well.

[0070] If the distribution of well flow data is symmetrical, the skewness coefficient SK equals 0; if the skewness coefficient SK is significantly non-zero, it indicates that the distribution of well flow data is asymmetrical. A positive skewness coefficient SK indicates a right-skewed distribution, while a negative skewness coefficient SK indicates a left-skewed distribution. If SK... i If ∈[-0.5,0.5], then the dispersion of the i-th well is small; if SK i ∈(-∞, -1) or (1, +∞), is called a highly skewed distribution; if SK i If the distribution is ∈(-1,-0.5) or (0.5,1), then it is considered to be a moderately skewed distribution.

[0071] S105: Dynamically adjust sampling rate and well location:

[0072] This invention uses a skewness coefficient to dynamically adjust the sampling rate and well location. Assuming the current location is the i-th well, the initial sampling rate is p. init The adjusted sampling rate p is then expressed as:

[0073]

[0074] The three distinct classes obtained from k-means clustering of the well data are denoted as Class0, Class1, and Class2 in this invention. Specifically, there are three cases: if SK... i For all classes ∈ [-0.5, 0.5], the sampling rate p = p init If SK i ∈(-∞, -1) or (1, +∞), increase the sampling rate p = 2 × p for the two classes with fewer samples. init ×|SK i |;If SK i ∈(-1, -0.5) or (0.5, 1), increase the sampling rate p = 2 × p for the class with the fewest samples. init ×|SKi |

[0075] This invention uses the skewness coefficient to dynamically adjust the well spacing, where the i-th well spacing WI i It is expressed as follows:

[0076]

[0077] In the formula, WIS init This indicates the initial well spacing. Specifically, there are two cases: if SK... i ∈[-0.5,0.5], increase well interval WI i =2×WI init If SK i ∈(-∞, -1)or(1, +∞)or(-1, -0.5)or(0.5, 1), reduce well spacing.

[0078] S106: Dynamically adjust the size of the well:

[0079] Streaming data peaks and troughs exhibit three distinct characteristics: 1. Very large slope variations; 2. Periodic variations; 3. Low dispersion. Based on these characteristics, the corresponding peaks are categorized as: shock-peek (SP), oscillation-peek (OP), and buffer-peek (BP), as shown in Figure 4(a); and the corresponding troughs are categorized as: shock-trough (ST), oscillation-trough (OT), and buffer-trough (BT), as shown in Figure 4(b). Shock-peek and shock-trough are characterized by very large slopes, resulting in very high dispersion when these types of peaks or troughs are present in the well. Oscillating peaks and troughs exhibit periodic variations, leading to relatively high dispersion when these types of peaks or troughs are present in the well. Buffer-peek and buffer-troughs exhibit low dispersion, resulting in lower dispersion in the well when these types of peaks and troughs are present in the well compared to the other two types. Furthermore, when two different wells contain the same type of peak or trough, the two wells exhibit a certain degree of self-similarity, and the degree of dispersion of the two wells will be very close.

[0080] Based on the above findings, in order to accurately locate the range of discrete data in streaming data, this invention proposes an algorithm that combines Pearson correlation coefficient and coefficient of variation to dynamically adjust well size.

[0081] The Pearson correlation coefficient is used to represent the degree of linear correlation between two sets of variables X and Y, and its value ranges from -1 to 1. A coefficient close to 1 indicates a large autocorrelation between X and Y. A coefficient close to -1 indicates an inverse autocorrelation between X and Y. A coefficient close to 0 indicates no significant autocorrelation between the two sets of related variables. The formula is shown below:

[0082]

[0083] The above formula is the covariance formula. The following formula is obtained by dividing the covariance by the standard deviation σ of the two related variables. X σ Y This compensates for the weak expressiveness of covariance data:

[0084]

[0085] The coefficient of variation (CV) = standard deviation σ / mean μ describes the relative dispersion of data and does not require the mean of the reference data. Therefore, it can be used to compare the dispersion of two sets of data, as shown in the formula below:

[0086]

[0087] S107: Algorithm for dynamically adjusting well size:

[0088] This invention first records representative wells in a well set, then uses a sliding window to receive new flow data, traverses the well set, sets the size of the sliding window to the size of different wells in the well set, and determines the size of the well by calculating the self-similarity and coefficient of variation of the wells in the well set and the sliding window.

[0089] S108: Dynamic Drilling Sampling Algorithm

[0090] This invention proposes a dynamic drilling sampling algorithm that performs unbiased intra-class sampling and biased inter-class sampling within the well. The intra-class sampling uses a classic reservoir sampling algorithm, while the inter-class sampling dynamically increases the sampling rate for minority classes based on the skewness coefficient. Within the well interval, equidistant sampling is employed to reduce the access rate of the sampled stream data. An algorithm for dynamically adjusting the well size accurately locates the discrete data range, and then the well interval size is dynamically adjusted based on the skewness coefficient to dynamically locate the discrete data position.

[0091] This invention proposes a novel streaming data value assessment model. It evaluates the value characteristics of the original dataset from three dimensions: discrete, centralized, and holistic. The model utilizes a sample set obtained through the large-scale streaming data sampling method. The specific assessment model is as follows: Figure 2As shown, the streaming data value assessment model uses discrete mean accuracy, discrete coefficient of variation accuracy, discrete sampling accuracy, lumped mean accuracy, lumped coefficient of variation accuracy, overall mean accuracy, overall coefficient of variation accuracy, and the JSD index.

[0092] Discrete Mean Accuracy:

[0093] Discrete Mean Accuracy (DMA) refers to the accuracy of estimating the mean of the discrete attribute values ​​of the original streaming dataset using the mean of the discrete attribute values ​​of the sample set. The formula for DMA is as follows:

[0094]

[0095] In the formula, This represents the mean of the discrete dataset attribute `value` in the original streaming dataset. This represents the mean of the discrete dataset attribute `value` in the sample set.

[0096] Accuracy of Discrete Coefficient of Variation:

[0097] This refers to the accuracy of estimating the coefficient of variation of the discrete attributes of the original streaming dataset using the coefficient of variation of the discrete attributes of the sample dataset. The formula for calculating the Discrete Coefficient of Variation (ADCV) is as follows:

[0098]

[0099] Discrete sampling accuracy:

[0100] Discrete Sampling Accuracy (DSA) is the ratio of the number of intersections between the discrete datasets of the sample set and the discrete datasets of the original streaming dataset to the length of the discrete datasets of the sample set. The formula for calculating DSA is as follows:

[0101]

[0102] In the formula, DDSS represents the discrete dataset of the sample set, DDRD represents the discrete dataset of the original stream dataset, and len(·) represents the computation length.

[0103] Central mean accuracy:

[0104] Centralized Mean Accuracy (CMA) refers to the accuracy of estimating the mean of the attribute values ​​of the original streaming dataset using the mean of the attribute values ​​of the centralized dataset in the sample set. The formula for calculating CMA is as follows:

[0105]

[0106] In the formula, This represents the mean of the dataset attribute `value` in the set of the original streaming dataset. This represents the mean of the attribute value in the central dataset of the sample set.

[0107] Accuracy of the coefficient of variation:

[0108] Accuracy of Centralized Coefficient of Variation (ACCV) refers to the accuracy of estimating the coefficient of variation of the centralized attribute values ​​of the original streaming dataset using the coefficient of variation of the centralized attribute values ​​of the sample dataset. The formula for calculating ACCV is shown below:

[0109]

[0110] In the formula, CV CDRD The coefficient of variation of the set attribute value of the original streaming dataset, CV CDSS The coefficient of variation of the attribute value in the dataset.

[0111] Overall mean accuracy:

[0112] This refers to the accuracy of estimating the mean of the attribute values ​​in the original streaming dataset RD (Raw Data) using the mean of the attribute values ​​in the sample set SS (Sample Set). The formula for calculating Overall Mean Accuracy (OMA) is as follows:

[0113]

[0114] In the formula, This represents the mean of the attribute `value` in the original stream dataset. This represents the mean of the attribute value in the sample set.

[0115] Overall coefficient of variation accuracy:

[0116] This refers to the accuracy of estimating the coefficient of variation of attribute values ​​in the original streaming dataset using the coefficient of variation of attribute values ​​in the sample set. The formula for calculating the overall coefficient of variation (AOCV) is as follows:

[0117]

[0118] In the formula, CV RD The coefficient of variation of the attribute value in the original streaming dataset, CV SS This represents the coefficient of variation of the attribute value in the sample set.

[0119] JSD Metrics:

[0120] The JSD metric, also known as JS divergence, is used to measure the difference between two probability distributions. It measures the distance between them by calculating their mean distributions and their KL divergence, with a value ranging from [0, 1]. A smaller KL divergence indicates a greater similarity between the sample set and the original streaming data distribution, while a larger JSD value indicates a greater difference between the sample set and the original streaming dataset distribution. Let the probability distribution of the sample set be P(x) and the probability distribution of the original streaming dataset be Q(X). The formula for calculating the KL divergence is as follows:

[0121]

[0122] The formula for calculating JS divergence is as follows:

[0123]

Claims

1. A method for large-scale streaming data sampling based on dynamic drilling, characterized in that, A method for sampling stream data S, the stream data S is represented as: S={(id i ,time i ,value i )|1≤i≤Nandi∈N + , wherein id i is the order of the i-th stream data arriving, time i is the time of the i-th stream data arriving, value i is the value of the i-th stream data, the method comprising the following steps: Step 1, determine the discrete data position and range in the flow data with well as analysis unit, wherein the i-th well is denoted as W i , the size of the i-th well is denoted as WS i , the number of sampling wells in the original flow data set is denoted as WN, and the data in W i is represented as: W i = {(id j , time j , value j )|1≤j≤WS i andj∈N +},1≤i≤WN Step 2, calculate the well intervals, where the ith well interval is denoted as WI i , the ith well interval size is denoted as WIS i , and the ith well interval WI i is expressed as: WI i = {(id j ,time j ,value j )|id wi_max +1≤id j ≤id wi+1_min -1} where id wi_max is the maximum id of all flow data in the ith well, id wi+1_min is the minimum id of all flow data in the ith+1 well; Step 3, the dispersion degree of the well is determined by using the skewness coefficient SK, if the distribution of the flow data in the well is symmetrical, the skewness coefficient SK is equal to 0, if the skewness coefficient SK is obviously not equal to 0, it indicates that the distribution of the flow data in the well is asymmetrical, wherein, if the skewness coefficient SK is positive, it is right-skewed distribution, if the skewness coefficient SK is negative, it is left-skewed distribution; If the distribution of the well flow data of the ith well is asymmetric, and the skewness coefficient of the ith well is denoted as SK i , then there is: if SK i ∈[-0.5,0.5], the dispersion degree of the ith well is small; if SK i ∈(-∞,-1)or(1,+∞), the dispersion degree of the ith well is large, which is called highly skewed distribution; if SK i ∈(-1,-0.5)or(0.5,1), it is considered to be moderately skewed distribution. Step 4, dynamically adjust the sampling rate and well interval using skewness coefficient, assuming that the current is the i th well, the initial sampling rate is p init The adjusted sampling rate p is represented as: adjusted ith well spacing size WIS i is represented as follows: where WIS init represents the initial well spacing size for the i-th well; Step 5, the size of the well is dynamically adjusted by combining the Pearson correlation coefficient and the coefficient of variation: Representative wells are recorded in the well set, then the newly obtained flow data is accepted by the sliding window, the well set is traversed, the size of the sliding window is set as the size of different wells in the well set, the size of the well is determined by calculating the Pearson correlation coefficient and the coefficient of variation of the well set and the sliding window; Step 6, dynamic well sampling: In-well unbiased sampling and inter-class biased sampling are performed, wherein, the sampling rate of the minority class is dynamically increased according to the skewness coefficient in the inter-class; in the inter-well interval, equidistance sampling is used to reduce the access rate of the sampled flow data, the dispersion data range is accurately positioned according to the dynamic adjustment of the size of the well in the algorithm of step 5, then the inter-well interval size is dynamically adjusted according to the skewness coefficient, and the dispersion data position is dynamically positioned.

2. The method for large scale streaming data sampling based on dynamic drilling of claim 1, wherein, In step 3, the i-th well internal skewness coefficient SK i is expressed as: wherein denotes the mean value of the data in the i-th well.

3. The method of claim 1, wherein, In step 5, the Pearson correlation coefficient ρ (X, Y) of the two groups of variables X and Y is represented as: where σ X , σ Y are the standard deviations of the two groups of variables X and Y, respectively, and cov(X, Y) is the Pearson correlation coefficient before adjustment, which is expressed by the following formula: where n represents the amount of data in the variables X and Y, x i represents the i-th variable in the current set of variables X, y i represents the i-th variable in the current set of variables Y, represents the mean of the current set of variables X, represents the mean of the current set of variables Y.

4. The method for large scale streaming data sampling based on dynamic drilling of claim 1, wherein, In step 5, the coefficient of variation CV is represented as: In the formula, σ is the standard deviation of the current group of variables, and μ is the mean of the current group of variables.

5. A method for large-scale streaming data evaluation based on dynamic drilling, characterized in that, The sample set is obtained by sampling from the original flow data set by using the large-scale flow data sampling method in claim 1, and the value characteristics of the original data set are evaluated based on the sample set by using the discrete mean accuracy, the discrete coefficient of variation accuracy, the discrete sampling accuracy, the concentrated mean accuracy, the concentrated coefficient of variation accuracy, the overall mean accuracy, the overall coefficient of variation accuracy and the JSD index, wherein: The discrete mean accuracy is the accuracy of estimating the mean of the discrete data set attribute value of the original flow data set by using the mean of the discrete data set attribute value of the sample set; The discrete coefficient of variation accuracy is the accuracy of estimating the coefficient of variation of the discrete data set attribute value of the original flow data set by using the coefficient of variation of the discrete data set attribute value of the sample set; The discrete sampling accuracy is the ratio of the intersection number of the discrete data set of the sample set and the discrete data set of the original flow data set to the length of the discrete data set of the sample set; The concentrated mean accuracy is the accuracy of estimating the mean of the concentrated data set attribute value of the original flow data set by using the mean of the concentrated data set attribute value of the sample set; The concentrated coefficient of variation accuracy is the accuracy of estimating the coefficient of variation of the concentrated data set attribute value of the original flow data set by using the coefficient of variation of the concentrated data set attribute value of the sample set; The overall mean accuracy is the accuracy of estimating the mean of the attribute value of the original flow data set by using the mean of the attribute value of the sample set; The overall coefficient of variation accuracy is the accuracy of estimating the coefficient of variation of the attribute value of the original flow data set by using the coefficient of variation of the attribute value of the sample set; The JSD index measures the distance between the two probability distributions by calculating the average distribution of the two probability distributions and the KL divergence thereof. The smaller the JSD index is, the greater the similarity between the sample set and the original stream data distribution is. The greater the JSD value is, the greater the difference between the sample set and the original stream data set distribution is. The JSD index is calculated by using the following formula: where P(x) is the probability distribution of the sample set, Q(X) is the probability distribution of the original stream data set, D(P||Q) represents the calculation of the KL divergence of P(x) and Q(X), and then

Images