A method and apparatus for estimating elements in a distributed environment

Abstract

The application discloses an element estimation method and device in a distributed environment, and the method comprises the following steps: extracting a sample from each node respectively; the node is a distributed database; performing frequency statistics on each sample to obtain a data dictionary; maintaining at least two data abstracts on each node, wherein the data abstracts comprise data frequency features obtained based on the data dictionary; the data frequency features represent the features of elements in the sample; and sending the data abstracts in each node to a master node, and merging the data abstracts in all nodes by the master node to obtain a first data abstract and estimating the elements based on the first data abstract can avoid the problems of excessively large data scale and excessively high communication cost between machines in the distributed environment, thereby improving the efficiency of estimating the number of elements in the distributed environment.

Description

Technical Field

[0001] This invention relates to the field of computer science and technology, and in particular to a method and apparatus for estimating elements in a distributed environment. Background Technology

[0002] With the advent of the big data era, the scale of data has grown dramatically, making it difficult for traditional databases to maintain the massive amounts of data generated in reality. Real-world big data applications distribute data across different machines, making traditional computational problems extremely challenging at this scale. Beyond traditional time and space complexity analysis, the communication cost of data in a distributed environment requires careful consideration. In traditional data computation models, data resides in memory, typically requiring only consideration of space and time costs. However, in a distributed environment, the communication cost must also be taken into account. This communication cost can lead to increased I / O pressure and potentially impact computational efficiency.

[0003] In databases, given a column of data, data items with different values ​​can be considered as different elements. Estimating the number of distinct elements in the data is one of the core problems in this field. Using the estimated number of distinct elements, query efficiency and compression algorithms can be optimized, making this problem particularly important. When the data volume is very large, estimating the number of distinct elements presents new challenges. Traditional estimation methods for centralized databases on a single machine require a full table scan, resulting in very high I / O costs, making this method unsuitable for extremely large data volumes. While sampling avoids high I / O costs, it introduces significant errors, especially when distributed databases are necessary for very large datasets. Under statistical assumptions, the error in approximating the true value through data sampling can be analyzed theoretically. Existing theoretical analysis shows that for effective estimation, the sampling size cannot be too small; generally, in practical applications, the sampling rate needs to be set between 0.01 and 0.001. A sampling rate that is too high becomes unnecessary, while a sampling rate that is too low cannot provide an effective estimate and introduces bias. For example, when the data volume reaches the hundreds of terabytes level, it means that the sample size will also reach the terabyte level. Transmitting terabytes of data between machines puts enormous pressure on them. Therefore, due to the inherent limitations of estimation error in sampling algorithms, merging data samples from different machines to obtain the final data frequency leads to significant communication costs and places a heavy burden on the server.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for estimating elements in a distributed environment, thereby reducing the communication cost in the process of estimating the number of different elements based on sampling methods, and thus improving the efficiency of estimating elements in a distributed environment.

[0006] To achieve the above objectives, the present invention provides a method for estimating elements in a distributed environment, the method comprising:

[0007] Extract one sample from each node; the nodes are a distributed database.

[0008] Frequency statistics are performed on each of the samples to obtain a data dictionary;

[0009] At least two data summaries are maintained on each node, the data summaries including data frequency features obtained based on the data dictionary; the data frequency features characterize the features of elements in the sample.

[0010] The data digest from each node is sent to the master node, which then merges the data digests from all nodes to obtain a first data digest and estimates the elements based on the first data digest.

[0011] Preferably, extracting a sample from each node includes:

[0012] When the elements in the node are evenly distributed, the sample is extracted by block sampling.

[0013] Otherwise, the samples extracted from the block sampling are corrected.

[0014] Preferably, the number of data summaries maintained on each node is related to the statistics.

[0015] Preferably, when the space of the data dictionary is smaller than the data digest, the data dictionary is sent to the master node.

[0016] Preferably, the method further includes:

[0017] The estimation method is corrected by the data frequency characteristics, which include low-frequency information, high-frequency information, and integrated data information.

[0018] In one embodiment of the present invention, a method for estimating elements in a distributed environment is provided, applied to a master node, the method comprising:

[0019] Receive data digests from each node; the data digests include data frequency features obtained based on a data dictionary; the data dictionary is a frequency statistic of samples extracted from each node, and the data frequency features characterize the features of elements in the samples; the data frequency features include: low-frequency information, high-frequency information, and data integration information;

[0020] The received data digests are merged to obtain a first data digest;

[0021] The elements are estimated based on the first data digest.

[0022] Preferably, the method further includes:

[0023] Obtain a data summary of data frequency characteristics from other nodes as data integration information; the other nodes are nodes other than the node to be estimated.

[0024] Merge a data digest of the other nodes into a second data digest and obtain a second estimate based on the second data digest;

[0025] The data summary and the second data summary, which are low-frequency information in the data frequency characteristics of the node to be estimated, are merged into a third data summary.

[0026] A third estimate is obtained based on the third data summary;

[0027] The difference between the third estimate and the second estimate is used to obtain the fourth estimate; the fourth estimate is the number of elements in the node to be estimated whose data frequency characteristics are low-frequency information.

[0028] In one embodiment of the present invention, an element estimation apparatus for a distributed environment is provided, the apparatus comprising:

[0029] An extraction unit is used to extract one sample from each node; the node is a distributed database.

[0030] A statistical unit is used to perform frequency statistics on each of the samples to obtain a data dictionary;

[0031] An acquisition unit is configured to maintain at least two data digests on each node, the data digests including data frequency features obtained based on the data dictionary; the data frequency features characterize the features of elements in the sample.

[0032] A sending unit is configured to send the data digest from each of the nodes to a master node, so that the master node can merge the data digests from all nodes to obtain a first data digest and estimate the elements based on the first data digest.

[0033] In one embodiment of the present invention, an element estimation apparatus for a distributed environment is provided, the apparatus comprising:

[0034] A receiving unit is configured to receive data digests sent from each node; the data digests include data frequency features obtained based on a data dictionary; the data dictionary is a frequency statistic of samples extracted from each node, and the data frequency features characterize the features of elements in the samples; the data frequency features include: low-frequency information, high-frequency information, and data integration information;

[0035] A merging unit is used to merge the received data digests to obtain a first data digest;

[0036] An estimation unit is used to estimate elements based on the first data digest.

[0037] In one embodiment of the present invention, a computer-readable storage medium is provided, on which a program is stored, which, when executed, implements the steps of an element estimation method in a distributed environment as described in any of the preceding claims.

[0038] Compared with existing technologies, the present invention avoids the problems of excessively large data scale, too many sampled elements, and excessive communication costs between machines in a distributed environment, thereby improving the efficiency of estimating the number of elements in a distributed environment. By improving the efficiency of estimating the number of elements, the efficiency of all problems that depend on the estimation of the number of elements can be further improved under large-scale data conditions, such as query optimization and data compression. Attached Figure Description

[0039] Figure 1 This is a flowchart of an element estimation method in a distributed environment according to an embodiment of the present invention;

[0040] Figure 2 This is a flowchart of an element estimation method in a distributed environment according to an embodiment of the present invention;

[0041] Figure 3 This is a structural diagram of an element estimation device in a distributed environment according to an embodiment of the present invention;

[0042] Figure 4 This is a structural diagram of an element estimation device in a distributed environment according to an embodiment of the present invention. Detailed Implementation

[0043] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0044] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0045] In a distributed environment, there is currently no method specifically designed to optimize communication costs for estimating the number of elements using sampling algorithms. While existing algorithms for estimating the number of different elements have their applications, there is still room for improvement in distributed environments when using sampling algorithms to estimate the number of different elements. Improving the efficiency of estimating the number of different elements in a distributed environment can, to some extent, ensure that the additional burden generated by query optimization and compression optimization is not too large. Otherwise, if calculating the number of different elements incurs excessive costs during the query optimization and compression optimization process, the optimization loses its intended meaning. Therefore, the following specific embodiments of the present invention are proposed.

[0046] like Figures 1 to 2 As shown, according to a preferred embodiment of the present invention, a method for estimating elements in a distributed environment includes:

[0047] Step 101: Extract a sample from each node; the node is a distributed database. In practice, this means obtaining a sample of the data stored on each physical machine. In a distributed environment, a machine can be considered a physical node of a distributed database.

[0048] Step 102: Perform frequency statistics on each sample to obtain a data dictionary. Specifically, perform frequency statistics on the samples extracted from each node. Frequency refers to the number of times data appears in a sample. For example, extracting samples from node X and performing frequency statistics on the data in those samples yields the following data dictionary: Where frequency Data item in the corresponding sample i The frequency at node X is the process of counting the frequency of the same data in the sample and forming a data dictionary.

[0049] Step 103: Maintain at least two data summaries on each node. Each data summary includes data frequency features obtained based on the data dictionary; the data frequency features characterize the features of the elements in the sample. Specifically, maintain two or more data summaries on each node. Depending on the features used for the statistics, different data summaries are needed to extract features. These data summaries can be used to return the features of the sample, thereby supporting the estimation of different elements, without needing to send a complex data dictionary.

[0050] Specifically, data summarization is a practical data structure in the field of streaming data. Such data structures occupy little physical space and are often used to extract data features and maintain key data information. In distributed scenarios, processing complete data often leads to excessively high communication costs. Transmitting data summaries can avoid excessive data communication while theoretically ensuring the accuracy of the estimation results. Therefore, in a distributed environment, utilizing such a small data structure can improve computational efficiency and reduce the heavy burden on the server. The data summarization embodiments of this invention typically involve several core operations:

[0051] Update: Used to read data one by one and put the data into a data digest. The space of the data digest is constant after the parameters are fixed and will not increase due to update operations.

[0052] Merging: Two data digests of the same type can be merged. The merged data digest incorporates the information from both, and the space of the merged data digest is equal to that of the original two digests.

[0053] Estimation: This operation returns an estimate of the information stored in the summary. The type of estimate returned for different data items varies depending on the summary. For example, the Hyperloglog summary returns an estimate of how many different elements were input into this data summary.

[0054] Step 104: The data digest from each node is sent to the master node. The master node merges the data digests from all nodes to obtain a first data digest and estimates the element based on the first data digest. In a distributed environment, a central machine is typically used as the master node to distinguish it from other nodes. The data digests from each node other than the master node are sent to the master node, which merges all data digests and uses the merged data digest as the first data digest. Since the data digests used here are very small, the merged first data digest is also relatively small, thus avoiding high communication costs. Subsequently, the master node can estimate the element based on the first data digest to obtain the estimated value of the element.

[0055] The method described in this embodiment of the invention can avoid the problems of excessive data size, too many sampled elements, and excessive communication costs between machines in a distributed environment, thereby improving the efficiency of estimating the number of elements in a distributed environment. By improving the efficiency of estimating the number of elements, the efficiency of all problems that depend on the estimation of the number of elements can be improved under large-scale data conditions, such as query optimization and data compression.

[0056] Preferably, extracting a sample from each node includes:

[0057] When the elements in the node are evenly distributed, the sample is extracted by block sampling to minimize the amount of I / O occupied.

[0058] Otherwise, the samples extracted from block sampling are corrected. When the data is not uniformly distributed, after obtaining samples through block sampling, the data in the samples needs to be corrected to a certain extent. The specific correction method can adopt feasible solutions in the existing technology.

[0059] In a preferred embodiment, the number of data summaries maintained on each node is related to the statistic. Specifically, at least two data summaries are typically maintained on each node to meet different later usage requirements. The specific requirements for the data summaries depend on the statistic used later. Depending on the features used by the statistic, different data summaries are needed to extract features. However, the number of data summaries maintained on different nodes needs to be kept consistent so that these data summaries can be used to return the features of samples on different nodes and support the estimation of different elements.

[0060] In a preferred embodiment of the present invention, an element estimation method in a distributed environment is described in which, when the space of the data dictionary is smaller than the data digest, the data dictionary is sent to the master node without needing to obtain the data digest from the data dictionary and then send the data digest to the master node.

[0061] In a preferred embodiment of the present invention, Hyperloglog can be used to return the number of distinct elements in a sample, and CountSketch can be used to return the l2 norm. Specifically, the estimation of the l0 norm in data frequency features (the physical meaning of the l0 norm is the number of distinct elements) can typically be done using Hyperloglog or KMV. The estimation of the l2 norm of data frequencies can be done using CountSketch, etc. The two basic data summaries maintained are both l0 norm data summaries, denoted as H1 and H2. d All elements that have appeared in the sample are input into H. d In the first instance, elements that appear only once are input into H1. When more complex frequency features are needed, such as the l2 norm of the frequency, the samples can be input into CountSketch.

[0062] The method for estimating elements in a distributed environment according to an embodiment of the present invention, preferably, further includes:

[0063] The estimation method is modified using the data frequency characteristics, which include low-frequency information, high-frequency information, and integrated data information. Low-frequency information refers to the number of elements that appear only once in the sample; this indicates the number of elements with relatively low frequency in the estimated data. High-frequency information refers to the number of elements that appear frequently; preferably, the data frequency characteristics of elements appearing more than 100 times are defined as high-frequency information, which can be represented by a norm combination of frequencies. Integrated data information is the number of different elements in the sample, i.e., how many types of elements are present in the sample.

[0064] Specifically, after collecting the required data frequency features using data summaries, the estimation method needs to be modified to facilitate subsequent estimations. Only some examples are described here. The statistical examples here can be replaced by any statistical measure that can be transformed into this framework, and are not limited to those exemplified in the embodiments of this invention.

[0065] This invention provides three examples of statistical measures for estimating the number of different elements, modifying the estimation method for each example to enable estimation of the number of different elements using the method described in this invention. The calculation expressions for the three different element counts are as follows:

[0066] Only the number of distinct elements and the number of elements that appear only once in the sample are needed to directly adapt the method described in this invention, without requiring modification of the following statistics:

[0067]

[0068] Where, D GEE Let represent the number of distinct elements guaranteed by the error estimation (Guaranteed-Error Estimator), where q is the sampling rate, and fi is the number of elements in the sample that appear i times in total. Since D... GEE The formula can be rewritten as This method only requires the sum of the number of distinct elements d and the number of elements that appear only once in the sample to complete the estimation.

[0069] With slight modifications, the statistic can be estimated using only the number of distinct elements and the number of elements that appear only once in the sample:

[0070] D Chao =d+(f1(f1-1)) / 2(f2+1)

[0071] This statistic was proposed by Chao, hence it is denoted as D. Chao Although D ChaoThere exists a term f2 that is difficult to estimate, which is the number of elements that appear twice in the sample. However, in reality, when the data is highly skewed and f1 plays a dominant role, the frequency of other elements becomes very small, and the number of high-frequency elements is not very large. In such cases, f2 can be approximated using d-f1, thus allowing the original D to be reduced. Chao Transform into the following form:

[0072]

[0073] For a more complex class of statistics, the most typical one can be represented as l of the data. p Norm, its expression is ∑ i i p f i For example, the more complex D proposed by Chao-Lee CL Statistic:

[0074]

[0075]

[0076] Where n represents the total number of sample elements, f i Let d be the number of elements that appear i times in the sample, and let d represent the number of distinct elements in the sample. CL The statistic also requires d,f1, which is an approximation of the combination of other frequencies, i.e., ∑ i i 2 f i The L2 norm here can be calculated using the CountSketch algorithm in traditional dataflow models.

[0077] In a preferred embodiment, the information obtained from the above embodiments regarding the estimation of the number of different elements in the sample, the estimation of the number of elements that appear only once, and the l2 norm of the frequency are input into the modified estimation method, and the estimation of the number of different elements can be obtained from the statistics obtained by the transformation.

[0078] The present invention provides an element estimation method in a distributed environment, applied to a master node, the method comprising:

[0079] Step 201: Receive data summaries from each node; the data summaries include data frequency features obtained based on each sample; the data frequency features characterize the features of elements in the sample; the data frequency features include: low-frequency information, high-frequency information, and data integration information. Specifically, in a distributed environment, a machine can be considered a physical node of a distributed database, and a central machine is used as the master node to distinguish it from other nodes. Nodes other than the master node send data summaries to the master node, and each node maintains its own data summaries, which include data frequency features obtained based on the data dictionary; the data frequency features characterize the features of elements in the sample. Specifically, each node maintains two or more data summaries. Depending on the features used for the statistics, different data summaries are needed to extract features. These data summaries can be used to return the features of the sample, thereby supporting the estimation of different elements.

[0080] The data dictionary is derived by extracting samples from node X and performing frequency statistics on the data in those samples, resulting in the following data dictionary: Where frequency Data item in the corresponding sample i The frequency at node X is the frequency of occurrence of the same data in the sample.

[0081] Step 202: The received data digests are merged to obtain a first data digest. Specifically, the master node integrates the data digests of all nodes and uses the integrated data digest as the first data digest. Since the data digests used here are very small, the integrated first data digest is also relatively small, thus avoiding high communication costs.

[0082] Step 203: Estimate the elements based on the first data digest. The master node can estimate the elements based on the first data digest to obtain the estimated values ​​of the elements. Specifically, it estimates the number of different elements d in the sample, the number of elements f1 that appear only once in the sample, and the l2 norm of the data frequency. For the number of different elements d in the sample, it is only necessary to estimate the data digest H. d By merging the results and then using the merged summary for estimation, we can obtain the final result. The L2 norm of the sample data frequency can be estimated by merging the transmitted CountSketch data summary.

[0083] The method described in this embodiment of the invention can avoid the problems of excessive data size, too many sampled elements, and excessive communication costs between machines in a distributed environment, thereby improving the efficiency of estimating the number of elements in a distributed environment. By improving the efficiency of estimating the number of elements, the efficiency of all problems that depend on the estimation of the number of elements can be improved under large-scale data conditions, such as query optimization and data compression.

[0084] The method for estimating elements in a distributed environment according to an embodiment of the present invention, preferably, further includes: estimating the number of elements that appear only once. Let the two data digests on the i-th node include... To calculate the number of elements that appear only once in the sample, we only need to estimate how many elements appear only once at each node. The number of elements that appear only once at each node can be calculated using the following method:

[0085] Data summaries of data frequency characteristics from other nodes are obtained as data integration information; the other nodes are nodes other than the node to be estimated. In this embodiment of the invention, a node i to be estimated is first determined, and the nodes other than the node i to be estimated are referred to as other nodes.

[0086] The data digests of the other nodes are merged into a second data digest, and a second estimate is obtained based on the second data digest. In one specific embodiment, the H values ​​from the nodes other than the node to be estimated (i) are... d Data summary merging, H d A data summary is a data digest whose frequency characteristics are integrated information from the data; it is referred to as the second data summary. And record the second data summary. The second estimate d res , where d res This represents the estimated value of the element that has appeared except for the node i to be estimated.

[0087] The data digest of node i, whose data frequency characteristics are low-frequency information, and the second data digest are merged into a third data digest, which is the data digest of node i to be estimated. With the second data summary Merged into a third data summary.

[0088] Obtain the third estimate based on the third data digest; return the third estimate d from the third data digest. all Here is the third estimate d. all This represents the estimated number of elements that appear only once at node i or have appeared at other nodes.

[0089] Obtain the third estimate d all Compared with the second estimate dres The difference is used to obtain a fourth estimate; the fourth estimate is the number of elements in the node to be estimated whose data frequency characteristics are low-frequency information. Specifically, using formula d all -d res This gives an estimate of the number of elements at node i that appear only once in the final sample. It cannot be used directly. To perform statistical analysis, we need to use a merge subtraction method to obtain an estimate of the element that appears only once at node i.

[0090] Using the method described above, we can calculate the number of elements that appear only once at each node, and then sum these values ​​together to obtain an estimate of the number of elements that appear only once.

[0091] This invention provides an element estimation device in a distributed environment, such as... Figure 3 As shown, the device includes:

[0092] Extraction unit 301 is used to extract one sample from each node; the node is a distributed database;

[0093] Statistical unit 302 is used to perform frequency statistics on each sample to obtain a data dictionary;

[0094] The acquisition unit 303 is configured to maintain at least two data digests on each node, the data digests including data frequency features obtained based on the data dictionary; the data frequency features characterize the features of elements in the sample;

[0095] The sending unit 304 is configured to send the data digest of each of the nodes to the master node, so that the master node can merge the data digests of all nodes to obtain a first data digest and estimate the elements based on the first data digest.

[0096] This invention provides an element estimation device in a distributed environment, such as... Figure 4 As shown, the device includes:

[0097] The receiving unit 401 is configured to receive a data digest from each node; the data digest includes data frequency features obtained based on each sample; the data frequency features characterize the features of elements in the sample; the data frequency features include: low-frequency information, high-frequency information, and data integration information;

[0098] Merging unit 402 is used to merge the received data digests to obtain a first data digest;

[0099] The estimation unit 403 is used to estimate the elements based on the first data digest.

[0100] This invention provides a computer-readable storage medium storing a program that, when executed, implements the steps of an element estimation method in a distributed environment as described in any of the preceding specific embodiments.

[0101] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for estimating elements in a distributed environment, characterized in that, The method includes: Extract one sample from each node; the nodes are a distributed database. Frequency statistics are performed on each of the samples to obtain a data dictionary; At least two data summaries are maintained on each node, the data summaries including data frequency features obtained based on the data dictionary; the data frequency features characterize the features of elements in the sample. The data digest in each node is sent to the master node, which then merges all received data digests of the same type. The merging operation combines two data digests of the same type, and the merged data digest incorporates the information of both to obtain a first data digest. An estimation operation is then performed based on the first data digest. Specifically, the estimation operation involves the master node obtaining an estimate of the number of different elements in the sample based on the first data digest as data integration information, and obtaining an estimate of the number of elements that appear only once in the local sample in each node as low-frequency information; and using the data integration information and the low-frequency information to estimate the element. The estimation is based on the number of distinct elements and the number of elements that appear only once in the sample, as shown in the following formula: ； in, This represents the estimator of the number of distinct elements that guarantee the estimation error, where q is the sampling rate. This represents the number of elements in the sample that appear a total of i times. Specifically, estimating the number of elements that appear only once includes: Obtain a data summary of the data integration information from the data frequency characteristics of all nodes except node i; The data digests of the data integration information of the other nodes are merged to form a second data digest, and a second estimate is obtained based on the second data digest. The second estimate represents an estimate of the number of elements that have appeared in the other nodes. The data digest of node i with low frequency information is merged with the second data digest to form a third data digest, and a third estimate is obtained based on the third data digest; Calculate the difference between the third estimate and the second estimate to obtain an estimate of the number of elements at node i whose data frequency characteristics are low-frequency information; The estimated values ​​obtained for each node are summed to obtain an estimate of the number of elements that appear only once in the entire distributed environment.

2. The method for estimating elements in a distributed environment according to claim 1, characterized in that, Extracting a sample from each node includes: When the elements in the node are evenly distributed, the sample is extracted by block sampling. Otherwise, the samples extracted from the block sampling are corrected.

3. The method for estimating elements in a distributed environment according to claim 1, characterized in that, The number of data summaries maintained at each node depends on the features required for the statistics.

4. The method for estimating elements in a distributed environment according to claim 1, characterized in that, When the space of the data dictionary is smaller than the data digest, the data dictionary is sent to the master node.

5. The method for estimating elements in a distributed environment according to claim 1, characterized in that, The method further includes: The estimation method is corrected by the data frequency characteristics, which include low-frequency information, high-frequency information, and integrated data information.

6. A method for estimating elements in a distributed environment, applied to a master node, characterized in that, The method includes: Receive data digests from each node; the data digests include data frequency features obtained based on a data dictionary; the data dictionary is a frequency statistic of samples extracted from each node, and the data frequency features characterize the features of elements in the samples; the data frequency features include: low-frequency information, high-frequency information, and data integration information; All received data digests of the same type are merged. The merging operation is to merge two data digests of the same type. The merged data digest incorporates the information of the two to obtain a first data digest. An estimation operation is performed based on the first data digest, wherein the estimation operation is used to estimate the element by returning an estimated value of the information stored in the first data digest; Specifically, the estimation operation involves the master node obtaining an estimate of the number of different elements in the sample based on the first data digest as data integration information, and obtaining an estimate of the number of elements that appear only once in the local sample in each node as low-frequency information; and using the data integration information and the low-frequency information to estimate the element. The estimation is based on the number of distinct elements and the number of elements that appear only once in the sample, as shown in the following formula: ； in, This represents the estimator of the number of distinct elements that guarantee the estimation error, where q is the sampling rate. This represents the number of elements in the sample that appear a total of i times. Specifically, estimating the number of elements that appear only once includes: Obtain a data summary of the data integration information from the data frequency characteristics of all nodes except node i; The data digests of the data integration information of the other nodes are merged to form a second data digest, and a second estimate is obtained based on the second data digest. The second estimate represents an estimate of the number of elements that have appeared in the other nodes. The data digest of node i with low frequency information is merged with the second data digest to form a third data digest, and a third estimate is obtained based on the third data digest; Calculate the difference between the third estimate and the second estimate to obtain an estimate of the number of elements at node i whose data frequency characteristics are low-frequency information; The estimated values ​​obtained for each node are summed to obtain an estimate of the number of elements that appear only once in the entire distributed environment.

7. The method for estimating elements in a distributed environment according to claim 6, characterized in that, The method further includes: Obtain a data summary of data frequency characteristics from other nodes as data integration information; the other nodes are nodes other than the node to be estimated. Merge a data digest of the other nodes into a second data digest and obtain a second estimate based on the second data digest; The data summary and the second data summary, which are low-frequency information in the data frequency characteristics of the node to be estimated, are merged into a third data summary. A third estimate is obtained based on the third data summary; The difference between the third estimate and the second estimate is used to obtain the fourth estimate; the fourth estimate is the number of elements in the node to be estimated whose data frequency characteristics are low-frequency information.

8. An element estimation device in a distributed environment, characterized in that, The device includes: An extraction unit is used to extract one sample from each node; the node is a distributed database. A statistical unit is used to perform frequency statistics on each of the samples to obtain a data dictionary; An acquisition unit is configured to maintain at least two data digests on each node, the data digests including data frequency features obtained based on the data dictionary; the data frequency features characterize the features of elements in the sample. The sending unit is used to send the data digest in each of the nodes to the master node, so that the master node can perform a merging operation on all received data digests of the same type. The merging operation is to merge two data digests of the same type. The merged data digest integrates the information of the two to obtain a first data digest, and an estimation operation is performed based on the first data digest. Specifically, the estimation operation involves the master node obtaining an estimate of the number of different elements in the sample based on the first data digest as data integration information, and obtaining an estimate of the number of elements that appear only once in the local sample in each node as low-frequency information; and using the data integration information and the low-frequency information to estimate the element. The estimation is based on the number of distinct elements and the number of elements that appear only once in the sample, as shown in the following formula: ； in, This represents the estimator of the number of distinct elements that guarantee the estimation error, where q is the sampling rate. This represents the number of elements in the sample that appear a total of i times. Specifically, estimating the number of elements that appear only once includes: Obtain a data summary of the data integration information from the data frequency characteristics of all nodes except node i; The data digests of the data integration information of the other nodes are merged to form a second data digest, and a second estimate is obtained based on the second data digest. The second estimate represents an estimate of the number of elements that have appeared in the other nodes. The data digest of node i with low frequency information is merged with the second data digest to form a third data digest, and a third estimate is obtained based on the third data digest; Calculate the difference between the third estimate and the second estimate to obtain an estimate of the number of elements at node i whose data frequency characteristics are low-frequency information; The estimated values ​​obtained for each node are summed to obtain an estimate of the number of elements that appear only once in the entire distributed environment.

9. An element estimation device in a distributed environment, characterized in that, The device includes: A receiving unit is configured to receive data digests sent from each node; the data digests include data frequency features obtained based on a data dictionary; the data dictionary is a frequency statistic of samples extracted from each node, and the data frequency features characterize the features of elements in the samples; the data frequency features include: low-frequency information, high-frequency information, and data integration information; The merging unit is used to merge all received data digests of the same type. The merging operation is to merge two data digests of the same type. The merged data digest incorporates the information of the two to obtain a first data digest. An estimation unit is configured to perform an estimation operation based on the first data digest, wherein the estimation operation is configured to estimate the element by returning an estimated value of the information stored in the first data digest; Specifically, the estimation operation involves the master node obtaining an estimate of the number of different elements in the sample based on the first data digest as data integration information, and obtaining an estimate of the number of elements that appear only once in the local sample in each node as low-frequency information; and using the data integration information and the low-frequency information to estimate the element. The estimation is based on the number of distinct elements and the number of elements that appear only once in the sample, as shown in the following formula: ； in, This represents the estimator of the number of distinct elements that guarantee the estimation error, where q is the sampling rate. This represents the number of elements in the sample that appear a total of i times. Specifically, estimating the number of elements that appear only once includes: Obtain a data summary of the data integration information from the data frequency characteristics of all nodes except node i; The data digests of the data integration information of the other nodes are merged to form a second data digest, and a second estimate is obtained based on the second data digest. The second estimate represents an estimate of the number of elements that have appeared in the other nodes. The data digest of node i with low frequency information is merged with the second data digest to form a third data digest, and a third estimate is obtained based on the third data digest; Calculate the difference between the third estimate and the second estimate to obtain an estimate of the number of elements at node i whose data frequency characteristics are low-frequency information; The estimated values ​​obtained for each node are summed to obtain an estimate of the number of elements that appear only once in the entire distributed environment.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed, implements the steps of an element estimation method in a distributed environment as described in any one of claims 1-7.

Images