Probability filtering method and device for distributed data, equipment and storage medium

Abstract

The invention relates to a probability filtering method and device for distributed data, equipment and a storage medium. The method comprises the following steps: collecting the highest score of each local node in the distributed system, generating an original data set, and clipping the original data set by adopting a value-based filter to obtain a clipped data set; for the clipped data set, adopting a Poisson filter to generate a random sample, and utilizing the random sample to send a response message to a query initiator; after the query initiator receives a response message, constructing a first candidate list C1 according to the rule of the Poisson filter, and refining the first candidate list C1 by using the clipped data set to generate a second candidate list C2; and after the second candidate list C2 is verified through the original data set, outputting a query result. According to the method, the Poisson filter is utilized to process the top-k aggregation query of the distributed system with low communication cost, so that the overall calculation cost and communication cost are reduced.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a method, device, equipment and storage medium for probabilistic filtering of distributed data. Background Art

[0002] When querying large-scale data, top-k aggregation queries are often used to provide users with concise and meaningful query results. Given a distributed system consisting of m local nodes, the goal of a top-k aggregation query is to find a subset T of size k. This type of query aggregates the scores of the same set of items from different sources and identifies the top k items with the highest combined scores. When the amount of data is small, the top-k aggregation question on distributed data can be simply answered by retrieving all the data and processing them centrally. However, when processing larger amounts of data, limited communication bandwidth and computing resources require more complex query schemes.

[0003] To address the above problems, many methods enhance the query process by using a set of conditions as filters to prevent local nodes from sending low-quality data. This method enhances the query process and reduces communication overhead and the computational cost of the query initiator (center) by adopting a set of conditions as filters to limit the amount of information transmitted by each local node in the distributed system. These filters include value-based filtering, position-based filtering, and generated information-based filtering. However, simple value-based and position-based filtering methods for TPUT, TA, and their variants still allow too much irrelevant data to pass, resulting in high execution costs. In contrast, generated information-based filtering such as KLEE and Bernoulli sampling are much more efficient in limiting communication costs, but they require more computing resources in each local node and there is still much room for improvement in effectiveness. Summary of the invention

[0004] The present application provides a distributed data probability filtering method, device, equipment and storage medium, aiming to solve at least one of the above technical problems in the prior art to a certain extent.

[0005] In order to solve the above problems, this application provides the following technical solutions:

[0006] A probabilistic filtering method for distributed data, comprising:

[0007] After receiving the top-k aggregation query request sent by the query initiator, the highest score of each local node in the distributed system is collected to generate an original data set, and the original data set is pruned by a value-based filter to obtain a pruned data set;

[0008] For the trimmed data set, a Poisson filter is used to generate a random sample, and a response message is sent to the query initiator using the random sample;

[0009] After receiving the response message, the query initiator constructs a first candidate list C1 according to the rule of the Poisson filter, and uses the trimmed data set to refine the first candidate list C1 to generate a second candidate list C2;

[0010] After verifying the second candidate list C2 through the original data set, the query result is output.

[0011] The technical solution adopted by the embodiment of the present application also includes: the original data set is trimmed by using a value-based filter to obtain a trimmed data set, specifically:

[0012] Replace the missing scores in the original dataset with zeros, calculate the partial total scores, and set the k-th highest partial total score of the top-k aggregation query to τ;

[0013] Set the value τ1 = F -1 (τ) is broadcast to all local nodes, and a value-based filter is used to filter out the scores of all local nodes that are lower than the value τ1 to obtain the pruned data set; where F -1 is the inverse function of the monotone aggregate function F. If the monotone aggregate function is summation, then m is the number of local nodes in the distributed system.

[0014] The technical solution adopted by the embodiment of the present application also includes: for the trimmed data set, a Poisson filter is used to generate a random sample, and a response message is sent to the query initiator using the random sample, specifically:

[0015] For all local nodes L in the pruned dataset j , through the Poisson distribution Pois(s ij λ) Generate random samples r ij , and sending the generated non-zero positive random samples through the local node.

[0016] The technical solution adopted by the embodiment of the present application also includes: after the query initiator receives the response message, the first candidate list C1 is constructed according to the rule of the Poisson filter, specifically:

[0017] Calculate the response message at the local node L j The aggregate sample fraction for each item stored in r (k) ←The kth highest aggregate sample score;

[0018] is the quantile function of the gamma distribution;

[0019] is the quantile function of the Poisson distribution;

[0020] Add all items with aggregate sample scores not less than α2 to the first candidate list C1; where, when there is s (k) When a lower threshold τ is available, the value of λ is set to The value of α2 is

[0021] The technical solution adopted by the embodiment of the present application further includes: before using the cropped data set to refine the first candidate list C1 to generate the second candidate list C2, it also includes:

[0022] It is determined whether the size of the first candidate list C1 exceeds a set upper limit. If the size exceeds the set upper limit, the step of using the cropped data set to refine the first candidate list C1 is performed.

[0023] The technical solution adopted by the embodiment of the present application also includes: the said using the cropped data set to refine the first candidate list C1 to generate the second candidate list C2, specifically:

[0024] Retrieve all items in the first candidate list C1 whose scores are not less than τ1, calculate the upper limit of the total scores of the items in the first candidate list C1, remove items whose upper limit of the total scores is less than the lower limit of the kth highest aggregated sample score, and generate a second candidate list C2.

[0025] Another technical solution adopted by the embodiment of the present application is: a probabilistic filtering device for distributed data, comprising:

[0026] Data trimming module: after receiving the top-k aggregation query request sent by the query initiator, collects the highest score of each local node in the distributed system, generates an original data set, and trims the original data set using a value-based filter to obtain a trimmed data set;

[0027] A random sample generation module: used to generate a random sample for the trimmed data set using a Poisson filter, and send a response message to the query initiator using the random sample;

[0028] List generation module: used for constructing a first candidate list C1 according to the rule of the Poisson filter after the query initiator receives the response message, and using the trimmed data set to refine the first candidate list C1 to generate a second candidate list C2;

[0029] List verification module: used to verify the second candidate list C2 through the original data set and output the query result.

[0030] The technical solution adopted by the embodiment of the present application also includes: after the query initiator receives the response message, the list generation module constructs a first candidate list C1 according to the rule of the Poisson filter, specifically:

[0031] Calculate the response message at the local node L j The aggregate sample fraction for each item stored in r (k) ←The kth highest aggregate sample score;

[0032] is the quantile function of the gamma distribution;

[0033] is the quantile function of the Poisson distribution;

[0034] Add all items with aggregate sample scores not less than α2 to the first candidate list C1; where, when there is s (k) When a lower threshold τ is available, the value of λ is set to The value of α2 is

[0035] Another technical solution adopted by the embodiment of the present application is: a device, the device includes a processor and a memory coupled to the processor, wherein:

[0036] The memory stores program instructions for implementing the probabilistic filtering method for distributed data;

[0037] The processor is used to execute the program instructions stored in the memory to control a probabilistic filtering method for distributed data.

[0038] Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the probabilistic filtering method for distributed data.

[0039] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: the probability filtering method, device, equipment and storage medium of distributed data in the embodiments of the present application utilize the characteristics of Poisson random samples to process top-k aggregation queries of distributed systems at low communication costs, and utilize random samples generated from raw data to effectively filter low-quality data transmitted by local nodes, thereby reducing overall communication costs while maintaining the probability guarantee of query results, and allowing users to control the trade-off between accuracy and query efficiency, thereby increasing the flexibility of the method. In addition, value-based filters are integrated in the initial pruning stage, further reducing the computational and communication costs of the random sample generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of a probabilistic filtering method for distributed data according to an embodiment of the present application;

[0041] Figure 2 A schematic diagram of the general framework of the top-k aggregate query filtering method;

[0042] Figure 3 A schematic diagram of the structure of a probabilistic filtering device for distributed data according to an embodiment of the present application;

[0043] Figure 4 This is a schematic diagram of the device structure of an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] The terms "first", "second", "third" in this application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the embodiments of this application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the steps or units listed, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0047] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] Specifically, see Figure 1 , is a flow chart of a probabilistic filtering method for distributed data in an embodiment of the present application. The probabilistic filtering method for distributed data in an embodiment of the present application comprises the following steps:

[0049] S100: after receiving the top-k aggregate query request sent by the query initiator, collect the highest score of each local node in the distributed system, generate an original data set, and use a value-based filter to trim the original data set to obtain a trimmed data set;

[0050] In this step, if Figure 2 As shown in Figure 1, it is a general framework diagram of the top-k aggregate query filtering method. Given a distributed system L consisting of m local nodes, L = {L1, L2, …, L m}, vertically partition the itemset E = {e1, e2, …, e n}, at the local node L j Items stored in i The feature is represented by a non-negative number called the local score, denoted by s ij ,s i =G(s i1 ,s i2 ,...,s im ) is project e i The goal of the top-k aggregation query is to find a subset T of size k such that if e1∈T, Then s1≥s2.

[0051] In the embodiment of the present application, both the value-based filter and the Poisson filter are based on s (k) The value-based filter has a lower cost and processes the entire original data set to reduce the size of the data set, thereby limiting the computational cost and reducing the communication cost. Specifically, the value-based filter is used to trim the original data set to obtain a trimmed data set, which is:

[0052] Replace the missing scores in the original dataset with zeros and calculate the partial total scores of the local nodes, setting the kth highest partial total score of the top-k aggregation query to τ;

[0053] Set the value τ1 = F -1 (τ) is broadcast to all local nodes, and a value-based filter is used to filter out the scores of all local nodes that are lower than the value τ1 to obtain the pruned data set; where F -1 is the inverse function of the monotone aggregate function F. If the monotone aggregate function is summation, then m is the number of local nodes in the distributed system.

[0054] S110: for the remaining fraction in each local node in the pruned data set, a Poisson filter (PF) is used to generate a random sample, and a response message is sent to the query initiator using the random sample;

[0055] In this step, in order to limit the irrelevant information sent by local nodes, the present invention uses information generated by Poisson random distribution. If the rate is small enough, most of the random samples generated are zero. Subsequently, each local node sends the generated non-zero positive random samples instead of the actual score, thereby reducing the size of the response message. When random samples are used in the filter to minimize the size of the response message, the selected distribution for generating random samples must allow for variables of minimum size to provide important insights about the global score. If the Poisson distribution Pois(s ij λ) is each score s ij Generate sample r ij , then the sum of random samples r i =∑r ij It is from Pois(s i λ) is a random sample of Poisson distribution, where λ is the expectation and variance of Poisson distribution. Specifically, the filtering algorithm of Poisson filter is shown in Table 1 below:

[0056] Table 1 Filtering algorithm of Poisson filter

[0057]

[0058] In terms of communication, when applying the Poisson filter, the average number of samples sent in the system is no greater than ∑s ij λ. Therefore, the cost of the Poisson filter depends strongly on the value of λ. If the value of λ is low, most of the random samples r generated ij will be zero and can be safely filtered out, thereby reducing the size of the response message from the local node, that is, the lower the λ value, the smaller the size of the response message.

[0059] S120: After receiving the response message, the query initiator constructs a first candidate list C1 according to the rule of the Poisson filter, and the first candidate list C1 includes all items that may become part of the final output;

[0060] In this step, the first candidate list C1 is constructed in the following manner:

[0061] Calculate the aggregate sample fraction, r, for each item present in the response message (k) ←The kth highest aggregate sample score;

[0062] is the quantile function of the gamma distribution, and δ is the amount of change;

[0063] is the quantile function of the Poisson distribution;

[0064] All items whose aggregate sample scores are not less than α2 are added to the first candidate list C1.

[0065] Since the variance of the Poisson distribution is relatively small, random samples generated from items with smaller total scores usually do not exceed samples generated from items with higher total scores. This property helps the query initiator to reduce the size of the first candidate list C1 more effectively. Considering the accuracy of the final output, any item in the top-k is in the first candidate list C1 of the Poisson filter with a probability of at least 1-δ, ensuring that the final output of the query still maintains the probability guarantee. However, if the value of λ is too low, it will also reduce r (k) The value of , which leads to a decrease in the quantile function α2 of the Poisson distribution, increases the size of the first candidate list C1, and thus increases the verification cost. The optimal value of λ varies depending on the data set. According to experience, it is set The cost of the response message and the verification step can be effectively balanced, so when there is s (k) When a lower threshold τ is available, the value of λ is set to approximately In addition, when τ holds, the value of α2 is

[0066] S130: Determine whether the size of the first candidate list C1 exceeds a set upper limit, if yes, execute S140;

[0067] The most advantageous feature of Poisson filters in this step is that the total size of the response message does not depend on the number of local nodes, which makes them particularly beneficial in distributed systems with many data points. However, as the amount of data stored in each local node increases, their computational cost and the size of the response message will increase accordingly. In addition, the computational cost of exponential and Poisson filters is hundreds to thousands of times higher than simpler filters such as value-based filters or position-based filters, mainly due to the process of generating random samples. Therefore, it is crucial to find a way to limit the number of scores processed by these filters.

[0068] S140: Retrieve all items in the first candidate list C1 with a score not less than τ1, calculate the upper limit of the total score of the items in the first candidate list C1, remove the items whose upper limit of the total score is less than the lower limit of the kth highest aggregated sample score, and generate a second candidate list C2;

[0069] In this step, if the first candidate list C1 is too large, the first candidate list C1 is refined using the trimmed data set according to the rule of the value-based filter, thereby obtaining a more manageable second candidate list C2.

[0070] S150: After verifying the second candidate list C2 through the original data set, output the final query result;

[0071] In this step, the actual aggregate score of the item can also be approximated using the aggregate random sample of the item to ensure that the query result output in the end still maintains the probability guarantee. In most cases, the computational overhead and communication cost of value-based filters and Poisson filters are significantly reduced due to the reduction in the number of scores that need to be processed, and the bandwidth usage of response messages and verification steps is reduced, unless all scores in each local node are greater than τ1.

[0072] In order to verify the feasibility and effectiveness of the embodiments of the present application, experimental evaluations were carried out on two real data sets, HEPMAS and HIGGS, through the following embodiments. Among them, the HEPMASS data set includes 7 million data points and 27 features (excluding labels and quality), and its size is 5GB. The HIGGS data set includes 11 million data points and 28 features (excluding labels), and its size is 7.4GB. In order to increase the size of the two data sets, each of their fractions is divided and assigned to 10 local nodes according to the Zipf law, and the final generated two data sets are 50GB and 74GB in size. In these two data sets, query the top k items with the highest Euclidean distance to point 0.

[0073] In the experiment, a scenario was simulated in which the data set was distributed on virtual nodes, each virtual node was directly connected to a virtual edge server, the medium bandwidth of each virtual node was 50Mbps, and the bandwidth of the server was 10Gbps. The average delay between the server and the virtual node was 20ms. It was assumed that the computing power of each node was equivalent to one core in the platform, and the computing power of the edge server was eight times that of the core. The experimental results show that the bandwidth consumed by the embodiment of the present application is at least three times less than that of the existing method, and the running speed is at least twice that of the existing method, while still maintaining the probability guarantee of the output.

[0074] The probabilistic filtering method for distributed data in the embodiment of the present application utilizes the characteristics of Poisson random samples to process top-k aggregate queries of distributed systems at low communication costs, and utilizes random samples generated from raw data to effectively filter low-quality data transmitted by local nodes, thereby reducing overall communication costs while maintaining the probability guarantee of query results, and allowing users to control the trade-off between accuracy and query efficiency, increasing the flexibility of the method. In addition, a value-based filter is integrated in the initial pruning stage, further reducing the computational and communication costs of the random sample generation process.

[0075] See also Figure 3 , is a schematic diagram of the structure of a probabilistic filtering device for distributed data in an embodiment of the present application. The probabilistic filtering device 40 for distributed data in an embodiment of the present application includes:

[0076] Data trimming module 41: after receiving the top-k aggregation query request sent by the query initiator, collect the highest score of each local node in the distributed system, generate an original data set, and trim the original data set using a value-based filter to obtain a trimmed data set;

[0077] Random sample generation module 42: used to generate random samples for the trimmed data set using a Poisson filter, and send a response message to the query initiator using the random samples;

[0078] List generation module 43: configured to construct a first candidate list C1 according to the rule of the Poisson filter after the query initiator receives the response message, and to refine the first candidate list C1 using the trimmed data set to generate a second candidate list C2;

[0079] List verification module 44: used to verify the second candidate list C2 through the original data set and output the query result.

[0080] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0081] The device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0082] See also Figure 4 , is a schematic diagram of the device structure of an embodiment of the present application. The device 50 includes:

[0083] A memory 51 storing executable program instructions;

[0084] A processor 52 connected to the memory 51;

[0085] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: after receiving the top-k aggregation query request sent by the query initiator, collect the highest score of each local node in the distributed system, generate an original data set, and use a value-based filter to trim the original data set to obtain a trimmed data set; for the trimmed data set, use a Poisson filter to generate a random sample, and use the random sample to send a response message to the query initiator; after receiving the response message, the query initiator constructs a first candidate list C1 according to the rules of the Poisson filter, and uses the trimmed data set to refine the first candidate list C1 to generate a second candidate list C2; after verifying the second candidate list C2 through the original data set, output the query result.

[0086] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0087] See also Figure 5, is a schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores a program instruction 61 that can implement the following steps: after receiving the top-k aggregation query request sent by the query initiator, collect the highest score of each local node in the distributed system, generate an original data set, and use a value-based filter to trim the original data set to obtain a trimmed data set; for the trimmed data set, use a Poisson filter to generate a random sample, and use the random sample to send a response message to the query initiator; after the query initiator receives the response message, it constructs a first candidate list C1 according to the rules of the Poisson filter, and uses the trimmed data set to refine the first candidate list C1 to generate a second candidate list C2; after verifying the second candidate list C2 through the original data set, the query result is output. Among them, the program instruction 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the various implementation methods of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program instructions, or terminal devices such as computers, servers, mobile phones, tablets, etc. Among them, the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0088] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0089] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A probabilistic filtering method for distributed data, characterized in that: include: After receiving the top-k aggregation query request sent by the query initiator, the highest score of each local node in the distributed system is collected to generate an original data set, and the original data set is pruned by a value-based filter to obtain a pruned data set; For the trimmed data set, a Poisson filter is used to generate a random sample, and a response message is sent to the query initiator using the random sample; After receiving the response message, the query initiator constructs a first candidate list C1 according to the rule of the Poisson filter, and uses the trimmed data set to refine the first candidate list C1 to generate a second candidate list C2; After verifying the second candidate list C2 through the original data set, the query result is output.

2. The probabilistic filtering method for distributed data according to claim 1, characterized in that: The original data set is trimmed by using a value-based filter to obtain a trimmed data set, specifically: Replace the missing scores in the original dataset with zeros, calculate the partial total scores, and set the k-th highest partial total score of the top-k aggregation query to τ; Set the value τ1 = F -1 (τ) is broadcast to all local nodes, and a value-based filter is used to filter out the scores of all local nodes that are lower than the value τ1 to obtain the pruned data set; where F -1 is the inverse function of the monotone aggregate function F. If the monotone aggregate function is summation, then m is the number of local nodes in the distributed system.

3. The probabilistic filtering method for distributed data according to claim 2, characterized in that: For the trimmed data set, a Poisson filter is used to generate a random sample, and a response message is sent to the query initiator using the random sample, specifically: For all local nodes L in the pruned dataset j , through the Poisson distribution Pois(s ij λ) Generate random samples r ij , and sending the generated non-zero positive random samples through the local node.

4. The probabilistic filtering method for distributed data according to claim 3, characterized in that: After receiving the response message, the query initiator constructs a first candidate list C1 according to the rule of the Poisson filter, specifically: Calculate the response message at the local node L j The aggregate sample fraction for each item stored in r (k) ←The kth highest aggregate sample score; is the quantile function of the gamma distribution; is the quantile function of the Poisson distribution; Add all items with aggregate sample scores not less than α2 to the first candidate list C1; where, when there is s (k) When a lower threshold τ is available, the value of λ is set to The value of α2 is 5. The probabilistic filtering method for distributed data according to any one of claims 1 to 4, characterized in that: Before the step of using the cropped data set to refine the first candidate list C1 to generate the second candidate list C2, the method further includes: It is determined whether the size of the first candidate list C1 exceeds a set upper limit. If the size exceeds the set upper limit, the step of using the cropped data set to refine the first candidate list C1 is performed.

6. The probabilistic filtering method for distributed data according to claim 5, characterized in that: The using the cropped data set to refine the first candidate list C1 to generate the second candidate list C2 is specifically: Retrieve all items in the first candidate list C1 whose scores are not less than τ1, calculate the upper limit of the total scores of the items in the first candidate list C1, remove items whose upper limit of the total scores is less than the lower limit of the kth highest aggregated sample score, and generate a second candidate list C2.

7. A probabilistic filtering device for distributed data, characterized in that: include: Data trimming module: after receiving the top-k aggregation query request sent by the query initiator, collects the highest score of each local node in the distributed system, generates an original data set, and trims the original data set using a value-based filter to obtain a trimmed data set; A random sample generation module: used to generate a random sample for the trimmed data set using a Poisson filter, and send a response message to the query initiator using the random sample; List generation module: used for constructing a first candidate list C1 according to the rule of the Poisson filter after the query initiator receives the response message, and using the trimmed data set to refine the first candidate list C1 to generate a second candidate list C2; List verification module: used to verify the second candidate list C2 through the original data set and output the query result.

8. The probabilistic filtering device for distributed data according to claim 7, characterized in that: After the query initiator receives the response message, the list generation module constructs a first candidate list C1 according to the rule of the Poisson filter, specifically: Calculate the response message at the local node L j The aggregate sample fraction for each item stored in r (k) ←The kth highest aggregate sample score; is the quantile function of the gamma distribution; is the quantile function of the Poisson distribution; Add all items with aggregate sample scores not less than α2 to the first candidate list C1; where, when there is s (k) When a lower threshold τ is available, the value of λ is set to The value of α2 is 9. A device, characterized in that: The device comprises a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the probabilistic filtering method for distributed data according to any one of claims 1 to 6; The processor is used to execute the program instructions stored in the memory to control a probabilistic filtering method for distributed data.

10. A storage medium, characterized in that: The device stores program instructions executable by a processor, wherein the program instructions are used to execute the probabilistic filtering method for distributed data as described in any one of claims 1 to 6.

Images