Massive data top-k skyline group query method

Abstract

The application provides a mass data Top-kskyline group query method, including stage one, applying a verification theorem of G-skyline to scan data points in a pre-sequencing table in sequence until all data points are scanned, and a candidate point set containing all candidate points and child sets of skyline points is obtained; stage two, candidate groups are generated from the candidate point set, and judgment of enumeration rounds is completed by a left part theorem, a right part theorem and an approximate theorem; scores of the candidate groups are calculated, and a Top-k score threshold is maintained; child G-skyline groups of Top-k candidate groups are generated, and a final result is obtained by updating the Top-k candidate groups. The pre-sequencing table is used as an index structure, and only once construction can be applied to all attribute combination scenes. Early termination is realized by using pre-sequencing, and the data scale that needs to be scanned for screening candidate points is reduced. Based on three calculation theorems of the left part theorem, the right part theorem and the approximate theorem, an efficient calculation method is provided, the upper bound of G-skyline group scores is reduced, and the number of generated G-skyline groups and the calculation amount are greatly reduced.

Description

Technical Field

[0001] This application relates to the field of big data query technology, and in particular to a method for querying Top-k skyline groups of massive data. Background Technology

[0002] Skyline queries can retrieve Pareto optimal points in a dataset, which are points that are not dominated by any other points. This is important in fields such as multi-objective decision-making. For two points p1 and p2 in a dataset, p1 is said to dominate p2 if and only if every attribute of p1 is no worse than that of p2, and at least one attribute of p1 is better than that of p2.

[0003] However, skyline queries can only return a single data point, while real-world applications often require group results composed of multiple points. This has led to research and exploration of the skyline grouping problem, where skyline group queries aim to obtain the optimal results for a fixed number of combinations of points. Among these, G-Skyline has gained widespread acceptance, aiming to obtain all group results that are not dominated by other groups. The key to the G-Skyline problem lies in its definition of group dominance: for two groups G and G1 of the same size, if there exists a permutation such that every point in G is no worse than any point in G1, and at least one point in G1 dominates any point in G1, then G is said to dominate G1.

[0004] In scenarios with massive datasets, the G-Skyline sets obtained from G-Skyline queries are often extremely large. For example, on a 3D inverse correlation dataset containing 10,000 points, there are approximately 100,000 G-Skyline sets with a set size of 3, making them difficult to use directly for decision-making. Furthermore, G-Skyline queries cannot control the number of output sets, failing to meet diverse user needs. Top-k G-Skyline queries combine the advantages of skyline set queries and Top-k queries, using the number of dominant points in a G-Skyline set as the set score and returning the k G-Skyline results with the highest set scores. Top-k G-Skyline queries can be applied to G-Skyline problems while providing a finite and controllable set of results, facilitating user decision-making and meeting diverse user needs. Therefore, Top-k G-Skyline problems are of great significance in the computation and application of skyline sets.

[0005] Existing Top-k G-Skyline query algorithms are all based on the DSG index structure. However, this requires building a separate DSG for each combination of data dimensions. For example, selecting 3 attributes as measures from 30 attributes could result in C(30,3) possibilities, each requiring the construction of its corresponding DSG. This obviously incurs significant overhead, which is unacceptable to users. Furthermore, when generating Top-k G-Skyline group results, all G-Skyline groups consisting of skyline points can be used as candidate groups, or all G-Skyline groups of size l consisting of skyline points can be used as candidate groups. However, this means that the size of the candidate groups that need to be generated and evaluated is enormous. This clearly leads to substantial overhead and severely impacts performance.

[0006] In summary, existing Top-k G-Skyline query methods suffer from high computational cost and long processing time, failing to provide a fast and efficient way to obtain Top-k G-Skyline groups. This makes them unsuitable for practical applications, especially those involving massive amounts of data. Summary of the Invention

[0007] The purpose of this application is to provide a method for querying Top-k skyline groups in massive datasets, aiming to solve the problems in the prior art.

[0008] This application provides a method for querying Top-k skyline groups in massive datasets. In the first stage, the verification theorem of G-Skyline is applied to sequentially scan the data points in the pre-sorted table to determine whether the current data point is a candidate point. This process continues until all data points have been scanned or the early termination condition is met, thereby filtering to obtain a candidate point set containing all candidate points and their dominance relationships. If any data points have not been scanned after the early termination condition is triggered, the scanning of all data points continues to obtain a complete set of children of the skyline points.

[0009] Phase Two: The first step is to generate Top-k G-Skyline candidate groups based on the candidate point set, by enumerating the skyline points in the candidate point set. During the enumeration process, the determination of the enumeration round is completed by three computational theorems: the left-hand theorem, the right-hand theorem, and the approximation theorem. The candidate group score is calculated based on the number of out-of-group points dominated by points within the group, and the Top-k score threshold is maintained. The second step is to generate child G-Skyline groups of the Top-k candidate groups, update the Top-k candidate groups, and obtain the final Top-k G-Skyline result.

[0010] Furthermore, the method for constructing the presorted table is as follows: For a d-dimensional dataset P containing n points, the presorted table PT is a combination of d sorted sets of size n; the presorted table PT(MPI, PI)T ,PI1…PI d There are n rows and d+2 columns, where each row represents a point p in the dataset P, and the 3rd...d+2nd columns (PI1...PI2 ... d ) represents the sorting index (1≤i≤d) of the i-th attribute value of point p among all i-th attribute values ​​of all points, and the value of MPI in the first column is the value of the i-th attribute value in that row (PI1…PI). d The minimum value of ), i.e., MPI = min (1≤i≤d) PI i , Column 2 PI T The value represents the position of the data point in dataset P.

[0011] Furthermore, the method for determining whether the current data point is a candidate point in Phase 1 is as follows: According to the verification theorem of G-Skyline, if a G-Skyline group of size s is required, then the candidate points that make up the G-Skyline are dominated by at most s-1 points; compare the current data point with all candidate points. If the current point is dominated by more than s-1 points, then the point is definitely not a candidate point and will not appear in the candidate point set; in addition, if a candidate point is dominated by more than s-1 points in subsequent scanning processes, it will also be removed from the candidate point set.

[0012] Furthermore, the early termination condition described in Stage 1 is as follows: When scanning the pre-sorted table, maintain a max-heap MH of size s, where s is the group size of the desired Top-k G-Skyline. The heap stores the maximum values ​​of each attribute of the scanned data points. If the max-heap MH has stored s elements and the maximum value in MH does not exceed the minimum value of each attribute of the current scanned point, it proves that the current scanned point is dominated by the s data points represented by MH. Subsequent data points are definitely not candidate points, and Stage 1 can terminate early. Subsequent data points only need to be compared with the skyline points without needing to determine whether they are candidate points.

[0013] Furthermore, the specific method for enumerating the Top-k G-Skyline candidate groups described in Phase 2 is as follows: take the skyline points from the candidate point set and sort the skyline points in descending order according to the number of their dominating points; if the required Top-k G-Skyline group size is s, then use the first s points of the sorted skyline points to form the first candidate group, and enumerate the last point in the candidate group to generate new candidate groups in turn, thus obtaining a candidate group composed entirely of skyline points; then, based on the candidate groups composed of skyline points, generate their child groups, and thus obtain all candidate groups.

[0014] Furthermore, the specific method for calculating candidate group scores and maintaining the Top-k score threshold in Phase 2 is as follows: the score of a G-Skyline group is the number of out-of-group points dominated by points within the group, and the upper bound of the score is the sum of the number of points dominated by points within the group. That is, a G-Skyline group G of size s in dataset P is defined as {p1, p2, ..., p...}. s The group score is denoted as DP(G) = |C1∪C2…∪C s |-|C1∪C2…∪C s ∩G|=|∪ p∈G {p'∈PG, p<p'}|, the upper bound of the group score is denoted as UDP(G)=|C1|+|C2|+…+|C s |

[0015] Furthermore, in Phase Two, when judging the candidate group G obtained through enumeration, the first step is to determine whether the group triggers the left-hand theorem: First, the skyline points are sorted in descending order according to their number of dominating points. In the same enumeration round, the left-hand side of G is fixed. If there is a currently generated group G that satisfies UDP(G)≤temp (temp is the score threshold of the Top-k group), then the candidate groups generated in subsequent enumeration rounds will necessarily not meet the threshold condition, and the group is directly pruned and enters the next enumeration round. If the left-hand theorem is triggered, the next step is to determine whether the right-hand theorem can be triggered: First, the skyline points are sorted in descending order according to their number of dominating points, and the candidate group G-Skyline group G={G L G R When {G}, if {G} R If {G} is continuous in the sorted skyline, and UDP(G) ≤ temp (temp is the score threshold for the Top-k groups), then it is no longer necessary to enumerate {G}. R After direct pruning, we then enumerate {G}. L}

[0016] Furthermore, in stage two, when judging the candidate group G obtained by enumeration, if the left-hand side theorem cannot be triggered, it is judged whether the group can trigger the approximation theorem: a G-Skyline group of size s, G = {p1, p2, ..., p...} s} can be viewed as being derived from the left part {G L} and the right part {G R} constitutes, that is, G = {G L G R},G L ={p1,p2,…,p i},G R ={p i+1 ,p i+2 …,p s The group dominance of group G is approximately bounded by the upper bound of the left part {G}. LThe size of the union of the child sets of interior points and the right part {G} R The sum of the sizes of the child sets of the interior points is denoted as ADP(G) = |C1∪…∪C i |+|C i+1 |+|C i+2 |+…+|C s |=DP(G L )+UDP(G R First, sort the skyline points in descending order based on the number of their dominating points, and enumerate to generate candidate groups G-Skyline groups G = {G L G R When ADP(G) ≤ temp (temp is the score threshold for the Top-k groups), and {G} R If {G} is continuous in the sorted skyline, then it is no longer necessary to enumerate {G}. R After direct pruning, we then enumerate {G}. L If only ADP(G)≤temp and no continuous {G} R If}, then the current enumeration round ends.

[0017] The beneficial effects of this invention are:

[0018] (1) A Top-k G-Skyline query method based on pre-sorting and enumeration computation theorem is proposed, which can efficiently and quickly obtain Top-k G-Skyline results on massive data.

[0019] (2) By using the pre-sorted continuation table as the index structure, it can be applied to all attribute combination scenarios with only one construction, avoiding the overhead of building the index multiple times.

[0020] (3) In stage one, early termination was achieved by using pre-sorting, which reduced the amount of data to be scanned for screening candidate points and the number of comparisons of non-candidate points.

[0021] (4) In Phase 2, an efficient calculation method was proposed based on three calculation theorems: the left-hand theorem, the right-hand theorem, and the approximation theorem. This method reduced the upper bound of the G-Skyline group score and greatly reduced the number of G-Skyline groups generated and the amount of computation. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 Flowchart of the method for querying Top-k G-Skyline data with massive amounts of data.

[0024] Figure 2 A schematic diagram of constructing the presorted table PT.

[0025] Figure 3 This is a flowchart of Phase 1.

[0026] Figure 4 This is a schematic diagram of the Phase 2 process.

[0027] Figures 5(a)-5(c) The figure shows the experimental results of how data size affects query time on a synthetic dataset.

[0028] Figures 6(a)-6(c) The experimental results are shown in the figure showing the impact of data dimension on query time on the synthetic dataset.

[0029] Figures 7(a)-7(c) The figure shows the experimental results of how group size affects query time on a synthetic dataset.

[0030] Figures 8(a)-8(c) The figure shows the experimental results of how different values ​​of k affect query time on the synthetic dataset.

[0031] Figures 9(a)-9(d) Figure showing experimental results on a real dataset. Detailed Implementation

[0032] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0033] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0034] Please see Figure 1 For ease of explanation, only the parts relevant to this embodiment are shown, and are described in detail below:

[0035] A method for querying Top-k skyline groups in massive datasets includes the following steps:

[0036] S1. Preprocessing: Read the data and construct a pre-sorted table PT.

[0037] A pre-sorted table (PT) is a data structure that reassembles a dataset by sorting it according to its individual attribute values. It can be used to find candidate points that make up a G-Skyline and can be applied to various attribute combination scenarios by only building it once.

[0038] For a d-dimensional dataset P containing n points, a presorted table PT is a combination of d sorted sets of size n. The presorted table PT(MPI, PI) T ,PI1…PI d There are n rows and d+2 columns, where each row represents a point p in the dataset P, and the 3rd...d+2nd columns (PI1...PI2 ... d ) represents the sorting index (1≤i≤d) of the i-th attribute value of point p among all i-th attribute values ​​of all points, and the value of MPI in the first column is the value of the i-th attribute value in that row (PI1…PI). d The minimum value of ), i.e., MPI = min (1≤i≤d) PI i , Column 2 PI T The value represents the position of the data point in dataset P.

[0039] by Figure 2 Taking the data shown as an example: Construct a pre-sorted table PT from a dataset P(A) containing 12 3D points (1≤i≤3), where the i-th column of P(A) is... i Let be the values ​​of the i-th attribute of a point, and let be an implicit attribute PI. T Let A represent the position of each point in P. For each column (A... i Sort L separately in ascending order to get L i (PI T A i ), L i PI in T The value of L corresponds to the position of the point in dataset P, and L i PI also has implicit properties. i For each point at L i The position within. For L i According to PI T The values ​​are sorted in ascending order to obtain PL i (PI i ,PI T Finally, PL1…PL3 are merged to obtain the pre-sorted table PT(MPI,PI). T ,PI i The value of MPI in the first column is the minimum value of (PI1…PI3) in that row, i.e., MPI = min. (1≤i≤3) PI i .

[0040] S2, Stage 1: Scan the pre-sorted table and filter to obtain the candidate point set.

[0041] Phase 1 involves sequentially scanning the data points in the pre-sorted table, determining if the current data point is a candidate point, and adding it to the candidate point set if so. This process continues until all data points have been scanned or the early termination condition is met. According to the verification theorem of G-Skylines, if a G-Skyline group of size 's' is required, then the candidate points forming the G-Skyline are dominated by at most 's-1' points. The current data point is compared with all candidate points; if the current point is dominated by more than 's-1' points, then it is definitely not a candidate point and will not appear in the candidate point set. Furthermore, if a candidate point is subsequently dominated by more than 's-1' points, it will also be removed from the candidate point set. The candidate point set stores the data points that could potentially form a Top-k G-Skyline group and their dominance relationships. While scanning the pre-sorted table, a max-heap MH of size 's' is maintained, storing the maximum sorted values ​​of each attribute for the scanned data points. If the max-heap MH has stored s elements and the maximum value in MH does not exceed the minimum value of the sorted attributes of the current scan point, it proves that the current scan point is dominated by the s data points represented by MH. Subsequent data points are definitely not candidate points, and stage 1 can be terminated early. Subsequent data points only need to be compared with the skyline points.

[0042] When the group size s of the Top-k G-Skyline group is 3, Figure 2 The process of completing Phase 1 on the pre-sorted table PT is as follows: Figure 3 As shown: PT Point is the currently scanned point, and the candidate point set (CPCandidatePoint CP) and the current max heap MH (Current max heap MH) are both initialized to empty. In iterations No.1-3, PT(1), PT(2), and PT(3) are directly added to CP and MH, at which point MH.max is 12; in iteration No.4, because PT(4).PI i The maximum value of (1≤i≤3) is 11, which is less than MH.max=12. Therefore, after removing the root node PT(3) of MH, PT(4) is added to MH, and MH.max=11 is updated. In addition, PT(1)<PT(4), so PT(4) is added to the child set PT(1):C() of PT(1), and PT(1) is added to the parent set PT(4):P() of PT(4). In the No.12 iteration, MPI≥MH.max, which satisfies the early termination condition and PT(12) is the last data point. The iteration terminates and stage 1 ends.

[0043] S3, Phase Two: Enumerate and generate candidate groups, and calculate the Top-k G-Skyline group.

[0044] The goal of Phase 2 is to obtain the Top-k G-Skyline group based on candidate points. The Top-k G-Skyline group must be a G-Skyline group consisting entirely of skyline points or its child groups. Therefore, obtaining the Top-k G-Skyline group based on candidate points mainly involves two steps: 1. Generating all groups of size 's' consisting of skyline points and maintaining the Top-k candidate groups; 2. Generating the child G-Skyline groups of the Top-k candidate groups and updating the Top-k candidate groups to obtain the final result. This invention proposes an efficient computational method incorporating three computational theorems: the left-hand theorem, the right-hand theorem, and the approximation theorem, to quickly obtain the Top-k G-Skyline group result with as few candidate groups as possible.

[0045] The score of a G-Skyline group is the number of out-of-group points dominated by points within the group. The upper bound of the score is the sum of the number of points dominated by points within the group. That is, a G-Skyline group G of size s in dataset P is defined as G = {p1, p2, ..., p...}. s The group score is denoted as DP(G) = |C1∪C2…∪C s |-|C1∪C2…∪C s ∩G|=|∪ p∈G {p'∈PG, p<p'}|, the upper bound of the group score is denoted as UDP(G)=|C1|+|C2|+…+|C s From the candidate point set, select skyline points and sort them in descending order according to the number of their dominating points. If the required Top-kG-Skyline group size is s, then the first s points of the sorted skyline points form the first candidate group. New candidate groups are then generated sequentially from the last point in each candidate group, thus obtaining all candidate groups composed of skyline points. Then, based on the candidate groups composed of skyline points, their child groups are generated, resulting in all candidate groups.

[0046] When enumerating a G-Skyline group of size s, the process of enumerating the s-th point after determining the first s-1 points is called an enumeration round. When judging the candidate group G obtained through enumeration, first determine if the group triggers the left-hand theorem: In the same enumeration round, the left-hand side of G is fixed. If there is a currently generated group G that satisfies UDP(G) ≤ temp (temp is the score threshold of the Top-k group), then the candidate groups generated in subsequent rounds in this enumeration round will necessarily not meet the threshold condition, and are directly pruned into the next enumeration round. If the left-hand theorem is triggered, further determine if the right-hand theorem can be triggered: Enumerating the candidate group G-Skyline group G = {G L G R When {G}, if {G} RIf {G} is continuous in the sorted skyline, and UDP(G) ≤ temp (temp is the score threshold for the Top-k groups), then it is no longer necessary to enumerate {G}. R After direct pruning, we then enumerate {G}. L If the left-hand side theorem cannot be triggered, then determine whether the group can trigger the approximation theorem: A G-Skyline group of size s, G = {p1, p2, ..., p...} s} can be viewed as being derived from the left part {G L} and the right part {G R} constitutes, that is, G = {G L G R},G L ={p1,p2,…,p i},G R ={p i+1 ,p i+2 …,p s The group dominance of group G is approximately bounded by the upper bound of the left part {G}. L The size of the union of the child sets of interior points and the right part {G} R The sum of the sizes of the child sets of the interior points is denoted as ADP(G) = |C1∪…∪C i |+|C i+1 |+|C i+2 |+…+|C s |=DP(G L )+UDP(G R ). Enumerate to generate candidate groups G-Skyline groups G = {G L G R When ADP(G) ≤ temp (temp is the score threshold for the Top-k groups), and {G} R If {G} is continuous in the sorted skyline, then it is no longer necessary to enumerate {G}. R After direct pruning, we then enumerate {G}. L If only ADP(G)≤temp and no continuous {G} R If}, then the current enumeration round ends.

[0047] exist Figure 3 The process of continuing to run Phase 2 on the candidate point set obtained is as follows: Figure 4 As shown: When k=2 in Top-k, the skyline points are Skyline={PT(1),PT(2),PT(3),PT(5),PT(6)}. Sort them in descending order according to their dominating points and renumber them to get Skyline={p1,p2,p3,p4,p5}. Their child sets are:

[0048] C1 = {PT(4), PT(0), PT(11), PT(12)}, C2 = {PT(7), PT(9)}, C3 = {PT(11), PT(12)}, C4 = {PT(8), PT(10)}, C5 = {}. In the first enumeration round, the first two points in the group are determined to be p1 and p2. Then G1 = {p1, p2, p3} and G2 = {p1, p2, p4} are directly added to TGS (Top-k candidate group). Since DP(G1) = 6 and DP(G2) = 7, the threshold temp = 6 at this time. Continuing with the first round of enumeration, G3 = {p1, p2, p5} has UDP(G3) = 6 ≤ temp and no consecutive right-hand side, triggering the left-hand side theorem, and proceeding to the next enumeration round; in the second enumeration round, the first two points in the group are determined to be p1 and p3. For G4 = {p1, p3, p4}, UDP(G4) = 7 > temp, ADP(G4) = 6 ≤ temp, and because G4 has a consecutive right-hand side G... R If G5 = {p3, p4}, then ADP(G4) = 8 > temp is recalculated, triggering the approximation theorem and proceeding to the next enumeration round. In the third enumeration round, the first two points in the group are determined to be p1 and p4. For G5 = {p1, p4, p5}, UDP(G5) = 6 ≤ temp and G5 has a continuous right-hand side G. R ={p4,p5}, triggering the right-hand side theorem, proceeding to the next enumeration round; in the 4th enumeration round, determine the first two points in the group as p2,p3, for G6 = {p2,p3,p4}, UDP(G6) = 6 ≤ temp and G6 has a continuous right-hand side G. R ={p2,p3,p4}=G6, triggering the right-hand side theorem, ending the enumeration process, and returning TGS={G1,G2}. Since DP(G1) and DP(G2) differ by only 1, the DP of their child groups must not exceed the threshold temp, thus obtaining the final result TGS={G1,G2}.

[0049] Method performance evaluation:

[0050] To verify the performance advantages of the proposed method, multiple sets of experiments were conducted on synthetic and real datasets.

[0051] Experimental equipment: DELL Vostro 3681 (Intel(R) Core(TM) i7-10700 CPU@2.90GHz (8 cores) + 64GB RAM + 64-bit Windows 11 + JDK 1.8 + IntelliJ IDEA 2022.1.4)

[0052] Datasets: The dataset information used in the experiment is shown in Table 1. The synthetic dataset contains three data distributions: independent distribution (INDE), correlation distribution (CORR), and inverse correlation distribution (ANTI). The real dataset is the HIGGS dataset.

[0053] The experiment mainly evaluated performance from the following aspects: data size n, data dimension |SC|, G-Skyline group size s, and the k value in Top-k. Specific settings for these parameters are shown in Table 2.

[0054] Table 1 Dataset Information Table

[0055]

[0056] Table 2 Experimental Parameter Settings

[0057]

[0058] Three methods were tested in the experiment: topkG p The algorithm and PR algorithm are existing Top-kG-Skyline query algorithms, serving as a control group; and TGPE (the complete version of the method of this invention, which applies the left-hand theorem, right-hand theorem and approximation theorem in stage 2).

[0059] Based on the aforementioned dataset and parameters, this invention sets up four sets of experiments on both synthetic and real datasets, respectively, to investigate the impact of data size n, dimension |SC|, group size s, and the value of k on query time. The experimental results on the synthetic dataset are as follows: Figures 5(a)-5(c) , Figures 6(a)-6(c) , Figures 7(a)-7(c) , Figures 8(a)-8(c) As shown in the figures, Figure (a) corresponds to a correlated distribution dataset, Figure (b) corresponds to an independently distributed dataset, and Figure (c) corresponds to an inversely correlated distribution dataset; the experimental results on the real dataset are as follows. Figures 9(a)-9(d) As shown.

[0060] The algorithm performance was evaluated on synthetic datasets with different data sizes n. The experimental results are as follows: Figures 5(a)-5(c) As shown. Overall, topkG p The algorithm and the PR algorithm take similar time. The three TGPE methods take similar time and all three TGPE methods consistently have an advantage of about two orders of magnitude over the existing algorithms.

[0061] The performance of the method was evaluated on synthetic datasets under different data dimensions |SC|. The experimental results are as follows: Figures 6(a)-6(c) As shown, with the increase in data dimensionality, the query time of each algorithm on the three data distribution datasets gradually increases. The increasing trend of the three TGPE methods is greater than that of topkG. pThe algorithms TGPE1 and PR are relatively slow, with TGPE1 showing a more pronounced increasing time trend compared to TGPE2 and TGPE. When the data dimension reaches 6, the top kG resolution is [not specified] on the three data distribution datasets. p Both the algorithm and the PR algorithm took more than 12 hours, while TGPE was 1-3 orders of magnitude faster.

[0062] The performance of the method was evaluated on a synthetic dataset with different group sizes s. The experimental results are as follows: Figures 7(a)-7(c) As shown. topkG p The time taken by the PR algorithm increases exponentially with the group size, while the TGPE method's time increases much more slowly. The TGPE method shows a significant advantage when the group size exceeds 4.

[0063] The performance of the method was evaluated on a synthetic dataset with different Top-k values ​​k. The experimental results are as follows: Figures 8(a)-8(c) As shown in the experimental results, the time performance of each method was not significantly related to the change in the value of k, but the time difference was significant on different data distribution datasets. On all data distribution datasets, the TGPE method significantly outperformed topkG. p Algorithms and PR algorithms.

[0064] Experimental results on real datasets are shown in Figure 9. The performance evaluation of the methods under different data sizes n is shown in Figure 9(a). The time taken by each method increases with the increase of the data size, and the growth trend is basically consistent. Among them, topkG... p The TGPE algorithm and the PR algorithm have similar time takeoffs, but the TGPE method consistently outperforms the topkG algorithm. p Algorithms and PR algorithms were used to evaluate the performance of the methods under different data dimensions |SC|. The experimental results are shown in Figure 9(b). As the dimension increases, the time taken by each method gradually increases, with topkG... p The time taken by the algorithm and the PR algorithm increased significantly, while the TGPE method showed relatively slow growth. When the dimension reaches 6, the topkG time... p The algorithm and the PR algorithm both took more than 12 hours, while the TGPE method took approximately 700 seconds. The performance of the methods was evaluated under different group sizes s, and the experimental results are shown in Figure 9(c). The time increase of the TGPE method compared to topkG... p The TGPE algorithm is much slower than the PR algorithm. When the group size is greater than or equal to 5, the TGPE method is slower than the topkG algorithm. p The algorithm is more than 10 times faster than the PR algorithm. The performance of the method was evaluated under different Top-k values ​​k, and the experimental results are shown in Figure 9(d). The time performance of each method is not significantly related to the change of k value. On all data distribution datasets, the TGPE method significantly outperforms topkG. p Algorithms and PR algorithms.

[0065] In summary, the TGPE method demonstrates a significant performance advantage over existing algorithms. While the query time of TGPE increases with increasing experimental parameters such as data size, data dimensionality, and group size, the growth trend of TGPE query time is much slower than that of existing algorithms. TGPE consistently maintains a performance advantage over existing algorithms, especially when the data dimensionality and group size are large, often resulting in a performance advantage of 1-2 times the data size.

[0066] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for querying Top-k skyline groups in massive datasets, characterized in that, Includes the following steps: Phase 1: Apply the verification theorem of G-Skyline to sequentially scan the data points in the pre-sorted table, determining whether the current data point is a candidate point, until all data points have been scanned or the early termination condition is met. This process filters out the candidate point set containing all candidate points and their dominance relationships. If any data points are not scanned after the early termination condition is triggered, the scanning continues until all data points are obtained to obtain the complete child set of the skyline points. The early termination condition in Phase 1 is as follows: When scanning the pre-sorted table, maintain a max-heap MH of size s, where s is the group size of the desired Top-k G-Skyline. The heap stores the maximum values ​​of each attribute of the scanned data points. If the max-heap MH has stored s elements, and the maximum value in MH does not exceed the minimum value of each attribute of the currently scanned point, it proves that the current scanned point is dominated by the s data points represented by MH. Subsequent data points are definitely not candidate points, and Phase 1 can terminate early. Subsequent data points only need to be compared with the skyline points without needing to determine whether they are candidate points. Phase Two: The first step is to generate Top-k G-Skyline candidate groups based on the candidate point set, by enumerating the skyline points in the candidate point set. During the enumeration process, the determination of the enumeration round is completed by three computational theorems: the left-hand theorem, the right-hand theorem, and the approximation theorem. The candidate group score is calculated based on the number of out-of-group points dominated by points within the group, and the Top-k score threshold is maintained. The second step is to generate child G-Skyline groups of the Top-k candidate groups, update the Top-k candidate groups, and obtain the final Top-k G-Skyline result.

2. The method for querying Top-k skyline groups in massive data according to claim 1, characterized in that, The method for constructing a presorted table: For a d-dimensional dataset P containing n points, the presorted table PT is a combination of d sorted sets of size n; the presorted table PT(MPI, PI) T ,PI1…PI d There are n rows and d+2 columns, where each row represents a point p in the dataset P, and the 3rd...d+2nd columns (PI1...PI2 ... d ) represents the sorting index (1≤i≤d) of the i-th attribute value of point p among all i-th attribute values ​​of all points, and the value of MPI in the first column is the value of the i-th attribute value in the row (PI1…PI). d The minimum value of ), i.e., MPI = min (1≤i≤d) PI i , Column 2 PI T The value represents the position of the data point in dataset P.

3. The method for querying Top-k skyline groups in massive data according to claim 1, characterized in that, The method for determining whether the current data point is a candidate point in Phase 1 is as follows: According to the verification theorem of G-Skyline, if a G-Skyline group of size s is required, then the candidate points that make up the G-Skyline are dominated by at most s-1 points; compare the current data point with all candidate points. If the current point is dominated by more than s-1 points, then the point is definitely not a candidate point and will not appear in the candidate point set; in addition, if a candidate point is dominated by more than s-1 points in subsequent scanning processes, it will also be removed from the candidate point set.

4. The method for querying Top-k skyline groups in massive data according to claim 1, characterized in that, The specific method for enumerating the Top-k G-Skyline candidate groups described in Phase 2 is as follows: take the skyline points from the candidate point set and sort the skyline points in descending order according to the number of their dominating points; if the required size of the Top-k G-Skyline group is s, then use the first s points of the sorted skyline points to form the first candidate group, and enumerate the last point in the candidate group to generate new candidate groups in turn, thus obtaining a candidate group composed entirely of skyline points; then, based on the candidate groups composed of skyline points, generate their child groups, and thus obtain all candidate groups.

5. The method for querying Top-k skyline groups in massive data according to claim 4, characterized in that, Phase 2 calculates candidate group scores and maintains Top-k score thresholds. The specific method is as follows: The score of a G-Skyline group is the number of out-of-group points dominated by points within the group. The upper bound of the score is the sum of the number of points dominated by points within the group. That is, a G-Skyline group G of size s in dataset P is defined as G = {p1, p2, ..., p...}. s The group score is denoted as DP(G) = |C1∪C2…∪C s |-|C1∪C2…∪C s ∩G|=|⋃ p∈G {p'∈PG, p≺p'}|, the upper bound of the group score is denoted as UDP(G)=|C1|+|C2|+⋯+|C s | 6. The method for querying Top-k skyline groups in massive data according to claim 5, characterized in that, In Phase Two, when judging the candidate group G obtained through enumeration, the first step is to determine whether the group triggers the left-hand theorem: First, sort the skyline points in descending order based on their number of dominators. In the same enumeration round, the left-hand side of G is fixed. If there is a currently generated group G that satisfies UDP(G) ≤ temp, where temp is the score threshold for the Top-k group, then the candidate groups generated in subsequent rounds in this enumeration round will also not meet the threshold condition, and are directly pruned into the next enumeration round. If the left-hand theorem is triggered, the next step is to determine whether the right-hand theorem can be triggered: First, sort the skyline points in descending order based on their number of dominators, and enumerate the generated candidate group G - Skyline group G = {G L G R When {G}, if {G} R If the {G} are continuous in the sorted skyline, and UDP(G) ≤ temp, where temp is the score threshold for the Top-k groups, then it is no longer necessary to enumerate {G}. R After direct pruning, we then enumerate {G}. L } 7. The method for querying Top-k skyline groups in massive data according to claim 6, characterized in that, In Phase Two, when evaluating the candidate group G obtained from the enumeration, if the left-hand side theorem cannot be triggered, then it is determined whether the group can trigger the approximation theorem: A G-Skyline group of size s, G={p1,p2,…,p…} s } can be viewed as being derived from the left part {G L } and the right part {G R } constitutes, that is, G={G L G R },G L ={p1,p2,…,p i },G R ={p i+1 ,p i+2 …,p s The group dominance of group G is approximately bounded by the upper bound of the left part {G}. L The size of the union of the child sets of interior points and the size of the right part {G} R The sum of the sizes of the child sets of the interior points is denoted as ADP(G) = |C1∪…∪C i |+|C i+1 |+|C i+2 |+⋯+|C s |=DP(G L )+UDP(G R First, sort the skyline points in descending order based on the number of their dominant points, and enumerate to generate candidate groups G-Skyline groups G={G L G R When ADP(G) ≤ temp, temp is the score threshold for the Top-k groups, and {G} R If {G} is continuous in the sorted skyline, then it is no longer necessary to enumerate {G}. R After direct pruning, we then enumerate {G}. L If only ADP(G)≤temp and no continuous {G} R If}, then the current enumeration round ends.

Images