Skyline query method, device and equipment based on MapReduce model and computer program product

Abstract

The invention discloses a Skyline query method, device and equipment based on a MapReduce model and a computer program product. The method comprises the steps of obtaining query preference data of a user; performing data block dimension filtering on the data tuple stored in the HDFS data set by utilizing a Skyline query algorithm according to the query preference data to obtain a data block dimension filtering result; according to the query preference data and the filtering result of the data block dimension, performing inter-grid and in-grid filtering on the filtering result of the data block dimension by using a Skyline query algorithm to obtain a filtering result of a grid dimension; and performing global processing on the filtering result of the grid dimension by utilizing a Skyline query algorithm to obtain a Skyline query result of the user. According to the method for conducting Skyline query through the MapReduce model under the big data condition, the query result can better meet the user requirement according to the user preference data, and the achievement of a traditional Skyline query algorithm is fully utilized.

Description

Technical Field

[0001] The present application relates to the technical field of data query, and particularly to a Skyline query method, device, equipment and computer program product based on the MapReduce model. Background Art

[0002] As a common algorithm for multi-criteria decision-making, Skyline query is widely used in fields such as personalized recommendation, market detection, and auxiliary decision-making.

[0003] Traditional Skyline query directly obtains the global Skyline result according to the given data set, and cannot utilize the user's preference data. The query result has nothing to do with the user's preference. In addition, in order to obtain the global optimal solution, Skyline query mostly uses a single-machine method for calculation at present, and the query scheme is not suitable for application scenarios with a large amount of data. Summary of the Invention

[0004] Embodiments of the present application provide a Skyline query method, device, equipment and computer program product based on the MapReduce model, providing the possibility of performing Skyline query under a large amount of data, improving the efficiency of Skyline query, and meeting the query requirements of large-data scenarios.

[0005] Embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, embodiments of the present application provide a Skyline query method based on the MapReduce model, and the Skyline query method based on the MapReduce model includes:

[0007] Obtain the query preference data of the user;

[0008] According to the query preference data, use the Skyline query algorithm to filter the data tuples stored in the HDFS data set in terms of data block dimension to obtain the filtering result in terms of data block dimension;

[0009] According to the query preference data and the filtering result in terms of data block dimension, use the Skyline query algorithm to filter the filtering result in terms of data block dimension in terms of grid dimension to obtain the filtering result in terms of grid dimension, and the filtering in terms of grid dimension includes inter-grid filtering and intra-grid filtering;

[0010] Use the Skyline query algorithm to globally process the filtering result in terms of grid dimension to obtain the Skyline query result of the user.

[0011] Optionally, according to the query preference data, use the Skyline query algorithm to filter the data tuples stored in the HDFS data set in terms of data block dimension, and the obtained filtering result in terms of data block dimension includes:

[0012] Perform normalization processing on the data tuples in the HDFS data set to obtain normalized data tuples;

[0013] According to the query preference data and the normalized data tuples, perform numerical conversion processing on the normalized data tuples to obtain numerically converted data tuples;

[0014] According to the query preference data, use the Skyline query algorithm to filter the numerically converted data tuples in terms of data block dimension to obtain the filtering result in terms of data block dimension.

[0015] Optionally, according to the query preference data and the filtering result in terms of data block dimension, use the Skyline query algorithm to filter the filtering result in terms of data block dimension in terms of grid dimension, and the obtained filtering result in terms of grid dimension includes:

[0016] Perform grid processing on the filtering result in terms of data block dimension to obtain a grid processing result, where the grid processing result includes mapping information between grids and data tuples and information on whether a Skyline tuple is included in the grid;

[0017] According to the query preference data and the grid processing result, use the Skyline query algorithm to filter the filtering result in terms of data block dimension in terms of grid dimension to obtain the filtering result in terms of grid dimension.

[0018] Optionally, the performing grid processing on the filtering result in terms of data block dimension to obtain a grid processing result includes:

[0019] According to a predefined grid interval, perform grid division on the filtering result in terms of data block dimension to determine the grids to which the unfiltered data tuples in each data block belong, and obtain mapping information between grids and data tuples;

[0020] Count whether each grid contains at least one data tuple to obtain information on whether a Skyline tuple is included in the grid.

[0021] Optionally, according to the query preference data and the grid processing result, use the Skyline query algorithm to filter the filtering result in terms of data block dimension in terms of grid dimension, and the obtained filtering result in terms of grid dimension includes:

[0022] Determine the dominance relationship between grids according to the query preference data and the grid processing result;

[0023] Based on the dominance relationship between the grids and the information on whether the grids contain Skyline tuples, perform inter-grid filtering on the filtering result of the data block dimension to obtain the inter-grid filtering result;

[0024] Determine the dominance relationship between the data tuples in each filtered grid according to the query preference data and the inter-grid filtering result;

[0025] According to the dominance relationship between the data tuples in each filtered grid, use the Skyline query algorithm to perform intra-grid filtering on each filtered grid to obtain the intra-grid filtering result.

[0026] Optionally, the performing filtering on the data tuples stored in the HDFS data set in the data block dimension by using the Skyline query algorithm according to the query preference data to obtain the filtering result of the data block dimension includes:

[0027] According to the data block division situation, start the corresponding Map task to read the corresponding data block tuples;

[0028] According to the query preference data, use the Skyline query algorithm to perform parallel filtering on the data block tuples corresponding to all Map tasks in the data block dimension to obtain the filtering result of the data block dimension.

[0029] Optionally, the performing filtering on the filtering result of the data block dimension in the grid dimension by using the Skyline query algorithm according to the query preference data and the filtering result of the data block dimension to obtain the filtering result of the grid dimension includes:

[0030] In the Map stage, perform grid processing on the filtering result of the data block dimension to obtain the grid processing result;

[0031] In the Shuffle stage, transfer the filtering result of the data block dimension to the Reduce stage according to the grid processing result;

[0032] In the Reduce stage, use the Skyline query algorithm to perform filtering on the filtering result of the data block dimension in the grid dimension to obtain the filtering result of the grid dimension.

[0033] In a second aspect, an embodiment of the present application further provides a Skyline query device based on the MapReduce model. The Skyline query device based on the MapReduce model includes:

[0034] An acquisition unit, configured to acquire query preference data of a user;

[0035] A first filtering unit, configured to filter data tuples stored in an HDFS data set in terms of data block dimension by using a Skyline query algorithm according to the query preference data, so as to obtain a filtering result in terms of data block dimension;

[0036] A second filtering unit, configured to filter the filtering result in terms of data block dimension in terms of grid dimension by using the Skyline query algorithm according to the query preference data and the filtering result in terms of data block dimension, so as to obtain a filtering result in terms of grid dimension, where the filtering in terms of grid dimension includes inter-grid filtering and intra-grid filtering;

[0037] A global processing unit, configured to perform global processing on the filtering result in terms of grid dimension by using the Skyline query algorithm, so as to obtain a Skyline query result of a user.

[0038] In a third aspect, an embodiment of the present application further provides a device, including:

[0039] A processor; and a memory arranged to store computer-executable instructions, where the executable instructions, when executed, cause the processor to execute any one of the foregoing Skyline query methods based on a MapReduce model.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer program product, including a computer program / instructions, where the computer program / instructions, when executed by a processor, implement any one of the foregoing Skyline query methods based on a MapReduce model.

[0041] The above at least one technical solution adopted in the embodiments of the present application can achieve the following beneficial effects: The Skyline query method based on the MapReduce model in the embodiments of the present application first obtains the query preference data of the user; then, according to the query preference data, uses the Skyline query algorithm to filter the data tuples stored in the HDFS data set in terms of data block dimensions to obtain the filtering result of the data block dimensions; then, according to the query preference data and the filtering result of the data block dimensions, uses the Skyline query algorithm to filter the filtering result of the data block dimensions in terms of grid dimensions to obtain the filtering result of the grid dimensions, and the filtering of the grid dimensions includes the filtering between grids and the filtering within grids; finally, uses the Skyline query algorithm to globally process the filtering result of the grid dimensions to obtain the Skyline query result of the user. The embodiments of the present application provide a method for Skyline query using the MapReduce model under the condition of big data, which can not only make the query result more in line with the user's needs according to the user preference data, but also make full use of the achievements of the traditional Skyline query algorithm. In addition, by performing multiple filters on the full data set, the amount of data for global Skyline calculation is greatly reduced, thereby improving the Skyline query efficiency and meeting the query requirements of the big data volume scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0043] Figure 1 is a Skyline schematic diagram in the embodiments of the present application;

[0044] Figure 2 is a flowchart of a Skyline query method based on the MapReduce model in the embodiments of the present application;

[0045] Figure 3 is a flowchart of a Skyline query process based on the MapReduce model in the embodiments of the present application;

[0046] Figure 4 is a schematic diagram of numerical conversion regarding query points in the embodiments of the present application;

[0047] Figure 5 is a schematic diagram of filtering between grids in the embodiments of the present application;

[0048] Figure 6 is a structural schematic diagram of a Skyline query device based on the MapReduce model in the embodiments of the present application;

[0049] Figure 7 This is a schematic structural diagram of a device in an embodiment of the present application. Detailed implementation manners

[0050] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part rather than all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the scope of protection of the present application.

[0051] The following details the technical solutions provided by each embodiment of the present application with reference to the drawings.

[0052] The main technical terms involved in the present application include:

[0053] 1) Skyline query

[0054] Skyline query provides a multi-criteria decision-making algorithm for multi-dimensional data. Given a set P of data tuples, the Skyline of set P refers to the set of all tuples in P that are not dominated by other tuples. A tuple x dominates another tuple y if x is not worse than y in all dimensions and x is better than y in at least one dimension. For example, when a tourist goes to a scenic spot and needs to choose a hotel, hoping to get a hotel with a short distance and a low price, but hotels with a short distance often have a high price. Therefore, the Skyline query algorithm can be used to calculate all the hotels in the Skyline, and then make a further choice.

[0055] 2) Dominance relationship

[0056] A tuple x dominates another tuple y means that x is not worse than y in all dimensions and x is better than y in at least one dimension. The criteria for good and bad can be "greater than", "less than", "not greater than", "not less than", etc.

[0057] As Figure 1 shown, a Skyline schematic diagram in an embodiment of the present application is provided. Assume that the tuples in the data set are x1 - x 13 , representing 13 hotels. Since all other hotels are dominated by x1, x6, x9, x 11 , the Skyline only needs to be selected from x1, x6, x9, x 11 .

[0058] 3) MapReduce

[0059] MapReduce is a programming framework for distributed computing programs. Its core function is to integrate the business logic code written by users and the default components it comes with into a complete distributed computing program, which runs concurrently on a Hadoop cluster. This greatly facilitates programmers to run their programs on a distributed system without the need for distributed parallel programming.

[0060] 4) Grid

[0061] The grid in this application refers to the subspace formed by logically dividing the multi-dimensional space.

[0062] The embodiments of this application provide a Skyline query method based on the MapReduce model. As Figure 2 shown, it provides a flowchart of a Skyline query method based on the MapReduce model in the embodiments of this application. The Skyline query method based on the MapReduce model at least includes the following steps S210 to step S240:

[0063] Step S210, obtain the query preference data of the user.

[0064] Combined with Figure 3 , it provides a flowchart of a Skyline query process based on the MapReduce model in the embodiments of this application. When performing a Skyline query based on the MapReduce model, it is necessary to first obtain the query preference data of the user. The query preference data may include, but is not limited to, the preference values of the user for certain attributes in the data set, the data range that the user hopes to query, and any other query-related constraint conditions that the user may set. Obtaining these preference data of the user is the basis for accurately and efficiently returning the Skyline query results that meet the user's needs in the subsequent steps.

[0065] For example, in the personalized recommendation scenario of financial products, the query preference data of the user may include the user's risk tolerance, expected return, etc.

[0066] Step S220, according to the query preference data, use the Skyline query algorithm to filter the data tuples stored in the HDFS data set in terms of data block dimensions, and obtain the filtering result in terms of data block dimensions.

[0067] After obtaining the query preference data of the user, it is necessary to use the traditional Skyline query algorithm to perform a preliminary filter on the data in the data set in terms of data block dimensions. A data block is the data storage unit in HDFS, and the purpose is for data distributed storage and calculation.

[0068] The partitioning of data blocks is a process in the Hadoop Distributed File System (HDFS) that splits large files into smaller blocks for parallel processing and computing on multiple nodes. The default block size in HDFS is 128MB or 256MB, and this value can be adjusted according to the file size and cluster performance. The choice of block size needs to be balanced according to the actual situation. Overly large blocks may lead to uneven processing, while overly small blocks may increase management overhead. Of course, regarding how to set it specifically, those skilled in the art can flexibly set it according to actual needs, and no specific limitation is provided here.

[0069] Tuple filtering in the data block dimension filters out non-Skyline tuples within each data block using the traditional Skyline query algorithm. The principle is that if a tuple is not a Skyline within its own data block, then it must not be a global Skyline and does not affect whether other tuples are global Skyline. The purpose of tuple filtering in the data block dimension is to quickly exclude data tuples that obviously do not meet the user's preferences or query conditions. By filtering tuples within the data block, the amount of data for subsequent calculation and transmission can be greatly reduced, thereby improving query efficiency.

[0070] For example, if the user prefers products with low risk and stable returns, then through tuple filtering in the data block dimension, data tuples with high risk and high volatility within each data block can be excluded.

[0071] Step S230, according to the query preference data and the filtering result in the data block dimension, use the Skyline query algorithm to perform filtering in the grid dimension on the filtering result in the data block dimension to obtain the filtering result in the grid dimension. The filtering in the grid dimension includes inter-grid filtering and intra-grid filtering.

[0072] After the rough filtering in the data block dimension, it is necessary to further filter the remaining data tuples in the grid dimension. A grid can be regarded as a pre-planned multi-dimensional space, and the remaining data tuples after filtering in the previous steps can be mapped into the grid, thereby performing tuple filtering in the grid dimension.

[0073] Tuple filtering in the grid dimension includes inter-grid filtering and intra-grid filtering. Inter-grid filtering needs to consider the dominance relationship between grids to filter out all dominated grids. Intra-grid filtering is to further perform tuple filtering within each grid based on the result of inter-grid filtering. The principle of tuple filtering within a grid is similar to that within a data block. By inter-grid filtering and intra-grid filtering, tuples that cannot be Skyline are further filtered out, further reducing the amount of data for subsequent global calculation.

[0074] Step S240: Use the Skyline query algorithm to globally process the filtering result of the grid dimension to obtain the Skyline query result of the user.

[0075] After the filtering result of the foregoing steps, continue to use the traditional Skyline query algorithm to perform global Skyline calculation on all the remaining data tuples, so as to obtain the Skyline query result that matches the user preference.

[0076] Consider a special case. When there is only one grid in the grid division, the results of all Map tasks are input into a single Reduce task for calculation. Therefore, the output result of the Reduce task is the final result, and there is no need to perform additional global Skyline calculation.

[0077] It should be noted that the traditional Skyline query algorithm adopted in the embodiments of the present application can be a basic nested loop algorithm, or a more advanced BBS algorithm, R-tree-based algorithm, etc. Specifically which traditional algorithm to adopt can be flexibly selected by those skilled in the art according to actual needs, and no specific limitation is made here.

[0078] The embodiments of the present application provide a method for performing Skyline query using the MapReduce model under big data conditions, which can not only make the query result more in line with the user's needs according to the user preference data, but also make full use of the achievements of the traditional Skyline query algorithm. In addition, by performing multiple filtering on the full amount of data set, the data volume for global Skyline calculation is greatly reduced, thereby improving the Skyline query efficiency and meeting the query requirements in the big data volume scenario.

[0079] In some embodiments of the present application, the filtering of the data tuples in the HDFS data set in terms of data block dimension according to the query preference data to obtain the filtering result in terms of data block dimension includes: performing standardization processing on the data tuples in the HDFS data set to obtain standardized data tuples; performing numerical conversion processing on the standardized data tuples according to the query preference data and the standardized data tuples to obtain numerically converted data tuples; using the Skyline query algorithm to filter the numerically converted data tuples according to the query preference data to obtain the filtering result in terms of data block dimension.

[0080] When performing tuple filtering in terms of data block dimension, dynamic space transformation can be first performed on the original data tuples in each data block stored in the HDFS data set. The dynamic space transformation mainly includes the unification of the domination criteria in each dimension of the data tuples and the numerical conversion with respect to the query point.

[0081] The unification of the domination criteria for each dimension of a tuple means that the values of certain dimensions of the given data are negated to make the domination criteria the same for each dimension. Suppose a d-dimensional tuple is given, and the domination criterion for dimensions 1 - i is "less than", while the domination criterion for dimensions i - d is "greater than". When "less than" is selected as the domination criterion, the values of dimensions i - d need to be negated so that the domination criterion for dimensions i - d becomes "less than".

[0082] In a Skyline query, the query point usually refers to a specific data point or location that the user is interested in and for which a Skyline query needs to be performed. Therefore, the query point can be determined based on the user's preferred data. The numerical transformation of the query point is the key to converting the Skyline query regarding the query point into a traditional Skyline query. Given a d-dimensional tuple p = {p1, p2, …, p d} and a d-dimensional query point q = {q1, q2, …, q d}, the result after the numerical transformation regarding the query point is: p' = {|p1 - q1|, |p2 - q2|, …, |p d - q d |}. Through the numerical transformation regarding the query point, the Skyline query regarding the query point can be transformed into a traditional Skyline query. Therefore, when performing data tuple filtering and global Skyline calculation, the traditional Skyline method can be directly called for calculation.

[0083] As Figure 4 shown, a schematic diagram of the numerical transformation regarding the query point in an embodiment of the present application is provided. Suppose there is a set of data a, b, c, d with unified domination criteria and a query point q in a two-dimensional space. After the numerical transformation regarding the query point, the data is transformed into a', b', c', d'.

[0084] Tuple filtering within a data block is to filter out non-Skyline tuples within this data block from the tuples after spatial transformation using the traditional Skyline calculation method. By tuple filtering within a data block, the amount of data for subsequent calculation and transmission can be greatly reduced.

[0085] In some embodiments of the present application, for the filtering result according to the query preference data and the data block dimension, the Skyline query algorithm is used to perform grid dimension filtering on the filtering result of the data block dimension, and the obtained grid dimension filtering result includes: performing grid processing on the filtering result of the data block dimension to obtain a grid processing result, where the grid processing result includes mapping information between grids and data tuples and information on whether a Skyline tuple is included in the grid; according to the query preference data and the grid processing result, using the Skyline query algorithm to perform grid dimension filtering on the filtering result of the data block dimension to obtain the grid dimension filtering result.

[0086] When performing tuple filtering in the grid dimension, grid processing can be first performed on the data tuples remaining after data block filtering processing. The grid processing mainly includes dividing the grid space for the data tuples and performing statistical calculations on all divided grid subspaces. Through grid processing, the mapping information between grids and data tuples and the information on whether a tuple is included in the grid can be finally obtained.

[0087] Based on the above grid processing result, grid - to - grid filtering can be performed according to the dominance relationship between grids and whether a tuple is included in the grid, and at the same time, the traditional Skyline query algorithm is called again to perform tuple filtering within the grid, thereby further reducing the amount of data to be queried.

[0088] In some embodiments of the present application, the performing grid processing on the filtering result of the data block dimension to obtain a grid processing result includes: performing grid division on the filtering result of the data block dimension according to a predefined grid interval, determining the grid to which the unfiltered data tuples in each data block belong, to obtain the mapping information between grids and data tuples; counting whether at least one data tuple is included in each grid to obtain the information on whether a Skyline tuple is included in the grid.

[0089] Continue to refer to Figure 3 , the grid processing includes calculating the grid to which the Skyline tuples in the data block belong according to the pre - planned grid interval, and counting whether each grid includes the Skyline tuples in the data block. After grid calculation, the Skyline tuples in all data blocks can be recorded in the form of (G i , {p1, p2, …, p d}) as the output M i of the Map task in the MapReduce model, where G i is used as the key to represent the grid id, and {p1, p2, …, p d} is used as the value to represent the tuple data. At the same time, it is counted whether each grid includes the Skyline tuples in the data block CF i, in the format of [(G1, true), (G2, false), …, (G m , true)], by aggregating the CFs of all Map tasks i to obtain the global CF for subsequent inter-grid filtering.

[0090] In some embodiments of the present application, the filtering of the data block dimension filtering result in the grid dimension by using the Skyline query algorithm according to the query preference data and the grid processing result includes: determining the dominance relationship between grids according to the grid processing result; performing inter-grid filtering on the data block dimension filtering result according to the dominance relationship between grids and the information on whether the grid contains Skyline tuples to obtain an inter-grid filtering result; determining the dominance relationship between data tuples within each filtered grid according to the query preference data and the inter-grid filtering result; and performing intra-grid filtering on each filtered grid by using the Skyline query algorithm according to the dominance relationship between data tuples within each filtered grid to obtain an intra-grid filtering result.

[0091] Inter-grid filtering uses the globally statistically CF information to filter out grid regions dominated by other grids. The dominated grids can filter out all the tuples they contain and do not need to perform intra-grid tuple filtering. Therefore, one of the prerequisites for inter-grid tuple filtering is to determine the dominance relationship between grids.

[0092] Considering that a grid may contain multiple data tuples, when determining the dominance relationship between grids, it is necessary to consider the dominance relationship between all data tuples in the grid and all data tuples in other grids. For example, if all data tuples in grid G1 are dominated by all data tuples in grid G2, then it can be determined that grid G1 is dominated by grid G2. If at least one data tuple in grid G1 is not dominated by all data tuples in grid G2, then grid G1 must not be dominated by grid G2. More generally, if the best tuple in a grid G i is dominated by the worst data tuple in another grid G j , then grid G i must be dominated by G j .

[0093] Another prerequisite for grid - to - grid filtering based on the dominance relationship is that there should be comparable data tuples in two grids. However, in actual situations, there may be a grid that does not contain data tuples. In this case, the grid containing data tuples cannot be directly filtered out. Therefore, after determining the dominance relationship between grids in the embodiments of the present application, it is also necessary to perform grid - to - grid filtering on the filtering results of the data block dimension in combination with the global CF information calculated in the above - mentioned embodiments, so as to obtain the grid - to - grid filtering result.

[0094] For the convenience of understanding the above - mentioned embodiments, as Figure 5 shown, a schematic diagram of grid - to - grid filtering in the embodiments of the present application is provided. Assume that in a certain global CF information, G2, G3, G4, G5, G6, G7, and G8 are grids containing tuples, and G1 and G9 are grids without tuples. According to the grid - to - grid dominance relationship, it can be determined that G1 must dominate G5, G6, G8, and G9 (assuming that the smaller the dimension value, the better), but since there are no tuples in G1, the grids G5, G6, G8, and G9 cannot be directly filtered out; G2 must dominate G6, and since G2 contains tuples, the grid G6 can be directly filtered out. Similarly, G4 must dominate G8, and the grid G8 can be filtered out. Therefore, the grid - to - grid filtering result is G7, G4, G5, G2, G3, that is, only the filtering tasks corresponding to G7, G4, G5, G2, and G3 need to be performed for subsequent in - grid tuple filtering.

[0095] For the grids remaining after grid - to - grid filtering, it is necessary to further use the traditional Skyline query algorithm to filter out non - Skyline tuples in each grid. Its principle, calculation process, and the in - data - block tuple filtering process are the same, and will not be elaborated here.

[0096] In some embodiments of the present application, the filtering of data tuples stored in the HDFS data set in the data block dimension by using the Skyline query algorithm according to the query preference data includes: starting corresponding Map tasks to read the corresponding data block tuples according to the data block division situation; using the Skyline query algorithm to perform parallel filtering of the data block tuples corresponding to all Map tasks in the data block dimension according to the query preference data to obtain the filtering result of the data block dimension.

[0097] The Skyline query process of the embodiments of this application can be implemented based on the MapReduce model, which mainly consists of three stages: Map, Shuffle, and Reduce. The Map stage is mapping, which converts the original data into key-value pairs. The Map operation requires the user to define the corresponding Map class by themselves to complete the required simplification operation. The Shuffle stage is to enable the Reduce to process the results of the Map in parallel. It is necessary to sort and split the output of the Map to a certain extent and then hand it over to the corresponding Reduce. This process of further organizing the Map output and handing it over to the Reduce is Shuffle. The Reduce stage is merging, which processes the values with the same key value and then outputs new key-value pairs as the final result. The Reduce operation requires the user to define the corresponding Reduce class by themselves to complete the required merging operation.

[0098] Continue to refer to Figure 3 , in the Map stage of the embodiments of this application, filtering processing in the data block dimension is implemented. When the Map task is started, the data in HDFS is loaded into multiple Maps in chunks for parallel computing to obtain the processing results of each Map task.

[0099] On the one hand, by combining the parallel processing ability of the MapReduce model, it is possible to process the data tuples in multiple data blocks simultaneously, thus significantly improving the efficiency of the Skyline query. In addition, the filtering in the data block dimension can exclude a large number of data tuples that do not meet the conditions in advance, further reducing the amount of data to be processed subsequently. On the other hand, the MapReduce model natively supports distributed processing, so it can be easily extended to large-scale data sets and complex query scenarios. By increasing the number of Map tasks, the processing capacity and response speed can be linearly improved. On the other hand, by combining the Skyline query algorithm with the MapReduce model, the data processing flow becomes more concise and efficient. The user does not need to care about the underlying data storage and parallel processing details, and only needs to provide the query preference data and the target data set.

[0100] In some embodiments of this application, filtering the filtering result in the data block dimension in the grid dimension by using the Skyline query algorithm according to the query preference data and the filtering result in the data block dimension to obtain the filtering result in the grid dimension includes: in the Map stage, performing grid processing on the filtering result in the data block dimension to obtain the grid processing result; in the Shuffle stage, transmitting the filtering result in the data block dimension to the Reduce stage according to the grid processing result; in the Reduce stage, filtering the filtering result in the data block dimension in the grid dimension by using the Skyline query algorithm to obtain the filtering result in the grid dimension.

[0101] Continue to refer to Figure 3 , after the filtering of the data block dimension is completed in the Map stage, grid processing can be further performed to obtain the grid processing result M of each Map task i and CF i . Then the grid processing results M of each Map task i and CF i enter the Shuffle stage. In the Shuffle stage, data tuples of the same grid are transmitted to the same Reduce task for calculation through the grid ID. Then in the Reduce stage, each Reduce task performs parallel filtering of the grid dimension, obtains the filtering results of the grid dimension output by each Reduce task and summarizes the filtering results. Finally, global Skyline calculation is performed according to the summarized filtering results of the grid dimension

[0102] The embodiments of the present application make full use of the parallel processing ability of the MapReduce model, enabling the filtering of the grid dimension to process data tuples in multiple grid cells simultaneously, thereby significantly improving the processing efficiency

[0103] The embodiments of the present application also provide a Skyline query device 600 based on the MapReduce model, as Figure 6 shown, which provides a structural schematic diagram of a Skyline query device based on the MapReduce model in the embodiments of the present application. The Skyline query device 600 based on the MapReduce model at least includes: an acquisition unit 610, a first filtering unit 620, a second filtering unit 630, and a global processing unit 640, where

[0104] The acquisition unit 610 is used to acquire the query preference data of the user

[0105] The first filtering unit 620 is used to filter the data tuples stored in the HDFS data set in the data block dimension according to the query preference data by using the Skyline query algorithm to obtain the filtering result of the data block dimension

[0106] The second filtering unit 630 is used to filter the filtering result of the data block dimension in the grid dimension according to the query preference data and the filtering result of the data block dimension by using the Skyline query algorithm to obtain the filtering result of the grid dimension. The filtering of the grid dimension includes inter-grid filtering and intra-grid filtering

[0107] The global processing unit 640 is used to perform global processing on the filtering result of the grid dimension by using the Skyline query algorithm to obtain the Skyline query result of the user

[0108] In some embodiments of the present application, the first filtering unit 620 is specifically configured to: perform normalization processing on the data tuples in the HDFS data set to obtain normalized data tuples; perform numerical conversion processing on the normalized data tuples according to the query preference data and the normalized data tuples to obtain numerically converted data tuples; and filter the numerically converted data tuples in terms of data block dimension by using the Skyline query algorithm according to the query preference data to obtain a filtering result in terms of data block dimension.

[0109] In some embodiments of the present application, the second filtering unit 630 is specifically configured to: perform grid processing on the filtering result in terms of data block dimension to obtain a grid processing result, where the grid processing result includes mapping information between grids and data tuples and information on whether a Skyline tuple is included in a grid; and filter the filtering result in terms of data block dimension in terms of grid dimension by using the Skyline query algorithm according to the query preference data and the grid processing result to obtain a filtering result in terms of grid dimension.

[0110] In some embodiments of the present application, the second filtering unit 630 is specifically configured to: perform grid division on the filtering result in terms of data block dimension according to a predefined grid interval to determine the grids to which the unfiltered data tuples in each data block belong, so as to obtain mapping information between grids and data tuples; and count whether at least one data tuple is included in each grid to obtain information on whether a Skyline tuple is included in the grid.

[0111] In some embodiments of the present application, the second filtering unit 630 is specifically configured to: determine the dominance relationship between grids according to the query preference data and the grid processing result; perform inter-grid filtering on the filtering result in terms of data block dimension according to the dominance relationship between grids and the information on whether a Skyline tuple is included in the grid to obtain an inter-grid filtering result; determine the dominance relationship between the data tuples in each filtered grid according to the query preference data and the inter-grid filtering result; and perform intra-grid filtering on each filtered grid by using the Skyline query algorithm according to the dominance relationship between the data tuples in each filtered grid to obtain an intra-grid filtering result.

[0112] In some embodiments of the present application, the first filtering unit 620 is specifically configured to: start corresponding Map tasks to read corresponding data block tuples according to the data block division situation; and filter the data block tuples corresponding to all Map tasks in terms of data block dimension in parallel by using the Skyline query algorithm according to the query preference data to obtain a filtering result in terms of data block dimension.

[0113] In some embodiments of the present application, the second filtering unit 630 is specifically configured to: in the Map stage, perform grid processing on the filtering result in the data block dimension to obtain a grid processing result; in the Shuffle stage, transmit the filtering result in the data block dimension to the Reduce stage according to the grid processing result; in the Reduce stage, use the Skyline query algorithm to perform filtering in the grid dimension on the filtering result in the data block dimension to obtain a filtering result in the grid dimension.

[0114] It can be understood that the above Skyline query device based on the MapReduce model can implement each step of the Skyline query method based on the MapReduce model provided in the foregoing embodiments. The relevant explanations regarding the Skyline query method based on the MapReduce model are applicable to the Skyline query device based on the MapReduce model, and will not be elaborated herein.

[0115] Figure 7 is a schematic structural diagram of a device in an embodiment of the present application. As Figure 7 shown, the device includes one or more processors (or processing units), and may further include one or more memories coupled to the processor, and may further include a communication module coupled to the processor.

[0116] The communication module can be used to communicate with other devices or apparatuses, such as sending or receiving data and / or signals. The communication module may have at least one communication module for communication. The communication module may include any interface necessary for communicating with other devices. Exemplarily, the communication module may be a transceiver, a circuit, a bus, a module, or other types of communication modules.

[0117] The processor may include, but is not limited to, at least one of the following: a general-purpose computer, a special-purpose computer, a microcontroller, a digital signal controller (Digital Signal Processor, DSP), or one or more in a multi-core controller architecture based on a controller. The device may have multiple processors, such as an application-specific integrated circuit chip, which is subordinate to a clock synchronized with the main processor in time.

[0118] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: Read-Only-Memory (ROM), Electrically Programmable Read-Only-Memory (EPROM), flash memory, hard disk, Compact Disc (CD), Digital Video Disk (DVD), or other magnetic storage and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: Random Access Memory (RAM), or other volatile memories that do not persist during a power-off duration.

[0119] The computer program includes computer-executable instructions executed by an associated processor. The program may be stored in the ROM. The processor may perform any suitable actions and processes by loading the program into the RAM.

[0120] Possible implementations of the present application may be implemented by means of a program such that the communication device can perform any process discussed in the foregoing embodiments. Possible implementations of the present application may also be implemented by hardware or by a combination of software and hardware.

[0121] In some embodiments, the program may be tangibly embodied in a computer-readable storage medium, which may be included in the device (such as in the memory) or other storage devices accessible by the device. The program may be loaded from the computer-readable storage medium into the RAM for execution. The computer-readable storage medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc.

[0122] The embodiments of the present application also provide a computer-readable storage medium, on which computer instructions or program codes are stored. When the processor runs the instructions or the program codes, the processor is caused to execute the methods and functions involved in any of the above embodiments. The computer-readable medium can be any tangible medium that contains or stores a program for or related to an instruction execution system, apparatus, or device. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or a data center that includes one or more integrated available media. More specific examples of the computer-readable storage medium include electrical connections with one or more wires, magnetic media (such as disks, floppy disks, hard disks, magnetic tapes, magnetic storage devices), optical media (such as optical storage devices, DVDs), semiconductor media (such as solid-state drives), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or any suitable combination thereof, etc.

[0123] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The embodiments of the present application also provide at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes one or more computer-executable instructions, such as instructions included in program modules, which are executed in a device on a target real or virtual processor to execute the processes, methods, and functions involved in any of the above embodiments. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, fiber optic, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.).

[0124] The embodiments of the present application also propose a computer program product, including a computer program or instruction. When the computer program or instruction runs on a computer, it enables the computer to execute the processes, methods, and functions in the above embodiments. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules can be combined or divided as needed. The machine-executable instructions for program modules can be executed within local or distributed devices. In a distributed device, program modules can be located in local and remote storage media.

[0125] Generally, the various embodiments of the present application can be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although the various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that the blocks, devices, systems, technologies, or methods described herein can be implemented as, by way of non-limiting example, hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0126] It should be noted that although the embodiments of the present application are described above in conjunction with the accompanying drawings respectively, the above embodiments are not independent of each other, and they can also be combined to obtain other embodiments. The ways, situations, categories, and the division of embodiments in the embodiments of the present application are only for the convenience of description and should not constitute a special limitation. The features in various ways, categories, situations, and embodiments can be combined with each other under logical conditions. The various embodiments of the present application can be combined arbitrarily to achieve different technical effects. The embodiments of the present application will no longer list various combinations.

[0127] In addition, although the operations of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be changed in the order of execution. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution. It should also be noted that the features and functions of two or more devices according to the present disclosure can be embodied in one device. Conversely, the features and functions of one device described above can be further divided and embodied by multiple devices.

[0128] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising said element.

[0129] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A Skyline query method based on the MapReduce model, characterized in that, The Skyline query method based on the MapReduce model includes: Obtain the query preference data of the user; According to the query preference data, use the Skyline query algorithm to filter the data tuples stored in the HDFS data set in terms of data block dimension, and obtain the filtering result of the data block dimension; According to the query preference data and the filtering result of the data block dimension, use the Skyline query algorithm to filter the filtering result of the data block dimension in terms of grid dimension, and obtain the filtering result of the grid dimension, where the filtering of the grid dimension includes inter-grid filtering and intra-grid filtering; Use the Skyline query algorithm to globally process the filtering result of the grid dimension, and obtain the Skyline query result of the user.

2. The Skyline query method based on the MapReduce model according to claim 1, wherein The step of using the Skyline query algorithm to filter the data tuples stored in the HDFS data set in terms of data block dimension according to the query preference data and obtain the filtering result of the data block dimension includes: Perform normalization processing on the data tuples in the HDFS data set to obtain the normalized data tuples; According to the query preference data and the normalized data tuples, perform numerical conversion processing on the normalized data tuples to obtain the numerically converted data tuples; According to the query preference data, use the Skyline query algorithm to filter the numerically converted data tuples in terms of data block dimension, and obtain the filtering result of the data block dimension.

3. The Skyline query method based on the MapReduce model according to claim 1, characterized in that The filtering result of the data block dimension includes the Skyline tuples within the data block. The step of using the Skyline query algorithm to filter the filtering result of the data block dimension in terms of grid dimension according to the query preference data and the filtering result of the data block dimension and obtain the filtering result of the grid dimension includes: Perform grid processing on the filtering result of the data block dimension to obtain the grid processing result, where the grid processing result includes the mapping information between the grid and the data tuples and the information on whether the grid contains Skyline tuples; According to the query preference data and the grid processing result, use the Skyline query algorithm to filter the filtering result of the data block dimension in terms of grid dimension, and obtain the filtering result of the grid dimension.

4. The Skyline query method based on the MapReduce model according to claim 3, wherein The step of performing grid processing on the filtering result of the data block dimension to obtain the grid processing result includes: According to the predefined grid interval, perform grid division on the filtering result of the data block dimension, determine the grid to which the unfiltered data tuples within each data block belong, and obtain the mapping information between the grid and the data tuples; Count whether each grid contains at least one data tuple to obtain the information on whether the grid contains Skyline tuples.

5. The Skyline query method based on the MapReduce model according to claim 3, wherein The step of using the Skyline query algorithm to filter the filtering result of the data block dimension in terms of grid dimension according to the query preference data and the grid processing result and obtain the filtering result of the grid dimension includes: Determine the dominance relationship between the grids according to the query preference data and the grid processing result; Filter the filtering result of the data block dimension among grids according to the dominance relationship between the grids and the information on whether the grids contain Skyline tuples, so as to obtain the filtering result among grids; Determine the dominance relationship between the data tuples in each filtered grid according to the query preference data and the filtering result among grids; According to the dominance relationship between the data tuples in each filtered grid, use the Skyline query algorithm to filter each filtered grid within the grid to obtain the filtering result within the grid.

6. The Skyline query method based on the MapReduce model according to claim 1, characterized in that The filtering of the data tuples stored in the HDFS data set in the data block dimension by using the Skyline query algorithm according to the query preference data to obtain the filtering result of the data block dimension includes: According to the data block division situation, start the corresponding Map task to read the corresponding data block tuples. According to the query preference data, use the Skyline query algorithm to filter the data block tuples corresponding to all Map tasks in parallel in the data block dimension to obtain the filtering result of the data block dimension.

7. The Skyline query method based on the MapReduce model according to claim 6, characterized in that, The filtering of the filtering result of the data block dimension in the grid dimension by using the Skyline query algorithm according to the query preference data and the filtering result of the data block dimension to obtain the filtering result of the grid dimension includes: In the Map stage, perform grid processing on the filtering result of the data block dimension to obtain the grid processing result; In the Shuffle stage, transfer the filtering result of the data block dimension to the Reduce stage according to the grid processing result; In the Reduce stage, use the Skyline query algorithm to filter the filtering result of the data block dimension in the grid dimension to obtain the filtering result of the grid dimension.

8. A Skyline query device based on the MapReduce model, characterized in that, The Skyline query device based on the MapReduce model includes: An acquisition unit, configured to acquire the query preference data of the user; A first filtering unit, configured to filter the data tuples stored in the HDFS data set in the data block dimension by using the Skyline query algorithm according to the query preference data to obtain the filtering result of the data block dimension; A second filtering unit, configured to filter the filtering result of the data block dimension in the grid dimension by using the Skyline query algorithm according to the query preference data and the filtering result of the data block dimension to obtain the filtering result of the grid dimension, and the filtering of the grid dimension includes filtering among grids and filtering within grids; A global processing unit, configured to perform global processing on the filtering result of the grid dimension by using the Skyline query algorithm to obtain the Skyline query result of the user.

9. A device, comprising: A processor; And a memory arranged to store computer-executable instructions, and the executable instructions, when executed, cause the processor to execute the Skyline query method based on the MapReduce model according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by a processor, the Skyline query method based on the MapReduce model according to any one of claims 1 to 7 is implemented.

Images