Data query method, device, computing device and storage medium

Abstract

The present application provides a method, apparatus, computing device, and storage medium for querying data, and belongs to the field of big data technology. The method includes: when performing a data query, receiving a data query request, wherein the data query request specifies a data range of multiple dimensions, obtaining spatial distribution information of the multi-dimensional data, and the spatial distribution information records the data ranges of multiple subspace objects associated with the multi-dimensional data in each dimension; based on the spatial distribution information of the multi-dimensional data, determining blocks in the multi-dimensional data that meet the data ranges of the multiple dimensions specified by the data query request, and finding data in the determined blocks that meet the data ranges of the multiple dimensions specified by the data query request. The use of the present application can improve the efficiency of data queries.

Description

[0001] This application claims priority to Chinese patent application No. 201910907480.2, filed on September 24, 2019, entitled “A Data Processing Method and Device Therefor,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of big data technology, and in particular to a method, apparatus, computing device, and storage medium for querying data. Background Art

[0003] The development of cloud computing and big data technologies has led to explosive data growth. Data volumes have grown from gigabytes (GB) and terabytes (TB) to petabytes (PB) and exabytes (EB). This has led to an even more pressing need for near-real-time data analysis. When analyzing data, the first step is to find the desired data within the vast amount of data.

[0004] In related technologies, the process for querying data to be analyzed is as follows: when storing a file, the server divides the file into blocks, determines the maximum and minimum values ​​of the data in each dimension of each block, and then stores the maximum and minimum values ​​of the data in each dimension of each block. When performing a data query, the server uses the data range of the data to be queried to match the maximum and minimum values ​​of the data in the dimension to which the data belongs, and filters out the blocks containing the data in the data range. The server then searches for data in the data range within the filtered blocks.

[0005] In related techniques, because the server only uses the maximum and minimum values ​​of each dimension to filter data, it is possible that a large number of blocks do not contain the requested data and are therefore filtered out. For example, if the data range of the d1 dimension of a block is [10, 45], and the data to be queried is also in the d1 dimension and the range is [27, 38], then according to the related technique, even if the data in the d1 dimension of the block does not contain data in the range [27, 38], it will be filtered out. Therefore, query methods based on related techniques can lead to relatively low data query efficiency. Summary of the Invention

[0006] In order to solve the problem of low data query efficiency, the present application provides a method, apparatus, computing device and storage medium for querying data.

[0007] In a first aspect, the present application provides a method for querying data. In the method, a data query request is received, wherein the data query request specifies data ranges in multiple dimensions, spatial distribution information of the multi-dimensional data is obtained, the spatial distribution information recording the data ranges in each dimension of multiple subspace objects associated with the multi-dimensional data, and blocks that meet the data ranges in the multiple dimensions specified in the data query request are determined in the multi-dimensional data based on the spatial distribution information of the multi-dimensional data. Within the determined blocks, data that meets the data ranges in the multiple dimensions specified in the data query request is searched for.

[0008] For example, a subspace object is formed by clustering multidimensional data in space.

[0009] In the scheme shown in the present application, the execution subject of the method can be a device for querying data, and the device for querying data can receive a data query request, and the data query request specifies the data range of multiple dimensions, that is, the data to be queried belongs to multiple dimensions. The device for querying data can obtain the spatial distribution information of the multi-dimensional data, and the spatial distribution information records the data range of multiple subspace objects associated with the multi-dimensional data in each dimension. Then, the device for querying data can determine the blocks that meet the data range of multiple dimensions to be queried in the multi-dimensional data based on the spatial distribution information. The device for querying data can use the data range of multiple dimensions to be queried in the determined blocks to find the data indicated by the data range of multiple dimensions. In this way, for data that are adjacent in space but are more dispersed in physical memory, by learning the spatial distribution information of these data, the spatial distribution information of the multi-dimensional data is constructed, which can effectively filter the query, reduce the invalid reading and analysis of a large number of blocks, and improve the data query speed.

[0010] In one possible implementation, based on the spatial distribution information of the multidimensional data, a block that satisfies the data range of multiple dimensions specified by a data query request is determined in the multidimensional data, including: based on the spatial distribution information of the multidimensional data, one or more subspace objects that satisfy the data range of multiple dimensions specified by the data query request are determined from multiple subspace objects; and the block to which the determined subspace object belongs is used as the block that satisfies the data range of multiple dimensions specified by the data query request.

[0011] In the solution shown in the present application, the device for querying data matches the data range of each dimension of the multiple dimensions with the spatial distribution information of the multi-dimensional data to obtain the subspace objects satisfied by the data range of each dimension. The device for querying data takes the intersection of the subspace objects satisfied by the data range of each dimension to obtain the subspace objects that satisfy the data ranges of the multiple dimensions. The device for querying data determines the data blocks indicated by the identifiers of these subspace objects as blocks that satisfy the data ranges of the multiple dimensions. In this way, the possibility that the found blocks do not include the data to be searched can be reduced as much as possible.

[0012] In a possible implementation, when no subspace object that satisfies the data ranges of multiple dimensions specified in the data query request is found from multiple subspace objects according to the spatial distribution information of the multidimensional data, feedback is given that no data that satisfies the data query request exists.

[0013] In the solution shown in the present application, when the device for querying data fails to find a subspace object that satisfies the data range of multiple dimensions among multiple subspace objects of multi-dimensional data, it can feedback to the device that sent the data query request that there is no data that satisfies the data query request, so that the device that sent the data query request can understand the query result in a timely manner.

[0014] In one possible embodiment, obtaining spatial distribution information of multidimensional data includes: obtaining spatial distribution information of first-level blocks of the multidimensional data, the spatial distribution information of the first-level blocks recording the data ranges of multiple subspace objects associated with the data of the first-level blocks in each dimension; obtaining spatial distribution information of second-level blocks of target blocks, the spatial distribution information of the second-level blocks recording the data ranges of multiple subspace objects associated with the data of the second-level blocks in each dimension; determining, in the multidimensional data, blocks that satisfy the data ranges of multiple dimensions specified by a data query request based on the spatial distribution information, including: determining, in the first-level blocks, a target block that satisfies the data ranges of multiple dimensions specified by the data query request based on the spatial distribution information of the first-level blocks; and determining, in the target blocks, a second-level block that satisfies the data ranges of multiple dimensions specified by the data query request based on the spatial distribution information of the second-level blocks.

[0015] In the solution shown in the present application, the multi-dimensional data may be divided into multiple levels. For example, the multi-dimensional data is divided into a first level and a second level, and the corresponding first level may include multiple first-level blocks, and the second level may include multiple second-level blocks. The second-level blocks are obtained by dividing the first-level blocks, and multiple second-level blocks constitute a first-level block. The device for querying data can obtain the spatial distribution information of the first-level blocks and the spatial distribution information of the second-level blocks of the multi-dimensional data. The spatial distribution information of the first-level blocks can record the data ranges of multiple sub-space objects associated with the data of the first-level blocks in each dimension. The spatial distribution information of the second-level blocks can record the data ranges of multiple sub-space objects associated with the data of the second-level blocks in each dimension.

[0016] The device for querying data can first determine one or more subspace objects in the first-level block that meet the data range of multiple dimensions in the spatial distribution information of the first-level block. The device for querying data determines the block corresponding to the identifier of the one or more subspace objects as the target block. The device for querying data then determines the second-level blocks included in the target block, and uses the spatial distribution information of these second-level blocks to determine one or more subspace objects in these second-level blocks that meet the data range of multiple dimensions, and determines the second-level blocks corresponding to the identifier of the one or more subspace objects as the second-level blocks that meet the data range of multiple dimensions specified by the data query request. In this way, the first-level blocks are first filtered to filter out the first-level blocks of data that do not include the data range of multiple dimensional data to be queried, and then the second-level blocks in the remaining first-level blocks are filtered, which can speed up the data query.

[0017] In a possible implementation, the method further includes: determining a plurality of subspace objects of the multidimensional data according to distribution characteristics and constraints of the multidimensional data in the multidimensional space.

[0018] In the solution presented in this application, constraints may include one or more of the following: the number of subspace objects, the amount of data per subspace object, incremental information about subspace objects, or computational performance. The number of subspace objects refers to the maximum and minimum number of subspace objects; the amount of data per subspace object refers to the maximum and minimum amount of data per subspace object; incremental information about subspace objects refers to the maximum increment in the volume of a subspace object after adding data; and computational performance refers to the maximum time required to calculate spatial distribution information. Distribution characteristics refer to the intervals between data, etc.

[0019] The data query device first divides the multidimensional data into multiple blocks (the multiple blocks can include blocks of only one level or blocks of multiple levels). Based on the distribution characteristics and constraints of the multidimensional data in the multidimensional space, the data query device determines multiple subspace objects for each block of the multidimensional data, thereby obtaining spatial distribution information of the multidimensional data. Each subspace object corresponds to the data range in each dimension.

[0020] In one possible implementation, multiple subspace objects of multidimensional data are determined based on the distribution characteristics and constraints of the multidimensional data in the multidimensional space, including: performing dimensionality reduction processing on the multidimensional data to obtain the data after dimensionality reduction processing; and determining multiple subspace objects based on the distribution characteristics and constraints of the data after dimensionality reduction processing in the low-dimensional space.

[0021] Among them, dimensionality reduction processing is dimensionality reduction processing. For example, if multi-dimensional data is three-dimensional data, the data after dimensionality reduction processing is one-dimensional, and the low-dimensional space is one-dimensional space.

[0022] According to the solution shown in the present application, for any block of multi-dimensional data, the device for querying data can obtain the data after dimension reduction by encoding and reducing the dimension of the multi-dimensional data included in the block. The device for querying data can then use the distribution characteristics and constraints of the data after dimension reduction in the low-dimensional space to determine the multiple subspace objects corresponding to the block. The device for querying data then merges the subspace objects of all blocks, that is, obtains multiple subspace objects of the multi-dimensional data. In this way, due to the dimension reduction processing of the multi-dimensional data, clustering the data after dimension reduction will reduce the amount of calculation, improve the speed of clustering determination, and thus improve the speed of determining the subspace objects.

[0023] In a possible implementation, multiple subspace objects of the multidimensional data are determined based on the distribution characteristics and constraints of the multidimensional data in the multidimensional space, including: clustering the multidimensional data by volume increments to obtain multiple subspace objects.

[0024] In this embodiment, for any block of multidimensional data, the device for querying data divides the multiple data in the block into a cluster (which can be called the first cluster), and then adds data to the first cluster. If the difference between the volume of the first cluster after adding the data and the volume of the first cluster without adding the data is less than a certain value, the data is added to the first cluster. If the difference between the volume of the first cluster after adding the data and the volume of the first cluster without adding the data is greater than or equal to the value, the first cluster is established. The device for querying data creates a new cluster (which can be called the second cluster) and performs the same processing as the first cluster until all the data in the multidimensional data are traversed to obtain the final cluster. The device for querying data regards each cluster as a subspace object. Then, the data range of each dimension of each subspace object is determined, that is, the subspace object corresponding to the block is obtained.

[0025] In one possible implementation, multiple subspace objects of the multidimensional data are determined based on the distribution characteristics and constraints of the multidimensional data in the multidimensional space, including: dividing the multidimensional data into subspaces, and obtaining multiple subspace objects based on the distribution characteristics and constraints of the subspaces obtained by the division.

[0026] The solution shown in the present application is that when the data range of each dimension of the multidimensional data is relatively large, there is a wide range of data values ​​or a large floating-point precision, which results in a large amount of data calculation or high computational complexity during clustering. The device for querying data divides the data range of each dimension of the block into N intervals (N can be preset by the user) at equal intervals. Then, for each dimension of any block in the multidimensional data, the device for querying data maps all data in the first interval of the dimension to 0, the device for querying data maps all data in the second interval of the dimension to 1, the device for querying data maps all data in the i-th interval of the dimension to i-1, and so on, mapping all data of the dimension to 0 to N-1 in sequence. In the same way, the device for querying data can map each dimension of data to 0 to N-1.

[0027] Then, the device for querying data maps the N numerical values ​​of the data in each dimension into data point coordinates, thereby obtaining a subspace. The device for querying data merges the subspaces, obtains the maximum and minimum values ​​of the data in each dimension of the merged subspace, and each merged subspace is a subspace object. For each subspace object, the maximum and minimum values ​​of the data of the subspace object in each dimension constitute the data range of the subspace object in each dimension. In this way, the subspace object of each block of multi-dimensional data can be determined. In this way, after the data is dimensionally compressed into a subspace, the amount of data that needs to be calculated will be reduced, so the filtering performance of the query data can be improved. Moreover, it is only required that the data of each dimension of the block can be sorted, that is, the subspace can be obtained by dimensionally compressing the encoding, so the scope of application is relatively wide.

[0028] In a possible implementation, the data ranges of the multiple subspace objects in each dimension constitute indexes of the multiple subspace objects.

[0029] In a second aspect, the present application provides a device for querying data, which includes one or more modules, and the one or more modules are used to implement the method for querying data provided by the above-mentioned first aspect or possible implementation methods of the first aspect.

[0030] In a third aspect, the present application provides a computing device for querying data, which includes a processor and a memory, and the processor executes instructions stored in the memory, so that the computing device implements the method for querying data provided by the above-mentioned first aspect or a possible implementation method of the first aspect.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium storing instructions for instructing a computing device to execute the method for querying data provided by the first aspect or a possible implementation of the first aspect.

[0032] In a fifth aspect, the present application provides a computer program product comprising instructions, which instruct a computing device to execute the method for querying data provided by the above-mentioned first aspect or a possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic diagram of query data provided by an exemplary embodiment of the present application;

[0034] Figure 2 is a schematic structural diagram of a computing device provided by an exemplary embodiment of the present application;

[0035] Figure 3 is a schematic diagram of an application scenario provided by an exemplary embodiment of the present application;

[0036] Figure 4 is a schematic diagram of an application scenario provided by an exemplary embodiment of the present application;

[0037] Figure 5 is a flowchart of a method for determining spatial distribution information provided by an exemplary embodiment of the present application;

[0038] Figure 6 is a schematic diagram of data blocks obtained by division provided by an exemplary embodiment of the present application;

[0039] Figure 7 is a schematic diagram of a subspace object provided by an exemplary embodiment of the present application;

[0040] Figure 8is a schematic diagram of a dimensionality reduction process provided by an exemplary embodiment of the present application;

[0041] Figure 9 is a schematic diagram of equally spaced mapping data provided by an exemplary embodiment of the present application;

[0042] Figure 10 is a schematic diagram of mapped data provided by an exemplary embodiment of the present application;

[0043] Figure 11 is a schematic diagram of a subspace object of a single block provided by an exemplary embodiment of the present application;

[0044] Figure 12 is a schematic diagram of a plurality of block subspace objects provided by an exemplary embodiment of the present application;

[0045] Figure 13 is a schematic diagram of a segment tree provided by an exemplary embodiment of the present application;

[0046] Figure 14 This is a schematic diagram of storing subspace objects of multiple blocks provided by an exemplary embodiment of the present application;

[0047] Figure 15 is a flowchart of a method for querying data provided by an exemplary embodiment of the present application;

[0048] Figure 16 It is a structural diagram of a device for querying data provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0050] To facilitate understanding of the embodiments of this application, the following first introduces the concepts of the terms involved:

[0051] Data filtering refers to the process of filtering out blocks that don't contain the requested data during a data query, eliminating subsequent queries within those blocks. Data filtering can, to a certain extent, filter out blocks that don't contain the requested data. This improves query efficiency by only searching for the requested data in blocks that are likely to contain it.

[0052] Blocking refers to breaking down a data table in a database into multiple smaller, more manageable parts according to certain rules. Each part can be a data block or a file block. It should be noted here that each data block can be a data block with equal data volume (this is because the data blocks are obtained by dividing the file blocks, and are generally divided into equal sizes), and each file block may have unequal data volume (this is because the data of a file imported at a certain time has not yet reached the data volume requirement of a file block, and the file will not be divided. The data of this file is a file block). Of course, block division can also be other division methods, which are not limited in the embodiments of the present application.

[0053] Single-dimensional data only includes data of one dimension. For example, if a data table only includes the age dimension, then the data included in this data table is single-dimensional data.

[0054] Multidimensional data, as opposed to single-dimensional data, refers to data that includes multiple dimensions. For example, if a data table includes age, salary, and name dimensions, then the data in this data table is multidimensional data.

[0055] In a big data scenario, the amount of data built for the index of each piece of data is too large, and the index cannot be stored in memory, so part of the index also needs to be stored on disk. Therefore, big data analysis systems with distributed architecture tend to store both data and indexes on disk for block processing. During block processing, if the amount of data in a file is relatively large, the device that performs data storage will divide the file into file blocks according to the partition column, and then further divide it into data blocks according to the primary key column. The device that performs data storage then determines the data range of each dimension of each data block. The data range of any dimension of a data block is: the range consisting of the maximum and minimum values ​​of the data of that dimension. In related technologies, such as Figure 1As shown, the data table has five attribute columns (i.e., four dimensions of data; the identifier (id) is not considered a dimension of the data), namely, id, name, height, age, and salary. The device performing data storage blocks the data table according to id, that is, every 32,000 data items are divided into a data block. The data table is divided into four data blocks, namely block 1, block 2, block 3, and block 4. Then, based on the statistical information at the data block level, the minimum and maximum values ​​(MINMAX) of each dimension of the data in each data block are constructed. For example, the MINMAX index of the age column in block 1 is [10, 40], the MINMAX index of the age column in block 2 is [10, 34], the MINMAX index of the age column in block 3 is [40, 50], and the MINMAX index of the age column in block 4 is [15, 37]. In this way, querying the data of height ≥ 1.62 and height ≤ 1.64, age ≥ 27 and age ≤ 38, salary ≥ 200 and salary ≤ 300 in the data table will hit the three data blocks 1, block 2 and block 4. In this way, the required data will be queried in the three data blocks 1, block 2 and block 4 subsequently.

[0056] In the related art, although blocks 1, 2, and 4 are all filtered out, because filtering is performed only based on the MINMAX index, the following situation may occur: the data range of the age dimension data of block 1 is [10, 40], which does not contain the data of [27, 38]. Therefore, even if the data of the age dimension of block 1 does not include the data of [27, 38], it will be filtered out. This results in ineffective scanning of a large number of data blocks, and data query efficiency is relatively low. Therefore, a more efficient data query method is needed.

[0057] The present application provides a method for querying data. The execution subject of the method can be a device for querying data. The device for querying data can be a hardware device, such as a server, terminal or other computing device, or a software device, such as a set of software programs running on a hardware device.

[0058] The device for querying data is a computing device, Figure 2 The following example provides a possible architecture diagram of the computing device. The computing device may include a processor 201, a memory 202, a communication interface 203, and a bus 204. In the computing device, the number of processors 201 may be one or more. Figure 2Only one processor 201 is shown. Optionally, the processor 201 may be a central processing unit (CPU). If the computing device has multiple processors 201, the types of the multiple processors 201 may be different or the same. Optionally, the multiple processors of the computing device may be integrated into a multi-core processor.

[0059] The memory 202 stores computer instructions and data. The memory 202 may store computer instructions and data required to implement the method for querying data provided in this application. For example, the memory 202 stores instructions for receiving module execution steps in the method for querying data provided in this application. For another example, the memory 202 stores instructions for determining module execution steps in the method for querying data provided in this application. The memory 202 may be any one or any combination of the following storage media: non-volatile memory (such as read-only memory (ROM), solid-state disk (SSD), hard disk (HDD), optical disk, etc.), volatile memory.

[0060] The communication interface 203 can be any one or any combination of the following devices: a network interface (such as an Ethernet interface), a wireless network card, or other devices with a network access function.

[0061] The communication interface 203 is used for the computing device to communicate data with other computing devices or terminals.

[0062] Figure 2 Also shown is a bus 204, which can connect the processor 201 to the memory 202 and the communication interface 203. In this way, through the bus 204, the processor 201 can access the memory 202 and can also use the communication interface 203 to exchange data with other computing devices or terminals.

[0063] In the present application, the computing device executes computer instructions in memory 202 to implement the data query method provided in the present application. For example, the computing device executes the steps performed by the receiving module in the aforementioned data query method. Another example is the computing device executes the steps performed by the determining module and the searching module in the aforementioned data query method.

[0064] The data query method in this application can be applied to but not limited to the following scenarios, which include Yet another resource negotiator (Yarn), Spark and Hadoop Distributed File System (HDFS) (Spark is a fast and general computing engine designed for large-scale data processing), such as Figure 3 The deployment methods can be:

[0065] Client nodes connect to the master node and the coordination server (Zookeeper). The master node deploys Yarn, which connects to the Spark cluster and Zookeeper. A Spark context is deployed on the client node (used for interacting with the program and the Spark cluster). Yarn operates in a master / slave mode. Within the resource management framework, the YARN Resource Manager is the master node, and the YARN NodeManager is the slave node. The Resource Manager is responsible for unified management and scheduling of resources on each Node Manager. A Spark cluster consists of multiple distributed worker nodes, each of which has a YARN Node Manager, a Spark executor, a cache, and HDFS (Data Node). HDFS data nodes contain multiple files, each of which consists of multiple data blocks (Blocklets). The file header and footer also contain essential information, such as mode, offset, and index.

[0066] This application can be Figure 4 (4a) and Figure 4 The data query is performed in the manner shown in (4b). Figure 4 (4a) neutralization Figure 4 In (4b), the Spark driver divides the multidimensional data into first-level blocks (e.g., file blocks). The Spark executor controls tasks to divide the first-level blocks into second-level blocks (e.g., data blocks). The second-level blocks are stored according to the distributed file system. Multiple second-level blocks constitute a first-level block.

[0067] exist Figure 4 In the data query process shown in (4a), filtering of multi-dimensional data is implemented in the second-level block (such as the data block), that is, filtering of multi-dimensional data is implemented at the second level.

[0068] exist Figure 4 In the data query process shown in (4b), multi-dimensional data can be filtered only within the first-level block (such as the file block), that is, multi-dimensional data filtering is implemented at the first level.

[0069] In one possible implementation, Figure 4 In the data query process shown in (4b), the data query device can also first filter the multi-dimensional data at the first level, and then implement the multi-dimensional data filtering at the second level.

[0070] like Figure 5 As shown, before querying the data, multi-dimensional data is imported and the spatial distribution information of the multi-dimensional data is learned (since the multi-dimensional data will be divided into blocks, the spatial distribution information of the blocks is learned). The method steps can be:

[0071] Step 501: The device for querying data creates a data table.

[0072] In this embodiment, for multidimensional data (multidimensional data is a data table or multiple data tables), when a user (who may be an operations and maintenance personnel) wishes to store the multidimensional data, the user may input a data table creation request corresponding to the multidimensional data into a data query device through an operation interface or by writing a Structured Query Language (SQL) statement. The data query device receives the data table creation request, which carries information about the data table to be created. Specifically, this may include attribute columns and identifiers for sorting columns. The data query device may then create a data table corresponding to the creation request.

[0073] For example, consider a multidimensional dataset with N data items and the dimensions name, age, height, and salary. The query device creates a table called t_salary (in this case, the multidimensional dataset is a single table) containing five attribute columns: id, name, age, height, and salary. The id column is the sorting column used to identify different data items, while the non-sorting columns are name, age, height, and salary.

[0074] The statements used can be:

[0075]

[0076]

[0077] In this example, "CREATE TABLE t_salary(id string, name string, age int, height double, salary int)" creates the table t_salary. "id" and "name" are in string format, "age" and "salary" are in integer format, and "height" is in double-precision floating-point format. "STORED BY 'name'" stores the data by name, and "TBLPROPERTIES('SORT_COLUMNS'='id')" uses "id" as the sort column.

[0078] In step 502, the data query device specifies the dimension of the spatial distribution information to be learned.

[0079] In this embodiment, since the spatial distribution information of multi-dimensional data must be learned before querying data, the device querying data must specify the specific dimensions of the multi-dimensional data to be learned. Specifically, for the above data table t_salary, the device querying data can use the following statement to specify the spatial distribution information of the non-sorted columns age, height, and salary in the data table t_salary. The statement can be:

[0080]

[0081] "CREATE DATAMAP t_salary_filter ON TABLE t_salary" creates a specific datamap in the t_salary table. "USING 'MultiDimensionalFilter'" further indicates that this DATAMAP performs multidimensional data filtering. "('MDF_COLUMNS' = 'age, height, salary') in DMPROPERTIES" further indicates that the specific dimensions involved in the multidimensional data filtering are age, height, and salary.

[0082] Step 503: The data query device receives the multi-dimensional data and performs block processing on the multi-dimensional data.

[0083] In this embodiment, if a user (who may be an operations and maintenance personnel) wants to import multidimensional data into a data table, he or she may input a data import command into the data query device (the data import command may be input through the interface or through an SQL statement). After receiving the data import command, the data query device imports the multidimensional data into the data table created in step 501. The data query device determines whether the amount of the multidimensional data is greater than a preset threshold. If the amount of the multidimensional data is greater than the preset threshold, the data query device may divide the data table into blocks according to the partition column. In this way, the multidimensional data is ultimately divided into multiple blocks.

[0084] In one possible implementation, the device for querying data divides the multi-dimensional data into hierarchical blocks, and the process is as follows: after receiving the data import command, the device for querying data imports the multi-dimensional data into the data table created in step 501. The device for querying data determines whether the amount of the multi-dimensional data is greater than a preset threshold. If the amount of the multi-dimensional data is greater than the preset threshold, the device for querying data may divide the data table into first-level blocks according to the partition column. Then the device for querying data may divide each first-level block into second-level blocks according to the primary key column. For example, Figure 6 As shown, for the data table t_salary above, the partition column is "city." The data query device partitions the file by "city," resulting in first-level blocks. The primary key column is id. The data query device partitions IDs 1 to 32000 into a second-level block, IDs 32001 to 64000 into a second-level block, and so on. In this way, the data table containing multidimensional data is ultimately partitioned into N second-level blocks (blocks 1 to N).

[0085] It should be noted here that when multi-dimensional data is divided into blocks, it can be divided into blocks of one level, or it can be divided into multiple levels, and the multi-dimensional data can be divided into blocks of multiple levels (such as the first-level blocks are file blocks, and the second-level blocks are data blocks). This embodiment of the present application does not limit this.

[0086] Step 504: The data query device learns the spatial distribution information of the multi-dimensional data.

[0087] In this embodiment, the device for querying data learns the spatial distribution information of multidimensional data for multidimensional data. Specifically, based on the distribution characteristics and constraints of the multidimensional data in the multidimensional space, multiple subspace objects of each block of the multidimensional data are determined, thereby obtaining the spatial distribution information of the multidimensional data. The constraints may include one or more of the number of subspace objects, the amount of data of each subspace object, the incremental information of the subspace objects, or the computing performance. The number of subspace objects refers to the maximum and minimum number of subspace objects, the amount of data of the subspace objects refers to the maximum and minimum amount of data included in each subspace object, the incremental information of the subspace objects refers to the maximum increment of the volume of the subspace object after adding one data, and the computing performance refers to the maximum time required to calculate the spatial distribution information.

[0088] For example, Figure 7 As shown in the figure, when the dimensions of the multidimensional data are age (d1 dimension), height (d2 dimension), and salary (d3 dimension), for a block in the multidimensional data, the square represents the division of spatially adjacent data in the block into the same cluster, ultimately generating n clusters. Each cluster can be represented as a subspace object, namely the S1 subspace object to the Sn subspace object (which can be called a multidimensional subspace object group). In this way, multiple subspace objects corresponding to a block can be obtained. The data range of each subspace object in each dimension of all blocks constitutes the spatial distribution information of the multidimensional data. Here, the data range can be represented by an interval from the minimum value to the maximum value. For example, for the S1 subspace object, the data range of the d1 dimension is [a1min, a1max], where a1min refers to the minimum value of the data of the d1 dimension in the S1 subspace object, and a1max refers to the maximum value of the data of the d1 dimension in the S1 subspace object.

[0089] In the above step 504, in this embodiment, the spatial distribution information of the multi-dimensional data can be learned in the following three ways, but not limited to:

[0090] Method 1: The data query device performs dimensionality reduction processing on the multi-dimensional data to obtain the reduced-dimensional data; and determines multiple subspace objects based on the distribution characteristics and constraints of the reduced-dimensional data in the low-dimensional space.

[0091] Among them, dimensionality reduction processing is dimensionality reduction processing. For example, if multi-dimensional data is three-dimensional data, the data after dimensionality reduction processing is one-dimensional, and the low-dimensional space is one-dimensional space.

[0092] In this embodiment, for any block of multidimensional data, the data query device can encode and reduce the dimensionality of the multidimensional data included in the block to obtain the reduced-dimensional data. The data query device can then use the distribution characteristics and constraints of the reduced-dimensional data in the low-dimensional space to determine the multiple subspace objects corresponding to the block. The data query device then merges the subspace objects of all blocks of the multidimensional data to obtain multiple subspace objects of the multidimensional data.

[0093] Specifically, the data query device arranges the data after dimensionality reduction in ascending order, and approximately assumes that the original data with similar values ​​after dimensionality reduction are also spatially adjacent. Figure 8 As shown, a block includes three-dimensional data d1, d2, and d3. The data query device encodes and reduces the dimensionality of the three-dimensional data to obtain one-dimensional data, see Figure 8 As shown in 8(a) in , this one-dimensional data can be called the coded value. Figure 8 The one-dimensional data shown in 8(a) is sorted in ascending order as follows Figure 8 As shown in 8(b).

[0094] The data query device can then merge data with adjacent code values ​​and calculate the volume of the polyhedron formed by the merged data, ultimately ensuring that the number of generated clusters and the cluster volume increment meet the constraints. Specifically, the data query device combines the data with the smallest code value with the data with adjacent code values ​​into a first cluster and calculates the maximum and minimum values ​​of the data corresponding to these two code values ​​in each dimension within the first cluster. Based on the calculated maximum and minimum values, the data query device then calculates the first volume of the polyhedron formed by the data corresponding to these two code values. The data query device then adds the data corresponding to the code value that is closest to these two code values ​​to the first cluster and determines the second volume of the polyhedron formed by the data corresponding to these three code values. The data query device calculates the difference between the second volume and the first volume. If the difference is less than or equal to a first preset threshold, the newly added code value is also added to the first cluster, and the process of adding the closest code value to the first cluster continues. If the difference is greater than a first preset threshold, the code value that is closest to the two code values ​​in the first cluster and the subsequent code value will be clustered instead of being added to the first cluster. At this time, the construction of the first cluster is completed, and the first cluster is a subspace object. The way of constructing a cluster with the code value that is closest to the two code values ​​in the first cluster is the same as the way of constructing the first cluster, which will not be repeated here. In addition, it should be noted that this way of constructing a cluster is only one possible implementation method, and other methods can of course be used. For example, the device for querying data can also consider determining the maximum time of the subspace object, so that the time for determining the subspace object cannot exceed the maximum time. When the device for querying data constructs a subspace object, the data volume of the subspace object cannot be less than the minimum data volume of the subspace object, etc. When the device for querying data constructs a subspace object, the time for determining the subspace object cannot exceed a certain value, etc.

[0095] After clustering, the maximum and minimum values ​​of each cluster's dimensional data are the data ranges for each dimensional data of the cluster, and thus the data ranges for each dimensional data of the subspace objects. This allows us to determine that a block contains multiple subspace objects, and that the block corresponds to the data range indicated by the maximum and minimum values ​​of each subspace object's dimensional data.

[0096] It should be noted that in Method 1, the coding dimensionality reduction can be performed using Morton coding, but other coding dimensionality reduction methods are also possible, and this embodiment does not limit this. Furthermore, in Method 1, since the dimensionality reduction process on multidimensional data leaves only one-dimensional data, clustering the one-dimensional data reduces the computational effort compared to clustering the multidimensional data, speeding up the clustering process and, in other words, speeding up the identification of subspace objects.

[0097] Method 2: The data query device divides the multi-dimensional data into subspaces, and obtains multiple subspace objects based on the distribution characteristics and constraint conditions of the subspaces obtained by the division.

[0098] In this embodiment, when the data range of each dimension of data in the multi-dimensional data is relatively large, there is a wide range of data values ​​or a large floating-point precision, which results in a large amount of data calculation or high calculation complexity during clustering. Therefore, in method 2, the data query device divides the data range of each dimension of the block in the multi-dimensional data into N intervals at equal intervals (N can be preset by the user).

[0099] Then, for each dimension of the block in the multidimensional data, the device querying the data maps all data in the first interval of that dimension to 0, the device querying the data maps all data in the second interval of that dimension to 1, the device querying the data maps all data in the i-th interval of that dimension to i-1, and so on, mapping all data in that dimension to 0 through N-1. In the same way, the device querying the data can map each dimension of data to 0 through N-1. It should be noted here that 0 represents the value 0, i represents the value i, and if i is 5, it represents the value 5.

[0100] For example, Figure 9 As shown, the multi-dimensional data is d1-dimensional, d2-dimensional and d3-dimensional. The data ranges of d1-dimensional, d2-dimensional and d3-dimensional are divided into N intervals at equal intervals. d1min represents the minimum value in the d1-dimensional data, d1max represents the maximum value in the d1-dimensional data, d2min represents the minimum value in the d2-dimensional data, d2max represents the maximum value in the d2-dimensional data, d3min represents the minimum value in the d3-dimensional data, and d3max represents the maximum value in the d3-dimensional data.

[0101] The data query device then maps the N values ​​from each dimension of the data into data point coordinates, thereby obtaining a subspace. The subspace is composed of the data point coordinates of the multidimensional data. The data query device then stores the data point coordinates and the ID of the data corresponding to the data point coordinates as a key-value pair, where the key is the data point coordinate and the value is the ID of the data corresponding to the data point coordinate.

[0102] For example, Figure 10As shown, 32,000 data items are considered a block, with data IDs ranging from 1 to 32,000. The three dimensions of this block are represented by d1, d2, and d3. The data values ​​in each dimension range from 0 to 9,000. With N set to 32, the interval is (9,000 - 0) / 32 = 281.25. This means that data values ​​in the range of 0 to 281.25 (which can be a closed interval on the left and an open interval on the right) are mapped to 0, data values ​​in the range of 281.25 to 281.25*2 are mapped to 1, and so on. After mapping, the data values ​​in each dimension range from 0 to 31. The data point coordinates are (0,0,0), (0,0,1), (0,1,0), ... (31,31,31). Data with IDs of 9, 256, 3150, 5460, etc. are all mapped to the subspace (0,0,0), and no data is mapped to the subspaces (0,0,2) and (0,0,7).

[0103] The data query device then merges the subspaces and obtains the maximum and minimum values ​​of the data in each dimension of the merged subspace. Each merged subspace is referred to as a subspace object. For each subspace object, the maximum and minimum values ​​of the data in each dimension constitute the data range of the subspace object in each dimension.

[0104] Specifically, the process of merging subspaces by the data query device can be as follows: For example, using three-dimensional data as the multi-dimensional data, the data query device determines the volume 1 of the polyhedron consisting of the data corresponding to subspace (0,0,0). The data query device then merges subspace (0,0,0) with subspace (0,1,0) to determine the volume 2 of the polyhedron consisting of the data corresponding to these two subspaces. The data query device determines the difference between volume 2 and volume 1 and determines the difference between the difference and a second preset threshold (the second preset threshold can be preset by the user). If the difference is less than or equal to the second preset threshold, subspace (0,0,0), subspace (0,1,0), and subspace (1,0,0) are merged (in the same manner as the merging of subspace (0,0,0) and subspace (0,1,0)). If the difference is greater than a second preset threshold, the device that queries the data will determine the subspace (0,0,0) as a multidimensional subspace object. The device that queries the data can merge the subspace (0,1,0) with its nearest subspace (the merging method is the same as the merging method of the subspace (0,0,0) and the subspace (0,1,0)), and merge all the subspaces according to this merging method to obtain the subspace objects of each block of the multidimensional data.

[0105] It should be noted that this is only one method for merging subspaces; other methods can also be used. For example, the maximum time required to calculate a subspace object can be used as a constraint, or the minimum and maximum data volumes of a subspace object can be used as constraints. Furthermore, the second and first preset thresholds are both preset by the user and may or may not be equal.

[0106] Compared to spatially clustering unmapped data in each block, using the second approach above, dimensional compression of the data into a subspace reduces the amount of data required for computation, thereby improving query data filtering performance. Furthermore, the second approach only requires that the data in each dimension of each block be sortable, meaning that dimensional compression encoding can be performed to obtain a subspace, making it widely applicable.

[0107] In a third approach, the data query device clusters the multi-dimensional data in terms of volume increments to obtain a plurality of subspace objects.

[0108] In this embodiment, the device for querying data divides multiple data in the multi-dimensional data into a cluster (which can be called the first cluster), and then adds data to the first cluster. If the difference between the volume of the first cluster after adding the data and the volume of the first cluster without adding the data is less than a certain value, the data is added to the first cluster. If the difference between the volume of the first cluster after adding the data and the volume of the first cluster without adding the data is greater than or equal to the value, the first cluster is established. The device for querying data creates a new cluster (which can be called the second cluster) and performs the same processing as the first cluster until all the data are traversed to obtain the final cluster. The device for querying data regards each cluster as a subspace object. Then the data range of each dimension of each subspace object is determined. Of course, other constraints can also be used here at the same time, such as computing performance, the amount of data of each subspace object, etc.

[0109] In addition, each subspace object should also include an identifier for indicating the block to which the subspace object belongs. The identifier is used to find the block in the subsequent data query process.

[0110] After the subspace objects of each block are determined in step 504, the subspace objects of a single block of the multi-dimensional data are as follows: Figure 11As shown, the subspace objects within a single block (P1) have dimensions of d1, d2, and d3, and the subspace objects are represented by S1, S2, S3, S4, etc. P1-S1 represents the first subspace object S1 in block P1, P1-S2 represents the second subspace object S2 in block P1, P1-S3 represents the third subspace object S3 in block P1, and P1-S4 represents the fourth subspace object S4 in block P1. In the d1 dimension, all subspace objects of block P1 can be filtered, in the d2 dimension, all subspace objects of block P1 can be filtered, and in the d3 dimension, all subspace objects of block P1 can be filtered.

[0111] exist Figure 11 On the basis of Figure 12 As shown, in an embodiment of the present application, all subspace objects (which can be called global subspace objects) of multiple blocks of multi-dimensional data are also provided, and the multiple blocks are block P1, block P2, block P3 and block P4. The dimensions of the data of block P1, block P2, block P3 and block P4 are d1 dimension, d2 dimension and d3 dimension. P1-S1 represents the first subspace object S1 in block P1, P1-S2 represents the second subspace object S2 in block P1, P1-S3 represents the third subspace object S3 in block P1, and P1-S4 represents the fourth subspace object S4 in block P1. P2-S1 represents the first subspace object S1 in block P2, P2-S2 represents the second subspace object S2 in block P2, P2-S3 represents the third subspace object S3 in block P2, and P2-S4 represents the fourth subspace object S4 in block P2. P3-S1 represents the first subspace object S1 in block P3, P3-S2 represents the second subspace object S2 in block P3, P3-S3 represents the third subspace object S3 in block P3, and P3-S4 represents the fourth subspace object S4 in block P3. P4-S1 represents the first subspace object S1 in block P4, P4-S2 represents the second subspace object S2 in block P4, P4-S3 represents the third subspace object S3 in block P4, and P4-S4 represents the fourth subspace object S4 in block P4.

[0112] In this way, when performing data search subsequently, all subspace objects of blocks P1, P2, P3, and P4 can be filtered in the d1 dimension, all subspace objects of blocks P1, P2, P3, and P4 can be filtered in the d2 dimension, and all subspace objects of blocks P1, P2, P3, and P4 can be filtered in the d3 dimension.

[0113] exist Figure 12 In the , each subspace object stores an identifier, which is used to indicate the block to which the subspace object belongs.

[0114] In one possible implementation, in order to increase the speed of querying data, the data ranges of multiple subspace objects in each dimension in the spatial distribution information of multidimensional data can constitute an index. For the subspace object of any dimensional data in the multidimensional data, the index of the data range of the subspace object in each dimension can be represented by a segment tree. Specifically, for any dimensional data, the value range of each subspace object of each block of the dimension will be attached to the segment tree corresponding to the dimension. The segment tree is a binary search tree that stores interval information. Each node of the segment tree includes information such as the left endpoint and the right endpoint of the interval. For example, Figure 13 As shown, Figure 12 The global subspace object in the d1 dimension is used as an example to construct a segment tree representing the data corresponding to the dimension. The data query device flattens the data range of all subspaces of the d1 dimension to obtain all endpoint information. All endpoint information constitutes some leaf nodes of the last layer of the segment tree, that is, Figure 13 The gray circles at the last level of the segment tree are shown. The values ​​of the other leaf nodes at the last level of the segment tree are determined based on these leaf nodes, specifically their adjacent left leaf nodes (which can be called sibling nodes). The data query device then constructs parent nodes based on the leaf nodes at the last level, sequentially upwards, until it reaches the root node.

[0115] For example, Figure 14 In the data range of each subspace object in dimension d1, the endpoints are expanded in ascending order, such as P4-S4: [5,10], P2-S3: [7,12], P1-S2: [15,19], P4-S3: [17,22], P4-S1: [24,30], P3-S4: [28,35], P1-S4: [32,42], P2-S2: [38,47], P1-S1: [49,55], P4-S2: [52,57], P2-S4: [59,65], P2-S1: [63,69], and P1-S3: [71,80]. The device that queries the data regards the endpoints expanded in ascending order as some leaf nodes of the last level of the segment tree, as shown by the nodes indicated by the shaded circles of the last level. The device that queries the data then determines another portion of the leaf nodes of the last level of the segment tree. For each leaf node in the other part of the leaf nodes ( Figure 14As shown by the white circle in the figure), the device for querying data determines that the value of the leaf node is a numerical value greater than the value of the left sibling node (the leaf node adjacent to the left of the leaf node) (the left endpoint 5 of P4-S4 is the first leaf node, and the value of the leaf node adjacent to the right is 6 (greater than 5), the left endpoint of P2-S3 is 7, and the value of the leaf node adjacent to the right is 8 (greater than 7), and all leaf nodes are generated in sequence). The device for querying data merges upwards by range to construct the second-to-last layer of nodes in the segment tree, and merges upwards in sequence until the root node. The segment tree of the global subspace object of the d1 dimension is constructed. In this way, when querying the d1 dimension, if the data is greater than 49, directly query the node on the right, otherwise query the node on the left.

[0116] In this way, through Figure 14 In the structure shown, the data query device can retrieve multiple matching subspace objects from one dimension, and then filter the multiple matching subspace objects in other dimensions without the need for a second query of subspace objects that do not match the first dimension, so data query can be performed efficiently.

[0117] like Figure 15 As shown, the embodiment of the present application also provides a process for querying data, which is as follows:

[0118] Step 1501: The data query device receives a data query request.

[0119] In this embodiment, when there is data that needs to be queried, the user inputs a data query request to the device for querying data (the data query request can be input through an interface or through an SQL statement), and the data query request specifies the data range of multiple dimensional data to be queried.

[0120] For example, for the above data table t_salary, the data query request can be:

[0121]

[0122] The above data query request can be expressed as: In the data table t_salary, retrieve the data with height>=1.62&&height<=1.64, age>=27&&age<=38, and salary>=200&&salary<=300. For this data query request, the data ranges for the multiple dimensions to be queried are height>=1.62&&height<=1.64, age>=27&&age<=38, and salary>=200&&salary<=300.

[0123] In step 1502 , the data query device obtains spatial distribution information of the multi-dimensional data. The spatial distribution information records the data range of multiple subspace objects associated with the multi-dimensional data in each dimension.

[0124] The multi-dimensional data comes from one or more data tables. The multi-dimensional data includes multiple blocks, each block corresponds to multiple subspace objects, and each subspace object corresponds to a data range in each dimension.

[0125] In this embodiment, the data query device can obtain spatial distribution information of the multidimensional data. This spatial distribution information records the data ranges of multiple subspace objects associated with the multidimensional data in each dimension. Specifically, the data query device can store the spatial distribution information of the multidimensional data itself, or it can obtain the spatial distribution information of the multidimensional data from another device.

[0126] In step 1503 , the data query device determines, in the multi-dimensional data, blocks that satisfy the data ranges of the multiple dimensions specified in the data query request based on the spatial distribution information.

[0127] In this embodiment, the device for querying data may determine blocks of data ranges of multiple dimensions to be queried in the multi-dimensional data according to the spatial distribution information.

[0128] In step 1504 , the data query device searches for data in the determined block that satisfies the data ranges of multiple dimensions specified in the data query request.

[0129] In this embodiment, the data query device can use the data ranges of the multiple dimensions to be queried in the determined block to find the data indicated by the data ranges of the multiple dimensions. The data query device can feed the found data back to the device that sent the data query request so that the device can obtain the queried data in a timely manner.

[0130] If the data indicated by the data ranges of the multiple dimensions is not found in the determined block, a feedback is sent to the device that sent the data query request that no data that meets the data query request exists.

[0131] In a possible implementation, in step 1503, the process of determining the block may be:

[0132] The data query device determines one or more subspace objects from multiple subspace objects that meet the data range of multiple dimensions specified in the data query request based on the spatial distribution information of the multi-dimensional data; and uses the block to which the determined subspace object belongs as the block that meets the data range of multiple dimensions specified in the data query request.

[0133] In this embodiment, the device for querying data can use the data range of each dimension of the data range of multiple dimensions specified in the data query request to match the spatial distribution information of the multi-dimensional data, and filter out the subspace objects that do not include the data range of the multiple dimensions. The remaining subspace objects are the subspace objects where the data range of the multiple dimensions may exist. The device for querying data determines the blocks to which the remaining subspace objects belong as blocks that meet the data range of the multiple dimensions. Specifically, the device for querying data can match the data range of each dimension of the multiple dimensions with the spatial distribution information of the multi-dimensional data to obtain the subspace objects satisfied by the data range of each dimension. The device for querying data takes the intersection of the subspace objects satisfied by the data range of each dimension to obtain the subspace objects that meet the data range of multiple dimensions. The device for querying data determines the blocks indicated by the identifiers of these subspace objects as blocks that meet the data range of multiple dimensions.

[0134] In a possible implementation, when no subspace object is found using the spatial distribution information of multi-dimensional data, the processing may be:

[0135] When no subspace object that satisfies the data ranges of multiple dimensions specified in the data query request is found from the multiple subspace objects according to the spatial distribution information of the multi-dimensional data, the data querying device feedbacks that no data that satisfies the data query request exists.

[0136] In this embodiment, when the device for querying data fails to find a subspace object that meets the data range of multiple dimensions among multiple subspace objects of the multi-dimensional data, it can feedback to the device that sent the data query request that there is no data that meets the data query request, so that the device that sent the data query request can understand the query results in a timely manner.

[0137] In one possible implementation, the multi-dimensional data may be divided into multiple levels. For example, the multi-dimensional data is divided into a first level and a second level. The corresponding first level may include multiple first-level blocks, and the second level may include multiple second-level blocks. The second-level blocks are obtained by dividing the first-level blocks, and multiple second-level blocks constitute a first-level block. The processing of step 1503 may be:

[0138] The device for querying data obtains spatial distribution information of first-level blocks of multi-dimensional data, where the spatial distribution information of the first-level blocks records the data ranges of multiple subspace objects associated with the data of the first-level blocks in each dimension; obtains spatial distribution information of second-level blocks of target blocks, where the spatial distribution information of the second-level blocks records the data ranges of multiple subspace objects associated with the data of the second-level blocks in each dimension; the device for querying data determines, in the first-level blocks, a target block that satisfies the data ranges of the multiple dimensions specified in the data query request based on the spatial distribution information of the first-level blocks; and determines, in the target blocks, a second-level block that satisfies the data ranges of the multiple dimensions specified in the data query request based on the spatial distribution information of the second-level blocks.

[0139] In this embodiment, the data query device can obtain spatial distribution information of first-level blocks and second-level blocks of multi-dimensional data. The spatial distribution information of the first-level blocks can record the data ranges of multiple subspace objects associated with the data of the first-level blocks in each dimension. The spatial distribution information of the second-level blocks can record the data ranges of multiple subspace objects associated with the data of the second-level blocks in each dimension.

[0140] The device for querying data can first determine one or more subspace objects in the first-level block that meet the data range of multiple dimensions in the spatial distribution information of the first-level block. The block corresponding to the identifier of the one or more subspace objects is determined as the target block. The device for querying data then determines the second-level blocks included in the target block, uses the spatial distribution information of these second-level blocks, and determines one or more subspace objects in these second-level blocks that meet the data range of multiple dimensions, and determines the second-level blocks corresponding to the identifier of the one or more subspace objects as the second-level blocks that meet the data range of multiple dimensions specified in the data query request.

[0141] In this way, the first-level blocks are first filtered to remove the first-level blocks that do not include the data range of multiple dimensional data to be queried, and then the second-level blocks in the remaining first-level blocks are filtered, which can speed up the data query.

[0142] It should be noted that the first-level blocks can be file blocks, and the second-level blocks can be data blocks. The above only divides into two levels. During the data query process, multiple levels can be divided according to actual needs.

[0143] In one possible implementation, after step 1504, if the spatial distribution information of the multi-dimensional data is no longer used subsequently, the spatial distribution information of the multi-dimensional data may be deleted in order to save storage space of the device for querying data. Specifically, for the above-mentioned data table t_salary, the device for querying data may receive a deletion request, which may be expressed as: delete the spatial distribution information of the multi-dimensional data of the data table t_salary. In addition, for the case where the spatial distribution information of the multi-dimensional data includes the spatial distribution information of the multi-dimensional data of the first-level block and the spatial distribution information of the multi-dimensional data of the second-level block, it is also necessary to indicate which type of spatial distribution information is to be deleted. For example, the deletion request may carry an identifier, which is used to indicate whether the spatial distribution information to be deleted is the first-level block or the second-level block.

[0144] In a possible implementation, when data in a data table is incrementally updated, it is only necessary to add the incrementally updated data to the original subspace object, and the cost of relearning the spatial distribution information of the multi-dimensional data is low.

[0145] Through the embodiments of the present application, for data that is spatially adjacent but relatively dispersed in physical memory, by learning the spatial distribution information of this data, the spatial distribution information of multi-dimensional data is constructed, which can effectively filter queries and reduce invalid reading and analysis of a large number of blocks, thereby reducing invalid reading of the entire block of disk input and output (IO), and improving storage performance.

[0146] It should be noted that, when describing the user in the above embodiment, the description refers to an operation and maintenance personnel. In fact, it may not be an operation and maintenance personnel, but refers to a user who has the authority to use the above-mentioned device for querying data.

[0147] It should also be noted that the multi-dimensional data is only logically divided into blocks, the spatial distribution information (i.e., subspace objects) corresponding to the logically divided blocks is learned, and the correspondence between the blocks and the subspace objects of the blocks is established. The data of the blocks after the multi-dimensional data is divided are not stored together.

[0148] Figure 16 This is a structural diagram of the device for querying data provided by the embodiment of the present application. The device can be implemented as part or all of the device through software, hardware or a combination of both. The device provided by the embodiment of the present application can implement the embodiment of the present application. Figure 15 The process includes: a receiving module 1610, an obtaining module 1620, a determining module 1630 and a searching module 1640, wherein:

[0149] Receiving module 1610 is used to receive a data query request, wherein the data query request specifies a data range of multiple dimensions, and can be specifically used to execute the receiving function of step 1501 and the implicit steps contained therein;

[0150] Acquisition module 1620 is used to acquire spatial distribution information of the multi-dimensional data. The spatial distribution information records the data ranges of multiple subspace objects associated with the multi-dimensional data in each dimension. Specifically, it can be used to execute the acquisition function of step 1502 and the implicit steps contained therein;

[0151] Determination module 1630 is used to determine, in the multi-dimensional data, blocks that satisfy the data ranges of the multiple dimensions specified in the data query request based on the spatial distribution information. Specifically, it can be used to perform the determination function of step 1503 and the implicit steps contained therein;

[0152] The search module 1640 is used to search for data that meets the data range of multiple dimensions specified in the data query request in the determined block, and can be specifically used to execute the search function of step 1504 and the implicit steps contained therein.

[0153] In one possible implementation, the determining module 1630 is configured to:

[0154] Determining, from the plurality of subspace objects, one or more subspace objects that satisfy the data ranges of the plurality of dimensions specified by the data query request according to the spatial distribution information of the multi-dimensional data;

[0155] The block to which the determined subspace object belongs is used as the block that satisfies the data range of multiple dimensions specified by the data query request.

[0156] In a possible implementation, the search module 1640 is further configured to:

[0157] When no subspace object that satisfies the data ranges of the multiple dimensions specified by the data query request is found from the multiple subspace objects according to the spatial distribution information of the multi-dimensional data, feedback is given that no data that satisfies the data query request exists.

[0158] In one possible implementation, the acquisition module 1620 is configured to:

[0159] Acquire spatial distribution information of a first-level block of the multi-dimensional data, where the spatial distribution information of the first-level block records data ranges of multiple subspace objects associated with the data of the first-level block in each dimension;

[0160] Acquire spatial distribution information of a second-level block of the multi-dimensional data, where the spatial distribution information of the second-level block records data ranges of multiple subspace objects associated with the data of the second-level block in each dimension;

[0161] The determination module 1630 is configured to:

[0162] determining, in the first-level blocks, a target block that satisfies the data ranges of the multiple dimensions specified by the data query request according to the spatial distribution information of the first-level blocks;

[0163] According to the spatial distribution information of the second-level blocks, second-level blocks that satisfy data ranges of multiple dimensions specified by the data query request are determined in the target block.

[0164] In a possible implementation, the determining module 1630 is further configured to:

[0165] The multiple subspace objects of the multidimensional data are determined according to the distribution characteristics and constraint conditions of the multidimensional data in the multidimensional space.

[0166] In a possible implementation, the determining module 1630 is further configured to:

[0167] Perform dimensionality reduction processing on multi-dimensional data to obtain data after dimensionality reduction processing;

[0168] According to the distribution characteristics and constraints of the data after dimensionality reduction in the low-dimensional space, multiple subspace objects are determined.

[0169] In a possible implementation, the determining module 1630 is further configured to:

[0170] Clustering of volume increments is performed on multi-dimensional data to obtain multiple subspace objects.

[0171] In a possible implementation, the determining module 1630 is further configured to:

[0172] The multi-dimensional data is divided into subspaces, and multiple subspace objects are obtained based on the distribution characteristics and constraints of the subspaces obtained by the division.

[0173] In a possible implementation manner, the data ranges of the multiple subspace objects in each dimension constitute indexes of the multiple subspace objects.

[0174] The division of modules in the above embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present application may be integrated into a single processor, exist physically separately, or two or more modules may be integrated into a single module. The above integrated modules may be implemented in the form of hardware or software functional modules.

[0175] It should be noted that the data query device provided in the above embodiment is only illustrated by the division of the above functional modules when querying data. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data query device provided in the above embodiment and the data query method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0176] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a server or terminal, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a server or terminal or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk and a tape, etc.), an optical medium (e.g., a digital video disk (DVD), etc.), or a semiconductor medium (e.g., a solid-state hard disk, etc.).

Claims

1. A method for querying data, characterized in that: The method comprises: receiving a data query request, wherein the data query request specifies a data range in multiple dimensions; Determine multiple subspace objects for each block based on the distribution characteristics and constraints of the multidimensional data in the multidimensional space to obtain spatial distribution information of the multidimensional data, wherein the spatial distribution information records the data ranges of the multiple subspace objects associated with the multidimensional data in each dimension, and the subspace objects are obtained by spatially clustering the multidimensional data; Determining, in the multi-dimensional data, blocks that satisfy data ranges of the multiple dimensions based on the spatial distribution information; In the determined block, data that satisfies the data ranges of the multiple dimensions is searched for.

2. The method according to claim 1, characterized in that The determining, in the multi-dimensional data, blocks that satisfy data ranges of the multiple dimensions according to the spatial distribution information includes: determining, from the plurality of subspace objects, one or more subspace objects that satisfy the data ranges of the plurality of dimensions according to the spatial distribution information; The block to which the determined subspace object belongs is used as the block that satisfies the data range of the multiple dimensions.

3. The method according to claim 2, characterized in that The method further comprises: When no subspace object that satisfies the data ranges of multiple dimensions is found from the multiple subspace objects according to the spatial distribution information, it is fed back that no data that satisfies the data query request exists.

4. The method according to claim 1, wherein The obtaining of spatial distribution information of multi-dimensional data includes: Acquire spatial distribution information of a first-level block of the multi-dimensional data, where the spatial distribution information of the first-level block records data ranges of multiple subspace objects associated with the data of the first-level block in each dimension; Acquire spatial distribution information of a second-level block of the multi-dimensional data, where the spatial distribution information of the second-level block records data ranges of multiple subspace objects associated with the data of the second-level block in each dimension; The determining, in the multi-dimensional data, blocks that satisfy data ranges of the multiple dimensions according to the spatial distribution information includes: determining, in the first-level blocks, a target block that satisfies the data ranges of the multiple dimensions according to the spatial distribution information of the first-level blocks; According to the spatial distribution information of the second-level blocks, a second-level block satisfying the data range of the multiple dimensions is determined in the target block.

5. The method according to claim 1, characterized in that The step of determining multiple subspace objects of each block according to the distribution characteristics and constraints of the multidimensional data in the multidimensional space includes: Performing dimensionality reduction processing on the multi-dimensional data to obtain data after dimensionality reduction processing; According to the distribution characteristics and constraints of the data after dimensionality reduction in the low-dimensional space, multiple subspace objects of each block are determined.

6. The method according to claim 1, characterized in that The step of determining multiple subspace objects of each block according to the distribution characteristics and constraints of the multidimensional data in the multidimensional space includes: Clustering of volume increments is performed on the multi-dimensional data to obtain multiple subspace objects of each block.

7. The method according to claim 1, characterized in that The step of determining multiple subspace objects of each block according to the distribution characteristics and constraints of the multidimensional data in the multidimensional space includes: Subspace division is performed on the multi-dimensional data, and multiple subspace objects of each block are obtained based on the distribution characteristics and constraint conditions of the subspace obtained by the division.

8. The method according to any one of claims 1 to 7, characterized in that The data ranges of the multiple subspace objects in each dimension constitute indexes of the multiple subspace objects.

9. A device for querying data, characterized in that: The device comprises: A receiving module, configured to receive a data query request, wherein the data query request specifies a data range of multiple dimensions; A determination module, configured to determine a plurality of subspace objects of each block according to distribution characteristics and constraints of the multidimensional data in the multidimensional space; an acquisition module, configured to acquire spatial distribution information of multidimensional data, wherein the spatial distribution information records the data ranges of multiple subspace objects associated with the multidimensional data in each dimension, wherein the subspace objects are obtained by spatially clustering the multidimensional data; The determination module is further configured to determine, in the multi-dimensional data, blocks that satisfy data ranges of the multiple dimensions based on the spatial distribution information; The search module is used to search for data that meets the data range of the multiple dimensions in the determined block.

10. The device according to claim 9, characterized in that The determining module is configured to: determining, from the plurality of subspace objects, one or more subspace objects that satisfy the data ranges of the plurality of dimensions according to the spatial distribution information; The block to which the determined subspace object belongs is used as the block that satisfies the data range of the multiple dimensions.

11. The device according to claim 10, characterized in that The search module is further used to: When no subspace object that satisfies the data ranges of the multiple dimensions is found from the multiple subspace objects according to the spatial distribution information, it is fed back that no data that satisfies the data query request exists.

12. The device according to claim 9, characterized in that The acquisition module is used to: Acquire spatial distribution information of a first-level block of the multi-dimensional data, where the spatial distribution information of the first-level block records data ranges of multiple subspace objects associated with the data of the first-level block in each dimension; Acquire spatial distribution information of a second-level block of the multi-dimensional data, where the spatial distribution information of the second-level block records data ranges of multiple subspace objects associated with the data of the second-level block in each dimension; The determining module is configured to: determining, in the first-level blocks, a target block that satisfies the data ranges of the multiple dimensions according to the spatial distribution information of the first-level blocks; According to the spatial distribution information of the second-level blocks, a second-level block satisfying the data range of the multiple dimensions is determined in the target block.

13. The device according to claim 9, characterized in that The determining module is further configured to: Performing dimensionality reduction processing on the multi-dimensional data to obtain data after dimensionality reduction processing; According to the distribution characteristics and constraints of the data after dimensionality reduction in the low-dimensional space, multiple subspace objects of each block are determined.

14. The device according to claim 9, characterized in that The determining module is further configured to: Clustering of volume increments is performed on the multi-dimensional data to obtain multiple subspace objects of each block.

15. The device according to claim 9, characterized in that The determining module is further configured to: Subspace division is performed on the multi-dimensional data, and multiple subspace objects of each block are obtained based on the distribution characteristics and constraint conditions of the subspace obtained by the division.

16. The device according to any one of claims 9 to 15, characterized in that The data ranges of the multiple subspace objects in each dimension constitute indexes of the multiple subspace objects.

17. A computing device, characterized in that The computing device includes a processor and a memory; The processor executes the instructions stored in the memory, so that the computing device implements the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and the instructions instruct a computing device to execute the method of any one of claims 1 to 8.

Images