A data processing system and apparatus
By introducing a multi-index node architecture into the database system, the coordinating node decomposes the query task into parallel processing, which solves the problem of low efficiency in filtering indexes by the coordinating node, and achieves efficient data query and stable index service.
Patent Information
- Application Number
- CN202211096488.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-06-10
- Filing Date
- 2022-09-06
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-09-06
AI Technical Summary
In existing technologies, database coordinating nodes need to filter a large number of indexes when querying data, resulting in low query efficiency. Furthermore, failure of the coordinating node may cause indexes to become unavailable, affecting system stability.
The system adopts a multi-index node architecture. The coordinating node decomposes the query task into multiple first query tasks and sends them to multiple index nodes in parallel. The index nodes perform index filtering operations in parallel, supporting multi-level index services, reducing read volume and filtering operations, and ensuring that index information is still available when the coordinating node fails.
It improves data query efficiency, increases index filtering efficiency by N times, reduces memory limitations of coordinating nodes, and enhances system stability and compatibility.
Smart Images

Figure CN117216083B_ABST
Abstract
Description
[0001] This application claims priority to Chinese Patent Application No. 202210657406.1, filed on June 10, 2022, entitled “A Data Processing System”, the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of computer technology, and in particular to a data processing system and apparatus. Background Technology
[0003] A database is a repository for data tables, used to store data tables. A data table is a common form of data storage, consisting of horizontal rows and vertical columns.
[0004] Figure 1 The diagram illustrates the architecture of an existing system, which includes a coordinating node, worker nodes, and storage nodes. The storage nodes store data tables. Before performing operations such as querying, modifying, and deleting data tables, the data to be manipulated must be retrieved from the data table. In existing technologies, the coordinating node first uses the data table's index to determine the region in the data table to be queried. Then, the coordinating node instructs the worker nodes to load that region into memory for querying, thereby retrieving the data to be manipulated from that region.
[0005] In the above methods, the coordinating node may need to filter a large number of indexes, resulting in low data query efficiency, and the failure of the coordinating node may cause all indexes to become unavailable. Summary of the Invention
[0006] This application provides a data processing system for improving index processing efficiency, thereby improving data query efficiency.
[0007] In a first aspect, embodiments of this application provide a data processing system, which includes a coordination node, multiple index nodes, and multiple working nodes.
[0008] The coordinating node can receive query requests from clients, which include information about the target object and query conditions. The target object is divided into multiple shards, and these shards are stored on at least one storage node.
[0009] The coordinating node generates multiple first query tasks based on the query request and sends these tasks to multiple index nodes. The target object is divided into multiple data ranges, and each index node manages one data range of the target object. Taking one index node as an example, the first index node, based on the received first query tasks, determines the target shards within its managed first data range that include the target data, and sends the target shard information to the coordinating node. The target data is the data that meets the query conditions. The coordinating node receives the target shard information from the multiple index nodes, generates second query tasks for each target shard, and schedules multiple worker nodes to execute these second query tasks.
[0010] Through the above design, the coordinating node generates multiple first query tasks based on query requests and sends these tasks to multiple index nodes. These index nodes then execute the index filtering operations in parallel, improving data processing efficiency. Furthermore, if the coordinating node fails, the index information remains available, and the index nodes' filtering operations based on this information are unaffected by the coordinating node. Moreover, compared to existing solutions, this system overcomes the memory limitations of a single node (coordinating node), potentially increasing indexing capacity by up to N times (if the system includes N or more index nodes).
[0011] In one possible implementation, the first index node corresponds to the index information of the corresponding shard within the first data range; when determining the target shard, the first index node is specifically used to determine the target shard based on the index information of the corresponding shard within the first data range.
[0012] Through the above design, since the index information retains a set of numerical characteristics of the data, during data querying, the index node can conveniently use the index information to determine whether all, some, or none of the rows of data corresponding to the shards of the target object meet the query conditions.
[0013] In one possible implementation, the target object is divided into multiple data blocks, each data block corresponding to a shard; the first query task includes information about the query object, and the data of the target object includes the data of the query object.
[0014] The first index node, within the first data range managed by the first index node, includes the target shard of the data queried by the query request. Specifically, it is used to: use the index information of the data block corresponding to the query object within the first data range to filter out at least one target data block, where the target data block is the target shard.
[0015] With the above design, multiple index nodes perform the filtering of index information of data blocks in parallel. Compared with the existing method of filtering index information by coordinating nodes, this system can improve the filtering efficiency by N times.
[0016] In one possible implementation, the target object is divided into multiple files, each file is divided into multiple data blocks, and each data block corresponds to a shard; the first query task includes information about the query object, and the data of the target object includes the data of the query object;
[0017] The first index node determines the target shards that include the data queried by the query request within the first data range managed by the first index node. Specifically, it is used to: use the index information of the files corresponding to the target objects within the first data range to filter out at least one target file, each target including the data queried by the query request; and use the index information of the data blocks corresponding to each target file to filter out at least one target data block, the target data block being the target shard.
[0018] The above design supports the creation of multi-level indexes, providing index services with various granularities and levels. Building upon the improved efficiency of index filtering by parallel execution of index information filtering across multiple index nodes, each index node can perform multi-granularity filtering operations through multi-level indexes, reducing the amount of index reads and filtering operations.
[0019] In one possible implementation, the plurality of files are divided into a plurality of partitions, and the target partition is the partition among the plurality of partitions that includes the target fragment;
[0020] The query object includes the target partition; or
[0021] The query object includes the multiple files corresponding to the target object; or
[0022] The query object includes at least one file corresponding to the target partition; or
[0023] The query object includes at least one file belonging to the target object within the first data range; or
[0024] The query object includes at least one file belonging to the target partition within the first data range.
[0025] The above design divides the target object into multiple partitions, and data management and storage are based on these partitions. The coordinating node can determine the target partition from one or more partitions corresponding to the target object. The target partition is the partition that satisfies some or all of the query conditions, or in other words, the target partition is the partition that includes the target shard. In this way, the index node can filter only the files within the target partition, without needing to filter all files corresponding to the target object, thus narrowing the range of files that need to be filtered and improving data query efficiency. Alternatively, the coordinating node can determine the files of the target object managed by each index node, which can save the computing power of the index nodes and improve the performance of the indexing service.
[0026] In one possible implementation, the plurality of files are divided into a plurality of partitions, and the target partition is the partition among the plurality of partitions that includes the target fragment;
[0027] The query object includes at least one data block contained in at least one file corresponding to the target object; or
[0028] The query object includes at least one data block contained in at least one file corresponding to the target partition; or
[0029] The query object includes at least one data block comprised of at least one file belonging to the target object within the first data range; or
[0030] The query object includes at least one data block comprised of at least one file belonging to the target partition within the first data range.
[0031] In one possible implementation, each shard and its corresponding index information are stored independently.
[0032] The above design decouples data write operations from index creation, creating the index only after the data is written, thus shortening the time required for data writing, improving data write efficiency, and providing flexibility in index creation.
[0033] In one possible implementation, the first worker node is one of a plurality of worker nodes; a data block comprises multiple row groups, and each row group comprises multiple data rows;
[0034] When the first working node executes the multiple second query tasks, it is specifically used for: the first working node sending a push-down task to the first index node, the push-down task indicating to query data that meets the query conditions in at least one target data block; the first index node using the index information of the row group corresponding to each target data block belonging to the first data range in one or more target data blocks to filter out at least one target row group, the target row group being a row group that includes data that meets the query conditions; the first index node reading the data of the target row group and filtering out the data that meets the query conditions.
[0035] Through the above design, the first working node pushes some query tasks down to the first index node for processing. The first index node determines the target row group, and only needs to read the data of the target row group and filter the data of the target row group. It does not need to read and filter all row groups of the target data block, thus reducing network transmission volume and filtering operations.
[0036] In one possible implementation, the first index node and the storage node are deployed on the same physical machine. Alternatively, the first index node and the worker node are deployed on the same physical machine.
[0037] With the above design, the first index node and storage node can be deployed on a physical machine or the first index node and worker node can be deployed on a physical machine without adding an additional host. This can be implemented on the hardware basis of the existing system architecture, and has stronger compatibility, wider application scope and applicability.
[0038] Secondly, embodiments of this application also provide a computing device that implements the function of the first index node in the method example of the first aspect described above. The beneficial effects are described in the first aspect description and will not be repeated here. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function. In one possible design, the device structure includes a receiving module, a processing module, and a sending module. These modules can perform the corresponding functions of the first index node in the method example of the first aspect described above; see the detailed description in the method example for details, which will not be repeated here.
[0039] Thirdly, this application also provides a computing device, the device including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor executing program instructions in the memory to execute the method of executing the first inode provided in the first aspect or any possible implementation of the first aspect. The memory is coupled to the processor and stores program instructions and data necessary for performing data processing.
[0040] Fourthly, this application also provides a computing device, which includes a processor and a memory, and may further include a communication interface. The processor executes program instructions in the memory to perform the method executed by the first inode in the first aspect or any possible implementation of the first aspect. The memory is coupled to the processor and stores program instructions and data necessary for performing data backup. The communication interface is used to communicate with other devices, such as receiving a first query task, or sending a first task result.
[0041] Fifthly, this application provides a computer-readable storage medium that, when executed by a computing device, allows the computing device to execute the method executed by the first inode in the first aspect or any possible implementation thereof. The storage medium stores a program. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).
[0042] Sixthly, this application provides a computing device program product, which includes computer instructions that, when executed by a computing device, enable the computing device to perform the method executed by the first inode in the first aspect or any possible implementation thereof. The computer program product can be a software installation package, and when the method provided in the first aspect or any possible implementation thereof is required, the computer program product can be downloaded and executed on the computing device.
[0043] In a seventh aspect, this application also provides a chip for executing a software program to implement the method of executing the first index node in the first aspect or various possible implementations of the first aspect.
[0044] For the beneficial effects of any of the implementation methods in aspects two through seven mentioned above, please refer to the description of aspect one, which will not be repeated here. Attached Figure Description
[0045] Figure 1 A schematic diagram of a system architecture;
[0046] Figure 2A This is a schematic diagram of a first possible system architecture provided for an embodiment of this application;
[0047] Figure 2B This is a schematic diagram of a second possible system architecture provided for an embodiment of this application;
[0048] Figure 2CThis is a schematic diagram of a third possible system architecture provided in the embodiments of this application;
[0049] Figure 3 A schematic diagram of a file system architecture provided for an embodiment of this application;
[0050] Figure 4 This is a schematic diagram of the structure of an ORC file;
[0051] Figure 5 This application provides a schematic diagram of an index creation process.
[0052] Figure 6 A schematic diagram of an index hierarchy provided for an embodiment of this application;
[0053] Figure 7 A flowchart illustrating a data processing method provided in an embodiment of this application;
[0054] Figure 8 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;
[0055] Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation
[0056] To facilitate understanding, some technical terms used in the embodiments of this application will be explained first.
[0057] (1) A data table is a collection of two-dimensional arrays, consisting of vertical columns and horizontal rows. Columns consist of similar information, each column is also called a field, and the title of each column is called the field name; rows include several columns of information items, and a row of data is called a record. For example, a sales table (as shown in Table 1 below) includes columns such as product identifier, sales volume, and sales date, where product identifier, sales volume, and sales date are the field names. A row of data in this sales table is a record, and a record includes a product identifier, sales volume, and sales date. Table 1 provides an example of some records in a sales table.
[0058] Table 1.saleTbl
[0059] ID (Product Identifier) Sales volume Data (Date) 1 100 2022.7.17 2 150 2022.7.17 1 100 2022.7.17 3 200 2022.7.18 … … …
[0060] (2) Table indexes are used to store the statistical characteristics (or numerical characteristics) of data in the table. For example, indexes of type min / max are mainly used to store the maximum (max) and minimum (min) values of records in a specified column of a certain storage area in the table.
[0061] (3) A file system is a structured form of data file storage and organization. All data in a computer consists of 0s and 1s, and the series of 0s and 1s stored on the hardware medium are completely indistinguishable and unmanageable to us. Therefore, I use the concept of "file" to organize this data. Data used for the same purpose is grouped into different types of files according to the structural requirements of different applications. Different suffixes are usually used to represent different types, and this application gives each file a name that is easy to understand and remember. When there are many files, this application groups these files according to a certain partitioning method, and each group of files is placed in the same directory (or folder). Furthermore, in addition to files, directories can also have subdirectories (called subdirectories or subfolders), and all files and directories form a tree structure. This tree structure has a specific name: File System. There are many types of file systems, the most common being Windows' FAT / FAT32 / NTFS, and Linux's EXT2 / EXT3 / EXT4 / XFS / BtrFS, etc. To facilitate searching, directories are listed level by level from the root node down to the file itself. The names of these directories, subdirectories, and files are concatenated using special characters (e.g., "\" in Windows / DOS, " / " in Unix-like systems). This string of characters is called a file path, such as " / etc / systemd / system.conf" in Linux or "C:\Windows\System32\taskmgr.exe" in Windows. A path is a unique identifier for accessing a specific file. For example, D:\data\file.exe in Windows is a file path, representing the file.exe file located in the data directory on the D partition.
[0062] Figure 2A This is an architecture diagram of a system that may be applicable to embodiments of this application.
[0063] The system includes a data management system 21 and a database 20. Database 20 is a collection of data organized, described, and stored according to a certain mathematical model; for example, database 20 may include one or more data tables (referred to as "tables"). Data management system 21 is used to establish, use, and maintain database 20.
[0064] Client 102 may include any type of hardware device or application configured to interact with data management system 21. For example, hardware devices may be devices deployed on the user side, such as application servers, desktop computers, laptops, mobile phones, iPads, etc. Users can send requests to data management system 21 by operating client 102, such as query requests (e.g., SQL queries), to request data in a table that meets the query conditions.
[0065] The data management system 21 supports Structured Query Language (SQL). Generally, SQL refers to a specialized programming language used to manage data stored in the relational database 20. SQL can refer to various types of data-related languages, including, for example, data definition languages and data manipulation languages. The scope of SQL can include data insertion, querying, updating and deleting, schema creation and modification, and data access control.
[0066] A query is a request to view, access, and / or manipulate data stored in database 20. As mentioned earlier, data management system 21 can receive query requests in SQL format (also known as SQL queries) from client 102. Typically, data management system 21 receives query requests from client 102 through a communication interface, such as an application programming interface (API) or a network interface like an Ethernet interface, accesses and manipulates relevant data from database 20 to generate query results corresponding to the query request, and returns the query results to client 102 through the aforementioned communication interface.
[0067] In one example, such as Figure 2A As shown, the data management system 21 includes at least one coordination node (for clarity, Figure 2A Only one coordinating node 203 is shown, but this application does not limit the number of coordinating nodes, and multiple index nodes are also shown. Figure 2A Only two index nodes, 301A and 301B, are shown, but this application does not limit the number of index nodes; multiple worker nodes (for clarity, Figure 2A Two working nodes, 204A and 204B, are shown, but this application does not limit the number of working nodes. Since the structure and function of each index node are basically the same, and the structure and function of each working node are also basically the same, for ease of explanation, the following description uses index node 301A and working node 204A as examples respectively:
[0068] (1) Coordinating node 203 can be used to perform syntax and semantic analysis on the query request submitted by client 102, and generate a set of execution plans that may be used for the query request. The execution plan may include the execution plan of the scheduling phase and the execution plan of the execution phase. The execution plan of the scheduling phase indicates the multiple coordinating nodes that execute the first query task (such as an index query task), and the index query task that each coordinating node needs to execute. The execution plan of the execution node can be generated after the index node completes the index query task.
[0069] The coordinating node 203 can be a worker node elected from among the worker nodes to assume the function of the coordinating node, or it can be a specific device. Furthermore, when there are multiple coordinating nodes 203 in the data management system 21, a query request sent by client 102 will be routed to one of the multiple coordinating nodes 203. In this way, the data management system 21 (multiple coordinating nodes 203) can simultaneously respond to multiple query requests, which can originate from one or more clients 102.
[0070] (2) Index node 301A can be used to provide index services, such as creating, deleting, updating, and querying indexes. In this embodiment, index node 301A and index node 301B can be used to execute the index creation method and data processing method provided in this application, such as to execute the aforementioned index query task.
[0071] (3) Worker node 204A can be used to execute data query tasks. For example, worker node 204A can obtain the data indicated by the data query task from database 20, perform various calculations on the obtained data, generate query results, and return the query results to coordinating node 203. The query results are finally returned to client 102 by coordinating node 203.
[0072] The coordinating node, index node, and worker nodes can all be implemented in software or hardware. For example, the timing mechanism of index node 301A will be described below. Similarly, the timing mechanisms of the coordinating node and worker nodes can be found in the implementation of index node 301A.
[0073] Index node 301A can be implemented by software (such as a process or application) or by hardware. When implemented by software, it may include code running on a computing instance. The computing instance can be at least one of a physical host (computing device, such as a server), a virtual machine, a container, or other computing devices. When index node 301A is implemented by hardware, it can also be a device implemented using an ASIC, a PLD, etc. The PLD can be a CPLD, FPGA, GAL, or any combination thereof.
[0074] To ensure compatibility with existing system hardware architecture, the index node 301A can be deployed on the same physical host as the storage node 310A (e.g., Figure 2A (As shown), or, index node 301A and worker node 204A are deployed on the same physical host (e.g. Figure 2B (As shown). Alternatively, index node 301A can also be deployed on a separate physical host.
[0075] The storage system 30 will be introduced next.
[0076] One or more databases may be established in storage system 30, including database 20. Figure 2A A schematic diagram of the structure of storage system 30 is shown as an example. Storage system 30 includes multiple storage nodes (for clarity, ...). Figure 2A Two storage nodes, 310A and 310B, are shown, but this application is not limited to these. The structure of each storage node is basically the same, and storage node 310A is used as an example for the following description.
[0077] Storage node 310A is used to store data, such as data in the tables of database 20. Storage node 310A can be a device with data processing and data storage capabilities, such as a server, desktop computer or storage array.
[0078] Storage system 30 can deploy file systems such as the Hadoop Distributed File System (HDFS). Therefore, a table is typically split into multiple files and stored on one or more storage nodes. In other words, a storage node can store the received table data as a single file or split it into multiple appropriately sized files for easy management and retrieval. Each file contains multiple records from the same table, and each file contains different records. The file format can be ORC (Optimized Row Columns) files, Parquet files, etc.
[0079] Assume the sales table shown in Table 1 is split into files 0 through 9. File 0 contains records 1 through 1500 of Table 1, file 1 contains records 1501 through 2500, file 2 contains records 2001 through 2500, and so on. These multiple files can be stored on a single storage node or multiple storage nodes. For example, storage node 310A writes to files 0 through 4, and storage node 310B writes to files 5 through 9. This allows for parallel writing across multiple storage nodes and parallel writing of multiple files within each storage node, improving data write efficiency. Note that the sales dates in each file are mixed; for example, in file 0 (see Table 1), the sales date for the first through third records is July 17, 2022, while the sales date for the fourth record is July 18, 2022.
[0080] To facilitate management, this application can also divide the table into multiple partitions according to certain rules. For example, Table 1 can be divided into multiple partitions based on the sales date, with each partition representing a sales date, such as partition 0 representing the sales date of July 17, 2022, partition 1 representing the sales date of July 18, 2022, etc. Multiple partitions can be read independently, improving query efficiency. This can be understood as dividing Table 1 into multiple partitioned tables, with different sales dates for different partitioned tables. All records within each partitioned table share the same sales date. When client 102 requests a query for sales records for a specific day, the data management system 21 only needs to query the partitioned table for that day, thereby improving query efficiency.
[0081] Similarly, all records within a partition table can be split into one or more files. In a file system, a partition table is simply an additional layer of directories indicating partitions compared to a non-partition table; files within the same partition are all placed in the directory indicating that partition. Figure 3 The partition table shows the file system directory; for example, the files stored in the folder of partition 0 (such as...). Figure 3 The sales date in files 10 to 14 is July 17, 2022. Files stored in the folder of partition 1 (such as...) Figure 3 The sales date in documents 15 to 19 is July 18, 2022. Figure 3 As shown, the files corresponding to Table 1 have an additional directory pointing to the partition. For example, the file path of file 10 is: D:\data\partition0\file10.orc.
[0082] The file system is also responsible for allocating logical storage space for all data. The unit of database space allocation is the stripe, also known as a logical block. The stripe is the smallest unit of storage in the file system, and a file can include one or more stripes. A stripe corresponds to several bytes or megabytes of physical space on the disk. It should be noted that the stripe here is different from the block in a block device. A block in a block device is usually a 512B (e.g., sector) or 4K block of physical storage space. In this embodiment, the stripe is a logical block with a specified size. For example, in HDFS, the default size of a stripe is 64MB. Assuming that file 10 is 192MB, file 10 can be split into three stripes, such as stripe 0 to stripe 2. Assuming that file 10 contains 1500 records, stripe 0 can include records 1 to 500 of file 10, stripe 1 can include records 501 to 1000 of file 10, and stripe 2 can include records 1001 to 1500 of file 10.
[0083] In a columnar database, the data within each block is stored in units of columns, which are called a data column. Figure 4This is a schematic diagram of an ORC file format within a columnar database. Assuming the ORC file is file 10, taking block 0 as an example, the index data portion includes the row group indexes corresponding to the data columns within the row data. The row data portion includes multiple data columns. For example, in table 1, column 1 represents the product identifier column, containing multiple rows of data for the product identifier column; column 2 represents the sales volume column, also containing multiple rows of data for the sales volume column. As mentioned earlier, block 0 can include records 1 through 500 of file 10; that is, within block 0, each data column contains 500 rows of data. The block script includes the offsets of each row group within the data block, indicating the starting address of each row group. The file script includes the offsets of each data block within the file; please refer to the introduction to existing ORC file structures, which will not be detailed here.
[0084] It should be noted that the data block size described above is merely an example, and the data block size may vary in different file systems. This embodiment can also set the data block size. Furthermore, the number of records in the table included in each data block is only an example, and this application does not impose any limitations on this. It should also be noted that... Figure 2A The storage system 30 shown is a distributed storage system. The storage system 30 applicable to the embodiments of this application can also be a centralized storage system. Any storage system capable of storing a database is applicable to the embodiments of this application.
[0085] The above text describes how to store tables within a storage system 30. It is worth noting that, in addition to storing data in the table, storage nodes can also be used to store indexes of the table data, metadata of the table data (such as the mapping relationship between the table and partitions / files), and metadata of the index nodes (which will be introduced below, but will not be elaborated here), etc., without any specific limitations.
[0086] The above Figure 2A and Figure 2B This application pertains to a storage-compute separation system architecture. The embodiments of this application are also applicable to storage-compute integrated system architectures. See [link to relevant documentation]. Figure 2C As shown, in Figure 2C In the system shown, worker nodes, index nodes, and storage nodes can be deployed in the same data center or the same server room. However, worker nodes, index nodes, and storage nodes can be deployed on different physical hosts, or any two or three nodes can be deployed on the same physical host.
[0087] It should be noted that, Figure 2A , 2B The systems shown in 2C are merely examples. The systems applicable to the embodiments of this application may have more or fewer components than the above system architecture, and this application does not limit them.
[0088] This application provides an index creation method in which table data and its indexes are stored independently, improving write efficiency. Furthermore, a coordinating node instructs multiple index nodes to create indexes for the table data, further enhancing index creation efficiency.
[0089] Next, combine Figure 5 The index creation method provided in the embodiments of this application is applied to Figure 2A The system shown is used as an example to introduce this index creation method. For ease of description, this method is described as being derived from... Figure 2A The coordinating node 203 and index nodes 301A and 301B in the process are executed.
[0090] like Figure 5 As shown, the method may include:
[0091] Step 501: Coordinating node 203 sends index creation requests to multiple index nodes respectively.
[0092] An index creation request is used to request the creation of an index for specified data.
[0093] In one implementation, the specified data may be data already stored in storage system 30. For example, after the data of Table 1 is stored in storage system 30, the coordinating node 203 instructs multiple index nodes to create indexes for the data of Table 1.
[0094] In another implementation, the specified data may also be data being written to storage system 30, such as when coordinating node 203 instructs multiple index nodes to create indexes for the data in table 1 during the process of writing data from table 1 to storage system 30.
[0095] It's worth noting that when writing table data to storage nodes, the storage nodes can store the table data in one or more files, such as ORC files or Parquet files. Taking ORC files as an example, currently, when writing to an ORC file, the table data and the table's indexes are typically written to the same file. Figure 4 As shown, "index data" represents the index portion, including the table's index. In this embodiment, it supports writing only the table data without writing the index when writing the ORC file, i.e., storing the data and index separately. After the table data is written, the coordinating node 203 instructs multiple index nodes to create indexes for the stored table data. Alternatively, during the process of the storage nodes writing the table data, the coordinating node 203 instructs multiple index nodes to create indexes for the table data.
[0096] Step 501 can be initiated proactively by the coordinating node 203. For example, after detecting that the sales table shown in Table 1 has been written to the storage system 30, or when the storage system 30 is writing data to Table 1, the coordinating node 203 generates multiple index creation requests for Table 1 to create indexes. Alternatively, step 501 can also be triggered by the client 102. For example, the client 102 first sends a first index creation request (create index) to the coordinating node 203, which requests the creation of an index for the target object (such as Table 1). After receiving the first index creation request, the coordinating node 203 generates multiple second index creation requests based on the first index creation request. Each second index creation request is used to create an index for a portion of the data in the target object, and different second index creation requests are used to create indexes for different data. It can be understood that the coordinating node 203 instructs different index nodes to create indexes for different data in the same target object. In this way, multiple index nodes 301 create indexes in parallel, thereby improving the index creation efficiency.
[0097] See Figure 5 As shown, coordinating node 203 sends index creation request 1 and index creation request 2 to index nodes 301A and 301B respectively. For example, index creation request 1 requests the creation of an index for data 1, and index creation request 2 requests the creation of an index for data 2. If the target object is table 1, please refer to... Figure 3 To understand, index creation request 1 requests to create an index for file 10, and index creation request 2 requests to create an index for file 11. Similarly, index creation request 1 requests to create indexes for files 10 to 14, and index creation request 2 requests to create indexes for files 15 to 19. Data 1 and data 2 can be stored on the same storage node, or they can be stored on different storage nodes. Furthermore, there is no limitation on the size of data 1 and data 2.
[0098] For example, an index creation request may include one or more of the following: data indication information, index indication information.
[0099] (1) Data indication information, used to indicate the specified data, that is, the data to be indexed.
[0100] For example, data indication information may include one or more of the following: the file path of the specified data, file handle, starting address (or offset of the specified data in the file), data length, and data column name (i.e., field name). For instance, when the specified data is file 10, the corresponding index creation request may include the file path of file 10. As another example, when the specified data is block 0 in file 10, the index creation request may include: the file path of file 10, the starting address of block 0 in file 10 (e.g., 0), and the length of block 0 (e.g., 64M). As yet another example, when the specified data is blocks 1 to 2 in file 10, the index creation request may include: the file path of file 10, the starting address of block 1 in file 10 (e.g., 64M), and the total length of blocks 1 and 2 (e.g., 128M). And as yet another example, when the specified data is the sales column in file 10, the index creation request includes the file path of file 10 and the field name of the sales column.
[0101] It should be noted that the above are only some examples. The data indication information in this embodiment may also include other information, such as the information of the storage node where the specified data is located, the table name, etc. This application does not limit the content included in the data indication information.
[0102] (2) Index indication information includes one or more of the following: index level, index type.
[0103] As mentioned earlier, in columnar databases, an index is used to indicate the numerical characteristics of a data column in several rows of records. There are various types of indexes, which will be introduced in detail below.
[0104] This application supports the creation of multi-level indexes, with different levels corresponding to different data granularities. That is, this application can generate data indexes at different data granularities. For example, data can correspond to a first level and a second level. The data at the first level includes the data at the second level. The first level includes multiple data objects (denoted as first data objects) divided according to a first granularity, and the second level includes multiple data objects (denoted as second data objects) divided according to a second granularity. Each first data object may include one or more second data objects. In this example, this application supports generating first-granularity indexes and second-granularity indexes. The first-granularity index is the index corresponding to the first data object, and the second-granularity index is the index corresponding to the second data object. Of course, it can also include a third level and a third-granularity index, a fourth level and a fourth-granularity index, etc. This application does not limit the index level.
[0105] For example, the index levels in this embodiment include, but are not limited to: file-level index (hereinafter referred to as file index), data block-level index (hereinafter referred to as block index), and row group-level index (hereinafter referred to as row group index). Typically, a file includes one or more blocks, a block includes multiple row groups, and a row group includes multiple rows of data. In this application, (1) one or more file indexes can be generated based on a file. Multiple file indexes can belong to different data columns. A file index is an index of a data column within the file, used to indicate the numerical characteristics of that data column within the file. Alternatively, multiple file indexes are indexes of different types corresponding to the same data column within the file. (2) one or more block indexes can be generated based on a block. A block index is an index of a data column within a block, used to indicate the numerical characteristics of that data column within the block. Alternatively, multiple block indexes are indexes of different types corresponding to the same data column within the data block. (3) one or more row group indexes can be generated for a row group. A row group index is an index of a data column within the row group, used to indicate the numerical characteristics of that data column within the row group. Alternatively, multiple row group indexes are indexes of different types corresponding to the same data column within the row group. Among them, row group index is the smallest unit for creating indexes, which will be introduced below.
[0106] In this application, the first granularity index can be a file index, the second granularity index can be a block index, and the third granularity index can be a row group index; alternatively, the first granularity index can be a partition index, the second granularity index can be a file index, the third granularity index can be a block index, and the fourth granularity index can be a row group index. For ease of explanation, the following description will use the example of the first granularity index being a file index, the second granularity index being a block index, and the third granularity index being a row group index. In the following description, the file index can be replaced with the first granularity index, the block index can be replaced with the second granularity index, and the row group index can be replaced with the third granularity index.
[0107] Index types include, but are not limited to: min / max (minimum / maximum value), Bloom, B-tree, bitmap, Sum, and Count, etc., with no specific restrictions. Min / max indicates the minimum and maximum values of a data column. Bloom indicates a fixed value in a data column. B-tree indicates the distribution characteristics of the data range within a data column. Sum indicates the sum of a data column. Count indicates the count of data items in a data column. One or more indexes of different types can be created for the same data column. For example, a sales column could include a min / max index to record the minimum and maximum values, a Bloom index to record a fixed value, a Sum index to record the sum of sales, and so on.
[0108] In other words, in this embodiment, an index creation request can request the creation of one or more levels of indexes corresponding to the data, and each level of index can include one or more different types of indexes. For example, the index creation request requests the creation of a file index, a block index, and a row group index corresponding to the sales column in file 10. Each level of index can include min / max, bloom, etc. That is, the file index includes the min / max index and bloom index of the sales column in file 10, the block index includes the min / max index and bloom index of the sales column within a block, and the row group index includes the min / max index and bloom index of the sales column within a row group. In this application, the specific index levels and index types to be created can also be specified by the user. In this case, the information such as the index level and index type can be carried in the index creation request.
[0109] In this embodiment, each index node corresponds to a data range of the target object and is responsible for managing the indexes corresponding to the data within that data range. In other words, the index node is responsible for creating indexes for the data within its corresponding data range.
[0110] The correspondence between index nodes and data ranges is included in the metadata of the index nodes. The metadata of the index nodes can be stored within the storage node, locally on the index node, or on other storage devices. Any device capable of storing data can be used to store the metadata of the index nodes. It should be noted that the metadata of the data and the metadata of the index nodes involved in this embodiment include different content.
[0111] In this embodiment, the index creation requests sent by the coordinating node 203 to multiple index nodes can be the same or different.
[0112] For example, the process of generating the above index creation request may include: (1) the coordinating node 203 obtains the metadata of the target object (such as a sales table), which includes the mapping relationship between the target object and partitions / files, and the metadata of each index node in multiple index nodes (index nodes 301A, 301B); (2) the coordinating node 203 generates an index creation request corresponding to each index node based on the metadata of the target object and the metadata of each index node. The data indicated in the index creation request corresponding to the index node is the data within the data range corresponding to that index node. For example, the data range corresponding to index node 301A includes file 10, and the data range corresponding to index node 301B includes file 11. This is just an example, and this application does not limit the unit of the data range, such as the data range can also be measured in data blocks, etc. Then, index creation request 1 can request to create an index for file 10, and correspondingly, index creation request 1 may include data indication information for indicating file 10; index creation request 2 can request to create an index for file 11, and correspondingly, index creation request 2 may include data indication information for indicating file 11. As can be seen, in this example, the index creation requests sent by the coordinating node 203 to different index nodes are different.
[0113] As another example, the index creation requests sent by the coordinating node 203 to different index nodes can also be the same. For instance, the index creation request may include the name of the target object, such as a table name, and instruct the index node to create an index on the data belonging to the target object within its own data scope. The index node determines which data of the target object to index based on its own metadata (this will be described in detail below and will not be repeated here). In this case, the index creation requests sent by the coordinating node 203 to different index nodes are the same.
[0114] It should be noted that the index creation request in any of the above embodiments may also include index indication information, or it may not carry index indication information. In this case, the index level and index type of the index to be created can be default values (i.e., default values).
[0115] Step 502: After receiving an index creation request, each of the multiple index nodes creates an index of the specified data according to the index creation request.
[0116] Suppose that index node 301A receives index creation request 1, which requests the creation of a file index, block index, and row group index corresponding to the sales column in file 10, with index type min / max. If file 10 includes multiple data columns without specifying any, the index node can create a corresponding index for each data column separately. Index node 301B receives index creation request 2, which requests the creation of a file index, block index, and row group index corresponding to the sales column in file 11, also with index type min / max.
[0117] The following uses the sales column in file 10 as an example to introduce the creation process of its row group index, block index, and file index:
[0118] (1) Row group index.
[0119] Combination Figure 3 Assuming file 10 includes blocks 0 to 2, taking block 0 as an example, please refer to [link to relevant documentation]. Figure 6 Assuming block 0 contains 500 rows of data in the sales column, and assuming a row group contains 100 rows of data, then within the 500 rows of data in the sales column of block 0, each 100 rows corresponds to a row group index, such as... Figure 6 As shown, the row group index corresponding to the sales column in block 0 includes 5 row group index entries, and each row group index entry records the minimum and maximum values of its corresponding 100 rows of data.
[0120] (2) Block index.
[0121] Continuing with block 0 as an example, see... Figure 6 The block index corresponding to the sales volume column in block 0 records the minimum and maximum values among the 500 rows of data corresponding to the sales volume column in block 0. For example, index node 301A can directly determine the minimum and maximum values among the 500 rows of data contained in the sales volume column in block 0, thereby determining the block index corresponding to the sales volume column in block 0. As another example, index node 301A can determine the block index based on all row group index entries in the row group index corresponding to the sales volume column in block 0. Figure 6 In the block index corresponding to the sales volume column in block 0, the minimum and maximum values are the minimum and maximum values among the five row group index items corresponding to the sales volume column in block 0.
[0122] (3) File index.
[0123] See also Figure 6The file index record corresponding to the sales column in file 10 contains the minimum and maximum values among the 1500 rows of data corresponding to the sales column in file 10. For example, an index node can directly determine the minimum and maximum values among the 1500 rows of data in the sales column of file 10, thereby determining the file index. As another example, index node 301A can also determine the file index based on block index 0 (the index corresponding to the sales column in block 0), block index 1 (the index corresponding to the sales column in block 1), and block index 2 (the index corresponding to the sales column in block 2) corresponding to file 10.
[0124] At this point, index node 301A has completed creating the index for the specified data. Index node 301B creates its index in the same way as index node 301A, and will not be described further here.
[0125] The creation order of the multi-level indexes described above is merely an example, and this application embodiment does not limit it. Furthermore, the index creation request includes data indication information. When the index creation request does not include data indication information, the index node can perform the operations of the coordinating node 203 described above, such as obtaining the index node's metadata and determining the data to be indexed based on the index node's own metadata, i.e., the data within the data range corresponding to the index node, as explained above, and will not be repeated here.
[0126] In one optional implementation, to avoid duplicate index creation, an index node can check if an index already exists for the data before creating an index for that data. If it does, it compares the data's timestamp (indicating the data's last modification time) with the timestamp of the already created index (indicating the index's creation time). If the data's timestamp is later than the index's timestamp, meaning the data was updated or modified after the index was created, a new index is created for that data, and the old index (i.e., the already created index) is deleted. If the data's timestamp is earlier than or the same as the index's timestamp, a new index does not need to be created; the old index can be retained.
[0127] Step 503: Each of the multiple index nodes 301 stores the created index.
[0128] Each index node writes the index it creates to a storage device for persistent storage. This storage device can be the index node's own storage device, such as a disk or hard drive, or the storage device within a storage node or other node, such as a third-party storage device. The specific storage device is not limited.
[0129] Taking the index storage in storage system 30 as an example, the physical storage space used to store data and the physical storage space used to store the index in storage system 30 are separate. For example, if each storage node in storage system 30 includes four storage units (such as storage units A to D), each storage unit can be a logical address space. Typically, the storage space occupied by the index is smaller than that of the data itself. Therefore, storage units A to C in each storage node 310 of storage system 30 can be set to store data, while storage unit D in each storage node can be set to store the index. In this case, the index node can store the data and the index of that data in the same storage node 310; that is, the index node writes the index of the data into the storage node used to store the data. As another example, if storage system 30 includes four storage nodes (such as storage nodes 1 to 4), all storage units in storage nodes 1 to 3 can be set to store data, while all storage units in storage node 4 can be set to store the index. In this embodiment, the storage units used to store data and the storage units used to store metadata are essentially the same, only the content stored in the storage units is different. Simply put, separate storage can be understood as storing data and its index in separate files. The data files and index files can be stored on the same storage node or on different storage nodes.
[0130] It is worth noting that in this embodiment, the index creation process (such as step 501) can be triggered after the data of the target object (such as in Table 1) has been written to the storage node. That is, in this embodiment, the data write operation can be performed first, and then the index can be created for the data. Alternatively, in this embodiment, the index can be created for the data that is being written to the storage node, that is, the index is created while the write operation is being performed. However, it should be understood that in this embodiment, the data write operation and the index creation operation are two independent processes, and the data write operation is not affected by the index creation operation.
[0131] Through the above design, the coordinating node 203 distributes multiple index creation requests to multiple index nodes 301, enabling parallel index creation across multiple index nodes to improve index creation efficiency. This avoids problems in the prior art, such as system instability or even unavailability caused by coordinating node 203 failure, insufficient memory / disk on coordinating node 203, or a large volume of index creation requests from client 102. Furthermore, this method decouples data write operations from index creation, creating the index only after the data write is complete, shortening the data write time, improving data write efficiency, and providing flexibility in index creation. Moreover, this embodiment supports the creation of multi-level indexes, providing index services with various granularities and multi-level structures.
[0132] The above text describes the index creation process.
[0133] Next, combine Figure 7 The data processing method provided in the embodiments of this application is applied to Figures 2A-2B Taking any of the systems shown as examples, the data processing method provided in this application embodiment will be described in detail. This data processing method includes two stages: a scheduling stage and an execution stage. In the scheduling stage, multiple index nodes use the index in parallel to determine the target shard in the target object. The target shard includes data that meets the query conditions. In the execution stage, worker nodes retrieve the data that meets the query conditions from the target shard and perform a series of operations on it. For ease of explanation, this method is described using... Figure 2A Taking the execution of coordinating node 203 and index nodes 301A and 301B as an example.
[0134] like Figure 7 As shown, the method may include:
[0135] Step 701: Client 102 sends a query request to coordinating node 203.
[0136] For example, the query request may include information about the target object and query conditions, used to request the search for data (denoted as target data) that meets the query conditions from the data of the target object. The target object may be a table (the information of the target object may be the table name) or a file (the information of the target object may be a file path or file handle, etc.), etc., and is not specifically limited.
[0137] Step 702: Coordinating node 203 generates multiple first query tasks (such as index query tasks) based on the received query request.
[0138] For example, the coordinating node 203 performs syntax and semantic parsing on the query request to obtain parsed information, such as the target object and query conditions indicated by the query request. The coordinating node 203 then generates an index query task based on the parsed information (such as the target object and query conditions).
[0139] For example, suppose the query request is an SQL statement, which includes:
[0140] SELECT saleTbl.ID
[0141] FROM saleTbl
[0142] WHERE saleTbl.data='20220717'
[0143] AND saleTbl.sales ≥ 300
[0144] This query request is used to retrieve products from Table 1 (named saleTbl) that were sold on July 17, 2022, and had a sales volume greater than 300. Based on syntax and semantic parsing, the target object is Table 1, and the query conditions are a sales date of July 17, 2022, and a sales volume greater than 300.
[0145] Coordinating node 203 generates an index query task based on the parsed information. This task instructs the index nodes to filter the shards of Table 1 to extract the target shards. The target shards are those containing data that meets the query conditions (i.e., the target data). These shards can be data objects of different granularities within the storage system 30, such as partitions, files, and data blocks corresponding to Table 1. The following explanation uses partitions, files, and data blocks as examples.
[0146] In this embodiment, each index node corresponds to a data range of the target object and is responsible for managing the indexes corresponding to the data within that range. In other words, the index node filters shards based on the indexes within its corresponding data range. It is worth noting that for the same index node, the data range corresponding to the index node during the index creation phase and the data range corresponding to the index node during the data processing phase can be the same or different; this embodiment does not impose any limitations on this.
[0147] In this embodiment, the index query tasks sent by the coordinating node 203 to multiple index nodes can be the same or different.
[0148] Example 1: The index query tasks sent by coordinating node 203 to multiple index nodes are the same.
[0149] For example, the generation process of this index query task may include: the coordinating node 203 obtaining the metadata of the target object (such as Table 1), which includes the correspondence between Table 1 and partitions / files. The coordinating node 203 generates an index query task based on the metadata of the target object and the query conditions. The index query task may include information about the query object and query conditions. The data of the target object includes the data of the query object. For example, the target object is Table 1, and the query object is one or more partitions corresponding to Table 1, or one or more files corresponding to Table 1, or one or more data blocks corresponding to Table 1.
[0150] Table 2 shows the correspondence between Table 1 and data objects of different granularities, which can be combined with... Figure 3 understand.
[0151] Table 2. Correspondence between Table 1 and data objects
[0152]
[0153] The metadata of Table 1 includes the correspondence between Table 1 and partitions 0 and 1, or the correspondence between Table 1 and files 10 to 19. It can be understood that if Table 1 is a non-partitioned table, its metadata may only include the correspondence between the table and the files.
[0154] Based on the different metadata, there are several examples of how index query tasks can be generated:
[0155] Example 1: The query object is any one of the files in at least one file corresponding to the target object (such as Table 1);
[0156] For example, if the metadata of Table 1 includes the correspondence between Table 1 and files, such as when Table 1 is a non-partitioned table, the query object can be all files corresponding to Table 1. For example, if the metadata of Table 1 includes that Table 1 corresponds to files 0 to 9, then the index query task can include information about files 0 to 9 (such as file paths) and query conditions to indicate which files containing the target data are to be queried from files 0 to 9.
[0157] Example 2: When the target object corresponds to multiple partitions, the query object can be the target partition corresponding to the target object. The target partition is the partition that includes the target data among the multiple partitions corresponding to the target object.
[0158] For example, coordinating node 203, based on the query condition (data = 20220717), can determine that partition 0 includes the target data and partition 1 does not. Coordinating node 203 then uses partition 0 as the query object and generates an index query task. Here, the query object is the partition that includes the target data. For instance, this index query task may include information about partition 0 (such as the pathname of partition 0) and query conditions, instructing the index node to query files in partition 0 that include the target data.
[0159] Example 3: The query object can be any file in at least one file corresponding to the target partition;
[0160] For example, the target partition is partition 0, and the query object is any one of at least one file (file 10 to file 14) corresponding to partition 0. Exemplarily, the index query task may include information from file 10 to file 14 (such as file names) and query conditions to indicate how to query files from file 10 to file 14 that contain the target data.
[0161] Example 4: The query object can be any one of the data blocks in at least one data block corresponding to the target partition (or target object);
[0162] For example, the target partition is partition 0, and the query object is any one of the at least one data block corresponding to partition 0 (see Table 2, which includes data blocks 0-2 of file 10, data blocks 0 and 1 of file 11, data block 0 of file 12, etc.). Exemplarily, this index query task may include information about the at least one data block (which may be the file path of the file to which the data block belongs and the offset of the data block within the file) and query conditions.
[0163] As can be seen, in this scenario, the index query tasks generated by coordinating node 203 for multiple index nodes are identical. Correspondingly, each index node performs a filtering operation based on the indexes of data belonging to the query object within its corresponding data range. These multiple index nodes could be all index nodes in the system, or any other subset of index nodes. If an index node determines that its corresponding data range does not include the query object's data, that index node will not execute subsequent process operations.
[0164] Example 2: The index query tasks sent by coordinating node 203 to multiple index nodes are different.
[0165] The coordinating node identifies multiple target index nodes corresponding to the target object. Each target index node is an index node whose data range includes the data of the target object. The coordinating node 203 generates a dedicated index query task for each target index node.
[0166] The process may include: Coordinating node 203 obtaining metadata of the target object and the metadata of each of the multiple index nodes, which indicates the data range corresponding to the index node. Coordinating node 203 determines the target index node based on the metadata of the target object and the metadata of each index node, and generates an index query task for each target index node based on the data range belonging to the target object corresponding to each target index node. Each index query task includes information about the query object and query conditions. At this point, the multiple index query tasks are different, such as the different query objects indicated by each index query task.
[0167] For example, the metadata of an index node includes one or more of the following:
[0168] The correspondence between inodes and partitions, the correspondence between inodes and files, and the correspondence between inodes and data blocks.
[0169] In one alternative approach, the data range corresponding to the index node includes data within the storage node deployed on the same computing device as the index node. For example, combining... Figure 2A and Figure 3It is understood that the data range corresponding to index node 301A includes the data within storage node 310A. The data range corresponding to index node 301B includes the data within storage node 310B. Table 3 provides an example of the metadata for index nodes 301A and 301B.
[0170] Table 3. Metadata of Index Nodes
[0171]
[0172] Based on Example 2, partition 0 includes the target data, while partition 1 does not. Coordinating node 203 determines the target data range based on the metadata of index node 301A and index node 301B as shown in Table 3. The data range corresponding to index node 301A includes files 10-12 in partition 0. The data range corresponding to index node 301B includes files 13 and 14 in partition 0. In this example, the target index nodes are index nodes 301A and 301B, and coordinating node 203 generates corresponding index query tasks for index nodes 301A and 301B respectively.
[0173] For example, the index query task may include information about query objects belonging to the target object (or target partition) within the data range corresponding to the index node. The index query tasks for different target index nodes may be different. The query objects here can be files, data blocks, etc.
[0174] For example, index query task 1 corresponding to index node 301A includes information and query conditions for files 10 to 12. Index query task 2 corresponding to index node 301B includes information and query conditions for files 13 and 14. Optionally, the information for each file may also include the storage information of the file's index, whereby the storage information of each index indicates the storage location of the index, facilitating the index node to retrieve the file's index from that storage location.
[0175] For example, index query task 1 corresponding to index node 301A includes information about blocks 0 to 2 in file 10, information about data 0 and block 1 in file 11, information about block 0 in file 12, and query conditions. Optionally, the information for each block may also include storage information indicating the storage location of the index for that block.
[0176] When there are multiple query objects, the index query task can include all query objects or a specified number of query objects. In other words, the coordinating node 203 can send multiple index query tasks to an index node to indicate all query objects.
[0177] It is worth noting that the file index can include the file index, block index, and row group index corresponding to the file. Similarly, the block index can include the block index and row group index. It should also be noted that Tables 2 and 3 are merely examples; the metadata of the data can include more or less information than in Table 2, and the metadata of the index node can include more or less information than in Table 3. For example, the metadata of the index node may not include partition information, or may not include block information, or may include index information. Index information includes, but is not limited to, one or more of the following: index level, index type, and index storage information. The data range corresponding to the index node can also be represented in other ways; this application does not limit these possibilities.
[0178] Step 703: Coordinating node 203 sends an index query task to each of the multiple index nodes 301.
[0179] In Example 1, coordinating node 203 can send index query tasks to each of any part or all of the index nodes in the system. In Example 2, coordinating node 203 only sends the corresponding index query task to each target index node.
[0180] The index query task requests to query the target shard that includes the target data in the query object.
[0181] Step 704: After receiving the index query task, each of the multiple index nodes determines the target shard that includes the target data within the data range managed by the index node.
[0182] Step 704 will be further explained using examples one and two from step 702 above.
[0183] In Example 1, since all index nodes receive the same index query tasks, each index node can obtain its own metadata and determine whether the data range corresponding to the index node includes the data of the query object. If it does, the index node executes the index query task. Alternatively, if the data range corresponding to the index node does not include the data of the query object, the index query task is not executed.
[0184] The index node executing the index query task uses the index of the specified object for filtering to determine the target shard. The specified object refers to an object within the data range corresponding to the index node that intersects with the query object; such as a target partition, file, or data block.
[0185] For example, in Example 1, suppose index query task 1 includes information about partition 0 and query conditions. After receiving index query task 1, index node 301A determines the specified objects belonging to partition 0 within the data range corresponding to index node 301A based on its metadata. Referring to Table 3, the specified objects corresponding to index node 301A include file 10, file 11, and file 12 in partition 0. Then, the index node performs filtering operations using the index of each specified object.
[0186] In Example 2, after the target index node receives the index query task, it does not need to perform the above judgment and can directly execute the index query task, using the index of the query object indicated in the index query task to perform the filtering operation. In other words, each query object indicated in the index query task here is the specified object.
[0187] The following describes the process of executing an index query task, using index node 301A as an example:
[0188] This embodiment provides multi-level indexes, such as file indexes, block indexes, and row group indexes. When performing index query tasks, index nodes can perform filtering at various data granularities based on multi-level indexes.
[0189] The index query task can include information about the query object and query conditions. For example, the query conditions can include information about the target column and predicate conditions. As mentioned above, the query condition is that the sales volume of a product in the sales volume column is ≥300. It can be seen that the target column is the sales volume column and the predicate condition is ≥300. For ease of explanation, the following example uses the query condition of products with sales volume ≥300 in the sales volume column.
[0190] In one implementation, inode 301A performs file index filtering (see step a) + block index filtering (see step b).
[0191] In step a, index node 301A can obtain the file index corresponding to the target column in each specified object, and use these file indexes to filter the files, resulting in one or more target files. The target file is a file containing data that meets the query conditions, such as when the numerical range indicated by the file index of the target column in the target file intersects with the numerical range indicated by the predicate conditions.
[0192] For example, taking the file index type as min / max, if the specified object includes file 10, file 11, etc., assuming that the file index corresponding to the sales column in file 10 is [0, 500], and the predicate condition is ≥300, it can be determined that file 10 includes products with sales ≥300, that is, part of the data in file 10 is the target data, and file 10 is the target file.
[0193] For example, suppose the file index of the sales column in file 11 is [10, 200], and the predicate condition is ≥300. Since 200 < 300, it can be known that file 11 does not include products with sales ≥300. Therefore, all the data in file 11 is not the target data, and file 11 is not the target file.
[0194] Each inode determines one or more target files using this method. At this point, the target is assigned as a target file.
[0195] There are several ways for index node 301A to obtain indexes, such as retrieving them from its own memory or from other storage media. It is worth noting that index node 301A can cache the indexes within the data range it manages in its own memory, thus enabling fast index lookup tasks. If the memory capacity of index node 301A is limited and cannot cache all indexes, it can use a portion of the indexes in memory first, and then read the remaining indexes from storage, such as storage nodes, into memory based on the storage information of those remaining indexes. For example, the storage information of an index includes one or more of the following: the storage node where the index file is located, the file path, the file handle, the offset, and the data length. The storage information of the index can be recorded in the metadata of the index node; see the aforementioned explanation, which will not be repeated here. The methods of obtaining indexes will not be described again below.
[0196] In step b, index node 301A uses the block index of each target file for filtering. If index node 301A determines the data blocks belonging to the target file within its corresponding data range, it uses the block index of the target column within each determined data block to filter out one or more target blocks. A target block is a data block that includes data that satisfies the query conditions, such as when the numerical range indicated by the block index of the target column in the target block intersects with the numerical range indicated by the predicate conditions.
[0197] For example, suppose the target files identified by index node 301A include file 10 and file 12. Taking file 10 as an example, suppose the block index corresponding to the sales column in block 0 of file 10 is [0, 200], and the predicate condition is sales ≥ 300. Therefore, it can be determined that block 0 does not include products with sales ≥ 300, that is, all the data in block 0 is not the target data, and block 0 is not the target block.
[0198] For example, suppose the block index corresponding to the sales column in block 1 of file 10 is [100, 400], and the predicate condition is sales ≥ 300. Therefore, it can be determined that block 1 includes products with sales ≥ 300, and part of the data in block 1 is the target data. Therefore, block 1 is the target block.
[0199] Each inode determines one or more target blocks using this method. At this point, the target fragment is the target block.
[0200] In another implementation, index node 301A performs only the filtering of the block index.
[0201] For example, index node 301 uses the block index corresponding to the target column in the specified object to filter out one or more target blocks. For detailed implementation methods, please refer to the foregoing description, which will not be repeated here.
[0202] It should be understood that different inodes determine different target files and target blocks.
[0203] In step 705, each index node sends its own determined first task result (i.e., the execution result of the index query task) to the coordinating node 203.
[0204] The first task result of index node 301A may include information about one or more target blocks determined by index node 301A.
[0205] For example, index node 301A returns first task result 1, and index node 301B returns first task result 2. First task result 1 may include information about each target block determined by index node 301A. First task result 2 may include information about each target block determined by index node 301B.
[0206] Step 706: Coordinating node 203 determines the second query task (such as a data query task) based on the results of multiple first tasks.
[0207] Based on the results of the multiple first tasks sent by each index node, the coordinating node 203 parses the query request into multiple data query tasks. Each query task includes operand information (denoted as first operand information) and operation information (denoted as first operation information).
[0208] The first operand information indicates the data to be operated on, such as a target block. Different data query tasks indicate different target blocks. The first operation information indicates the operation to be performed and may include one or more operator information. For example, for the aforementioned query request, the operator information may include a scan operator, a filter operator, and filter conditions for the filter operator, such as sales volume ≥ 300.
[0209] This data query task instructs the execution of corresponding operations on the target block's data using specified operators. As described above, the worker node will first use the `scan` operator to read each row of data in the target block, and then use the `filter` operator to filter each row to obtain the data that meets the filtering conditions, i.e., the target data. Alternatively, the `scan` and `filter` operators can be replaced by the `scanandfilter` operator.
[0210] The following describes the method for generating data query tasks: For example, coordinating node 203 can generate a data query task based on a first task result, both carrying the same target block information. For instance, if first task result 1 includes information about one or more target blocks determined by index node 301A, then data query task 1 generated based on first task result 1 may include information about one or more target blocks determined by index node 301A. If first task result 2 includes information about one or more target blocks determined by index node 301B, then data query task 2 generated based on first task result 2 may include information about one or more target blocks determined by index node 301B.
[0211] As another example, a data query task may also include a portion of the target block or a portion of the operator in the result of a first task, and this embodiment does not limit this.
[0212] Step 707: Coordinating node 203 sends data query tasks to multiple worker nodes.
[0213] Step 708: Each of the multiple worker nodes executes a data query task.
[0214] In one implementation, worker nodes can push down some operators from a data query task to other nodes, such as index nodes, for execution. Figure 2A In this architecture, worker node 204A can push the scanandfilter operator (included in the pushdown task) down to the index node 301A near the data source side for execution. In this way, index node 301A can return the filtered data to worker node 204A, reducing the amount of network data transmission and alleviating the pressure and overhead of network bandwidth.
[0215] Typically, a pushdown task includes a portion of the query task and may include second operand information and second operation information. The second operand information refers to part or all of the data to be operated on, as indicated by the first operand information in the data query task. The second operation information refers to part or all of the operator information carried in the data query task. Pushdownable operators can be set operators, such as scan operators, filter operators (or scanandfilter operators), etc.
[0216] Assume that the target blocks indicated by data query task 1 include: block 0 of file 10, block 2 of file 10, and block 1 of file 12; the first operation information includes: the scanandfilter operator and filtering conditions. Based on data query task 1, pushdown task 1 is generated, and the target blocks indicated by pushdown task 1 include: block 0 of file 10, block 2 of file 10, and block 1 of file 12; the second operation information includes: the scanandfilter operator and filtering conditions. In this case, the pushdown task is the same as the data query task.
[0217] The following uses inode 301A as an example to describe the process of executing the pushdown task for an inode:
[0218] Index node 301A receives pushdown task 1, obtains the row group index of the target column within the target block indicated in pushdown task 1, and uses the row group index of the target column within the target block indicated in pushdown task 1 to filter out one or more target row groups. Here, a target row group is a row group containing the target data. This is indicated if the numerical feature indicated by the row group index of the target column in the target row group intersects with the numerical range indicated in the predicate condition.
[0219] For example, suppose pushdown task 1 indicates that the target blocks include: block 0 and block 2 of file 10, and block 1 of file 12. See block 0 as an example. Figure 6 As shown, assume block 0 includes 5 row groups, namely row group 0 to row group 4. Taking row group 0 as an example, index node 301A uses the row group index 0 corresponding to the target column in row group 0 and the predicate condition to determine whether row group 0 is the target row group. For example, if the value range of the records in row group index 0 is [20, 350], and the predicate condition is sales ≥ 300, then row group 0 contains products with sales greater than 300, and part of the data in row group 0 is the target data; therefore, row group 0 is the target row group. As another example, if the value range of the records in row group index 1 is [20, 100], and the predicate condition is sales ≥ 300, then the records in row group 1 do not contain products with sales greater than 300, and the data in row group 1 is not the target data; therefore, row group 1 is not the target row group.
[0220] It's worth noting that the inode can obtain the row group index in the ways described above, such as caching the row group index in memory or reading it from other storage devices; these methods will not be elaborated upon here. In another possible scenario, if the file containing the target block that the inode is reading includes a row group index, such as... Figure 4 The ORC file shown can be used by the index node to filter rows using the row group index carried in the ORC file; this embodiment does not limit this. It should also be noted that if the ORC file includes a row group index, the pushdown task here can also be pushed down to the storage node for execution, where the storage node performs the functions of the index node to obtain one or more target row groups.
[0221] Each inode determines one or more target row groups using the method described above. It should be understood that different inodes determine different target row groups.
[0222] Then, the index node uses the scanandfilter operator to read the data in the target row group and filters the data in the target row group to obtain the target data.
[0223] For example, suppose inode 301A determines that the target row groups include: row groups 0, 1, and 5 in block 0 of file 10; row group 2 in block 2 of file 10; and row groups 1, 3, and 4 in block 1 of file 12. Taking row group 0 in block 0 of file 10 as an example, inode 301A uses the scan operator to read multiple rows of data included in the sales column of row group 0 in block 0, and uses the filter operator to filter the read multiple rows of data, filtering out one or more rows of data that meet the filtering conditions, i.e., the target data. For example, if the sales column in row group 0 includes 100 rows of data, inode 301A filters out the data in these 100 rows whose value (i.e., sales) is 300 or higher; these data are the target data that meet the filtering conditions.
[0224] Index node 301A reads the data in each target row group sequentially and filters it to obtain the target data in each target row group, and then sends the filtered target data to worker node 204A.
[0225] After receiving the target data sent by index node 301A, worker node 204A performs the corresponding operations using the remaining operators. Since pushdown task 1 is the same as data query task 1, worker node 204A does not need to perform any further operations here. Worker node 204A generates the second task result, which includes the received target data.
[0226] It should be noted that the above example only uses a data query task that includes the scanandfilter operator. In reality, different query requests may apply different operators. The specific operators used depend on the query conditions. If the data query task includes multiple operators (usually executed serially), the worker node will typically push some operators down to the index node. Correspondingly, the worker node 204 also needs to perform the remaining operator operations based on the push-down task results returned by the index node 301, and return the obtained operation results to the coordinating node 203 through the second task result.
[0227] For example, suppose a query request asks for the total sales volume of products with a sales date of July 17, 2022, and a sales volume greater than 300. The data query task carries operator information including the `scanandfilter` operator and the `aggregation` operator. Worker nodes can push the `scanandfilter` operator down to the index nodes. The index nodes use the `scanandfilter` operator to determine the target data that meets the filtering conditions. Then, the worker nodes use the `aggregation` operator to sum the target data and return the sum to the coordinating node 203. Finally, the coordinating node 203 sums the multiple sums returned by the multiple index nodes to obtain the total sales volume value of products with a sales volume greater than 300 on July 17, 2022. At this point, the data query result includes this total sales volume value.
[0228] In another implementation, the worker node performs all data query tasks. For example, in... Figure 2B In the architecture shown, when worker node 204A executes a data query task, it determines the target row group through index node 301A (index node 301A uses the row group index of the target block to filter the row groups within the target block to obtain one or more target row groups, as described above, and will not be repeated here). Then, worker node 204A uses the scanandfilter operator to read the data of the target row group from storage node 310A and filters the read data to determine the target data. In this method, worker node 204A reads the data of the target row group from storage node 310A, relative to... Figure 1 The way worker nodes read entire ORC files or entire stripes from storage nodes can reduce network data transmission volume and alleviate network bandwidth pressure and overhead.
[0229] Step 709: Each of the multiple worker nodes sends the result of the second task to the coordinating node 203.
[0230] Step 710: Coordinating node 203 generates data query results based on the received results of multiple second tasks.
[0231] In the example above, the query task results include the raw data that meets the filtering conditions as determined by the index node. The coordinating node 203 aggregates the raw data included in the results of multiple query tasks to generate a data query result, which includes the raw data carried in all the query task results received by the coordinating node 203.
[0232] Step 711: Coordinating node 203 sends the data query results to client 102.
[0233] Through the above design, the coordinating node 203 generates multiple index query tasks based on query requests and sends these tasks to multiple index nodes. These index nodes then execute index filtering operations in parallel, improving data processing efficiency. Each index node can perform multi-granularity filtering operations through multi-level indexes, reducing the amount of index readings and filtering operations. Furthermore, compared to existing methods, even if the coordinating node 203 fails, it can still execute index filtering operations and overcomes the memory limit of a single node (existing coordinating node), potentially increasing indexing capacity by up to N times compared to existing solutions.
[0234] Based on the same inventive concept as the method embodiments, this application also provides a computing device for performing the above-described... Figure 5 , Figure 7 The method executed by the inode in the method embodiment. For example... Figure 8 As shown, the computing device 800 includes a receiving module 801, a processing module 802, and a sending module 803; specifically, in the computing device 800, the modules are connected to each other through a communication path.
[0235] The receiving module 801 is used to receive an index query task from the coordinating node. The index query task includes information about the query object and query conditions. For details on the implementation, please refer to [link to implementation details]. Figure 7 The description of step 703 in the previous section will not be repeated here.
[0236] Processing module 802 is used to determine, based on the received index query task, the target shards within the data range managed by the computing device that belong to the query object and include data that meets the query conditions; for specific implementation details, please refer to [link to implementation details]. Figure 7 The description of step 704 in the previous section will not be repeated here.
[0237] The sending module 803 is used to send the information of the target fragment to the coordinating node. For details on its implementation, please refer to [link to implementation details]. Figure 7 The description of step 705 in the previous section will not be repeated here.
[0238] Figure 9 This is a schematic diagram of a computing device provided in an embodiment of this application. The computing device is used to perform the above-described... Figure 5 , Figure 7 The method described in this embodiment involves the execution of an inode. The computing device 900 includes a processor 901, a memory 902, and a communication interface 903. The processor 901, memory 902, and communication interface 903 can be connected via a bus 904.
[0239] The processor 901 is used to execute the instructions stored in the memory 902 so that the computing device 900 executes the data processing method provided in this application. The processor 901 can be any one or more of the following processors, including but not limited to: a central processing unit (CPU), a data processing unit (DPU), a system on chip (SOC), a field programmable gate array (FPGA), a graphics processing unit (GPU), and an application-specific integrated circuit (ASIC).
[0240] Memory 902 is used to store computer instructions and data, such as computer instructions and data required to implement the data processing method provided in this application. Memory 902 includes volatile memory, such as random access memory (RAM), dynamic random access memory (DRAM), etc. It can also be non-volatile memory, such as read-only memory (ROM), storage-class memory (SCM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0241] The memory 902 stores executable program code, and the processor 901 executes the executable program code to implement the functions of the aforementioned receiving module 801, processing module 802, and transmitting module 803, thereby realizing the data processing method. That is, the memory 902 stores instructions for the computing device 800 to execute the data processing method provided in this application.
[0242] Communication interface 903 is used to communicate with internal or external devices, such as receiving index query tasks sent by coordinating node 203, or communicating with storage node 310 to complete data access. For example, communication interface 903 may be a network interface card (NIC).
[0243] The 904 bus can be a Peripheral Component Interconnect Express (PCIe) bus, a Double Data Rate (DDR) bus, a Serial Advanced Technology Attachment (SATA) bus, a Serial Attached SCSI (SAS) bus, a Controller Area Network (CAN) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 904 is represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 904 may include a path for transmitting information between various components of the computing device 900 (e.g., memory 902, processor 901, communication interface 903).
[0244] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to perform the above-described... Figure 5 or Figure 7 The method performed by the index node in the embodiment is described in [reference]. Figure 5 or Figure 7 The details of each step will not be repeated here.
[0245] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the above-described... Figure 5 or Figure 7 The method performed by the index node in the embodiment is described in [reference]. Figure 5 or Figure 7 The details of each step will not be repeated here.
[0246] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.
[0247] Those skilled in the art will understand that the various numerical designations such as "first," "second," etc., used in this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application, nor do they indicate a sequential order. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. "At least one" means one or more. "At least two" means two or more. "At least one," "any one," or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. "Multiple" means two or more, and other quantifiers are similar. Furthermore, for elements appearing in the singular forms "a," "an," and "the," unless the context explicitly specifies otherwise, it does not mean "one or only one," but rather "one or more than one." For example, "a device" means one or more such devices.
[0248] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0249] The various illustrative logic units and circuits described in the embodiments of this application can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can also be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.
[0250] The steps of the methods or algorithms described in the embodiments of this application can be directly embedded in hardware, software units executed by a processor, or a combination of both. The software units can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC.
[0251] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0252] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely illustrative descriptions of the application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Thus, if such modifications and modifications fall within the scope of the claims and their equivalents, this application is also intended to include such modifications and modifications.
Claims
1. A data processing system, characterized in that, The system includes a coordination node, multiple index nodes, and multiple worker nodes; The coordination node is used to generate a first query task according to the query request and send the first query task to the index node. The query request is used to query data that meets the conditions in the target object. The target object is divided into multiple shards, and the multiple shards are stored in at least one storage node. The target object is divided into multiple data ranges, and each of the multiple index nodes manages a data range of the target object. The first index node among the plurality of index nodes is used to determine, based on the received first query task, the target shard of the data queried by the query request within the first data range managed by the first index node, and to send the information of the target shard to the coordinating node. The first index node is any one of the plurality of index nodes. The coordinating node receives information about multiple target shards sent by the multiple index nodes, generates a second query task for the multiple target shards, and schedules the multiple worker nodes to execute the second query task for the multiple target shards.
2. The system as described in claim 1, characterized in that, The first index node corresponds to the index information of the fragment corresponding to the first data range; When determining the target shard, the first index node is specifically used for: The target shard, including the data queried by the query request, is determined based on the index information of the shard corresponding to the first data range.
3. The system as described in claim 1 or 2, characterized in that, Each shard and its corresponding index information are stored independently.
4. The system as described in claim 2, characterized in that, The target object is divided into multiple files, each file is divided into one or more data blocks, and each data block corresponds to a shard; the first query task includes information about the query object, and the data of the target object includes the data of the query object; The first index node determines the target shard that includes the data queried by the query request within the first data range managed by the first index node, specifically for: Using the index information of the file corresponding to the query object within the first data range, at least one target file is filtered out, and each target file includes the data queried by the query request. Using the index information of the data block corresponding to each target file, at least one target data block is filtered out, and the target data block is the target fragment.
5. The system as described in claim 2, characterized in that, The target object is divided into multiple files, each file is divided into one or more data blocks, and each data block corresponds to a shard; the first query task includes information about the query object, and the data of the target object includes the data of the query object; The first index node determines the target shard that includes the data queried by the query request within the first data range managed by the first index node, specifically for: Using the index information of the data block corresponding to the query object within the first data range, at least one target data block is filtered out, and the target data block is the target fragment.
6. The system as described in claim 4 or 5, characterized in that, The multiple files are divided into multiple partitions, and the target partition is the partition that includes the target fragment among the multiple partitions; The query object includes the target partition; or The query object includes the multiple files corresponding to the target object; or The query object includes at least one file corresponding to the target partition; or The query object includes at least one file belonging to the target object within the first data range; or The query object includes at least one file belonging to the target partition within the first data range.
7. The system as described in claim 4 or 5, characterized in that, The multiple files are divided into multiple partitions, and the target partition is the partition that includes the target fragment among the multiple partitions; The query object includes at least one data block contained in at least one file corresponding to the target object; or The query object includes at least one data block contained in at least one file corresponding to the target partition; or The query object includes at least one data block comprised of at least one file belonging to the target object within the first data range; or The query object includes at least one data block comprised of at least one file belonging to the target partition within the first data range.
8. The system as described in claim 1, 2, 4 or 5, characterized in that, The first index node and storage node are deployed on the same physical machine.
9. The system as described in claim 1, 2, 4 or 5, characterized in that, The first index node and worker node are deployed on the same physical machine.
10. A computing device, characterized in that, include: The receiving module is used to receive a first query task from the coordinating node, the first query task including information about the target object and query conditions; The target object is divided into multiple data ranges, and each of the multiple data ranges corresponds to a multiple computing device; The processing module is configured to determine, based on the received first query task, a target fragment that belongs to the target object and includes data that satisfies the query conditions within the data range managed by the computing device itself; The sending module is used to send the information of the target shard to the coordinating node. The information of the target shard is used to generate a second query task. The second query task is a task executed by multiple worker nodes for multiple target shards.
11. The apparatus as claimed in claim 10, characterized in that, The computing device corresponds to the index information of the slice within the first data range; When determining the target shard, the processing module is specifically used to: determine the target shard including the data queried by the query request based on the index information of the shard corresponding to the first data range.
12. The apparatus as claimed in claim 11, characterized in that, The target object is divided into multiple files, each file is divided into multiple data blocks, and each data block corresponds to a shard; the first query task includes information about the query object, and the data of the target object includes the data of the query object; The processing module determines that the target fragments of the data queried by the query request are included in the first data range managed by the computing device. Specifically, it is used to: use the index information of the file corresponding to the query object in the first data range to filter out at least one target file, and each target file includes the data queried by the query request. Using the index information of the data block corresponding to each target file, at least one target data block is filtered out, and the target data block is the target fragment.
13. The apparatus as claimed in claim 11, characterized in that, The target object is divided into multiple files, each file is divided into multiple data blocks, and each data block corresponds to a shard; the first query task includes information about the query object, and the data of the target object includes the data of the query object; The processing module determines that the target shard of the data queried by the query request is included in the first data range managed by the computing device. Specifically, it is used to: use the index information of the data block corresponding to the query object in the first data range to filter out at least one target data block, wherein the target data block is the target shard.
14. The apparatus as claimed in claim 12 or 13, characterized in that, The multiple files are divided into multiple partitions, and the target partition is the partition that includes the target fragment among the multiple partitions; The query object includes the target partition; or The query object includes the multiple files corresponding to the target object; or The query object includes at least one file corresponding to the target partition; or The query object includes at least one file belonging to the target object within the first data range; or The query object includes at least one file belonging to the target partition within the first data range.
15. The apparatus as claimed in claim 13, characterized in that, The multiple files are divided into multiple partitions, and the target partition is the partition that includes the target fragment among the multiple partitions; The query object includes at least one data block contained in at least one file corresponding to the target object; or The query object includes at least one data block contained in at least one file corresponding to the target partition; or The query object includes at least one data block comprised of at least one file belonging to the target object within the first data range; or The query object includes at least one data block comprised of at least one file belonging to the target partition within the first data range.
16. A computing device, characterized in that, Including processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the function of the first index node in the system as described in any one of claims 1-9.
17. A computer-readable storage medium, characterized in that, When the computer-readable storage medium is executed by the storage device, the storage device performs the function of the first index node in the system according to any one of claims 1-9.
Citation Information
Patent Citations
Distributed system, data aggregation method and computer readable storage medium
CN113132267A
