A system, method, and apparatus for data querying using a network device

Abstract

The application provides a system, method and device for data query using a network device. A center node of the system is connected to worker nodes through a network device. The network device is, for example, a network card, a switch, a router, etc. The center node generates a plurality of tasks from a query request input by a user. When assigning execution devices to the plurality of tasks, the center node configures execution devices of some tasks as network devices and execution devices of some tasks as worker nodes, and then sends configuration instructions to the configured network devices and worker nodes to set the tasks configured for them on the network devices and worker nodes. After the configuration is completed, since data transmitted between the worker nodes and the center node all pass through the network devices, the network devices execute the set tasks on the passing data, thereby reducing the computational load of the worker nodes, reducing the burden on the processors of the worker nodes and speeding up data processing.

Description

[0001] This application is a divisional application. The original application has the application number 202011588814.3 and the original application date is December 29, 2020. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a system, method, and apparatus for data querying using network devices. Background Technology

[0003] In today's information age, with the rapid development of computer and information technology, the amount of data generated is also growing rapidly. The amount of data stored in databases has reached hundreds of terabytes (1TB = 1024GB) or even tens to hundreds of petabytes (1PB = 1024TB). These data come from numerous sources, are enormous in quantity, and vary in form, making it extremely important to know how to quickly find the target data in the database.

[0004] With the massive increase in data volume, the current main approach to ensure data query efficiency is to increase hardware resources for data processing, such as increasing the processing power and memory capacity of processors (e.g., central processing units, CPUs) at each node in the query system. However, increasing processor processing power and memory capacity leads to increased product costs. In addition, the growth potential of processor processing power is limited, so sometimes it is not possible to improve query efficiency by enhancing processor processing power. Summary of the Invention

[0005] This application provides a data query method and apparatus to accelerate data processing without increasing CPU performance and / or memory capacity.

[0006] Firstly, this application provides a data query system. The system includes a central node, worker nodes, and network devices, with the central node connected to the worker nodes via the network devices. The central node generates multiple tasks from user-input query requests. When assigning execution devices to these tasks, the central node configures some tasks as network devices and others as worker nodes, then sends configuration instructions to configure the corresponding tasks on the network devices and worker nodes. After the network devices and worker nodes are configured with their assigned tasks, the pre-configured tasks can be executed on the data as it flows through the network devices.

[0007] Through the above design, the central node can configure some tasks to be executed by network devices. In this way, when data flows through the network devices, the network devices execute the pre-configured tasks and then forward them to other execution devices. Compared with the scheme in related technologies where all tasks are handled by worker nodes, this reduces the amount of computation on worker nodes and alleviates the burden on the processors on worker nodes. Thus, data processing can be accelerated without increasing the processing power of the processors on worker nodes.

[0008] In one possible implementation, after generating multiple tasks, the central node is used to find unloadable tasks among the multiple tasks and set the execution device of the unloadable task as the network device, wherein the unloadable task is a preset task to be unloaded to the network device for execution.

[0009] Through the above design, suitable uninstallable tasks can be pre-configured for unloading to network devices, and uninstallable tasks can be quickly found from the multiple tasks.

[0010] In one possible implementation, the central node is used to send the setting instructions for the undeletable task to the network device; the network device is used to set the undeletable task according to the setting instructions.

[0011] In one possible implementation, the network device is a network interface card (NIC) of a worker node or a forwarding device, and the forwarding device can be a switch or a router.

[0012] In one possible implementation, the forwarding device includes a data port and a control port; the central node is used to send the setting instructions for the undeletable task to the forwarding device through the control port of the forwarding device; it is also used to send the setting instructions for tasks whose execution device is a network card or a worker node to the forwarding device through the data port; correspondingly, when the forwarding device receives the setting instructions through the control port, it sets the undeletable task indicated in the setting instructions; and when it receives the setting instructions from the data port, it forwards the setting instructions received from the data port.

[0013] With the above design, the forwarding device can quickly distinguish the data packets that need to be forwarded based on the data port and forward them to the corresponding devices without parsing the data packets, thus reducing transmission latency; and it can distinguish the setting instructions sent by the central node based on the control port, thus avoiding misforwarding and missing configurations.

[0014] In one possible implementation, when the network device executing the undeletable task is the network interface card (NIC) in a worker node, the central node sends the setting instructions for the undeletable task to the worker node, and the worker node sets the undeletable task on the NIC according to the setting instructions.

[0015] With the above design, the worker node sets up undoneable tasks on the network card according to the setting instructions. When the worker node has an integrated undone policy, it can also determine whether to undone to the network card based on the actual load of the network card. This makes the way the worker node controls the network card to execute undoneable tasks more flexible.

[0016] In one possible implementation, the configuration instruction for the undeletable task includes an undeletable flag; upon receiving the configuration instruction, if the worker node determines that the configuration instruction includes the undeletable flag, it configures the undeletable task on the worker node's network interface card.

[0017] In one possible implementation, after receiving a data packet, the network device executes the offloadable task on the data packet when it determines that the data packet contains an identifier of an offloadable task to be performed by the network device.

[0018] With the above design, network devices can monitor data packets of undeletable tasks based on the identifier of the undeletable task, and execute the undeletable task on the data packet. This eliminates the need for separate execution instructions, allowing for quick and accurate identification of undeletable tasks executed by the network device, saving overhead, and accelerating data processing.

[0019] In one possible implementation, the central node is further configured to send a setting instruction for the uninstallable task after determining that the uninstallable task conforms to the uninstallation strategy corresponding to the uninstallable task.

[0020] Through the above design, it is possible to further determine whether the unloadable task is suitable for execution on a network device based on actual factors such as the network environment, thereby further improving the efficiency of data query.

[0021] In one possible implementation, a task is used to indicate the operation to be performed and the operands, where the operands are the data of the operation to be performed; the task setting instructions may include a task identifier and operator information, wherein the task identifier is used to uniquely identify a task in a query request, and the operator information includes an operator identifier, which uniquely identifies an operator. An operation can be completed by running one or more operators, and the operators are run to perform the operation indicated by the task on the operands.

[0022] In one possible implementation, an unloadable task refers to any operator required to complete the task being an unloadable operator; wherein, the unloadable operator can be a preset operator; for example, unloadable operators include: filter operator, aggregation operator, distinct operator, TopN operator, Join operator, etc.; or unloadable operators are preset operators that satisfy the corresponding unloadable strategy; for example, preset operators and their corresponding unloadable strategies include: filter operator, whose corresponding unloadable strategy is that the selectivity of performing the filter operator on the filter column is not lower than a preset threshold (e.g., a first preset value); aggregation operator, whose corresponding unloadable strategy is that when performing the aggregation operator on the aggregation column, the cardinality of the data being aggregated on the aggregation column does not exceed a second preset value; distinct operator, whose corresponding unloadable strategy is that the cardinality of the data on the column that needs to be deduplicated does not exceed a third preset value; wherein, the first preset value, the second preset value, or the third preset value can be exactly the same value, or not exactly the same value, or completely different value.

[0023] Secondly, this application provides a data query method that can be applied to a central node connected to worker nodes via network devices. The method includes: the central node generating multiple tasks from a user-input query request; when assigning execution devices to the multiple tasks, the central node configures some task execution devices as network devices and some task execution devices as worker nodes, and then sends configuration instructions to configure the corresponding tasks on the network devices and worker nodes.

[0024] In one possible implementation, after a worker node or network device has completed setting the task indicated by the configuration command, it sends a feedback response to the central node to indicate that the configuration of the task issued by the central node has been completed. Subsequently, the central node can send an execution command for the query request, which is used to trigger the execution device to execute the set task.

[0025] In one possible implementation, determining the execution device for each of the plurality of tasks includes: after generating the plurality of tasks, the central node searches for unloadable tasks among the plurality of tasks and sets the execution device of the unloadable task as a network device, wherein the unloadable task is a preset task to be unloaded to the network device for execution.

[0026] In one possible implementation, when it is determined that the execution device of the unloadable task is a network device, the central node sends a setting instruction for the unloadable task to the network device.

[0027] In one possible implementation, the network device can be a network interface card (NIC) of a worker node or a forwarding device, which can be a switch or a router.

[0028] In one possible implementation, when the network device executing the offloadable task is determined to be the network card of the worker node, the central node sends the setting instruction for the offloadable task to the worker node, and the worker node controls the setting of the offloadable task in the network card.

[0029] In one possible implementation, when the central node determines that the network device performing the offloadable task is a forwarding device, it sends the setting instruction for the offloadable task to the forwarding device.

[0030] In one possible implementation, the setting instruction carries an unloadable flag.

[0031] In one possible implementation, the uninstallable task can be preset. After the central node determines the uninstallable task from among multiple tasks and confirms that the uninstallable task conforms to the uninstallation policy corresponding to the uninstallable task, it sends the setting instruction of the uninstallable task to the network device.

[0032] In one possible implementation, for an unloadable task that can be unloaded to multiple devices, the central node can determine the execution device according to the preset priority of each device corresponding to the unloadable task when determining the execution device for the unloadable task.

[0033] In one possible implementation, when determining the execution device for the unloadable task, the central node can also determine the execution device based on the priority of each device corresponding to the pre-defined unloadable task and the load status of each device.

[0034] Regarding the beneficial effects of this second aspect, please refer to the description of the beneficial effects of the central node execution method in the first aspect, which will not be repeated here.

[0035] Thirdly, this application provides a data query method that can be applied to a network device for connecting a central node and worker nodes. The method includes: the network device receiving a setting instruction sent by the central node, setting a corresponding task according to the setting instruction, and executing the task on data packets flowing through the network device.

[0036] In one possible implementation, the network device can be a network interface card (NIC) of a worker node or a forwarding device, such as a switch or router.

[0037] In one possible implementation, the forwarding device includes a data port and a control port. The forwarding device can receive configuration instructions from the control port, and the data received through the control port is used by the central node to configure the forwarding device. The forwarding device then configures the unloadable tasks according to the configuration instructions received through the control port. Alternatively, the forwarding device can also receive configuration instructions from the data port, and the data received through the data port is used by the central node to configure other devices besides the forwarding device. The forwarding device then forwards the data received from the data port.

[0038] In one possible implementation, after receiving a data packet, the network device executes the undeletable task based on the data packet if it determines that the data packet contains an identifier of the undeletable task to be performed by the network device.

[0039] Regarding the beneficial effects achieved by this third aspect, please refer to the description of the beneficial effects of the network device execution method in the first aspect, which will not be repeated here.

[0040] Fourthly, embodiments of this application also provide a data query interface, including: a query command input area, a task display area, and an execution device display area;

[0041] The query command input area is used to receive query requests input by the user.

[0042] The task display area is used to display multiple tasks generated based on the query request to execute the query request;

[0043] The execution device display area is used to display the execution device for each task, which includes worker nodes and network devices.

[0044] In one possible implementation, the query command input area, task display area, and execution device display area are displayed on the same interface.

[0045] In one possible implementation, the query command input area, task display area, and execution device display area are displayed on different interfaces.

[0046] Fifthly, this application also provides a data query interaction method. This method can utilize a central node, which serves as the server for the client. The method includes: a user inputting a query request on the client; the client forwarding the query request to the central node; the central node receiving the query request and generating multiple tasks based on it; further, the central node generating an execution plan for the query request, which includes information about the execution device for each task. The central node can assign tasks to worker nodes or network devices for execution; that is, the execution device can be either a worker node or a network device. The central node can display the execution plan for the multiple tasks locally, or it can send the execution plan to the client. Upon receiving the execution plan, the client can display it, including the multiple tasks and the execution device for each task.

[0047] In one possible implementation, the multiple tasks on the client are displayed in a tree structure according to the execution plan.

[0048] In one possible implementation, the execution progress of the multiple tasks is displayed.

[0049] The above design allows users to intuitively understand the execution plan and progress of the query request, thereby improving user engagement and user experience.

[0050] Sixthly, embodiments of this application also provide a central device comprising multiple functional units capable of performing the functions executed in the various steps of the method in the second aspect. These functional units can be implemented in hardware or software. In one possible design, the device includes a detection unit and a processing unit.

[0051] In a seventh aspect, embodiments of this application also provide a network device comprising multiple functional units capable of performing the functions executed in the various steps of the method in the third aspect. These functional units may be implemented in hardware or software. In one possible design, the device includes a detection unit and a processing unit.

[0052] Eighthly, embodiments of this application also provide a central device, which includes a processor, a memory, and a transceiver. The memory stores program instructions, and the processor executes the program instructions in the memory to communicate with other devices through the transceiver to implement the method provided in the second aspect.

[0053] In a ninth aspect, embodiments of this application also provide a network device including at least one processor and an interface circuit, the processor being configured to communicate with other devices via the interface circuit to implement the method provided in the third aspect.

[0054] The processor can be a field programmable gate array (FPGA), a data processing unit (DPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a system on chip (SOC).

[0055] In a tenth aspect, this application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in the second aspect or the method provided in the third aspect. Attached Figure Description

[0056] Figure 1 A schematic diagram of a system architecture provided for an embodiment of this application;

[0057] Figure 2 A schematic diagram of a query system architecture provided in this application embodiment;

[0058] Figure 3 This application provides a schematic diagram of the internal structure of a working node according to an embodiment of the present application.

[0059] Figure 4 A schematic diagram of a network architecture provided for an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of another network architecture provided in an embodiment of this application;

[0061] Figure 6 This is a schematic diagram corresponding to a data query method provided in an embodiment of this application;

[0062] Figure 7 A schematic diagram of an execution plan provided in an embodiment of this application;

[0063] Figure 8 A schematic diagram of an interface for another execution plan provided in an embodiment of this application;

[0064] Figure 9 A schematic diagram illustrating network interface card (NIC) resource allocation as provided in an embodiment of this application;

[0065] Figure 10 A flowchart illustrating another data query method provided in this application embodiment;

[0066] Figure 11 A schematic diagram of a device structure provided in an embodiment of this application;

[0067] Figure 12 This is a schematic diagram of the device structure of a network device provided in an embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0069] The network architecture and business scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0070] See Figure 1 This is a schematic diagram of a system architecture that may be applicable to embodiments of this application. The system includes a client 10, a query system 20, and a data source 30.

[0071] Client 10 is a computing device on the user side, such as a desktop computer or laptop. At the hardware level, client 10 includes a processor and memory. Figure 1 (Not shown in the image). At the software level, a client program runs on client 10. The client program is used to receive query requests triggered by the user and interact with query system 20, such as sending the query request to query system 20. Correspondingly, a server program runs on query system 20 to interact with the client program, such as receiving query requests sent by client 10. Query system 20 is also used to obtain the raw data requested in the query from data source 30, and to perform calculations or processing on the raw data to obtain the query results (or target data). Subsequently, query system 20 returns the obtained query results to client 10.

[0072] Data source 30 can refer to a database or database server. In this embodiment, it refers to a data source that the query system can analyze, such as a MySQL data source, an Oracle data source, or a HIVE data source. The data storage format can be HDFS (Hadoop Distributed File System) files, ORC (Optimized Row Columns) files, CSV (comma-separated values) files, or semi-structured data such as XML (eXtensible Markup Language) or JSON (JavaScript Object Notation). Of course, the above examples are only illustrations, and this application embodiment does not limit the data source or data storage format. The data source can adopt distributed storage. Correspondingly, at the hardware level, the data source can include one or more storage nodes, where the storage nodes can be storage servers, desktop computers, or the controllers and hard drives of storage arrays.

[0073] To improve query efficiency, query systems can employ a massively parallel processing (MPP) architecture. For example, the Presto query engine is an open-source MPP SQL (structured query language) engine, a distributed SQL query engine used to query large datasets distributed across one or more different data sources, suitable for interactive analytical queries. Specifically, an MPP architecture involves distributing tasks in parallel across multiple servers or nodes, with each server or node executing the tasks concurrently. For instance, a student information table contains student names, ages, student IDs, etc. If a user triggers a query request to find students with the name "Xiaoming" in the table, in an MPP-based query system, multiple nodes can perform queries based on subsets of rows from the student information table. This shortens the query time, reducing the overall query duration and improving efficiency. It should be understood that the more nodes involved in the query, the shorter the query time for the same request.

[0074] The following uses the MPP architecture as an example to specifically describe the query system provided in the embodiments of this application.

[0075] Please refer to Figure 2 The query system in this embodiment mainly includes a central node cluster and a worker node cluster. For example... Figure 2 As shown, the central node cluster includes one or more central nodes ( Figure 2Only two central nodes, 100 and 101, are shown in the diagram, but this application does not limit the number of central nodes. A worker node cluster includes one or more worker nodes. Figure 2 The image shows three working nodes 20a, 20b and 20c, but this application is not limited to three working nodes.

[0076] The central node receives query requests from clients, parses them into one or more tasks, and then distributes these tasks in parallel to multiple worker nodes. The worker nodes can process the assigned tasks concurrently. It should be understood that the central node can distribute tasks to some or all worker nodes in the query system. Furthermore, the tasks assigned to each worker node can be identical, different, or completely different; this embodiment does not limit this. It should be noted that the central node can be a node elected from among the worker nodes to assume the functions of a central node, or it can be a specific device. Additionally, when multiple central nodes exist in the query system, a query request sent by a client will be routed to any one of the central nodes. In this way, multiple central nodes in the query system can respond to multiple query requests simultaneously, which can be sent by multiple clients or by a single client.

[0077] Worker nodes receive tasks from the central node and execute them. For example, tasks include retrieving data from a data source and performing various calculations on the retrieved data. Since each task can be processed in parallel by different worker nodes, the results of the parallel processing are finally aggregated and fed back to the client.

[0078] See also Figure 2 In terms of hardware, the central node and worker nodes include at least a processor, memory, and network interface card (NIC). The following section will use worker node 20a as an example to describe the connection relationships and operating methods of the aforementioned hardware.

[0079] The following description uses working node 20a as an example. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of the internal structure of working node 20a. (See diagram below.) Figure 3 As shown, the working node 20a mainly includes a processor 201, a memory 202, and a network interface card 203. The processor 201, memory 202, and network interface card 203 communicate with each other through a communication bus.

[0080] The processor 201 can be a central processing unit (CPU), used for data calculation or processing. The memory 202 refers to a device for storing data, including RAM and hard disks. RAM can be read and written to at any time, and its read speed is very fast, serving as temporary data storage for running programs. RAM includes at least two types of memory, such as Random Access Memory (RAM) and Read Only Memory (ROM). Compared to RAM, hard disks have slower read and write speeds and are typically used for persistent data storage. Hard disk types include at least solid-state disks (SSDs), mechanical hard disks (HDDs), or other types of hard disks. Generally, data on the hard disk needs to be read into RAM first, and the processor 201 or computing unit 221 retrieves data from RAM. The memory resources of the processor 230 and the computing unit 221 can be shared or independent; this embodiment does not limit this.

[0081] Network interface card 203 is used for data interaction and data processing. At the hardware level, network interface card 203 includes at least a communication unit 220 and a computing unit 221. Figure 3 This example illustrates a computing unit, but is not limited to this specific unit. The communication unit 220 provides efficient network transmission capabilities for receiving data input from external devices or sending data output from the device itself. The computing unit 221 includes, but is not limited to, field-programmable gate arrays (FPGAs), data processing units (DPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and system-on-chips (SoCs). This embodiment uses an FPGA as an example. FPGAs possess the versatility and programmability of CPUs, but are more specialized, capable of efficiently operating on network packets, storage requests, or analysis requests. FPGAs differ from CPUs due to their higher degree of parallelism (requiring the processing of numerous requests).

[0082] In this application, worker nodes or central nodes can be deployed on at least one physical node. For example, a worker node and a central node can be deployed on the same server. Alternatively, a central node and a worker node can be deployed on two separate servers, and so on. Furthermore, a storage node can also be a separate device, such as a storage server. It should be noted that the aforementioned nodes can be deployed on physical machines or virtual machines, and this application does not impose any limitations on this.

[0083] In practical applications, Figure 1 The system shown also includes forwarding devices, such as switches or routers. For ease of description, a switch will be used as an example below. Switches can be used for data forwarding. In this embodiment, any two nodes, such as between a central node and a worker node, between a central node and a storage node, between worker nodes, and between a worker node and a storage node, can be interconnected to achieve collaborative computing.

[0084] Please see Figure 4 This is a schematic diagram of the physical architecture of the query system provided in the embodiments of this application in a real-world application scenario. Figure 4 In the query system shown, the worker node cluster, besides Figure 3 The cluster includes worker nodes 20a, 20b, and 20c, as well as worker nodes 20d and 20e; the central node cluster only includes central node 100; the data source includes storage nodes 30a, 30b, and 30c; and the forwarding devices include switch 10, switch 20, switch 30, and switch 40.

[0085] like Figure 4 As shown, the central node, worker nodes, and storage nodes are all independent physical machines. Among them, worker nodes 20a, 20b, central node 100, and switch 10 are installed on rack 1; worker nodes 20c, 20d, 20e, and switch 20 are installed on rack 2; and storage nodes 30a, 30b, 30c, and switch 30 are installed on rack 3.

[0086] Within the same rack, nodes can interact via switches within the rack. For example, within rack 1, the central node 100 can interact with any other node within rack 1, such as worker node 20a, via switch 10. Exemplarily, in this embodiment, the data exchanged between the central node and worker nodes includes at least a header and a data portion. The header includes a source IP address and a destination IP address, and the data portion is the data itself to be transmitted. Exemplarily, the process of the central node 100 sending data to worker node 20a can include: the central node 100 sending data to worker node 20a, where the source IP address is the central node 100's IP address and the destination IP address is the worker node 20a's IP address. Specifically, when the central node 100 sends the data, it is first routed to switch 10, which then forwards the data to worker node 20a based on the destination IP address carried in the data.

[0087] Figure 4 The system also includes switch 40, which is the core switch. Switches 10, 20, and 30 can be referred to as tor switches within their respective racks. The core switch enables data exchange between nodes on different racks. For example, data exchange between nodes on rack 1 and rack 3 can be achieved through switches 10, core switch 40, and switch 30. For instance, when worker node 20a sends data to storage node 30a, the data transmission path is as follows: the data is first routed to switch 10. Switch 10 checks that the destination IP address is not in the same network segment as its own IP address, and then forwards the data to core switch 40. Core switch 40 forwards the data to switch 30, which is in the same network segment as the destination IP address. Switch 30 then forwards the data to storage node 30a corresponding to the destination IP address. The installation location of switch 40 is not limited; for example, it can be installed in… Figure 4 On any shelf in the middle.

[0088] In addition to data forwarding, the switch in this embodiment also has computing and data processing capabilities, such as a programmable switch.

[0089] It should be noted that, Figure 4 The system architecture shown is for illustrative purposes only; please refer to [link / reference]. Figure 5 This provides another system architecture diagram for the embodiments of this application, in which the storage node can also be deployed on the worker node. For example, the data in the data source can be stored on the hard disk of the worker node. The embodiments of this application do not limit the system architecture or the deployment form of each node.

[0090] In general, to address the problems mentioned in the background art, this application provides a data query method. The central node receives a query request sent by the client and parses the query request into one or more tasks. Then, the central node can offload some of these tasks to network devices for processing, such as the network cards or forwarding devices of the worker nodes. Compared with the method of distributing tasks to worker nodes for processing, this reduces the computational load on the worker nodes and the CPU burden on the worker nodes, thereby improving the data processing speed without increasing the hardware resources of the worker nodes.

[0091] The data query method provided in this application will be described in detail below with reference to the accompanying drawings and embodiments.

[0092] Please see Figure 6 This is a flowchart illustrating the data query method provided in this embodiment. In this embodiment, the central node identifies unloadable tasks from multiple tasks and instructs worker nodes to unload these unloadable tasks to the network interface card (NIC) for processing. This method can be applied to... Figure 4 or Figure 5 In the system architecture shown, the method mainly includes the following steps:

[0093] Step 601: The client sends a query request to the central node, and the corresponding central node receives the query request sent by the client.

[0094] The query request is triggered by the user on the client. For example, if the query system is an MPP SQL engine, the query request can be an SQL statement. The following uses an SQL statement as an example to introduce the query request. In fact, the embodiments of this application do not limit the statement of the query request.

[0095] First, let's introduce the following two lists. These lists represent the raw data requested in the user-triggered query request. Table 1 (named factTbl) contains the sales records of a certain merchant, used to record the merchant's transaction history. Table 2 (named dimTbl) is the product name table, used to record the identifiers and names of the products sold by this merchant. It should be understood that Tables 1 and 2 only show partial data.

[0096] Table 1. factTbl

[0097] Id (Product Identifier) Sales (Amount) / Yuan Data (Date) 1 100 2020.01.01 2 150 2020.01.01 1 100 2020.01.02 3 200 2020.01.02 … … …

[0098] Table 2.dimTbl

[0099] Id (Product Identifier) Name (Product Name) 1 Water Cup A 2 Water Cup B 3 Water Cup C 4 Water Cup D … …

[0100] Suppose the query request is that a user wants to retrieve the total sales amount for each product in Table 2 on the date 2020 / 01 / 02, as shown in Table 1. For example, the corresponding SQL statement for this query request is as follows:

[0101] SELECT dimTbl.name,sum(factTbl.sale)

[0102] FROM factTbl JOIN dimTbl

[0103] ON factTbl.ID = dimTbl.ID

[0104] WHERE factTbl.day='20200102'

[0105] GROUP BY dimTbl.name.

[0106] Step 602: The central node parses the query request into one or more tasks.

[0107] In this embodiment, a query request can be broken down into one or more tasks. Specifically, a task includes some or all of the following information: information about the data to be operated on, operator information, and operation rules. The information about the data to be operated on indicates the data to be operated on, which is the object being operated on. The operator information includes the operator's identifier, which indicates the operator; one operator represents one type of operation. The operation rules refer to the rules for performing the operation, and can also be understood as the rules for the operator. For example, the SQL statement "WHERE factTbl.day='20200102'" can be parsed into a filtering task. In this filtering task, the data to be operated on is table 1 (factTbl), the operator identifier is the identifier of the filter operator, the operation represented by the filter operator is filtering, and the rule of the filter operator in this task is data=20200102.

[0108] The following explains some operators that may be used in the embodiments of this application.

[0109] 1. Tablescan (sequential list scan) operator

[0110] The `tablescan` operator represents a scan operation, used to read all rows in all pages of a table according to the order in which rows are stored in the database.

[0111] 2. Filter Operator

[0112] The `Filter` operator represents a filtering operation, used to filter rows in a table according to rules (or filtering conditions) to obtain rows that meet the filtering criteria. The filtering column refers to one or more columns that need to be filtered. For example, if the filtering condition is to filter data in table 1 where `Data` = 2020 / 01 / 02, then the `Data` column in table 1 is the filtering column. Specifically, when executing task 1, you can filter each row in table 1 according to the filtering conditions while reading each row, or you can read a portion of the rows and then perform filtering based on that portion.

[0113] 3. Join (table union) operator

[0114] The Join operator represents a table union operation, used to reorganize and combine two tables based on conditions in one or more columns. It is often used to filter data in a larger table (a table with a relatively large amount of data, such as Table 1) based on one or more data items in a smaller table (a table with a relatively small amount of data, such as Table 2). It can also combine data from a smaller table with data from a larger table.

[0115] For example, the process using the Join operator includes: maintaining a Bloom filter (BF) on the columns of the On condition in the smaller table to be joined (e.g., the ID column in Table 2); then scanning the larger table; while scanning the larger table, matching the value of the column of the On condition (e.g., the ID column in Table 1) in each scanned row against the BF; if it does not exist in the BF, discarding the row; if it exists, retaining it. The Join operator can be used to combine certain columns from a smaller table with certain columns from a larger table, for example, combining the name columns of Table 1 and Table 2 based on the same ID value in both tables.

[0116] Specifically, the join operator includes broadcast join and hash join. Assuming the data to be joined includes Table 2, the broadcast join operation proceeds as follows: one worker node reads the complete Table 2 and then broadcasts this complete Table 2 to each of the other worker nodes executing the join operator. Hash join, on the other hand, allows multiple worker nodes to read one or more shards of Table 2 (shards will be explained in detail below), and then send their respective shards to the other worker nodes. In this way, each worker node can obtain the complete Table 2 based on the shards read by other worker nodes, and then perform a join based on Table 2.

[0117] 4. group by operator

[0118] The `GROUP BY` operator represents a grouping operation, used to group products according to a certain condition, such as grouping them by product name.

[0119] 5. Aggregation operator

[0120] Aggregation operators represent aggregation operations and mainly include: Sum, Min, Max, Count, and AVG. The Sum operator sums the values ​​to be aggregated; the Min operator maintains the minimum value among the aggregated values; the Max operator maintains the maximum value among the aggregated values; the Count operator counts the number of values ​​to be aggregated; and the AVG operator maintains the average of the cumulative sums among the aggregated values. For example, the execution flow using the aggregation operator is as follows: first, group the data according to the groupby column, for example, group by dimTbl.name, which groups the data according to the name column in Table 2; then, perform operations such as sum, min, max, count, or AVG on the grouped data. In this example, the sum operation is performed.

[0121] 6. Distinct (non-empty and unique) operator

[0122] The `Distinct` operator performs a deduplication operation, selecting unique, non-empty columns, or removing duplicate data. Specifically, it removes duplicates from rows containing data (non-empty) in the `distinct` column. For example, to determine how many types of products are in Table 2, the `name` column is the `distinct` column. Each row's `name` column is scanned row by row; if a `name` does not appear in the table, it is recorded. If the `name` appears again later, it is not recorded again. That is, each product name in the `name` column is recorded only once, thus allowing the determination of the total number of products in Table 2.

[0123] 7. TopN Operator

[0124] The TopN operator represents the operation of maintaining the maximum N values. Specifically, it is used to maintain the current maximum N values. When a new value comes in, if the new value is greater than the minimum value among the current maximum N values, it replaces the minimum value among the current maximum values.

[0125] The following explanation uses the SQL statement from step 601 as an example. It should be understood that executing an SQL statement is done in a specific order, through operations (or steps). The combination of these operations / steps used to execute the SQL statement is called the execution plan. The execution plan can also represent the complete execution process of the SQL statement. At the software level, for example, after the central node receives the SQL statement sent by the client, it can perform syntax parsing on the SQL statement, generate the execution plan, and then parse one or more tasks based on the execution plan. For example, a task may include one or more operations that execute the SQL statement; that is, a task can be executed using one or more operators.

[0126] For example, the execution plan of the above SQL statement includes: (1) scanning table 2 and reading all rows in table 2; (2) scanning table 1 and reading all rows in table 1; (3) filtering out the row data in table 1 with the date 2020 / 01 / 02; (4) filtering out the rows in the ID column of table 1 that have the same ID value as the ID value in table 2; according to the same ID value, combining the row in table 1 with the name column in table 2; (5) based on the combined row data, grouping according to name to obtain multiple groups of products, and calculating the total sales of each group of products.

[0127] Furthermore, considering the work nodes assigned to tasks, the execution plan can be divided into multiple stages, and a stage can include one or more tasks. For example, stages can be divided according to whether the nodes need to interact with each other, and the tasks within the same stage do not depend on the results of other nodes.

[0128] Please see Figure 7 This is a schematic diagram of an execution plan generated from parsing the above SQL statement. It is generated after the user enters the query statement. Figure 7 The execution plan shown can be specifically accessed by the user in the query command input area of ​​the client's data query interface. Figure 7 (Not shown) Enter the query statement. For example... Figure 7The execution plan shown can be displayed to the user through the central node or the client's display interface. The execution plan can be displayed directly to the user after they input a query command, or it can be displayed to the user when they need to view the execution plan by inputting a command to display it. Furthermore, the execution plan will also display the execution device for each task, such as a network interface card (NIC), worker node, router, or switch. The execution plan can be displayed simultaneously with the task, or the execution device can be displayed when the user clicks on a task. The interface for inputting the query command and the interface for displaying the execution plan can be the same interface or different interfaces.

[0129] like Figure 7 As shown, the execution plan includes stage1, stage2, stage3, and stage4. Stage3 and stage4 are parallel and can be executed synchronously; stage2 is the next stage after stage3 (or stage4), and correspondingly, stage3 (or stage4) is the previous stage of stage2; stage1 is the next stage after stage2, and correspondingly, stage1 is the previous stage of stage2, and so on.

[0130] Based on the execution plan of the above SQL statement, the following tasks can be derived:

[0131] Task 1: Scan Table 1 and filter out rows from Table 1 with the date 2020 / 01 / 02. Task 1 can be completed using the `tablescan` and `filter` operators. The `tablescan` operator performs a scan, while the `filter` operator performs a filtering operation. Table 1 contains the data to be processed, and the filtering condition is the data in the `data` column of Table 1 where `data` = 2020 / 01 / 02.

[0132] Task 2: Read Table 2. Task 2 can be accomplished using the `tablescan` operator.

[0133] Task 3: Combine Table 1 and Table 2. Specifically, Task 3 involves filtering the rows in Table 1 with the date 2020 / 01 / 02 to find rows whose ID column values ​​match the ID values ​​in Table 2; then, based on these matching ID values, combine them with the name column in Table 2. Task 3 can be accomplished using the Join operator.

[0134] Task 4: Grouping task. Based on the results of Task 3, group the products by name. Task 4 can be completed using the `group by` operator.

[0135] Task 5: Partial Aggregation: Based on the grouping results of Task 4, sum the sales revenue of each group of products to obtain the total sales revenue for each group. Task 5 can be accomplished using the aggeration (aggregation) operator. It should be understood that each worker node is assigned to process one or more shards from Table 1 (sharding will be explained in detail below, but will not be emphasized here). That is, each worker node executing Task 5 only summarizes the sales revenue of a group of products based on a portion of the data in Table 1; therefore, Task 5 can also be understood as partial aggregation.

[0136] Task 6: Final aggregation, which determines the final query result based on the results of all partial aggregations. In the example above, the data to be operated on is the result of Task 5 executed on each worker node assigned to Task 5. The operation is summation, and the execution rule is to sum the sales of the same product based on the results of Task 5 executed on each worker node, thus obtaining the final query result.

[0137] Continue to refer to Figure 7 For example, the logical relationship between tasks is as follows: Task 3 is the next task after Task 1 and Task 2, and Task 1 and Task 2 are the previous tasks of Task 3. Task 4 is the next task after Task 3, and correspondingly, Task 3 is the previous task of Task 4. That is, the output data of Task 1 and Task 2 is the input data of Task 3, the output data of Task 3 is the input data of Task 4, and so on. Correspondingly, the node that executes the next task is the next-level node of the current node. For example, the node that executes Task 3 is the next-level node of the node that executes Task 1, and so on.

[0138] The task will be described in detail below.

[0139] In this embodiment, different tasks have different request IDs. A request ID uniquely identifies a task; among multiple tasks belonging to the same query request, each task's request ID is different. For the task execution device, the input data is the data to be calculated for this task, which can be identified based on the request ID. Specifically, data containing the request ID of this task is the input data. The result of executing the task based on the input data is the task's output data. The request ID of the output data becomes the request ID of the next task. Specifically, the data packet carrying the output data also carries the request ID of the next task.

[0140] A single task can be executed using one or more operators. When multiple operators are required, their order indicates the execution order. The execution device performs the corresponding operations according to the order of the operators. Different operators do not need to use task identifiers to pass execution results; for the result of the current operator, the execution device can directly use the next operator to process that result. For example, in Task 1, after the `tablecan` operator, comes the `filter` operator. The execution device for Task 1 first reads Table 1, then uses the `filter` operator to filter Table 1 based on its filtering conditions. The execution result of the last operator in the task is the output data for that task.

[0141] Step 603: The central node determines the unloadable task in the one or more tasks and determines the execution device of the unloadable task.

[0142] To achieve parallel processing, the central node generates a task scheduling plan, distributing the tasks to multiple execution devices. These devices then execute one or more tasks in parallel. For example, task 1 can be assigned to multiple worker nodes, each reading one or more shards of table 1. A shard refers to dividing the data to be queried into equal-sized partitions. For instance, if table 1 contains 10,000 rows, and each partition consists of 2,000 rows, then table 1 can be divided into 5 shards. This is what is known as parallel processing, which improves task execution efficiency.

[0143] In this embodiment, tasks include unloadable tasks and non-unloadable tasks, and the execution devices for different types of tasks can be different. For example, non-unloadable tasks can be processed by worker nodes, while unloadable tasks can be offloaded to network devices for processing, such as the network card of the worker node. This reduces the workload of the worker node and also alleviates the computational load and CPU burden on the worker node's CPU.

[0144] For example, in this embodiment, the unloadable task can be a task containing unloadable operators, which can be preset or pre-defined by a protocol. Unloadable operators include, but are not limited to, the `tablescan` operator, `filter` operator, `join` operator, `aggregation` operator, `TopN` operator, and `distinct` operator. It should be noted that the above-mentioned unloadable operators are merely examples, and this application embodiment does not limit the type or number of unloadable operators. Furthermore, if a task contains multiple operators, and some of these operators are not unloadable, then the task can be defined as a non-unloadable task. In practical applications, a task that requires execution using non-unloadable operators can be defined as a separate task. That is, all operators involved in the unloadable task in this embodiment are unloadable operators.

[0145] For example, based on the unloadable operators mentioned above, for tasks 1 to 6, task 3 contains an unloadable operator, therefore, task 1 is an unloadable task, and the execution device for task 1 can be the network interface card of the worker node. Task 4 uses a non-unloadable operator, therefore, task 4 is a non-unloadable task, and the execution device for task 4 can be the worker node itself.

[0146] The following explains how to generate shards:

[0147] It should be understood that not every task needs to retrieve data from the data source, meaning not every task needs to generate shard information. For tasks that require reading tables, such as those using the `tablescan` operator, the central node can allocate shards for each execution device when assigning execution devices. Taking Table 1 as an example, the process of generating shard information for Table 1 can be as follows: The central node obtains the storage information of Table 1 from the data source, such as which storage nodes Table 1 is stored on, the size of Table 1's data stored on each storage node, the storage location of Table 1's data on each storage node, and the IP address of that storage node. Based on this storage information, the central node generates shard information for Table 1. The shard information for each shard includes the IP address and storage location of the storage node where the shard resides.

[0148] For example, suppose Table 1 contains 10,000 rows. Rows 1 to 4,000 are stored on storage node 1, rows 4,001 to 8,000 are stored on storage node 2, and rows 8,001 to 10,000 are stored on storage node 3. Given that each shard has 2000 rows, Table 1 can be divided into 5 shards, for example, shard 1 to shard 5. Correspondingly, the sharding information of shard 1 includes, but is not limited to, some or all of the following: the identifier of shard 1, the IP address of storage node 1, and the storage location (rows 1 to 2000 of Table 1 are stored in the address space of storage node 1, for example, it can be represented as the starting address of the address space and the length of rows 1 to 2000); the sharding information of shard 2 includes, but is not limited to, some or all of the following: the identifier of shard 2, the IP address of storage node 1, and the storage location (rows 2001 to 4000 of Table 1 are stored in the address space of storage node 1, for example, it can be represented as the starting address of the address space and the length of rows 2000 to 4000), and so on. These will not be described in detail here. Of course, the above is only an example, and Table 1 may also be stored on only one storage node. This application embodiment does not limit this.

[0149] The following explains how to generate a task scheduling plan:

[0150] For example, the central node generates a task scheduling plan based on information about multiple tasks (or the aforementioned execution plan) included in the SQL, information about worker nodes, sharding information, etc.

[0151] The task information includes a task identifier, or a task identifier and an uninstallation flag. The uninstallation flag indicates whether the task corresponding to the task identifier carried in the first setting instruction is an uninstallable task. For example, the uninstallation flag can be 1 bit; a value of 1 indicates an uninstallable task, and a value of 0 indicates a non-uninstallable task. As another example, the uninstallation flag can be a fixed value; uninstallable tasks carry this uninstallation flag, while non-uninstallable tasks do not.

[0152] The information for worker nodes includes the number of worker nodes, their addresses (e.g., IP address, port), and their identifiers. The worker node identifier can be globally unique; this means that the identifier is unique within the query system, and each worker node and the central node know its meaning. This identifier can be the worker node's IP address, device identifier, device name, or a unique identifier generated by the central node for each worker node in the query system. The information for switches includes the switch's address (e.g., IP address, port), whether it has the capability to handle offloadable tasks, and its identifier. Fragmentation information is described in the previous section and will not be repeated here.

[0153] Specifically, the task scheduling plan includes some or all of the following: task identifier, unload flag, identifier of the worker node to which the task is assigned, sharding information corresponding to the task, etc. See Table 3, which is a specific example of a task scheduling plan provided by this application embodiment for the above SQL statement. Here, we take assigning all tasks to worker nodes for processing as an example. Assume that worker node 20a reads the complete table 2, and subsequently, worker node 20a broadcasts table 2 to every other worker node.

[0154] Table 3

[0155]

[0156] For example, based on Table 3, the central node assigns tasks 1 to 4 to worker nodes 20a to 20e respectively. Worker node 20a reads shard 1, worker node 20b reads shard 2, worker node 20c reads shard 3, worker node 20d reads shard 4, and worker node 20e reads shard 5. In this way, each worker node, while executing task 1, can read a portion of the rows in Table 1 in parallel without interference. Based on the read data, subsequent tasks are executed until task 5 yields a partial aggregation result. Finally, the node executing task 6 (worker node 20a) summarizes the partial aggregation results from worker nodes 20a to 20e to obtain the final query result.

[0157] As an feasible approach, users can also view the progress of the query request at any time during the execution process, such as... Figure 8 As shown, after selecting any task in the execution plan, the specific execution information of the current task can be displayed, such as which nodes the task is assigned to, whether it is a detachable task, the execution device for detachable tasks, information about the node or execution device (assuming the IP addresses of worker nodes 20a to 20e are 76.75.70.14-18 respectively), and the execution status (e.g., not started, executing, completed). It should be noted that... Figure 7 and Figure 8 The interface shown is only an illustration and may be displayed in other ways. This application does not limit the specific display method.

[0158] Step 604: The central node sends a first setting instruction for the unloadable task to the worker node, and the network card of the worker node is set as the execution device for the unloadable task.

[0159] According to the task scheduling plan, the central node sends the first setting instructions for each task to the worker node that is set to process the task.

[0160] For example, the central node can generate a first setting instruction for each worker node based on a task scheduling plan, using tasks as the granularity, and send a first setting instruction for unloadable tasks to the worker nodes. Correspondingly, the worker nodes receive the first setting instruction sent by the central node. For example, the first setting instruction includes, but is not limited to, some or all of the following information: task identifier, unload flag, and operator information. Table 4 below shows an example format of a first setting instruction provided in this embodiment.

[0161] Table 4

[0162]

[0163] The operator information includes, but is not limited to, some or all of the following:

[0164] The operator includes an operator identifier, operator execution rules, operator input information, and operator output information. The operator input information indicates the input data required to execute the task, including information about the node where this input data resides, such as address information, storage information, and table name; for example, the sharding information of Table 1 above is the input information for Task 1. The operator output information includes the task identifier of the next task corresponding to the Request ID, and information about the next-level node.

[0165] Similarly, the first setting instructions for non-uninstallable tasks and the first setting instructions for uninstallable tasks can be similar. The difference is that the uninstallation flag for uninstallable tasks indicates that the task is uninstallable, while the uninstallation flag for non-uninstallable tasks indicates that the task is non-uninstallable. Alternatively, only the first setting instructions for uninstallable tasks may carry the uninstallation flag, while the first setting instructions for non-uninstallable tasks may not carry the uninstallation flag. This is how the two can be distinguished.

[0166] Referring to Table 5, taking worker node 20a as an example, Table 5 lists specific examples of the first setup instructions sent to worker node 20a for each task. The execution rules of each operator are described above and will not be repeated in Table 5. In addition, it is assumed that Flags is 1 to indicate a task that can be unloaded and Flags is 0 to indicate a task that cannot be unloaded.

[0167] Table 5

[0168]

[0169] It should be noted that the format of the first setting instruction described above is merely an example. In practice, the first setting instruction may contain more or less information than that in Table 5, and this application embodiment does not limit this. For example, the central node may not be certain whether a task is an unloadable task, and each worker node may determine this based on a preset operator. In this case, the unload flag may not be included in the first setting instruction. As another example, the first configuration instruction may also include padding data (magic bytes), which can be data with known bits, such as 0 or 1, to make the length of the first setting instruction a preset length.

[0170] Step 605: The worker node determines whether the received task is an unloadable task. If it is, then proceed to step 606; otherwise, the task is processed by the worker node.

[0171] The following description uses a single working node as an example.

[0172] Referring to Table 5, one feasible approach is for the worker node to determine whether a task is uninstallable based on the unload flag carried in any received first setting instruction. If so, the worker node sets the uninstallable task information in the network interface card (NIC), and the NIC then processes the uninstallable task. Another feasible approach is for the worker node to distinguish between uninstallable and non-uninstallable tasks based on whether an unload flag is carried. This example uses the central node to identify uninstallable tasks. If, as described above, the central node does not identify uninstallable tasks, then the first setting instruction will not carry an unload flag, regardless of whether the task is uninstallable or non-uninstallable. In this case, the worker node can identify uninstallable tasks based on a preset uninstallable operator; this embodiment does not limit this approach.

[0173] Step 606: The worker node will unload the unloadable task to its own network interface card (NIC), that is, set the unloadable task information in the NIC.

[0174] Specifically, when a worker node sets the information of an unloadable task in the network interface card (NIC), it can send a second setting command for the task to the NIC. The NIC then obtains and records the information of the unloadable task based on the second setting command.

[0175] For example, the second setting instruction may include a header and a data portion, wherein the header may include control instructions and a task identifier, and the data portion may contain operator information for the task. For example, see Table 6, which shows a format of a second setting instruction provided in this embodiment.

[0176] Table 6

[0177]

[0178] In this context, `Command` represents the command type, or in other words, indicates what operation to perform. For example, `Command` can be, but is not limited to, the following types: `init command`, `readcommand`, and `end command`. The `init command` instructs the unloading of the task corresponding to the Request ID. The `execute command` instructs the starting of the `tablescan` task to read the input data of the task corresponding to the Request ID. Generally, reading the data to be queried is the starting point for SQL execution; therefore, this command can be called a read command or an execute command. The `end command` instructs the execution device to release the resources used to process the task corresponding to the Request ID after the task is completed; or it can be understood as indicating that the task has ended and the resources allocated to it can be released. For ease of description, the second setting instruction with `init command` will be referred to as the `init command`; the second setting instruction with `readcommand` will be referred to as the `execute command`; and the second setting instruction with `end command` will be referred to as the `end command`. The `payload` includes the task's operator information, which has been introduced previously and will not be repeated here.

[0179] The next step in configuring the information flow for offloadable tasks on the network interface card of the worker node may include:

[0180] Taking Task 1 as an example, after worker node 20a determines that Task 1 is an unloadable task, it sends an unload command for Task 1 to its own network card, as shown in Table 7 below.

[0181] Table 7

[0182]

[0183] For example, after receiving the uninstall command, the network card first checks the packet header of the uninstall command. If the packet header is an init command, the network card determines that task 1 (Request ID is 1) is an uninstallable task, allocates (or reserves) network card resources for task 1, and configures the network card resources to process task 1.

[0184] The following section introduces network interface card (NIC) resources. For example... Figure 9 As shown, in this embodiment, the network card resources (including computing units) and memory resources used to process offloadable tasks can be divided into multiple parts, each of which can be called a processing engine (PE). A PE can be configured to process an offloadable task.

[0185] Continue to refer to Figure 9 Taking Task 1 as an example, setting the information for unloadable tasks in the network interface card (NIC) can include the following process: After receiving the unload command for Task 1, if there is an idle PE (Preinstallation Equipment), the NIC assigns Task 1 to an idle PE. This PE records the Request ID and operator information of Task 1. Correspondingly, the NIC records a first correspondence between this PE and the unloadable tasks it handles, to record which task the PE is assigned to execute. Specifically, the first correspondence includes the PE identifier and the Request ID of the unloadable task. Subsequently, when the NIC receives a data packet containing the same Request ID, it can determine the PE corresponding to that Request ID based on the first correspondence and route the data packet to the corresponding PE for processing. For example, when executing task 1, the network card of worker node 20a sends a read request to the storage node corresponding to fragment 1 to read fragment 1. The read request carries the Request ID of task 1. The storage node sends the feedback data packet of task 1 to the network card, and these feedback data packets also carry the Request ID of task 1. In this way, the network card can determine the PE corresponding to the data packet returned by the storage node according to the first correspondence, and route the data packet to the determined PE. The PE processes the data packet using the corresponding operator according to the recorded operator information.

[0186] It should be understood that the number of PEs is limited. For example, a circular queue can be set up in the network interface card (NIC). In one feasible approach, the number of unloadable tasks that can be placed in the circular queue can be equal to the number of PEs. When a new unloadable task arrives and the circular queue is not full, the unloadable task is placed in the circular queue, and an idle PE is allocated to it. When the circular queue is full, the NIC sends a response to the device that sent the unload command. This response indicates that the NIC cannot process the unloadable task and may include the reason for the inability, such as the NIC lacking the resources to process the unloadable task. For instance, after the processor of worker node 20a sends an unload command for task 1 to the local NIC, and the circular queue is full, if the NIC determines that there is no idle PE to process the task, it sends a response to the processor of worker node 20a, indicating that the NIC cannot execute task 1. Subsequently, worker node 20a can execute task 1, thereby reducing latency and improving task processing speed. Another possible implementation is that the network card places all received unloadable tasks into a circular queue. All unloadable tasks can be placed into the circular queue. If the number of unloadable tasks is greater than the number of PEs, then when an idle PE appears, an unloadable task that has not yet been assigned a PE is selected from the circular queue and the idle PE is assigned.

[0187] The above describes the process of unloading unloadable tasks to the network card, using task 1 as an example. Similarly, worker node 20a unloads tasks 2, 3, 5, and 6 to its own network card in the same way. It should be understood that the unloading process for other worker nodes is similar to that of worker node 20a, and will not be repeated here.

[0188] Step 607a: The central node sends an execution command to the network interface card (NIC) of the worker node, and the corresponding NIC of the worker node receives the execution command.

[0189] In this application, some tasks require a received execution instruction before they can be executed. For example, tasks that require the use of the `tablescan` operator, i.e., tasks 1 and 2 in the example above. For such tasks, the central node can send the execution instruction to trigger the execution device to perform the task.

[0190] For example, the execution instruction here can be the read command described above, where the central node sends read commands for Task 1 and Task 2 to the worker nodes. In a first implementation, a read command can carry the Request IDs of multiple tasks; for example, the Request ID of the read command carries the Request ID of Task 1 and the Request ID of Task 2. That is, the read command for Task 2 and Task 1 can be the same. In another implementation, the read command for each task is independent; for example, the read command for Task 1 carries only the Request ID of Task 1, and similarly, the read command for Task 2 carries only the Request ID of Task 2. This application does not limit this implementation.

[0191] For ease of description, the second implementation method will be used as an example below. Taking the read command of Task 1 as an example, see Table 8, which provides a specific example of a read command for Task 1 in this application embodiment.

[0192] Table 8

[0193] Command Request ID Payload Read command 1 Fragment information for fragment 1

[0194] It should be noted that Table 8 is only an example. If the unload command for Task 1 carries the fragment information of Task 1, then the read command for Task 1 does not need to carry the fragment information of Task 1 again, thereby reducing the amount of data to be transmitted, avoiding duplicate transmission, and saving network resources. Alternatively, regardless of whether the unload command contains fragment information, the read command can carry the fragment information, and the executing device will use the fragment information in the read command as the standard, so as to achieve the effect of dynamically and flexibly adjusting the fragment information and improving the data hit rate.

[0195] Optionally, after a worker node sets the task completion information according to the first setting instruction, it can send a completion response to the central node. The central node then sends an execution instruction upon receiving the completion response. Alternatively, the central node can directly send an execution instruction, and after a worker node sets the task completion information according to the first setting instruction, it can directly start executing the corresponding task to achieve automatic execution of subsequent tasks.

[0196] In step 607b, the network interface card (NIC) of the worker node receives data from this worker node or other nodes. Similarly, these other nodes can be other worker nodes, central nodes, storage nodes, or forwarding devices. For example, for worker node 20a, the NIC receives the result of task 5 executed by other worker nodes. It should be noted that step 607b is optional, not mandatory, and there is no strict timing constraint between steps 607b and 607a.

[0197] Step 608: The network card determines whether the received data is a task offloaded to the network card itself. If so, proceed to step 609; otherwise, proceed to step 610.

[0198] Subsequently, after the task is started and executed, the network interface card monitor checks whether the data received is the input data for the unloadable task that it is configured to process. If so, the data is processed; otherwise, the data is forwarded to the worker node.

[0199] The data here includes various setting instructions and data to be processed. In conjunction with step 607a, when the network card receives an execution instruction, it determines whether the received instruction pertains to a task offloaded to the network card. If so, the network card initiates the execution of the corresponding task; otherwise, the network card sends the execution instruction to the worker node, which then initiates the execution of the corresponding task.

[0200] Similarly, each task execution device monitors the received data and determines whether the data belongs to the task offloaded to its network interface card (NIC). For example, if the data contains the request ID of the task on the NIC, then the data belongs to that task. If the data is an execution instruction, the NIC executes the instruction. If the data is input data for the task, the NIC processes the data using the task's operators and execution rules. If not, it is determined that the data does not belong to the task, and the data is sent to the worker node for processing.

[0201] Step 609: The network card executes the corresponding task and returns the result to the next level node of the execution plan or the execution device of the next task.

[0202] Step 610: The network card sends the data to the worker node.

[0203] Step 611: The working node executes the corresponding task based on the data and returns the result to the execution device or the next level node of the execution plan.

[0204] It should be noted that steps 608 to 611 may be executed cyclically until the final query result, i.e., the result of task 6, is obtained.

[0205] For example, referring to Table 3, taking worker node 20a as an example, assume that worker node 20a offloads tasks 1, 2, 3, 5, and 6 to PE0 to PE5 of the local network card for processing, respectively. That is, the first correspondence is: PE0 corresponds to task 1, PE1 corresponds to task 2, and so on, with PE4 corresponding to task 6. Task 4 is processed by worker node 20a. The following describes the process of worker node 20a executing tasks 1 to 6:

[0206] After receiving the execution instruction for Task 1, the network interface card (NIC) of worker node 20a determines the PE corresponding to Task 1, namely PE0, according to the first correspondence mentioned above. The execution instruction for Task 1 is then routed to PE0, and PE0 executes Task 1: that is, it sends a read request to the corresponding storage node based on the fragmentation information of Task 1 (fragment 1). For example, this read request can be a read request in the existing implementation mechanism or a read request in other formats. Specifically, PE0 can forward the read request for Task 1 to the storage node corresponding to fragment 1. After receiving the read command, the storage node returns the data of fragment 1 corresponding to Task 1 to the NIC. As mentioned earlier, the data returned by the storage node contains the same Request ID as the read request.

[0207] After receiving the data packet for Task 1 from storage node 1, the network card sends the data packet to the corresponding PE0 according to the first correspondence. After receiving the data packet, PE0 determines that the next operator of the tablescan operator is the filter operator based on the operator information of Task 1, and filters the filter column (data column) in fragment 1 carried in the data packet according to the execution rule of the filter operator (data = 2020 / 01 / 02).

[0208] Specifically, the filtering result can be represented by a bitmap corresponding to the filtering column. Each bit in the bitmap corresponds sequentially to each row in the read slice, and different bit values ​​indicate whether the row meets the filtering conditions. See Table 9 below, assuming that Table 9 is a portion of the data in slice 1 of Table 1.

[0209] Table 9

[0210] Id sale Data … … 2020.01.02 … … 2020.01.01 … … 2020.01.01 … … 2020.01.01 … … 2020.01.02

[0211] PE0 determines whether the read data column is a filter column. If it is a filter column, the filter conditions are used to filter the data in each row of the read filter column. According to the filter condition Data = 2020 / 01 / 02 in Task 1, it can be determined that Data is a filter column. Assuming that a bit value of 1 indicates that the filter condition is met and a bit value of 0 indicates that the filter condition is not met, the bitmap corresponding to the filter column shown in Table 9 is 10001.

[0212] The network interface card (NIC) can also store a second mapping between Request ID and bitmap, for example, Request ID = 1, bitmap = 10001. Then, based on the bitmap of Task 1, it can be quickly determined that only the first and fifth rows in Table 9 meet the conditions. This allows the filtered data to be sent to the next level node, reducing the amount of data transmitted. Simultaneously, the worker nodes do not need to execute this offloadable task, further reducing their workload and processing time.

[0213] For the output data of Task 1, the output data carries the request ID of the next task of Task 1, namely request ID 3, as well as the filtered data of Task 1. The network card continues to determine the PE corresponding to request ID 3, namely PE2, according to the first correspondence, and routes the output data of Task 1 to PE2.

[0214] Optionally, when the task is completed, the network card can send an indication message to the worker node. This indication message indicates that the task has been completed. After receiving the indication message, the worker node can send a termination command to the network card to release the corresponding network card resources (such as PE), memory resources, etc., that were used to process the unloadable task.

[0215] The following example uses the end command for Task 1. Please refer to Table 10 for a specific example of an end command provided by this application embodiment for Task 1.

[0216] Table 10

[0217] Command Request ID Payload end command 1 NULL

[0218] For example, after task 1 on the network interface card (NIC) is completed, worker node 20a sends a termination command as shown in Table 10 to the NIC. Upon receiving this termination command, the NIC releases the PE (Processing Equipment) used to process task 1, as well as memory resources. The released resources can then be used to process other unloadable tasks offloaded to the NIC. Alternatively, the NIC can decide when to release the PE. For instance, when the storage node sends the last data packet of task 1 to the NIC, it includes an identifier indicating that this is the last data packet. After determining that the PE has finished processing the last data packet of task 1, the NIC releases the corresponding resources used to process task 1. Similar scenarios will not be repeated below.

[0219] Subsequently, similarly, the central node sends the execution instruction for Task 2 to the network interface card (NIC) of worker node 20a. The NIC determines the PE corresponding to request ID2, namely PE1, based on the first correspondence. PE1 can retrieve the complete Table 2 based on the storage information of Table 2 (including the IP address and storage location of the storage node) according to the first setting instruction of Task 2 shown in Table 5, and send the retrieved Table 2 to worker nodes 20b, 20c, 20d, and 20e respectively. At the same time, Table 2 is routed to the next task of Task 2, namely PE2 corresponding to Task 3.

[0220] When all the input data for Task 3 arrives, PE2 executes Task 3. That is, PE2 processes the output data of Task 1 and Task 2 based on the operators and execution rules corresponding to Task 3 to obtain the output data of Task 3.

[0221] PE2 sends the output data of Task 3 (including request ID 4) to the execution device of Task 4. Since Task 4 is a non-unloadable task and the execution device of Task 4 is the worker node itself, the network card can send the output data of Task 3 to the processor of worker node 20a for processing.

[0222] Subsequently, worker node 20a sends the output data of task 4 (including request ID 5) to the execution device of task 5. Specifically, worker node 20a sends the output data of task 4 to the network card. The network card determines the PE corresponding to request ID 5, i.e., PE3, according to the first correspondence. The network card routes the output data of task 4 to PE3, and so on, until worker node 20a obtains the final query result.

[0223] It should be noted that other worker nodes, such as worker node 20b, also need to send the output data packets of Task 5 to the network card of worker node 20a. After all the output data packets of Task 5 from worker node 20a to worker node 20e have arrived, PE4 executes Task 6 to obtain the final query result. When the output data is transmitted in multiple data packets, the last data packet can also carry an end marker. This end marker indicates whether the data packet is the last data packet of the current request ID, allowing the receiving end to determine whether the data transmission on the peer worker node is complete.

[0224] It should be noted that the format of the second setting instruction described above is merely an example. In practice, the second setting instruction may contain more or less content than in the example above, and this application embodiment does not limit this. For example, the second setting instruction may include padding data to make the length of the second setting instruction a preset length.

[0225] Another feasible approach is that, for unloadable tasks, the central node can directly send the unload command for the unloadable task to each execution device configured to execute the task, such as the network interface card or forwarding device of the worker node, according to the task scheduling plan. Another data query method of this embodiment is provided below.

[0226] Please see Figure 10 , Figure 10 This is a flowchart illustrating another data query method provided in this embodiment. In this embodiment, besides the network interface card (NIC) of the worker node described in the first embodiment acting as an execution device for offloadable tasks, forwarding devices in the network, such as switches and routers, can also be configured as execution devices for offloadable tasks. The method for setting offloadable tasks in the NIC of the worker node is different from... Figure 6 The embodiments shown are the same, that is, steps 1001 to 1006 and steps 1013 to 1016 in this embodiment can be referred to respectively. Figure 6 The descriptions of steps 601 to 606 and steps 608 to 611 will not be repeated here; the following will only explain the differences.

[0227] The following section will focus on describing how to configure undeletable tasks when the undeletable task is a forwarding device. For ease of explanation, the following text will use a switch as an example for illustration.

[0228] This embodiment also uses Figure 6The SQL statement in the embodiment is used as an example for description. As shown in Table 11 below, another task scheduling plan is generated for the SQL statement in step 602 above. The execution device of the unloadable task in this task scheduling plan includes a switch. For example, the execution device of task 1 is switch 30, and the execution device of task 6 is switch 40.

[0229] Table 11

[0230] Task identifier Uninstallable tasks Actuating device Stored information Task 1 yes Switch 30 NULL Task 2 yes Network interface card of worker node 20a Storage information in Table 2 Task 3 yes Network interface card (NIC) from worker node 20a to worker node 20e NULL Task 4 no Working nodes 20a to 20e NULL Task 5 yes Network interface card (NIC) from worker node 20a to worker node 20e NULL Task 6 yes Switch 40 NULL

[0231] It should be understood that Table 11 is merely an example and does not constitute a limitation on the task scheduling plan of this application.

[0232] When the execution device for the unloadable task determined in step S1003 is a switch, please refer to the description of steps 1007 and 1008 for the settings of the central node for the unloadable task.

[0233] Step 1007: The central node sends the configuration instructions for the unloadable task to the switch, which is then configured as the execution device for the unloadable task.

[0234] In practice, data packets sent by the central node, storage node, or worker node are first routed to the switches on their respective network segments. To distinguish between data packets to be forwarded and data packets configured for the switches, in this embodiment, the switch includes at least two ports: a data port and a control port. When the switch receives a data packet through the data port, it indicates that the data packet needs to be forwarded, and the switch forwards it to the device corresponding to the destination IP address based on the destination IP address. If the switch receives a data packet through the control port, it means that the data packet is a data packet configured for the switch by the central node, and the switch needs to configure it based on this data packet, such as setting instructions for tasks sent by the central node to the switch.

[0235] As shown in Table 11, the central node will send the configuration instructions for Task 1 to switch 30; the central node will send the configuration instructions for Task 6 to switch 40.

[0236] For example, the central node can send a first configuration command for the uninstallable task to the control port of the switch, instructing the switch to configure the information of the uninstallable task according to the first configuration command. As another example, the central node can send a second configuration command (uninstall command) for the uninstallable task to the control port of the switch, instructing the switch to configure the information of the uninstallable task according to the uninstall command. The following description uses the uninstall command as an example of the configuration command being configured as an uninstall command.

[0237] Step 1008: Configure the switch with information on tasks that can be unloaded.

[0238] Correspondingly, when switch 30 receives the unload command for task 1 through its control port, it records the information of task 1 (including the operator information and request ID 1, etc.) based on the unload command. Similarly, when switch 40 receives the unload command for task 6 through its control port, it records the information of task 6 (including the operator information and request ID 6, etc.) based on the unload command. Subsequently, the switch configured to handle unloadable tasks monitors whether the Request ID of each received data packet belongs to a task to be unloaded locally. If so, the switch processes the data; otherwise, the switch forwards the data to the destination IP address of the data.

[0239] Step 1009a: The central node sends execution instructions for Task 1 and Task 2.

[0240] The execution command here can be the startup command mentioned above.

[0241] As mentioned earlier, data packets sent by the central node, storage node, or worker node are first routed to the switch on their respective network segment. In other words, execution commands sent by the central node are also first routed to the switch.

[0242] Step 1009b: The switch receives data from other nodes.

[0243] Other nodes can be worker nodes, central nodes, storage nodes, or other forwarding devices.

[0244] Step 1010: The switch determines whether the received data is a task offloaded to the switch. If so, proceed to step 1011; otherwise, proceed to step 1012a.

[0245] The data received by the switch includes setup commands, execution commands, fragmented data from Table 1 or Table 2 sent by the storage node, or output data obtained by other worker nodes executing tasks. Step 1010 can be referenced from the network card operation in step 608, and will not be repeated here.

[0246] It should be understood that the switch will first receive the execution instructions for Task 1 and Task 2 sent by the central node. The switch will determine whether Task 1 and Task 2 are tasks offloaded to the switch. If not, the switch will forward the execution instructions for Task 1 and Task 2 to the devices corresponding to the destination IP addresses respectively.

[0247] It should be understood that step 1010 may also be a step that is executed repeatedly until the query system obtains the final query result.

[0248] Step 1011: The switch executes the task and returns the execution result to the next level node in the execution plan.

[0249] The following is a brief explanation of the process by which the switch performs tasks:

[0250] Taking Task 1 as an example, first refer to the following configuration: (Worker node 20a, shard 1), (Worker node 20b, shard 2), (Worker node 20c, shard 3), (Worker node 20d, shard 4), (Worker node 20e, shard 5).

[0251] Based on the above configuration, the central node sends the first execution instruction of Task 1 to the worker node 20a, instructing the worker node 20a to read the data of shard 1; similarly, it sends the second execution instruction of Task 1 to the worker node 20b, instructing the worker node 20b to read the data of shard 2, and so on.

[0252] Taking worker node 20a as an example, upon receiving the first execution instruction, worker node 20a sends a read request (including request ID 1) to the storage node corresponding to shard 1 based on the sharding information of shard 1. Please refer to... Figure 4 Understood, the transmission path for this read request is worker node 20a → switch 10 → switch 40 → switch 30 → storage node.

[0253] Correspondingly, the storage node responds to the read request by sending a feedback data packet for fragment 1 (containing request ID 1). The destination IP address of this feedback data packet is worker node 20a. Similarly, after the storage node sends the packet, it is first routed to switch 30. Switch 30 checks whether the feedback data packet is data for a task offloaded to switch 30, i.e., data for task 1. If so, switch 30 executes task 1 based on the feedback data packet; otherwise, switch 30 sends the data packet to the destination IP address corresponding to the data packet.

[0254] Obviously, after receiving the data packet of fragment 1 sent by the storage node, the switch 30 determines that the data packet is the data packet of task 1. Then, according to the operator information of task 1, based on the execution rules of the filter operator, the filter operator is used to perform a filtering operation on the data in the data packet to obtain the filtering result, which is the output data of task 1. The switch 30 encapsulates the output data in an output data packet. As mentioned above, the output data packet carries the request ID of the next task of task 1, namely request ID3. According to the destination IP address carried in the data packet of fragment 1 received from the storage node, the output data packet is sent to the destination IP address, namely the working node 20a.

[0255] Similarly, the interaction method between switch 30 and any other working node will not be described in detail here. It should be noted that the above method of setting operator information by executing instructions is only an example. In this embodiment, the correspondence between each working node and the shard information can also be sent to switch 30 through the setting instructions of task 1, such as the unload command, so that switch 30 can distribute the filtering results of each shard in Table 1 to achieve the same effect as the above example.

[0256] Step 1012a: The switch forwards the data to the network interface card of the corresponding worker node.

[0257] Step 1012b: The network interface card (NIC) of the worker node receives data from this worker node. It should be noted that step 1012b is optional, not mandatory, and there is no strict timing constraint between steps 1012b and 1012a. Subsequent steps S1013–S1016… Figure 6 Steps S608 to S611 are the same and will not be repeated here.

[0258] Regarding the execution flow shown in Table 11, the execution processes for Task 1 and Task 5 can be found in the relevant descriptions above, and will not be repeated here. Switch 40, which executes Task 6, is the next-level node after worker nodes 20a to 20e. Worker nodes 20a to 20e each send their respective Task 5 output data (carrying request ID 6) to switch 40. If switch 40 receives the data, it determines that Task 6 needs to be executed on the data. Then, it processes the data according to the operator and execution rules corresponding to Task 6 to obtain the final query result. Afterwards, switch 40 sends the obtained query result to the central node, which returns the query result to the client.

[0259] The above design offloads tasks to network devices, reducing the burden on the processors of worker nodes and further reducing the amount of data transmitted over the network.

[0260] It should be noted that the above-described method for determining unloadable tasks is merely an example, and this application embodiment does not limit it. For tasks suitable for offloading to network interface cards (NICs) or forwarding devices, offloading to the NICs or forwarding devices of worker nodes can reduce the burden on the processors of worker nodes and reduce the amount of data transmitted over the network. However, in some scenarios, offloading tasks to NICs or switches may affect execution efficiency. Therefore, in another embodiment of the present invention, for the preset unloadable tasks in the central node, an unloading strategy can be further set. The unloading strategy is used to determine whether the unloadable task can be unloaded, and only tasks that meet the unloading strategy are set as unloadable tasks.

[0261] To facilitate understanding of the uninstallation strategy for the following tasks, one point should be noted first:

[0262] Data sources can analyze stored data tables to obtain data distribution information. This data distribution information includes the total number of data (n) in a certain column of the data table, as well as information indicating the distribution of data in a certain column across different intervals. For example, if a personnel registration table contains an age column, the data distribution of the age column in the table could be the number of people aged 1-10 (denoted as quantity a), the number of people aged 11-20 (denoted as quantity b), and the number of people aged 21-30 (denoted as quantity c).

[0263] The central node can request data distribution information for the table to be queried from the data source. Correspondingly, the storage node sends the requested data distribution information back to the central node. Based on this data distribution information, the central node can roughly calculate the selectivity of the filter operator, the cardinality of the aggregation column of the aggregation operator, etc. For example, if the execution rule of a filter operator is for people aged 1-30, then the selectivity of this filter operator is (a+b+c) / n. Of course, this data distribution is a relatively rough statistic, and the selectivity or cardinality of the aggregation column obtained from it is not an exact selectivity. This will not be repeated in subsequent sections.

[0264] The following examples illustrate specific tasks.

[0265] 1. Filter operator;

[0266] When a filter operator is set as an unloadable operator at the central node, the corresponding unloading strategy set for the filter operator can be: if the selectivity of the filter operator is low, for example, below a first preset value, the task to which the filter operator belongs is an unloadable task. The selectivity can be determined based on the ratio of rows that meet the filtering conditions to all rows contained in the original data to be queried. For example, taking task 1 as an example, assuming table 1 has 10,000 rows, of which only 10 rows meet the date 2020 / 01 / 02, the selectivity is 10 / 10000*100% = 0.1%. If the first preset value is 1%, and the selectivity of task 1 is lower than the first preset value, then task 1 is determined to be an unloadable task. It should be understood that the first preset value is only an example, and all specific values ​​listed in this embodiment are examples and are not intended to limit the scope.

[0267] If the selectivity is low, the central node can offload the task to the network interface card (NIC) of the storage node or a switch, thus reducing the amount of data transmitted over the network. For example, when the storage node and worker nodes are deployed on different devices, the central node can offload Task 1 to the NIC of the storage node. The storage node only needs to send 10 rows of data to the worker node, instead of sending the entire table 1 it reads. This reduces the amount of data sent from the storage node to the worker node, resulting in less data transmission over the network. This reduces the CPU load and avoids consuming excessive network bandwidth. Alternatively, Task 1 can be offloaded to the NIC of the worker node, where the filtering is performed. The worker node's CPU does not need to execute the task, further reducing the CPU load. Additionally, the NIC does not need to send a large amount of data to the CPU, reducing data interaction within the worker node.

[0268] 2. Aggregation operator;

[0269] When the aggregation operator is set as an unloadable operator in the central node, the corresponding unloading strategy set for the aggregation operator can be: if the cardinality of the columns to be aggregated is relatively small, for example, not exceeding the second preset value, then the task to which the aggregation operator belongs is determined to be an unloadable task.

[0270] For example, when calculating the sales revenue of product A in Table 1, the "sale" column is the column that needs to be aggregated. If product A has 10 sales records, then the "sale" values ​​of these 10 rows need to be aggregated. These 10 rows can be understood as the cardinality of the column that needs to be aggregated. For instance, if the second preset value is 100, then the task executed using the aggregation operator can be determined to be an unloadable task.

[0271] For offloadable tasks of the aggregation operator type, they can be offloaded to the network interface card of the worker node or to the switch for processing. For example, when the cardinality of the columns to be aggregated in the aggregation task is relatively small, it can be offloaded to the switch for processing. This way, it will not occupy more computing resources of the switch and can also reduce the amount of data transmitted in the network.

[0272] 3. Distinct operator

[0273] The unloading strategy for the `DISTINCT` operator is similar to that of the `Aggregation` operator. Specifically, the unloading strategy for the `DISTINCT` operator is based on the cardinality of the `DISTINCT` column. For example, if the cardinality of the `DISTINCT` column does not exceed a third preset value, the task to which the `DISTINCT` operator belongs is an unloadable task. The difference between the unloading strategies of the `DISTINCT` and `Aggregation` operators is that the `Aggregation` operator groups based on the `GROUP BY` column and then performs operations such as `SUM`, `MIN`, `MAX`, `CONUT`, or `AVG` on the grouped data. The `DISTINCT` operator, however, only needs to group the `DISTINCT` column.

[0274] For unloadable tasks of the distinct operator type, they can be unloaded to the network interface card of the worker node or to the switch for processing.

[0275] 4. Dynamic filtering operator

[0276] Dynamic filtering operators refer to the filtering of the larger table by using the smaller table when two tables are joined in the join operator.

[0277] The following example uses Task 3, combined with... Figure 4 and Figure 5 The two scenarios shown illustrate the execution flow of the dynamic filtering operator in Task 3.

[0278] As mentioned earlier, the central node allocates a shard for reading Table 1 to the worker nodes. The process of each worker node executing Task 1 is described above and will not be repeated here.

[0279] Since Table 2 needs to be joined with Table 1—that is, the values ​​in the ID column of Table 1 need to be filtered based on the values ​​in the ID column of Table 2 to check if the values ​​in the ID column of Table 1 exist in the ID column of Table 2—each worker node needs to know the value of the ID column in each row of Table 2. Therefore, the central node can choose not to divide Table 2 into different shards, allowing each worker node to read the complete Table 2 separately. Alternatively, to improve query efficiency, the central node can divide Table 2 into different shards, allowing each worker node to read one or more shards. Then, each worker node can send the ID column data from its shard to the central node, which then combines the ID columns returned by each worker node to obtain the complete ID column data for Table 2.

[0280] Scenario 1: A scenario where storage nodes and worker nodes are integrated, i.e., storage nodes and worker nodes are deployed on the same physical node, such as... Figure 5 The scene shown.

[0281] In this scenario, the dynamic filtering operator is more suitable for offloading to the network interface card (NIC) of the worker node; that is, the NIC of the worker node is the execution device for the dynamic filtering operator. For example, please refer to... Figure 5 In this scenario, the process of joining Table 1 and Table 2 using the Join operator includes the following steps:

[0282] 1) Each working node assigned to Task 2 reads a shard from Table 2 and creates a BF column on the column of the ON condition of the data in that shard. The BF column is then sent to the central node. The BF column may contain duplicate ID values, or the ID column may be deduplicated using the distinct operator. This application embodiment does not limit this.

[0283] Assuming Table 2 also has 5 shards, for example named shards 11 to 12, worker nodes 20a to 20e each read one shard. In the example above, the specific process could be: worker node 20a reads shard 11, creates a BF in the column of the ON condition of the read shard 11 data, and sends the obtained BF to the central node 100; worker node 20b reads shard 12, creates a BF in the column of the ON condition of the read shard 12 data, and sends the obtained BF to the central node 100; worker node 20c reads shard 13, creates a BF in the column of the ON condition of the read shard 13 data, and sends the obtained BF to the central node 100; and so on.

[0284] 2) The central node merges the BF columns sent by each worker node to obtain the ID column of the complete Table 2. Specifically, the central node 100 receives the BF columns of each shard of Table 2 from worker nodes 20a to 20e, merges the BF columns of each shard to obtain the BF column of the complete Table 2, i.e., the ID column.

[0285] 3) The central node sends the complete ID column of Table 2 to the network card of the worker node.

[0286] Since the output of Task 2 is the input data of Task 3, the central node can send the BF column of the complete Table 2 to each execution device of Task 3. For example, if the execution devices of Task 3 are the network cards of worker nodes 20a to 20e, then the central node can send the ID column of the complete Table 2 to the network cards of worker nodes 20a, 20b, 20c, 20d, and 20e respectively.

[0287] It should be noted that if the execution device for Task 3 in this scenario is one or more switches, then the central node 100 can send the BF column of the complete Table 2 to those switches. For ease of description, the following description will use a network interface card (NIC) as the execution device for Task 3.

[0288] 4) The network card of the worker node executes task 1, reads one or more shards from table 1, and filters the read data to obtain the row with data=20200102.

[0289] It should be understood that step 4) and steps 1) to 3) are parallel tasks, and the steps here do not indicate a temporal relationship.

[0290] 5) The network card of the working node is filtered again according to the BF column of Table 2, and the row with ID that exists in the ID column of Table 2 is in the row with data=20200102.

[0291] Scenario 2: Storage nodes and worker nodes are deployed separately, meaning the storage nodes and worker nodes are deployed on different physical nodes, such as... Figure 4 The scene shown.

[0292] In this scenario, since the working node must pass through the switch between the two to interact with the storage node, the switch can filter the data passing through, reducing the amount of data transmitted over the network. In other words, the switch is the execution device for this dynamic filtering operator.

[0293] For example, please combine Figure 4 Understandably, assuming Task 1 is offloaded to the network interface card of the storage node, and the dynamic filtering in Task 3 is offloaded to switch 30, in... Figure 4 In the scenario shown, the process of joining Table 1 and Table 2 using the Join operator includes the following steps:

[0294] 1) The worker node sends read requests for Task 2 and Task 1 to the storage node;

[0295] 2) The storage node sends the data of Task 2 read by the worker node to the switch 30, and filters the data of Task 1 read by the worker node before sending it to the switch 30;

[0296] 3) Switch 30 creates BF based on the data of Task 2 returned by each storage node, obtains the BF column of complete Table 2, and filters the received data after filtering Table 1 again based on the BF column, to obtain the row where the ID column of data=20200102 is equal to the BF column of Table 2.

[0297] 4) Switch 30 will send the row in the data=20200102 where the ID column is equal to the BF column of Table 2 to the corresponding worker node.

[0298] 5) The working node uses the filtered data from switch 30 for joining.

[0299] The above steps constitute the dynamic filtering process of the join operator. In the above scenario, the dynamic filtering in the join operator can be offloaded to the network interface card of the worker node, or it can be offloaded to the switch through which the interaction between the central node and the worker nodes occurs. The offloading strategy is determined by the selectivity when filtering large tables.

[0300] In this embodiment, for an unloadable task that can be offloaded to multiple devices, the central node can determine the execution device for the unloadable task according to the preset priority of each device corresponding to the unloadable task. For example, the execution device can be the device with the highest priority among the devices corresponding to the unloadable task. For instance, for an unloadable task including a filter operator, the priority order of the corresponding devices is: the network card of the storage node, the switch of the rack where the storage node is located, the switch of the rack where the worker node is located, the core switch, the network card of the worker node, etc. Based on this order, it can be determined that the network card of the storage node can be the execution device for task 1.

[0301] It should be noted that the above method for determining the execution device is only an example. It can also be determined by considering priority and device load. For instance, although the storage node's network interface card (NIC) has the highest priority among the undeletable tasks, if the storage node's performance is low or its load is high, task 1 may not be undeletable to the storage node's NIC, and the next device may be determined according to priority. Furthermore, the method for undeletable tasks to be undeletable to the storage node's NIC can be found in the detailed process described above for worker nodes undeletable tasks to their local NICs; it will not be repeated below.

[0302] Of course, in order to reduce the load on the network card, the worker node can also selectively offload some unloadable tasks to the network card based on a preset offloading policy. For example, the preset offloading policy on the worker node can be based on the principle of load balancing, etc., which will not be discussed in detail here.

[0303] Based on the same inventive concept as the method embodiments, this application also provides an apparatus for performing the above-described method embodiments. Figure 6 or Figure 10 The functions performed by the central node, such as Figure 11 As shown, the device includes a generation unit 1101, a processing unit 1102, and a communication unit 1103.

[0304] The generation unit 1101 is used to receive query requests sent by the client and parse the user-input query request into multiple tasks. For details on the implementation, please refer to [link to implementation details]. Figure 6 Steps 601, 602 and Figure 10 The descriptions of steps 1001 and 1002 are not repeated here.

[0305] Processing unit 1102 is used to determine the unloadable tasks among multiple tasks and to determine the execution device of the unloadable tasks. The network device can be the network interface card (NIC) of a worker node or a forwarding device, including switches and routers. For specific methods of determining the unloadable tasks and their execution devices, please refer to [reference needed]. Figure 6 Step 603 and Figure 10 The description of step 1003 in the previous section will not be repeated here.

[0306] Communication unit 1103 is also used to send setting instructions for each task, so as to set the task to be executed on each execution device. Figure 6 For specific implementation details of the embodiments described in step S604, please refer to the relevant description. Figure 10 For an embodiment, please refer to the relevant description of steps S1004 and S1007.

[0307] After setting the corresponding task on each execution device by sending the setting instruction for each task, the communication unit 1103 can send the execution instruction for the query request. This execution instruction is used to trigger the execution device to execute the set task. For details, please refer to [link to relevant documentation]. Figure 6 Step S607a and Figure 10 The description in step 1009a is not repeated here.

[0308] Based on the same inventive concept as the method embodiments, this application also provides a network device for executing the above-described method embodiments. Figure 6 or Figure 10 The functions performed by network devices (switches, routers, or network interface cards of worker nodes), such as... Figure 12 As shown, the device includes a communication unit 1201 and a processing unit 1202.

[0309] Communication unit 1201 is used to receive configuration commands sent by the central node. These commands are used to configure the network device to perform tasks when a query request is executed. For details on the process when the network device is a network interface card (NIC), please refer to [link to relevant documentation]. Figure 6 The relevant descriptions of steps 607a, 610, 1012a, and 1015 are provided below. When the network device is a forwarding device, please refer to the detailed procedure. Figure 6 The relevant descriptions of steps 1009a and 1012a.

[0310] Figure 10 The description of step 1004 will not be repeated here.

[0311] Processing unit 1202 is configured to set tasks according to setting instructions and execute the tasks on data flowing through the network device. When the network device is a network interface card (NIC), for Figure 6 For specific implementation details of the embodiments described in steps S608 and 609, please refer to the relevant descriptions. Figure 10 For embodiments, please refer to the relevant descriptions of steps S1013 and S1014. When the network device is a forwarding device, for Figure 10 For an embodiment, please refer to the relevant description of steps S1010 and S1011.

[0312] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely illustrative descriptions of the application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Thus, if such modifications and modifications fall within the scope of the claims and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A data query system, characterized in that, include: Central node, used for An execution plan for the query request is generated based on the query request, and multiple tasks are generated based on the execution plan; Determine the uninstallable and non-uninstallable tasks among the plurality of tasks, wherein the uninstallable tasks are those that need to be uninstalled to a network device for execution, and the non-uninstallable tasks are those that are executed through a worker node. Identify the network device that will perform the offloadable task; Send a setting instruction for the uninstallable task, the setting instruction being used to uninstall the uninstallable task to the network device; The network device is configured to execute the unloadable task according to the settings of the setting instructions; The working node is used to execute the non-unloadable task.

2. The system as described in claim 1, characterized in that, The uninstallable task is a pre-defined task that is uninstalled to the network device for execution.

3. The system as described in claim 1 or 2, characterized in that, The network device is the network card or forwarding device of the working node in the data query system, and the forwarding device includes: switch and router.

4. The system as described in claim 3, characterized in that, The forwarding device includes a data port and a control port. The central node is used to send the setting instructions for the unloadable task executed by the forwarding device to the forwarding device through the control port. The setting instruction for the task to be executed by the network card or the worker node is sent to the forwarding device through the data port; The forwarding device is used to set the unloadable task according to the setting instructions received from the control port, and to forward the setting instructions received from the data port.

5. The system as described in claim 3, characterized in that, When the network device executing the undeletable task is the network card of the working node, the central node is used to send the setting instructions of the undeletable task to the working node; The worker node sets the unloadable task on its network card according to the setting instructions.

6. The system as described in claim 5, characterized in that, The configuration instructions for the uninstallable task include an uninstallable flag; Upon receiving the setting instruction, if the setting instruction includes the uninstallable flag, the working node sets the uninstallable task on the network card of the working node.

7. The system as described in claim 1 or 2, characterized in that, After receiving a data packet, if the network device determines that the data packet contains an identifier of an offloadable task to be performed by the network device, then the network device executes the offloadable task based on the data packet.

8. The system according to any one of claims 4-6, characterized in that, The central node, when sending the setting instruction for the uninstallable task to the network device, specifically performs the following: after determining the uninstallable task, and when determining that the uninstallable task conforms to the uninstallation policy corresponding to the uninstallable task, it sends the setting instruction for the uninstallable task to the network device.

9. A data query method applied to a central node, characterized in that, The method includes: Obtain a query request, generate an execution plan for the query request based on the query request, and generate multiple tasks based on the execution plan; Determine the uninstallable and non-uninstallable tasks among the plurality of tasks, wherein the uninstallable tasks are those that need to be uninstalled to a network device for execution, and the non-uninstallable tasks are those that are executed through a worker node. Identify the network device that will perform the offloadable task; Send a setting instruction for the uninstallable task, the setting instruction being used to uninstall the uninstallable task to the network device.

10. The method as described in claim 9, characterized in that, The uninstallable task is a pre-defined task that is uninstalled to the network device for execution.

11. The method as described in claim 10, characterized in that, When it is determined that the network device executing the undeletable task is the network card of the worker node, the step of sending the setting instruction for each task includes: sending the setting instruction for the undeletable task to the worker node, instructing the worker node to set the undeletable task for the network card of the worker node.

12. The method as described in claim 11, characterized in that, The setting instruction includes an uninstallable flag, which is used to mark the task set by the setting instruction as an uninstallable task.

13. The method according to any one of claims 9-12, characterized in that, Sending the setting instruction for the uninstallable task to the network device includes: after determining the uninstallable task, and when determining that the uninstallable task conforms to the uninstallation policy corresponding to the uninstallable task, sending the setting instruction for the uninstallable task to the network device.

14. A data query method, applied to network devices, characterized in that, The method includes: The central node receives a configuration instruction; the configuration instruction is used to offload an unloadable task to the network device; the unloadable task belongs to multiple tasks generated by the central node according to the execution plan of the query request; the multiple tasks include the unloadable task and the non-unloadable task; the non-unloadable task is a task executed through a worker node; The undeletable task to be executed on the network device when executing a query request is configured according to the configuration instructions. The offloadable task is performed on the data flowing through the network device.

15. The method as described in claim 14, characterized in that, The network device is a network interface card (NIC) or a forwarding device for a working node. The forwarding device includes a switch and a router.

16. The method as described in claim 15, characterized in that, The forwarding device includes a data port and a control port; the method further includes: Based on the setting instructions received from the control port, configure the offloadable task indicated by the setting instructions on the forwarding device; forward the setting instructions received from the data port.

17. The method according to any one of claims 14-16, characterized in that, Performing the task on the data flowing through the network device includes: Upon receiving a data packet, if it is determined that the data packet contains an identifier of the offloadable task executed by the network device, then the offloadable task is executed based on the data packet.

18. A device, characterized in that, It includes a memory and a processor; the memory stores program instructions, and the processor executes the program instructions to perform the method according to any one of claims 9-13.

19. A network device, characterized in that, The device includes a processing unit and a storage unit, wherein the storage unit stores executable code, and the processing unit executes the executable code to implement the method according to any one of claims 14-17.

20. A network device, characterized in that, The device includes: The communication interface is used for data transmission with the central node and worker nodes. A processing unit is configured to process the data received by the communication interface to perform the method described in any one of claims 14-17.

Images