Semantic-aware computation-storage coordination method

Abstract

A computer implementation method for data analysis in a distributed computer system includes: receiving a directed acyclic graph (DAG) plan from a user application by a distributed computing engine; performing operations to transform the DAG plan using a query analyzer module of the distributed computing engine, wherein the transformed DAG operations include at least one semantic caching operation and at least one near data processing (NDP) pushdown operation; performing the NDP pushdown operation using an NDP executor module of the computer system's data cluster; performing the semantic caching operation on a distributed semantic cache memory of the distributed computing engine; completing the transformed DAG operations; and returning the result of the completed DAG plan to the user application.

Description

Technical Field

[0001] This application belongs to the field of storage technology, and in particular relates to a semantically aware computing storage coordination method and a distributed computer system. Background Technology

[0002] In big data processing environments, multiple data processing engines are used to process a shared set of data. These engines comprise both computing and data resources, and data processing typically involves a pipeline between computing and data storage systems. As data moves along this pipeline, different resources operate on the data at different stages. The amount of data these systems must process continues to grow. As data scales, the data processing pipeline becomes more complex, potentially involving multiple layers of computing and storage. To address the coordination challenges of multiple data processing engines, computing and data storage systems are often decomposed. Optimizing the performance of these data processing engines is a challenge due to the complexity of allocating multiple data processing resources. Summary of the Invention

[0003] Various examples will now be described to briefly introduce some concepts, which will be further described in the detailed embodiments. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] According to one aspect of the present invention, a computer implementation method for data analysis in a distributed computer system is provided. The computer system includes at least one data cluster of a data storage device. The method includes receiving a directed acyclic graph (DAG) plan from a user application by a distributed computing engine; transforming the DAG plan using a query analyzer module of the distributed computing engine, wherein the transformed DAG operation includes at least one semantic caching operation and at least one near data processing (NDP) pushdown operation; executing the at least one NDP pushdown operation using an NDP executor module of the data cluster; performing the at least one semantic caching operation on a distributed semantic cache memory of the distributed computing engine; completing the transformed DAG operation; and returning the result of the completed DAG plan to the user application.

[0005] Optionally, in the foregoing aspect, another implementation of this aspect provides: using the query analyzer module to merge the NDP pushdown primitive operation into the transformed DAG operation; using the query executor module of the distributed computing engine to execute the transformed DAG operation; pushing the NDP pushdown primitive operation to the data cluster for execution, and returning the result to the distributed computing engine.

[0006] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides: identifying NDP pushdown operations executable by the at least one data cluster in a subgraph; determining a cost function value for performing the identified NDP operations; and including the identified NDP pushdown operations in the transformed DAG operations based on the determined cost function value.

[0007] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides: searching for semantic information of the contents of the distributed semantic cache memory; and, based on the semantic information, using the query analyzer module to incorporate semantically aware cache commands into the transformed DAG operation.

[0008] Optionally, in the foregoing aspect, another implementation of this aspect provides: determining a cost function value for executing the semantic-aware caching command; and including the semantic-aware caching command in the transformed DAG operation based on the determined cost function value.

[0009] Optionally, in either of the above two aspects, another implementation of these aspects provides: metadata identifying the data stored in the distributed semantic cache memory; and, based on the identified metadata, at least one of a data skip cache operation or an adaptive partition cache operation.

[0010] Optionally, in any of the three aspects described above, another implementation of these aspects provides semantic information identifying intermediate data stored in the distributed semantic cache memory, wherein the semantically aware cache command includes loading the intermediate data.

[0011] Alternatively, in any of the four aspects described above, another implementation of these aspects provides a distributed semantic cache as a distributed multi-level semantic cache, wherein the semantic information indicates the contents of the distributed multi-level semantic cache.

[0012] According to another aspect of the present invention, a distributed computer system is provided. The system includes: at least one data cluster, comprising multiple data storage devices and a near data processing (NDP) executor module; a distributed semantic cache memory; and a distributed computing engine. The distributed computing engine includes multiple processing nodes, a query analyzer module, and at least one query executor module. The query analyzer module is configured to receive a directed acyclic graph (DAG) plan from a user application, wherein the DAG plan includes data flow operations and data loading operations; convert the operations of the DAG plan into a transformed DAG operation including at least one semantic cache operation and at least one near data processing (NDP) pushdown operation. The at least one query executor module is configured to propagate the at least one NDP pushdown operation to the NDP executor module for execution by the data cluster; execute the at least one semantic cache operation on the distributed semantic cache memory of the distributed computing engine; complete the transformed DAG operation; and return the result of the completed DAG plan to the user application.

[0013] Optionally, in the foregoing aspect, another implementation of this aspect provides: a query analyzer module for merging NDP pushdown primitive operations into the transformed DAG operations; and a query executor module for pushing the NDP pushdown primitive operations to the data cluster for execution and receiving the results of the NDP pushdown primitive operations.

[0014] Optionally, in any of the foregoing aspects, another implementation of these aspects provides a query analyzer module configured to: identify NDP pushdown operations executable by the data cluster in a subgraph of the DAG plan; determine a cost function value for performing the identified NDP operations; and include the identified NDP pushdown operations in the transformed DAG operations based on the determined cost function value.

[0015] Optionally, in any of the foregoing aspects, another implementation of these aspects provides a query analyzer module, which is configured to: search for semantic information of the contents of the distributed semantic cache memory; and, based on the searched semantic information, use the query analyzer module to incorporate semantically aware cache commands into the transformed DAG operation.

[0016] Optionally, in any of the foregoing aspects, another implementation of these aspects provides a query analyzer module configured to: determine a cost function value for executing the semantically aware caching command; and include the semantically aware caching command in the transformed DAG operation based on the determined cost function value.

[0017] Alternatively, in any of the foregoing aspects, another implementation of these aspects provides a distributed semantic cache as a distributed multi-level semantic cache, wherein the semantic information indicates the contents of the distributed multi-level semantic cache.

[0018] Optionally, in the foregoing aspect, another implementation of this aspect provides a distributed semantic cache memory, the distributed semantic cache memory including a semantic cache manager module for providing cache hit query results received from the query analyzer module.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided comprising executable instructions that, when executed by one or more processing nodes of a distributed computing engine, cause the distributed computing engine to perform the following actions: receiving a directed acyclic graph (DAG) plan from a user application; converting the operation of the DAG plan into an operation including at least one semantic caching operation and at least one near data processing (NDP) pushdown operation; pushing the at least one NDP pushdown operation to a data cluster for execution and receiving the result of the NDP pushdown operation; executing the at least one semantic caching operation on a distributed semantic caching memory of the distributed computing engine; completing the converted DAG operation; and returning the result of the completed DAG plan to the user application.

[0020] Optionally, in the foregoing aspect, another implementation of this aspect provides a computer-readable storage medium including executable instructions that cause the distributed computing engine to perform the following actions: identifying NDP pushdown primitive operations executable by the data cluster in a subgraph; determining a cost function value for performing the identified NDP primitive operations; and incorporating the identified NDP pushdown primitive operations into the transformed DAG operations based on the determined cost function value.

[0021] Optionally, in any of the foregoing aspects, another implementation of this aspect provides a computer-readable storage medium including executable instructions that cause the distributed computing engine to perform the following actions: searching for semantic information in the contents of the distributed semantic cache memory; and incorporating a semantically aware cache command into the transformed DAG operation based on the semantic information.

[0022] Optionally, in any of the foregoing aspects, another implementation of this aspect provides a computer-readable storage medium including executable instructions that cause the distributed computing engine to perform the following actions: determining a cost function value for executing the semantic-aware caching command; and including the semantic-aware caching command in the transformed DAG operation based on the determined cost function value. Attached Figure Description

[0023] The document includes some figures illustrating exemplary embodiments, and the textual description is provided in the detailed implementation.

[0024] Figure 1 A diagram illustrating a distributed computing system for implementing one or more exemplary embodiments.

[0025] Figure 2 A block diagram of an example of a data processing system for implementing one or more exemplary embodiments.

[0026] Figure 3 A flowchart illustrating an example of a data analysis method for implementing one or more exemplary embodiments of a distributed computer system.

[0027] Figure 4 A block diagram of an example query analyzer module for implementing one or more exemplary embodiments.

[0028] Figures 5A-5C This is a sequence diagram illustrating communication between computing resources of a distributed computing engine that implements one or more exemplary embodiments.

[0029] Figure 6 A schematic block diagram of a computer system for implementing one or more exemplary embodiments. Detailed Implementation

[0030] Reference is made in the following description to the accompanying drawings, which form a part of this document, in which specific embodiments that can be practiced are illustrated by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that other embodiments may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. Therefore, the following description of exemplary embodiments is not intended to be limiting, and the scope of the invention is defined by the appended claims.

[0031] In one embodiment, the functions or algorithms described herein may be implemented in software. This software may consist of computer-executable instructions stored in a computer-readable medium or storage device, such as one or more non-transitory memories or other types of hardware-based local or networked storage devices. Furthermore, these functions correspond to modules that may be software, hardware, firmware, or any combination thereof. Multiple functions may be executed in one or more modules as needed; the described embodiments are merely illustrative. The software may execute on a digital signal processor, an application-specific integrated circuit (ASIC), a microprocessor, or other type of processor running on a computer system, such as a personal computer, server, or other computer system, thereby turning such a computer system into a specifically programmed machine.

[0032] This function can be used to perform operations using, for example, software, hardware, firmware, etc. For example, the phrase "for" can refer to the logic circuit structure of a hardware element used to implement the associated function. The phrase "for" can also refer to the logic circuit structure of a hardware element used in a coded design to implement the associated function of firmware or software. The term "module" refers to a structural element that can be implemented using any suitable hardware (e.g., a processor, etc.), software (e.g., an application program, etc.), firmware, or any combination of hardware, software, and firmware. The term "logic" includes any function that performs a task. For example, each operation shown in the flowchart corresponds to the logic that performs that operation. Operations can be performed using software, hardware, firmware, etc. The terms "component," "system," etc., can refer to computer-related entities, hardware, software in execution, firmware, or combinations thereof. A component can be a process, object, execution, program, function, subroutine, computer, or a combination of software and hardware running on a processor. The term "processor" can refer to a hardware component, such as the processing unit of a computer system.

[0033] Furthermore, the claimed subject matter can be implemented as a method, apparatus, or article of manufacture that uses standard programming and engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computing device to implement the disclosed subject matter. The term "article of manufacture" as used herein is intended to include computer programs accessible from any computer-readable storage device or medium. Computer-readable storage media may include, but is not limited to, magnetic storage devices such as hard disks, floppy disks, magnetic stripes, optical disks, compact disks (CDs), digital versatile disks (DVDs), smart cards, flash memory devices, etc. Conversely, computer-readable media (i.e., not limited to storage media) may also include communication media, such as transmission media for wireless signals.

[0034] A data processing system includes computing resources (e.g., processing units) and data storage resources (e.g., various forms of memory units). Multiple computing and data resources can be included in a cluster of devices. All data being processed is available to all devices in the cluster. As the amount of data to be processed continues to grow, some methods decompose the system into multiple clusters to reduce system size.

[0035] Figure 1 This is a diagram of a data processing system. The data processing system is a distributed computing system, comprising multiple analysis clusters or computing clusters, and one or more data clusters. Analysis cluster 102 includes analysis servers 104, cache servers 106, and a cluster network 108 for transferring data between servers. Data cluster 110 may be a data lake cluster. A data lake can be a centralized, large-scale repository of structured or unstructured data. Data cluster 110 includes storage servers 112.

[0036] Data pipelines are used to improve the efficiency of data processing, but as the volume of data to be processed increases, data processing pipelines become more complex. Complex data processing pipelines can include multiple layers, including compute, caching, and storage layers. Manually orchestrating the cluster roles in a data processing solution requires significant effort from the user, and the result may not be the most efficient or optimized solution. One improvement is to automatically optimize the overall performance of the data processing system from a holistic perspective. This approach should establish best practices for the automated decision-making process of selecting and prioritizing system cluster roles, and should define the interactions and information transfer between applications, compute clusters, data clusters, and storage.

[0037] Figure 2 A block diagram of an example of a data processing system. Figure 2 The example illustrates a portion of an analytics cluster 202, a data cluster 210, and a distributed semantic cache memory 206. The system adds a logical layer 220, including a distributed computing engine, between the analytics cluster and the data cluster. System components in 220 may be implemented within the logical layer 220, or may be implemented or deployed upstream or downstream of the system. The distributed computing engine includes multiple processing nodes (e.g., a master node 222 and one or more worker nodes 224). Each processing node includes processing circuitry (e.g., one or more processing units), memory, and one or more application programs with instructions executed by the processing circuitry. Processing nodes may include modules that perform the functions described above.

[0038] The logical layer 220 possesses semantic knowledge to enable end-to-end optimization of query analysis and execution for big data processing applications. Semantic knowledge is knowledge about how data content is formed, not necessarily the value of the content itself. The logical layer 220 supports various big data applications sending queries to it.

[0039] The distributed computing engine in logical layer 220 uses the collection and storage of semantic knowledge of data to identify the near data processing (NDP) capabilities and semantic caching capabilities of user queries. The distributed computing engine divides queries from user applications into tasks and analysis tasks, and derives transformed plans by dividing tasks into subtasks. These subtasks collectively utilize system capabilities, such as NDP and semantic caching, to achieve global optimization of the tasks. For example, a query request may be in the form of a directed acyclic graph (DAG) plan that limits the acyclic flow of data in data processing, and the distributed computing engine can generate subgraphs (e.g., directed trees) from the DAG plan. The received DAG plan is automatically optimized based on the overall knowledge of semantic caching and NDP.

[0040] Figure 3 This is a flowchart illustrating an example of a data analysis method for a distributed computer system. The method can use... Figure 2 This is implemented using a distributed computer system. In box 302, the DAG plan is received from the user application. In box 304, the DAG plan is analyzed. Figure 2 The system's distributed computing engine includes a query or task analyzer module in the master node 222. The query analyzer module 226 analyzes the DAG plan to identify subgraphs and collects data access information for these subgraphs. Data access refers to access to the distributed semantic cache memory 206 and the system memory 212.

[0041] Figure 4 In the master control processing node (e.g., Figure 2 A block diagram of an example query analyzer module 426 implemented in the master node 222), which can be used to analyze queries about Figure 3 The DAG plan described in box 304. Query analyzer module 426 analyzes the DAG plan passed from the user application, generates an execution plan to optimize the DAG plan (e.g., using one or both of query plan generator 445 and DAG processor 447) and complete the DAG operation. Optimization is based on one or both of the computing power of the computing and data cluster and the cost of the plan (e.g., in terms of completion time, processing resources used, storage resources used, etc.).

[0042] The generated optimization plan identifies subqueries or subgraphs with data that should be cached, which subgraphs have data that should be cached, and how long the data should be cached. The query analyzer module 426 includes a semantic caching interface 442 and maintains consistency with... Figure 2The semantic cache manager 242 is connected to the query analyzer module 426. The semantic cache manager 242 provides a list of results or possible semantic cache entries for semantic cache hit queries from the query analyzer module 426. Data payloads, including subgraph identifiers (IDs) and subgraph results, can be exchanged with the semantic cache manager and the query analyzer. Control of the payload may include the direction of the subgraph ID cache.

[0043] The optimized plan also identifies subgraphs that can be offloaded to the system storage side for near-data processing. In addition to semantic caching and near-data processing, some optimization methods include the query analyzer module 426 using the accelerator manager 444 to determine which subgraphs should use processing accelerators. The query analyzer module 426 stores optimization strategies 448 and a knowledge base 446. NDP information can be received via the NDP interface 443. The knowledge base 446 can indicate the NDP capabilities and distributed semantic caching capabilities of the data cluster.

[0044] Return to Figure 3 In decision box 306, the query analyzer module determines whether a semantic caching operation should be used. Some examples of semantic caching operations include adaptive partitioning, data skipping, and intermediate data storage and loading (e.g., from a distributed semantic cache store). Adaptive partitioning is the process of dynamically reorganizing data based on workload characteristics. For example, if there are many jobs with filtering operations that involve data with specific attribute values, it may be beneficial to sort and split the data into new partitions based on the attribute values. The initial data is reordered into new partitions that can be stored in a distributed cache. Since all data in the new partition is known to have that attribute, the filtering and mapping operations required to identify data with that attribute are no longer performed, thus saving processing resources. However, the additional overhead caused by repartitioning data can be considerable in terms of additional storage space and computation.

[0045] Data skipping is the process of maintaining secondary partition attribute information for stored data. Some examples of data structures used for secondary partition attribute values ​​include:

[0046] 1. Minimum / maximum values ​​of the numeric attributes for each partition;

[0047] 2. A list of all values ​​for the category attribute within a single partition;

[0048] 3. Bloom filter for attribute values ​​within a single partition.

[0049] If data skipping information ensures that removing a partition will not result in different DAG or subgraph outcomes, then the partition can be removed (e.g., by the query analyzer). For example, if some partitions are known to have no attributes of interest, these partitions can be removed from data processing. Data skipping information can be stored as data skipping metadata. Because removed partitions do not require filtering and mapping, processing resources are saved.

[0050] Intermediate data caching is a process by which a distributed computing engine stores the results of its operations. Data may not have been fully processed by the distributed computing engine, but other operations can use this intermediate result, allowing the data to be reused. For example, a first DAG or subgraph can use a filtering operation to retrieve records from stored unprocessed data. A second DAG or subgraph can use the same filtering operation. If the result of the filtering operation is stored in the cache, the second operation can load the data without performing the filtering operation. Storing and loading intermediate results from a semantic cache eliminates the processing associated with re-executing operations.

[0051] Furthermore, in box 306, the query analyzer can use a cost function to determine whether a semantically aware caching operation should be included in the optimized DAG plan or the optimized subgraph. The cost can be based on parameters such as the time required to complete the DAG and the resources used by the subgraph (e.g., storage, number of processors required, etc.).

[0052] In the example, the cost function can be determined as follows:

[0053] Cost = Storage cost estimate (push plan) + Computation cost estimate (push plan) + α·size estimate (push plan)

[0054] Here, the push plan is a portion of the DAG plan or subgraph pushed to the query analyzer for optimization, and α is a customizable parameter (e.g., customizable by the system administrator). In box 308, the query analyzer queries the semantic cache to look for opportunities to transform traditional cache storage operations into corresponding semantically aware cache operations that convey semantic information, and implements techniques including at least one of the three techniques mentioned above: adaptive partitioning, data skipping metadata, and intermediate data. Some examples of semantically aware operations include:

[0055] Storage → (Repartitioning (source path, attributes, output path, layer))

[0056] Data skips metadata (source path, attributes, output path, layer).

[0057] Intermediate data (input DAG, output path, layer)

[0058] Load → Optimize (Enter DAG)

[0059] Evict → Delete (path, layer).

[0060] like Figure 2 As shown in the example, the semantic cache memory can be a distributed, multi-tiered semantic cache memory. These layers include a memory cache 230 as the top layer, a storage cache 232 as the middle layer, and a memory 234 as the bottom layer. The top layer provides the fastest results, and the bottom layer provides the slowest results. As shown in the example above, semantic information may include layer information.

[0061] In box 310, the query analyzer determines whether all the data in the distributed semantic cache is required to complete the transformed DAG plan. If so, data is collected from the distributed semantic cache storage 206 in box 312. The transformed subgraph operations and DAG plan operations are completed, and the result of the DAG plan is returned to the user. These operations can be performed by... Figure 2 The query executor module 236 of one or more worker nodes 224 completes the task. If it is determined at box 310 that the data in the distributed semantic cache is insufficient to complete the DAG plan or to adequately optimize the DAG plan, then at 314, the query analyzer module 226 collects metadata on the cached data and transforms the subgraph into a task that includes semantically aware data skipping and adaptive partitioning techniques.

[0062] In box 316, query analyzer module 226 examines the knowledge base of the data to identify NDP pushdown opportunities. Query analyzer module 226 may examine the registry of the computer services of data cluster 210. Depending on the capabilities of data cluster 210, query analyzer module 226 may convert operations in the subgraph (e.g., projection, aggregation, grouping, user-defined functions, etc.) into NDP pushdown operations for processing by the data cluster. Optimization plans may include decisions to leverage NDP opportunities or simply retrieve the data. For example, query analyzer module 226 may want to perform non-NDP operations to retrieve data from storage and create a materialized view of the data in a distributed semantic cache for future use.

[0063] The query analyzer module 226 may include NDP pushdown primitive operations (e.g., filtering, mapping, reduction, projection, aggregation grouping, etc.) in the transformed subgraph operations. The query executor module 236 executes the optimized subgraph and includes an NDP client application or NDP client 238 to propagate the NDP operations to the data cluster for execution. The data cluster 210 includes an NDP executor 240, which includes processing circuitry capable of executing the NDP pushdown operations received from the query executor module 236.

[0064] Pushdown operations reduce the amount of data processed near the data source, thus reducing the amount of data that needs to be transferred between the storage and compute clusters. This also reduces the amount of memory storage required on the compute side.

[0065] Back Figure 3 In box 318, the query analyzer module can use a cost function to determine whether an NDP pushdown operation that can be performed on the data cluster should be included in the optimized DAG plan or the optimized subgraph. In the example, the cost function value of the NDP pushdown operation is calculated using the same method used to determine the cost of a semantically aware cache operation. If the determined cost function value is not too large and is below a predetermined cost threshold, the identified NDP pushdown operation will be included in the transformed DAG plan operation.

[0066] At position 320, query analyzer module 226 forms an execution plan. This execution plan is an optimized DAG plan, incorporating a holistic approach to the capabilities of the data processing system. The optimized plan coordinates the execution of NDP pushdown primitives and semantic-aware caching operations such as data skipping and adaptive partitioning. The optimized plan can be an optimized subgraph of the DAG plan or a complete optimized DAG plan.

[0067] At 322, query analyzer module 226 sends an optimization plan to one or more query executor modules 236 according to its optimization strategy. At 324, the results of subgraph operations are merged and any final operations on the DAG plan are completed to generate the DAG plan result. In some aspects, the NDP operation results of the subgraph are merged with the semantically aware cache operation results of the subgraph; these subgraph results are intermediate results for further operations on the DAG plan. The completed DAG plan result is returned to the user application.

[0068] Figures 5A-5C To show Figure 2 The system's distributed computing engine has a sequence diagram of communication between its computing resources. These resources include a query analyzer 526, a distributed semantic cache 506, a query executor 536, an NDP client 538, and an NDP service 540 provided by the data cluster. The NDP service 540 performs computing services that offload operations from the compute side to the storage side. The NDP service 540 may include... Figure 2 In the NDP executor 240.

[0069] Figure 5A The example illustrates an optimization process for optimizing caching operations in a DAG plan. Driver 502 can be... Figure 2 The analysis cluster 202 analyzes any driver. Driver 502 determines candidate query plans 550 to be sent to the query analyzer 526 of the distributed computing engine and sends a logical DAG plan 552 to the query analyzer 526. The query analyzer parses the logical DAG plan. In some respects, the query analyzer 526 optimizes the logical DAG plan by converting traditional caching operations into semantically aware caching operations. Figure 5AIn the example, query analyzer 526 sends the parsed logical DAG plan to distributed semantic cache store 506. The semantic cache manager of distributed semantic cache store 506 optimizes caching operations on the parsed logical DAG plan and transforms traditional caching operations into semantically aware caching operations (e.g., including tasks based on data skipping and adaptive partitioning, such as those related to...). Figure 3 (As illustrated in the example). The resulting optimized logical DAG plan may include data partitioning information. The optimized logical DAG plan 554 is returned to the query analyzer 526.

[0070] Figure 5B and 5C This is a sequence diagram illustrating the optimization process of optimizing a DAG plan by incorporating NDP pushdown operations into the DAG plan. The query analyzer 526 receives the optimized logical DAG plan from the distributed semantic cache 506, in... Figure 5B The optimized logical DAG plan is converted into a physical DAG plan including NDP pushdown operations, and then into a physical DAG plan including multiple tasks. The physical DAG plan 556 is sent to the query executor 536 for execution. The query executor 536 can be located remotely from the query analyzer 526, and communication between the query analyzer 526 and the query executor 536 is conducted via the cloud. The physical DAG plan may include acceleration instructions and NDP DAG operations. The query executor 536 parses the physical DAG and sends the NDP DAG operations 558 to the NDP client 538.

[0071] exist Figure 5C In this context, NDP client 538 sends request 560 to NDP service 540. These requests include requests for NDP operations. Figure 5B and 5C In the example, the transport layer of query executor 536 sends the request as a Hypertext Transfer Protocol (HTTP) request to the NDP service of the data cluster. The NDP service sends a response 562 to the request, including processed data. NDP client 538 parses and transforms the response and sends the result 564 of the NDP operation and status to query executor 536. Query executor 536 executes the remainder of the physical DAG plan and sends the result 566 to query analyzer 526. Query analyzer 526 accumulates the results from semantically aware cached operations and the subgraph from query executor 536 and sends the result 568 of the query plan to driver 502.

[0072] The methods, systems, and devices described in this paper provide an intelligent logic layer for optimizing big data queries while minimizing required user interactions. User applications send semantic DAG plans to the logic layer, where semantic knowledge of the data is used to provide end-to-end optimization of the DAG plan. Semantic knowledge provides a holistic approach that utilizes semantic caching, near-data processing, hardware acceleration, and other optimizations to subdivide the DAG planning task into subtasks to best leverage these capabilities.

[0073] Figure 6 This is a schematic block diagram of a computer 600 for performing methods and algorithms according to exemplary embodiments. Not all components need to be used in all embodiments. One example is a computing device that may include a processing unit 602, a memory 603, a removable memory 610, and a non-removable memory 612. Although the exemplary computing device is illustrated and described as computer 600, the computing device may take different forms in different embodiments. For example, the computing device may be a server, a router, or a virtual router.

[0074] While various data storage elements are shown as part of computer 600, the memory may also include, or optionally include, cloud-based memory accessible via a network such as the Internet or server-based storage. Also note that the SSD may include a processor capable of running a parser, thereby allowing the transfer of parsed, filtered data through the I / O channels between the SSD and main memory.

[0075] Memory 603 may include volatile memory 614 and non-volatile memory 608. Computer 600 may include various computer-readable media, such as volatile memory 614, non-volatile memory 608, removable memory 610, and non-removable memory 612, or a computing environment that can access various computer-readable media. Computer memory includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD ROM), digital versatile disk (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

[0076] Computer 600 may include or have access to a computing environment including an input interface 606, an output interface 604, and a communication interface 616. Output interface 604 may include a display device, such as a touchscreen, which may also be used as an input device. Input interface 606 may include a touchscreen, touchpad, mouse, keyboard, camera, one or more device-specific buttons, one or more sensors integrated into computer 600 or coupled to the computer via a wired or wireless data connection, and one or more other input devices. Computer 600 may operate in a networked environment, using a communication connection to connect to one or more remote computers, such as a database server. Remote computers may include personal computers (PCs), servers, routers, network PCs, peer-to-peer devices, or other common data flow network switches. Communication connections may include local area networks (LANs), wide area networks (WANs), cellular networks, Wi-Fi, Bluetooth, or other networks. According to one embodiment, various components of computer 600 are connected to system bus 620.

[0077] Computer-readable instructions stored on a computer-readable medium can be executed by a processing unit 602 of computer 600, such as program 618. In some embodiments, program 618 includes software for implementing one or more methods described herein. Hard disk drives, CD-ROMs, and RAM are examples of articles of art that include non-transitory computer-readable media such as storage devices. The terms computer-readable media and storage devices do not include carrier waves that are considered too transient. Memory may also include networked memory, such as storage area network (SAN). Computer program 618 can be used to cause processing unit 602 to execute one or more methods or algorithms described herein. In some examples, computer 600 is the master processing node of a distributed computing engine, and the computer program implements a query analyzer module, for example... Figure 2 The query analyzer module 226.

[0078] While several embodiments have been described in detail above, other modifications are possible. For example, the logical flow depicted in the figures does not require the specific or sequential order shown to achieve the desired result. Other steps may be provided, steps may be removed from the described flow, and other components may be added to or removed from the described system. Other embodiments are within the scope of the appended claims.

Claims

1. A method executed by a distributed computer system, characterized in that, The distributed computer system includes at least one data cluster of a distributed computing engine and data storage devices, and the method includes: The distributed computing engine receives the directed acyclic graph (DAG) plan from the user application. The operation of transforming the DAG plan using the query analyzer module of the distributed computing engine includes at least one semantic caching operation and at least one near data processing (NDP) pushdown operation. The NDP pushdown primitive operation is pushed to the data cluster for execution, and the result is returned to the distributed computing engine; the processing node of the distributed computing engine is used to perform the at least one semantic caching operation on the distributed semantic cache memory of the distributed computing engine; The distributed computing engine is used to complete the operation of the transformed DAG plan, and the result of the completed DAG plan is returned to the user application.

2. The method according to claim 1, characterized in that, The operations for converting the DAG plan include: The query analyzer module converts the operations of the DAG plan into at least one semantic caching operation to obtain an optimized logical DAG plan. The query analyzer module is used to merge the NDP pushdown primitive operations into the optimized logic DAG plan to obtain the operations of the transformed DAG plan.

3. The method according to claim 2, characterized in that, The combined NDP pushdown primitive operations include: Identify the NDP pushdown operations that can be performed on the at least one data cluster in the subgraph; Determine the cost function value used to perform the NDP operation for the identifier; Based on the determined cost function value, the identified NDP push-down operation will be included in the DAG plan's operations.

4. The method according to any one of claims 1-3, characterized in that, Performing the at least one semantic caching operation includes: Search the semantic information of the contents of the distributed semantic cache memory; Based on the semantic information, the query analyzer module is used to merge semantically aware cache commands into the DAG plan operation.

5. The method according to claim 4, characterized in that, The semantic-aware cache commands to be merged include: Determine the cost function value used to execute the semantically aware caching command; Based on the determined cost function value, the semantically aware caching command is included in the operation of the DAG plan.

6. The method according to claim 5, characterized in that, include: Metadata that identifies the data stored in the distributed semantic cache memory; Based on the identified metadata, this includes at least one of the following operations: a data skip cache operation or an adaptive partition cache operation.

7. The method according to claim 5 or 6, characterized in that, The semantic information identifies intermediate data stored in the distributed semantic cache memory, and the semantic-aware cache command includes loading the intermediate data.

8. The method according to claim 5 or 6, characterized in that, The distributed semantic cache is a distributed multi-layer semantic cache, and the semantic information indicates the contents of the distributed multi-layer semantic cache.

9. A distributed computer system, characterized in that, The system includes: At least one data cluster, including multiple data storage devices and a near data processing (NDP) actuator module; Distributed semantic cache memory; Distributed computing engines, including: Multiple processing nodes; The query analyzer module is used for: Receive a directed acyclic graph (DAG) plan from the user application, wherein the DAG plan includes data flow operations and data loading operations; Transform the operations of a DAG plan into the operations of a transformed DAG plan, which includes at least one semantic caching operation and at least one near data processing (NDP) pushdown operation; At least one query executor module is used for: Used to push the NDP pushdown primitive operation to the data cluster for execution, and to receive the result of the NDP pushdown primitive operation; Perform the at least one semantic caching operation on the distributed semantic cache memory of the distributed computing engine; The operation of the transformed DAG plan is completed, and the result of the completed DAG plan is returned to the user application.

10. The system according to claim 9, characterized in that... The query analyzer module is used to convert the operation of the DAG plan into at least one semantic cache operation to obtain an optimized logical DAG plan. The query analyzer module is also used to merge NDP pushdown primitive operations into the optimized logic DAG plan to obtain the operations of the transformed DAG plan.

11. The system according to claim 10, characterized in that, The query analyzer module is used for: Identify the NDP pushdown operations that the data cluster can perform in the subgraph of the DAG plan; Determine the cost function value used to perform the NDP operation for the identifier; Based on the determined cost function value, the identified NDP push-down operation will be included in the DAG plan's operations.

12. The system according to any one of claims 9-11, characterized in that, The query analyzer module is used for: Search the semantic information of the contents of the distributed semantic cache memory; Based on the semantic information of the search, the query analyzer module is used to merge semantically aware cache commands into the DAG plan operation.

13. The system according to claim 12, characterized in that, The query analyzer module is used for: Determine the cost function value used to execute the semantically aware caching command; Based on the determined cost function value, the semantically aware caching command is included in the operation of the DAG plan.

14. The system according to claim 12, characterized in that, The distributed semantic cache is a distributed multi-layer semantic cache, and the semantic information indicates the contents of the distributed multi-layer semantic cache.

15. The system according to claim 14, characterized in that, The distributed semantic cache storage includes a semantic cache manager module for providing cache hit query results received from the query analyzer module.

16. A computer-readable storage medium including executable instructions, characterized in that, When the executable instructions are executed by one or more processing nodes of the distributed computing engine, the distributed computing engine performs the following actions, the actions including: Receive the directed acyclic graph (DAG) plan from the user application; Convert the DAG plan operations into operations that include at least one semantic caching operation and at least one near data processing (NDP) pushdown operation; The NDP pushdown primitive operation is pushed to the data cluster for execution. Receive the result of the NDP pushdown primitive operation; Perform at least one semantic caching operation on the distributed semantic cache memory of the distributed computing engine; Complete the operation of the transformed DAG plan to generate the result of the DAG plan; The result is returned to the user application.

17. The computer-readable storage medium according to claim 16, characterized in that, This includes instructions that cause the distributed computing engine to perform the following actions, the actions of which include: The operations of the DAG plan are converted into at least one semantic caching operation to obtain an optimized logical DAG plan; The NDP pushdown primitive operations are merged into the optimized logic DAG plan to obtain the operations of the transformed DAG plan.

18. The computer-readable storage medium according to claim 17, characterized in that, This includes instructions that cause the distributed computing engine to perform the following actions, the actions of which include: Identify the NDP pushdown primitive operations that the data cluster can perform in the subgraph; Determine the cost function value used to perform the NDP primitive operation on the identifier; Based on the determined cost function value, the identified NDP pushdown primitive operations are incorporated into the DAG program operations.

19. The computer-readable storage medium according to any one of claims 16-18, characterized in that, This includes instructions that cause the distributed computing engine to perform the following actions, the actions of which include: Search the semantic information of the contents of the distributed semantic cache memory; Based on the semantic information, semantically aware caching commands are merged into the DAG plan operation.

20. The computer-readable storage medium according to claim 19, characterized in that, This includes instructions that cause the distributed computing engine to perform the following actions, the actions of which include: Determine the cost function value used to execute the semantically aware caching command; Based on the determined cost function value, the semantically aware caching command is included in the operation of the DAG plan.

Images