Computing engine determination method and apparatus

Abstract

The application provides a computing engine determination method and device. In an embodiment, the method comprises: obtaining a data operation statement to be processed; determining a first memory consumption amount of a first engine in executing the data operation statement; wherein the first engine is a memory-based engine, and the first engine is used for processing target data; determining a current memory remaining amount of total memory allocated to the first engine; and determining that the first engine processes the data operation statement if the current memory remaining amount is greater than the first memory consumption amount. Thus, intelligent selection of the engine is realized, and a balance between efficiency and stability is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data query, and in particular to a computing engine determination method and device. BACKGROUND

[0002] In a big data business scenario, multiple computing engines are usually needed to meet various business requirements, which brings a problem: under the premise of ensuring stability requirements, different tasks need to use different computing engines to ensure optimal performance. It is more difficult for business personnel to select appropriate computing engines according to personal experience, which not only increases workload but also cannot guarantee the best selection according to experience.

[0003] Figure 1 A flowchart of the selection of a big data computing engine is shown. As shown in Figure 1 , the specific implementation process is as follows:

[0004] 1. The Hive engine serves as a unified SQL portal and receives SQL statements.

[0005] 2. After the SQL is parsed by the Hive engine, the data size of all tables involved in the SQL is obtained from the generated AST tree.

[0006] 3. Compare the table data size with the pre-set fixed threshold. If it is smaller, it is considered that the task data size is small, and it is submitted to Trino for execution to obtain the best performance. If it is larger, it is considered that the task data size is large, and the SQL continues to be executed by Hive to ensure stability under large data size.

[0007] However, for the above technical solution, considering that the pre-set fixed threshold cannot be applied to different sizes of clusters, the developer needs to manually adjust the threshold according to experience to improve the accuracy of engine selection, which cannot truly realize intelligent selection of computing engines.

[0008] Therefore, there is an urgent need for a method that can realize intelligent selection of computing engines.

[0009] The information disclosed in this BACKGROUND section is only intended to increase an understanding of the general context of the present application and is not intended to be taken as an acknowledgement that this information constitutes prior art that is already known in the art. SUMMARY

[0010] The embodiment of the present application provides a kind of computing engine determination method and cluster, can be compared by the remaining amount of total memory assigned to memory-based engine and the memory consumption amount of data operation statement executed by memory-based engine, select the engine of data operation statement from memory-based read-write engine and disk-based read-write engine, to further realize the intelligent selection of engine, realize the balance of efficiency and stability.

[0011] In the first aspect, the embodiment of the present application provides a kind of computing engine determination method, method includes: obtaining data operation statement to be processed;Determine the first memory consumption amount of data operation statement executed by the first engine;Wherein, the first engine is memory-based engine, and the first engine is used to process target data;Determine the current memory remaining amount of total memory assigned to the first engine;If the current memory remaining amount is greater than the first memory consumption amount, determine that the first engine processes data operation statement.

[0012] In the present scheme, the remaining amount of total memory assigned to memory-based engine and the memory consumption amount of data operation statement executed by memory-based engine can be compared to select the engine of data operation statement from the first engine based on memory read-write and the second engine based on disk read-write, to further realize the intelligent selection of engine, realize the balance of efficiency and stability.

[0013] In a possible implementation manner, if the current memory remaining amount is less than or equal to the first memory consumption amount, determine that the second engine processes data operation statement;Wherein, the second engine is disk-based engine;Second engine is used to process target data.

[0014] In the present scheme, when the first memory consumption amount is greater than the current remaining memory resource, on the one hand, it indicates that the current data volume is high, and on the other hand, it indicates that the memory-based engine does not have enough memory to execute data operation statement, so the disk-based engine is called to execute data operation statement.

[0015] Optionally, obtaining data operation statement to be processed includes: data operation statement received by the second engine is taken as data operation statement to be processed.

[0016] In the present scheme, the second engine serves as the interface for unified docking data operation statement, and can receive various data operation statements, so as to ensure the normal implementation of business.

[0017] In a possible implementation manner, determining the first memory consumption amount of data operation statement executed by the first engine includes: determining the logical execution tree of data operation statement;Wherein, the logical execution tree is used to indicate the logical flow of data processing represented by data operation statement;Based on the metadata of target data and the logical execution tree, determine the first memory consumption amount of data operation statement executed by the first engine.

[0018] In the scheme, based on the metadata of the target data and the logical execution tree, the analysis process of the first engine actually executing the data operation statement is simulated, the resource consumption in the process of the first engine actually executing the data operation statement is obtained, the real memory resource consumption of executing the data operation statement is analyzed, and the intelligent selection of the engine is ensured.

[0019] In one example, based on the metadata of the target data and the logical execution tree, the first memory consumption of the first engine executing the data operation statement is determined, including: for each node in the logical execution tree, based on the metadata of the target data, determining the data amount corresponding to the node; based on the data amount corresponding to the node, determining the second memory consumption corresponding to the node; wherein the second memory consumption is used to indicate the memory consumption of the first engine executing the task corresponding to the node; based on the second memory consumption of each node in the logical execution tree, determining the first memory consumption of the first engine executing the data operation statement.

[0020] In the scheme, based on the metadata of the target data and the memory resource consumption of each node in the logical execution tree, the analysis of the memory consumption is realized.

[0021] Optionally, based on the data amount corresponding to the node, the second memory consumption corresponding to the node is determined, including: taking the data amount that needs to be processed by the task corresponding to the node as the memory consumption, to obtain the third memory consumption corresponding to the node; determining the data processing operation corresponding to the node; based on the data processing operation corresponding to the node, determining the correction value corresponding to the third memory consumption; based on the correction value, correcting the third memory consumption to determine the second memory consumption corresponding to the node.

[0022] In the scheme, based on the correction value, the memory consumption is analyzed, and the memory consumption that can reflect the actual memory consumption is determined, so that the engine with better performance can be selected to execute the data operation statement.

[0023] In one implementation, the data processing operation is scanning, and the correction value is the reciprocal of the data parallelism.

[0024] In one implementation, the data processing operation is a network communication-based operation, and the correction value includes a first value and a second value, the first value is used to indicate the number of memory applications in the process of the first engine executing the task corresponding to the node, and the second value is used to indicate the proportion of memory applications and releases in the process of the first engine executing the task corresponding to the node.

[0025] In one implementation, the data processing operation is a local processing operation, and the correction value is the second value.

[0026] Optionally, the first memory consumption of executing the data operation statement is determined based on the respective second memory consumptions of each node in the logical execution tree, including: summing the respective second memory consumptions of each node in the logical execution tree, and taking the summed result as the first memory consumption of executing the data operation statement.

[0027] In one example, the logical execution tree of the data operation statement is determined, including: performing syntax analysis on the data operation statement to determine an abstract syntax tree; and performing semantic analysis on the abstract syntax tree to determine the logical execution tree.

[0028] In one possible implementation, the data operation statement to be processed is obtained, including: taking the data operation statement received by the terminal as the data operation statement to be processed.

[0029] In a second aspect, an embodiment of the present application provides a computing engine determination apparatus, which comprises a plurality of modules, each of which is configured to perform each step of the computing engine determination method provided in the first aspect of the present application. The division of the modules is not limited herein. The specific functions performed by each module in the computing engine determination apparatus and the beneficial effects achieved are described with reference to the functions of each step of the computing engine determination method provided in the first aspect of the present application, and will not be described herein again.

[0030] For example, the computing engine determination apparatus installs a first engine based on memory, and the first engine is configured to manage target data. The apparatus comprises:

[0031] A statement determination module is configured to obtain a data operation statement to be processed.

[0032] A consumed resource determination module is configured to determine a first memory consumption of the first engine in executing the data operation statement. The first engine is a memory-based engine, and the first engine is configured to process target data.

[0033] A remaining resource determination module is configured to determine a current memory remaining amount of a total memory allocated to the first engine.

[0034] An engine selection module is configured to determine that the first engine processes the data operation statement if the current memory remaining amount is greater than the first memory consumption.

[0035] In one possible implementation, the engine selection module is further configured to determine that a second engine processes the data operation statement if the current memory remaining amount is less than or equal to the first memory consumption. The second engine is a disk-based engine, and the second engine is configured to process target data.

[0036] Optionally, the statement determination module is configured to take a data operation statement received by a terminal as the data operation statement to be processed or take a data operation statement received by the second engine as the data operation statement to be processed.

[0037] In a possible implementation, the consumed resource determining module comprises an execution tree determining unit and a consumed amount determining unit; wherein,

[0038] The execution tree determining unit is configured to determine a logical execution tree of the data operation statement; wherein, the logical execution tree is configured to indicate a logical flow of data processing represented by the data operation statement.

[0039] The consumed amount determining unit is configured to determine, based on the metadata of the target data and the logical execution tree, a first memory consumption amount of the first engine executing the data operation statement.

[0040] In an example, the first consumed amount determining unit comprises a first consumed amount determining subunit and a second consumed amount determining subunit.

[0041] The first consumed amount determining subunit is configured to, for each node in the logical execution tree, determine, based on the metadata of the target data, a data amount corresponding to the node; determine, based on the data amount corresponding to the node, a second memory consumption amount of the first engine executing a task corresponding to the node; wherein, the second memory consumption amount is configured to indicate a consumption amount of memory of the first engine executing the task corresponding to the node.

[0042] The second consumed amount determining subunit is configured to determine, based on the respective second memory consumption amounts of each node in the logical execution tree, the first memory consumption amount of the first engine executing the data operation statement.

[0043] Optionally, the first consumed amount determining subunit specifically performs the following content:

[0044] take a data amount needed to be processed by the task corresponding to the node as a memory consumption to obtain a third memory consumption amount corresponding to the node; determine a data processing operation corresponding to the node; determine, based on the data processing operation corresponding to the node, a correction value corresponding to the third memory consumption amount; correct the third memory consumption amount based on the correction value to determine the second memory consumption amount corresponding to the node.

[0045] In an implementation, the data processing operation is scanning, and the correction value is an inverse of a data parallelism degree.

[0046] In an implementation, the data processing operation is a network communication-based operation, and the correction value comprises a first value and a second value, the first value is configured to indicate a number of times of applying memory in a process of the first engine executing the task corresponding to the node, and the second value is configured to indicate a proportion of applying and releasing memory in the process of the first engine executing the task corresponding to the node.

[0047] In an implementation, the data processing operation is a local processing operation, and the correction value is the second value, the second value is configured to indicate the proportion of applying and releasing memory in the process of the first engine executing the task corresponding to the node.

[0048] Optionally, a second consumption determining unit is configured to sum up the respective second memory consumptions of each node in the logical execution tree, and take the summed result as the first memory consumption of the execution of the data operation statement.

[0049] In one example, the execution tree determining unit comprises a syntax analysis unit and a semantic analysis unit, wherein,

[0050] The syntax analysis unit is configured to perform syntax analysis on the data operation statement and determine an abstract syntax tree.

[0051] The semantic analysis unit is configured to perform semantic analysis on the abstract syntax tree and determine a logical execution tree.

[0052] In a possible implementation, the statement determining module is configured to take the data operation statement of the receiving terminal as the data operation statement to be processed.

[0053] In a third aspect, an embodiment of the present application provides a computing engine determining device, which comprises a processor and a memory; the memory stores program instructions; and the processor is configured to execute the program instructions, so that the device performs the method in the first aspect.

[0054] In actual application, the first engine comprises a scheduling node and a plurality of working nodes, and the computing engine determining device comprises the scheduling node. Specifically, the computing engine determining device is a device in a device cluster that installs the first engine. Specifically, the device cluster comprises at least one electronic device, and the at least one electronic device installs the first engine.

[0055] Further, the second engine is also installed in the device cluster. In one example, the first engine and the second engine are installed on the same electronic device, for example, the computing engine determining device simultaneously installs the first engine and the second engine. In another example, the first engine and the second engine are installed on different electronic devices.

[0056] In a fourth aspect, an embodiment of the present application provides a computing engine determining apparatus, which comprises at least one memory configured to store a program and at least one processor configured to execute the program stored in the memory. When the program stored in the memory is executed, the processor is configured to perform the method provided in the first aspect. For example, the program is a program of the first engine, and the computing engine determining apparatus can be a device.

[0057] In a fifth aspect, an embodiment of the present application provides a computing engine determining apparatus, which is characterized in that the apparatus runs computer program instructions to perform the method provided in the first aspect. For example, the apparatus can be a chip or a processor. For example, the computer program instructions are a program of the first engine.

[0058] In one example, the device may include a processor that can be coupled to memory, read instructions from the memory, and execute the methods provided in the first aspect according to those instructions. The memory may be integrated into the chip or processor, or it may be independent of the chip or processor.

[0059] In a sixth aspect, embodiments of the present invention provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect. For example, the instructions are a program of a first engine.

[0060] In a seventh aspect, embodiments of the present invention provide a computer program product comprising instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect. For example, the instructions are a program of a first engine. Attached Figure Description

[0061] Figure 1 It is a flowchart defined by a computing engine;

[0062] Figure 2 This is a schematic diagram of the framework of a computational engine determination system provided in an embodiment of the present invention. Figure 1 ;

[0063] Figure 3 This is a flowchart illustrating a method for determining a computing engine according to an embodiment of the present invention;

[0064] Figure 4 yes Figure 3 The flowchart for step 320 is provided;

[0065] Figure 5 yes Figure 4 The flowchart for step 322 is provided;

[0066] Figure 6 This is a schematic diagram of a logic execution tree provided in an embodiment of the present invention;

[0067] Figure 7 This is a schematic diagram of the Trino engine and Hive engine selection method provided in the embodiments of the present invention;

[0068] Figure 8 This is a flowchart illustrating a computing engine determination device provided in an embodiment of the present invention;

[0069] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0070] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below with reference to the drawings.

[0071] In the description of the embodiments of the present application, the words such as "exemplary", "for example", or "for instance" are used to represent an example, an illustration, or a description. Any embodiment or design solution described as "exemplary", "for example", or "for instance" in the embodiments of the present application should not be interpreted as being more preferred or having more advantages than other embodiments or design solutions. Rather, the words such as "exemplary", "for example", or "for instance" are used in the specific manner to present the relevant concept.

[0072] In the description of the embodiments of the present application, the term "and / or" merely describes an association relationship of associated objects, and indicates that there can be three relationships, for example, A and / or B, which can represent three cases of existence of A alone, existence of B alone, and existence of A and B simultaneously. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems mean two or more systems, and multiple terminals mean two or more terminals.

[0073] In addition, the terms "first", "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features. The terms "include", "contain", "have" and their variants mean "include but are not limited to", unless otherwise specifically emphasized.

[0074] Since the concept of big data was proposed, enterprises have increasingly strong demands for data information to become data assets. In order to meet the needs of different big data application scenarios, various big data computing engine services appear in the technical field, including Hive / Spark for offline analysis scenarios, Flink / Storm for real-time computing scenarios, Trino / Impala for interactive query scenarios, etc. With the continuous maturity of the big data ecosystem and the widespread application of lake warehouse integration architecture, the trend of unified data lake storage at the bottom layer + compatibility with multiple computing engines at the upper layer has gradually become clear. Currently, Hive and Trino have become the most commonly used computing engines in offline analysis and interactive query scenarios, respectively.

[0075] Based on the Trino engine can avoid data disk, so that its interactive analysis performance is excellent in small and medium data, has gradually become the most commonly used interactive query engine, but the stability of Trino cannot be guaranteed in the analysis task of large data. Hive, as the most commonly used offline analysis engine, provides powerful data warehouse capability, and has become the de facto technical standard in the field of offline data warehouse, but the poor offline analysis performance of small and medium data is a problem that Hive has been facing.

[0076] In the big data business scenario, multiple computing engines are usually needed to meet various business needs, which brings a problem: under the premise of ensuring stability, in order to ensure the best performance, different tasks need to use different computing engines, and more difficultly, business personnel need to choose the appropriate computing engine according to personal experience, which not only increases the workload but also cannot guarantee the best choice according to experience.

[0077] Figure 1 The flowchart of the selection of the computing engine of big data is shown. As shown in Figure 1 The specific implementation process is as follows:

[0078] 1. The Hive engine receives the SQL statement as a unified SQL entry.

[0079] 2. After the SQL is parsed by the Hive engine, the data size of all tables involved in the SQL is obtained from the generated AST tree.

[0080] 3. Compare the table data size with the pre-set fixed threshold value. If it is less than the threshold value, it is considered that the task data size is small, and it is submitted to Trino for execution to obtain the best performance. If it is greater than the threshold value, it is considered that the task data size is large, and the SQL continues to be executed by Hive to ensure the stability of large data.

[0081] However, the above technical solution has the following two technical problems:

[0082] Firstly, the intelligent selection of the computing engine cannot be truly realized: considering that the pre-set fixed threshold value cannot be applied to different sizes of clusters, the developer needs to manually adjust the threshold value according to experience to improve the accuracy of engine selection.

[0083] Secondly, the accuracy of automatic selection of the computing engine is insufficient: the table data size obtained from the generated AST tree is the physical storage space of the table. In the actual calculation process, data is loaded into memory in batches, and memory occupation and release are performed simultaneously. Therefore, the method of judging the resource consumption during task running by the physical storage space of the table is also not accurate.

[0084] To solve the above technical problems, the embodiment of the present application provides a big data computing engine selection method.

[0085] Figure 2 An architecture example diagram of a computing engine selection system provided by the embodiment of the present application is shown. The computing engine determination method provided by the embodiment of the present application can be applied to the system architecture diagram as shown in Figure 2 Figure 2 The computing engine selection system includes a terminal device 101 and a device cluster 102.

[0086] The terminal device 101 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The exemplary embodiments of the terminal device involved in the present solution include, but are not limited to, electronic devices equipped with iOS, android, Windows, Harmony OS or other operating systems. The type of electronic device is not specifically limited in the embodiment of the present application.

[0087] The device cluster 102 can be implemented by an independent electronic device or a device cluster composed of multiple electronic devices. In some possible implementation manners, the electronic devices in the device cluster 102 can be terminals or computers, and can also be servers.

[0088] In an example, the server involved in the present solution can be used to provide cloud services, which can be a server or a super terminal that can establish a communication connection with other devices and provide computing and / or storage functions for other devices. The server involved in the present solution can be a hardware server, or can be implanted in a virtualization environment. For example, the server involved in the present solution can be a virtual machine executed on a hardware server including one or more other virtual machines.

[0089] ​The terminal device 101 communicates with the device cluster 102 through a network. The network can be a wired network or a wireless network. For example, the wired network can be a cable network, a fiber network, a digital data network (DDN), etc., and the wireless network can be a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public service telephone network (PSTN), a Bluetooth network, a ZigBee network, a global system for mobile communications (GSM) network, a code division multiple access (CDMA) network, a general packet radio service (CPRS) network, or any combination thereof. It can be understood that the network can use any known network communication protocol to realize communication between different client layers and gateways. The network communication protocol can be various wired or wireless communication protocols, such as an Ethernet, a universal serial bus (USB), a firewire, a global system for mobile communications (GSM), a general packet radio service (GPRS), a code division multiple access (CDMA), a wideband code division multiple access (WCDMA), a time-division code division multiple access (TD-SCDMA), a long term evolution (LTE), a new radio (NR), a Bluetooth, a wireless fidelity (Wi-Fi), or the like.

[0090] In the embodiment of the present application, the device cluster 102 can store big data (for the convenience of description and distinction, referred to as target data), and can also install multiple engines, and the multiple engines can read and calculate the target data. Here, the multiple engines include an engine based on memory for reading and writing (for the convenience of description and distinction, referred to as a first engine) and an engine based on a disk for reading and writing (for the convenience of description and distinction, referred to as a second engine). The first engine can be a Trino engine or an impala engine, and the second engine can be a Hive engine. Different engines have advantages and disadvantages in query process. For example, the Hive engine needs to perform multiple disk read and write operations in the query process, and each disk read and write operation will cause a delay, but the Hive engine is suitable for a large amount of data query process. The Trino engine is a distributed real-time query engine based on memory, and there is no delay problem caused by disk read and write in the query process, and it has a faster query speed, but the amount of query data is limited. Therefore, different engines need to be selected based on the actual data processing process.

[0091] It should be noted that the electronic devices in which the first engine, the second engine and the target data in the device cluster 102 provided by the embodiment of the present application can be different, or can partially overlap. For example, the electronic devices in the device cluster 102 can store part of the target data, can execute part of the functions of the first engine, and can also execute part of the functions of the second engine; for another example, the device cluster 102 can be divided into three clusters, one cluster stores the target data, one cluster installs the first engine, and the other cluster installs the second engine. The embodiment of the present application does not make specific limitation on this, and the server in which the first engine, the second engine and the target data are located can be set according to actual conditions.

[0092] Based on this, after the terminal device 101 obtains the query instruction triggered by the user, the data operation statement can be obtained, and the data operation statement is uploaded to the device cluster 102. The first engine in the device cluster 102 can pre-calculate the total memory consumption of executing the data operation statement, obtain the current memory remaining amount in the memory allocated to the first engine, compare the total memory consumption with the current memory remaining amount in the memory allocated to the first engine, if less, the first engine such as Trino engine is used to execute the data operation statement, if greater, the second engine such as Hive engine is used to execute the data operation statement, so as to select the best data processing engine.

[0093] It should be noted that when there are multiple first engines installed in the device cluster 102, the memory resource consumption of executing the data operation statement needs to be compared with the current memory remaining amount of each first engine. If the current memory remaining amount of all the first engines is less than the memory resource consumption of executing the data operation statement, the data operation statement is processed by the second engine.

[0094] The architecture diagram of the above computing engine selection system is only an example. In some possible implementations, the computing engine selection system further includes an engine selection server, the terminal device 101 communicates with the engine selection server through a network, and the device cluster 102 communicates with the engine selection server through the network. For details of the network, refer to the above description, and no further description is given. The engine selection server can be implemented by an independent server or a device cluster composed of multiple servers. In some possible implementations, after obtaining the query instruction triggered by the user, the terminal device 101 of the embodiment of the application can obtain the data operation statement and upload the data operation statement to the engine selection server. The engine selection server requests the first engine in the device cluster 102 to pre-calculate the total memory consumption of executing the data operation statement, obtains the current memory remaining amount in the memory allocated to the first engine, compares the total memory consumption with the current memory remaining amount in the memory allocated to the first engine, and if the total memory consumption is less than the current memory remaining amount, the data operation statement is executed by the first engine such as the Trino engine in the device cluster 102, and if the total memory consumption is greater than the current memory remaining amount, the data operation statement is executed by the second engine such as the Hive engine in the device cluster 102, so as to select the best engine for data processing.

[0095] Next, a computing engine determination method provided by the embodiment of the application is introduced. The computing engine can include a first engine based on memory read and write and a second engine based on disk read and write. Here, the first engine and the second engine are installed in the device cluster 102. The method can be executed by any device, apparatus, platform or device cluster having computing and processing capability. For example, the device cluster 102 described above can be used to execute the method. Figure 2In the device cluster 102, the technical solution provided by the embodiment of the present application involves the parsing and consumption estimation of the data operation statement, and the specific engine is required to determine the functions, and thus, the electronic device (for the convenience of description and distinction, referred to as a target device) in which the first engine is installed in the device cluster 102 is preferred to execute. It should be noted that the first engine includes a scheduling node and a plurality of working nodes, the scheduling node is used to convert the data operation statement into a plurality of tasks, and the plurality of working nodes are scheduled to execute. The scheduling node can be an entity electronic device, or a virtual electronic device, such as a virtual server, a virtual machine, etc. The working node is similar. For example, an entity electronic device can be provided with a scheduling node and a processing node at the same time, and the scheduling node and the processing node can share the physical resources of the electronic device. Correspondingly, the scheduling node and the processing node can be virtual machines. The scheduling node is generally used to parse the data operation statement and estimate the consumption to implement task scheduling, and thus, the electronic device in which the scheduling node is located is the target device. The following is described taking the target device as an execution subject.

[0096] It should be noted that the device cluster 102 stores target data. For example, the target data can be massive data stored in the storage device cluster 103, such as structured database tables, semi-structured text data, and unstructured voice, picture, video, etc. In addition, the device cluster 102 stores metadata of the target data, based on which the target data is managed. The metadata, also known as intermediate data or relay data, is data about data, mainly information describing data properties, and is used to support functions such as indicating storage location, historical data, resource search, file recording, etc. Metadata is information about the organization of data, data fields and their relationships. In short, metadata is data about data. In practical applications, the metadata of the target data is stored in a distributed file system, such as HDFS (Hadoop Distributed File System), NFS (Network File System), etc., but is not limited thereto. The deployed distributed file system can manage the target data. For example, the device cluster 102 can be deployed with a distributed file system, based on which the stored target data is managed.

[0097] Figure 3 A flowchart of a computing engine determination method provided by an embodiment of the present application is shown. The method includes the following steps:

[0098] Step 310, receiving a data operation statement to be processed.

[0099] In actual application, in one example, the target device receives the data operation statement sent by the terminal device 101, which is a data operation statement to be processed. In one example, if the second engine in the device cluster 102 serves as a unified interface for receiving data operation statements, the second engine in the device cluster 102 receives the data operation statement sent by the terminal device 101, and sends the data operation statement to the target device where the first engine is installed.

[0100] Here, the data operation statement indicates the logic of filtering the required data from the target data and performing data processing. The data operation statement can be embodied by a query language such as SQL (Structured Query Language), HQL (Hibernate Query Language), etc. For example, the method provided by the present application is mainly described in connection with the SQL statement, but the present application is not limited to the specific type of data operation statement in view of the diversity of data operation statements. It should be noted that the first engine and the second engine installed in the device cluster 102 can process the same data operation statement.

[0101] In one embodiment, the data operation statement can be generated according to the target operation of the user on the query interface provided by the terminal device 101.

[0102] For example, the query interface includes an input box, and correspondingly, the query operation includes but is not limited to the query content input by the user through the input box. The data operation statement can be generated from the input content in the input box, and optionally, the input content can be the data operation statement.

[0103] For example, the query interface includes a data operation statement component, which is a component encapsulating a data operation statement and can be used repeatedly. Correspondingly, the query operation is to assemble a complete data operation statement by the user dragging at least one data operation statement component on the query interface.

[0104] Further, the query interface includes a data operation statement generation control, and when the terminal device 101 detects a click operation on the data operation statement generation control, the data operation statement is generated according to the query operation in the above two example scenarios, and the data operation statement is uploaded to the device cluster 102.

[0105] Step 320, determining the first memory consumption of the first engine for executing the data operation statement.

[0106] It is worth noting that the first memory consumption indicates the memory consumed by the first engine in executing the data operation statement. It should be noted that, in order to determine the first memory consumption of the first engine in executing the data operation statement, the logic processed by different engines is not completely consistent, so the first engine in the device cluster 102 is usually evaluated, so that the first memory consumption of the first engine in executing the data operation statement can be accurately evaluated.

[0107] Step 330, determining the current memory remaining amount of the total memory allocated to the first engine.

[0108] In practical applications, the target device manages the use of resources allocated to the first engine in the device cluster 102, thereby determining the use of the total memory allocated to the first engine, and obtaining the current memory remaining amount.

[0109] Then, the target device determines whether the first memory consumption is less than the current remaining memory value. If yes, the data operation statement is executed by the first engine, in other words, step 340a is executed. Otherwise, it is executed by the second engine, in other words, step 340b is executed. As shown in the figure, after step 330, either 340a is executed or 340b is executed. The embodiments of the present application mainly take the processing of SQL statements by the Trino engine and the Hive engine as an example to illustrate the method provided by the present application, but considering the diversity of computing engines, the present application does not limit the specific type of computing engine. Figure 3

[0110] Step 340a, if the current memory remaining amount is greater than the first memory consumption, determining that the first engine processes the data operation statement.

[0111] For example, the first engine can be a trino engine. The working principle is as follows: the trino engine includes a scheduling node and a plurality of worker nodes. The scheduling node is used to parse the data operation statement after receiving the data operation statement, generate a logical execution plan, generate a plurality of actual tasks, and distribute the plurality of actual tasks to all worker nodes. The worker nodes are responsible for actually executing the actual tasks, and the plurality of worker nodes can perform data transmission between each other, and each worker node can interact with the distributed file system to read the data stored on the device cluster 102 corresponding to the distributed file system. The worker node notifies the scheduling node to end the query after the calculation is completed, and sends the query result to the scheduling node. It is worth noting that the calling node and the worker node represent a thread or a process.

[0112] ​In a specific implementation, for the trino engine, a thread pool can be pre-built, and a plurality of worker threads (created by a plurality of worker nodes) can be preloaded in the thread pool. After the trino engine receives a SQL statement, a SQL parser can be called to parse the SQL statement to obtain an AST tree. Then, the AST tree can be converted into a logical execution tree by a logical execution plan component. The logical execution tree is distributed and parsed by a distributed plan component to obtain a plurality of plans, and each plan is converted into a corresponding task. The plurality of tasks can be scheduled to the plurality of worker threads for execution by calling an algorithm. Here, the scheduling algorithm can be a random algorithm, a round-robin scheduling algorithm, a weighted round-robin algorithm, etc.

[0113] In one embodiment, when the first memory consumption is less than the current remaining memory resource, it indicates that the current data volume is not high, and it also indicates that the trino engine has sufficient memory to execute the data operation statement. At this time, the trino engine is called to execute the data operation statement to ensure execution efficiency.

[0114] It is worth noting that the logical execution tree is obtained by the logical execution plan component in the trino engine. When it is determined that the data operation statement is executed by the trino engine, the scheduling node can directly parse the logical execution tree to obtain a plurality of tasks, and schedule the plurality of tasks to the plurality of worker nodes for execution.

[0115] In step 340b, if the current memory remaining amount is less than or equal to the first memory consumption, it is determined that the first engine processes the data operation statement.

[0116] For example, the second engine can be a Hive engine. The Hive engine is a data warehouse tool based on Hadoop, which can map structured data files into a database table and provide simple sql query functions. The sql statement can be converted into a MapReduce (a programming model) task for running.

[0117] In practical implementation, for the Hive engine, after receiving an SQL statement, it calls the SQL parser to parse the SQL statement and obtain an AST tree. The logical execution plan component then transforms the AST tree into a logical execution tree. The distributed plan component performs distributed parsing of the logical execution tree, obtaining multiple plans, and each plan is transformed into a corresponding Map task. Each Map task reads data from disk, processes it, and then outputs the intermediate results to disk for storage. Because the Hive engine needs to perform multiple disk reads and writes, the query process has a relatively long latency. However, precisely because the Hive engine writes intermediate results to disk, it has few limitations on the amount of data. It is worth noting that in practical applications, the Hive engine typically also includes a scheduler node and multiple worker nodes. The scheduler node is used to parse the data operation statement after receiving it, generate a logical execution plan, generate multiple Map tasks, and distribute the Map tasks to all worker nodes.

[0118] In one embodiment, when the first memory consumption is greater than the current remaining memory resources, it indicates that the current data volume is high, and it also indicates that the Trino engine does not have enough memory to execute data operation statements. In this case, the Hive engine is called to execute the data operation statements.

[0119] Therefore, in this embodiment of the invention, by comparing the remaining amount of total memory allocated to the memory-based engine with the memory consumption of the memory-based engine in executing data operation statements, the engine for executing data operation statements can be selected from the memory-based read / write engine and the disk read / write engine. The memory-based read / write engine can quickly process the data operation statements, or the disk read / write engine can stably process the data operation statements, thereby achieving intelligent engine selection and achieving a balance between efficiency and stability.

[0120] Figure 4 As shown Figure 3 The flowchart of step 302 in the illustrated embodiment is shown.

[0121] like Figure 4 As shown above, in the above Figure 3 Based on the illustrated embodiment, in an exemplary embodiment of the present invention, step 320 may specifically include the following steps:

[0122] Step 321: Determine the logical execution tree of the data manipulation statements; wherein, the logical execution tree indicates the logical flow of data processing represented by the data manipulation statements.

[0123] First, the target device performs syntax analysis on the data operation statement to obtain an AST tree. Here, the AST (abstract syntax code) is a tree representation of the abstract syntax structure of the source code, and each node on the tree represents a structure in the source code. It is said to be abstract because the abstract syntax tree does not represent every detail of the real syntax, such as nested parentheses, which are implicitly included in the structure of the tree and are not presented in the form of a node. The abstract syntax tree is independent of the syntax of the source language, that is, the context-free grammar used in the syntax analysis stage.

[0124] Then, semantic analysis is performed on the AST tree to obtain a logical execution tree.

[0125] The logical execution tree is a tree composed of operators and indicates the processing logic of the database table. In actual applications, the logical execution tree is an optimized tree, and the optimization of the logical execution tree is a function of the first engine and the second engine themselves. The specific optimization method needs to be determined in combination with the design of the engine.

[0126] It should be noted that after the semantic analysis of the AST tree, the AST tree can be analyzed semantically to obtain the table name of the data table processed by the SQL statement, the data processing operation, and further obtain the logical execution tree. Figure 5 A schematic diagram of a logical execution tree provided by an embodiment of the present application is shown in FIG. 1. Figure 6 As shown in FIG. 1, the data processing operation includes data scanning (TableScan), filtering (Filter), projection (Project), hash join (HashJoin) for implementing table join, data aggregation (Aggreate), and the like.

[0127] The data scanning (TableScan) means that all data in the table are scanned to display the data result. In actual applications, the data result can be obtained by scanning a part of the data according to the index.

[0128] The filtering (Filter) means selecting data in the table that meets the condition.

[0129] The projection (Project) means converting the data in the table to another memory space for processing.

[0130] The table connection (JOIN) can be understood as associating between tables through certain connection conditions among multiple tables, and obtaining data from multiple tables. Since different tables are in different servers, the table connection process involves communication between servers in the device cluster 102. The hash connection (Hash join) is only one way to implement table connection, which can improve the connection efficiency. In actual application, there are other ways of table connection.

[0131] The data aggregation (Aggregete) can be understood as a process of collecting data in different servers in a table and representing the data in summary. It should be noted that the data of a table can be stored in different servers, and therefore data aggregation is required.

[0132] It should be noted that, in the embodiment of the present application, the first memory consumption of the first engine executing the data operation statement needs to be determined, and the logic of the logical execution tree generated by different engines is not completely consistent. Therefore, the logical execution tree is usually determined by the first engine in the device cluster 102, so as to ensure that the first memory consumption of the first engine executing the data operation statement can be accurately evaluated.

[0133] Then, the first memory consumption of the first engine executing the data operation statement can be determined based on the metadata of the target data and the logical execution tree.

[0134] Based on the metadata, the size of the data table, the number of rows, the data storage space of each row, and other information can be known, and then the data to be processed when executing the logical execution tree can be obtained, so that the memory consumption of executing the data operation statement can be more accurately analyzed. Figure 4 As shown in the following steps:

[0135] In step 322, for each node in the logical execution tree, the amount of data corresponding to the node is determined based on the metadata of the target data, and the second memory consumption corresponding to the node is determined based on the amount of data corresponding to the node.

[0136] The second memory consumption is used to indicate the memory consumption of the first engine executing the task corresponding to the node.

[0137] First, for each node in the logical execution tree, the target device determines the amount of data corresponding to that node based on the target data's metadata and the name of the data table processed by that node. Based on this data amount, it then determines the second memory consumption for that node. The second memory consumption indicates the memory consumption of the task corresponding to the first engine's execution node. In practical applications, a task corresponding to each node can be generated based on the logical execution tree, and this task indicates a plan for implementing the data operations indicated by the node. Specifically, based on the metadata, information such as the size of the data table, the number of rows, and the data storage space per row can be obtained, thus determining the amount of data processed by each node in the logical execution tree, and consequently, the second memory consumption for each node. Specifically, the data amount can be the product of the number of data rows and the data storage space per row.

[0138] Step 323: Based on the second memory consumption of each node in the logical execution tree, determine the first memory consumption of the first engine executing data operation statements.

[0139] Next, based on the second memory consumption of each node in the logical execution tree, the first memory consumption of executing the data operation statement is determined. Specifically, the first memory consumption of executing the data operation statement is obtained by summing the second memory consumption of each node in the logical execution tree.

[0140] Therefore, in this embodiment of the invention, by reconstructing the analysis process of the actual execution of data operation statements by the first engine, the resource consumption during the actual execution of data operation statements by the first engine is obtained, and the actual memory resource consumption of the executed data operation statements is analyzed, thereby ensuring the intelligent selection of the engine.

[0141] Figure 5 As shown Figure 4 The flowchart of step 322 in the embodiment shown is illustrated.

[0142] like Figure 5 As shown above, in the above Figure 3 Based on the illustrated embodiment, in an exemplary embodiment of the present invention, step 322, which involves determining the first memory consumption of executing data operation statements based on the metadata and logical execution tree of the target data, may specifically include the following steps:

[0143] Step 3221: Take the amount of data that the task corresponding to the node needs to process as the memory consumption, and obtain the third memory consumption of the node.

[0144] In practical implementation, the device cluster 102 can determine the amount of data that the task corresponding to the node needs to process, and use the amount of data as the third memory consumption of the node.

[0145] Step 3222, determining the data processing operation under the task corresponding to the node.

[0146] In the embodiment of the application, the data processing operations represented by the plurality of nodes in the logical execution tree have multiple types, and generally one node indicates one data processing operation. Specifically, there can be three types, Type 1: data scanning (TatleScan); Type 2: local processing, such as filtering (Filter) and projection (Project); and Type 3: network communication, such as table joining (JOIN) and data aggregation (Aggregete).

[0147] As shown in Figure 6 , the data processing operations include data scanning (TatleScan), filtering (Filter), projection (Project), hash joining (Hashjion) for implementing table joining, and data aggregation (Aggregete). Correspondingly, the third memory consumption amounts are cost(TatleScan), cost(Filter), cost(Project), cost(Hashjion), and cost(Aggregete).

[0148] Step 3223, determining a correction value corresponding to the third memory consumption amount based on the data processing operation under the task corresponding to the node.

[0149] For Type 1: data scanning.

[0150] It is considered that data is not loaded into memory all at once, but the proportion of data loaded into memory at the same time is determined according to maxPartition (maximum parallelism).

[0151] Therefore, when the data processing operation of the node is data scanning, that is, Type 1, the correction value corresponding to the node is used to represent the data parallelism. For example, the correction value can be W1, W1 = 1 / maxPartition, thereby reflecting the real memory occupation.

[0152] For Type 2: local processing.

[0153] It is considered that the task corresponding to the node processed by the first engine is parallel processing, and the occupation and release of memory resources are also performed at the same time, for example, for the trino engine, the task corresponding to the node is processed by multiple worker nodes in parallel. Therefore, when calculating the memory resource consumption, it is not accurate to only consider the total amount of memory resource application, and the memory release during the process of the task corresponding to the node processed by the first engine in parallel needs to be considered.

[0154] Therefore, when the data processing operation of the node is a local processing operation, i.e., type 2, the correction value corresponding to the node is used to represent the proportion of memory application and release in the process of executing the task corresponding to the node by the first engine, so as to reflect the real memory occupation. For example, the correction value can be W (also referred to as a second value), W = r1 / T1, where r1 represents the real memory consumption in the process of executing the task corresponding to the node by the first engine, and T1 represents the total amount of memory resource application in the process of executing the task corresponding to the node by the first engine. As shown in Figure 7 The correction value corresponding to the node for implementing filtering is W(Filter), and the correction value corresponding to the node for implementing projection is W(Project).

[0155] For type 3: network communication.

[0156] The actual task corresponding to the node is executed in parallel by multiple nodes, and on the other hand, considering that a large amount of data will be received through network communication, the actual occupied memory is N (also referred to as a first value) times the input data. Here, N can be represented by the number of times of memory application in the execution process of the task corresponding to the node.

[0157] Therefore, when the data processing operation of the node is a network communication-based operation, i.e., type 3, the correction value corresponding to the node is used to represent, on the one hand, the proportion of memory application and release in the process of executing the task corresponding to the node by the first engine, and on the other hand, the newly occupied memory (represented by the number of times of memory application) through network communication in the process of executing the task corresponding to the node by the first engine, so as to reflect the real memory occupation.

[0158] For example, the correction value can include a second value W and a first value N, where W is described above and will not be repeated. Specifically, the correction value can be W*N. As shown in Figure 7 The correction value corresponding to the node for implementing Hashjion is W(Hashjion)*N(Hashjion), and the correction value corresponding to the node for implementing Aggregete is W(Aggregete)*N(Aggregete).

[0159] Step 3224, correcting the third memory consumption based on the correction value to determine the second memory consumption corresponding to the node.

[0160] Specifically, the device cluster 102 takes the product of the correction value corresponding to the node and the third memory consumption as the second memory consumption corresponding to the node.

[0161] As shown in Figure 6As shown, the second memory consumption of the type 1 node is cost(TatleScan) / maxPartition; the second memory consumption of the type 2 node is cost(Filter)*W(Filter); cost(Project)*W(Project); the second memory consumption of the type 3 node is cost(HashJoin)*W(HashJoin)*N(HashJoin); cost(Aggreate)*W(Aggreate)*N(Aggreate).

[0162] Therefore, in the embodiments of the present application, the modified value is obtained by considering the memory resource consumption of each node in the actual execution process of the task, the memory resource consumption is modified based on the modified value, the resource consumption of each node in the actual execution process of the task is obtained, the real memory resource consumption of executing the data operation statement is ensured, and the intelligent selection of the engine is further ensured.

[0163] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0164] Based on the above-provided calculation engine determination method, the specific application of the method is described. Figure 6 A schematic diagram of a Trino engine and a Hive engine selection method provided for the implementation of the present application is shown in FIG. 1. Figure 6 As shown, the specific content includes:

[0165] 1. The Hive engine serves as a unified SQL portal and receives a SQL statement. The SQL statement is handed over to the Trino engine for processing.

[0166] 2. The Trino engine parses the SQL statement, generates an AST tree, and generates a logical execution tree based on the AST tree.

[0167] 3. The Trino engine counts the memory consumption of executing the SQL statement.

[0168] 4. The Trino engine compares the memory consumption with the remaining memory amount of the total memory allocated to itself. If it is smaller, it is considered that the task data amount is small, and the task is submitted to the Trino engine to obtain the best performance and efficiency; if it is larger, it is considered that the task data amount is large, and the SQL statement is handed over to the Hive engine for execution to ensure stability under large data amount.

[0169] In the embodiment of the present application, the engine for executing the SQL statement can be selected from the Trino engine and the Hive engine based on the comparison result of the remaining memory of the Trino engine and the memory consumption of the Trino engine for executing the SQL statement, the processing of the SQL statement can be quickly implemented based on the Trino engine, or the processing of the SQL statement can be stably implemented based on the Hive engine, thereby realizing intelligent selection of the engine and balancing efficiency and stability.

[0170] Based on the same concept as the method embodiment of the present application, the embodiment of the present application further provides a computing engine determination device. The device includes several modules, each module is used to execute each step in the computing engine determination method provided by the first aspect of the embodiment of the present application, and the division of the modules is not limited here. The specific functions performed by each module in the computing engine determination device and the beneficial effects achieved are described above with reference to the functions of steps 310 to 340b of the computing engine determination method provided by the embodiment of the present application, and will not be repeated here.

[0171] Figure 8 is a structural schematic diagram of a computing engine determination device provided by the embodiment of the present application.

[0172] As Figure 8 shown, the computing engine determination device provided by the embodiment of the present application, the computing engine determination device installs a first engine based on memory, the first engine is used to manage target data, and includes:

[0173] The statement determination module 801 is used to obtain a data operation statement to be processed.

[0174] The consumption resource determination module 802 is used to determine a first memory consumption of the first engine for executing the data operation statement; wherein the first engine is a memory-based engine, and the first engine is used to process target data.

[0175] The remaining resource determination module 803 is used to determine a current memory remaining amount of the total memory allocated to the first engine.

[0176] The engine selection module 804 is used to determine that the first engine processes the data operation statement if the current memory remaining amount is greater than the first memory consumption.

[0177] The detailed functions of each module of the statement determination module 801, the consumption resource determination module 802, the remaining resource determination module 803 and the engine selection module 804 are described above for steps 310 to 340b and will not be repeated here.

[0178] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of modules is taken as an example, and in actual application, the above function distribution can be completed by different modules according to needs, that is, the internal structure of the device is divided into different modules to complete all or part of the functions described above. Each module in the embodiment can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, such as the computing engine determination device can be a plurality of electronic devices in the device cluster 102, or in the form of a software function unit, such as the computing engine determination device can be deployed in the first engine. In addition, the specific names of each functional unit and module are only for convenient distinction, and do not limit the protection scope of the present application. The specific working process of the modules in the device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0179] Based on the same idea as the method embodiments of the present application, the embodiment of the present application also provides an electronic device.

[0180] Figure 9 is a structural schematic diagram of an electronic device provided by the embodiment of the present application.

[0181] As shown in Figure 9 , the electronic device 900 includes a processor 901, a memory 902 and a network interface 903.

[0182] The processor 901 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0183] The memory 902 can include one or more computer program products that can include various forms of computer-readable storage media that can be either volatile or non-volatile memory, or can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory, for example. The volatile memory can be random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). One or more computer programs can be stored on the computer-readable storage media of the memory 902, and the processor 901 implements the steps in the embodiments of the various engine determination methods described above when executing the computer programs, for example. Figure 3 The steps 310 to 340b shown above.

[0184] By way of example, the computer programs can be divided into one or more modules / units, which are stored in the memory 902 and executed by the processor 901 to complete the present application. The one or more modules / units can be a series of computer program instruction segments that can complete a specific function. For example, the computer programs can be divided into the sentence determination module 801, the consumed resource determination module 802, the remaining resource determination module 803, and the engine selection module 804, and the specific functions of each module are described above.

[0185] The network interface 903 is configured to transceive data, for example, transmitting the data processed by the processor 901 to other electronic devices, or receiving the data transmitted by other electronic devices, etc.

[0186] Of course, in order to simplify, Figure 9Only some of the components of the electronic device 900 related to the present application are shown, and components such as buses, input / output interfaces, and the like are omitted. In addition, the electronic device 900 can include any other appropriate components according to specific application cases. In addition, the electronic device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. Those skilled in the art can understand that Figure 9 The electronic device 900 is only an example and does not constitute a limitation on the electronic device, and can include more or fewer components than shown, or combine certain components, or different components, for example, the electronic device can also include input devices, output devices, network access devices, buses, and the like. For example, the input device can be a microphone array, and can also include, for example, a keyboard, a mouse, and the like. For example, the output device can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0187] In addition to the above method, device and electronic device, the embodiment of the present application can also provide a computer program product including computer program instructions, which, when executed by a processor, causes the processor to perform the steps in the computing engine determination method of the embodiment of the present application. Wherein the computer program product can be written in one or more program design languages in any combination for computer program code for performing the operations of the embodiment of the present application, including object-oriented program design languages such as Java, C++, and conventional procedural program design languages such as "C" language or similar program design languages. Wherein the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer program code can be executed entirely on the electronic device, partially on the electronic device, as an independent software package, partially on the electronic device and partially on a remote electronic device, or entirely on a remote electronic device.

[0188] In addition, the present application also provides a device cluster 102. The device cluster 102 includes a plurality of electronic devices. For example, the electronic device is the electronic device 900 described above. The device cluster 102 is used to perform the steps in the computing engine determination method of the embodiment of the present application.

[0189] In actual application, the processor 901 in the plurality of electronic devices 900 in which the first engine is installed in the device cluster 102 runs the program of the first engine to execute the computing engine determination method provided by the embodiment of the present application. More specifically, the program of the first engine is run by the processor 901 in the target device in which the scheduling node in the first engine is located to execute the computing engine determination method provided by the embodiment of the present application.

[0190] Optionally, the device cluster 102 installs the first engine and the second engine simultaneously. For example, the target device described above installs the first engine and the second engine simultaneously.

[0191] Optionally, the device cluster 102 can be divided into a first cluster and a second cluster, the first cluster installs the first engine, and the second cluster installs the second engine.

[0192] Here, the installation can be understood as storing the program of the engine in the memory 902, so that the processor 901 can run the program of the engine to realize the functions that the engine can realize.

[0193] In addition, the embodiment of the present application can also provide a computer readable storage medium, which stores computer program instructions, and the computer program instructions make the processor execute the steps in the computing engine determination method of the embodiment of the present application when the processor runs. The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. It should be noted that the computer readable medium contains contents which can be appropriately added or reduced according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0194] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0195] The basic principles of the present application are described above in combination with specific embodiments, but it should be noted that the advantages, advantages, effects and the like mentioned in the present application are only examples and not limitations, and these advantages, advantages, effects and the like cannot be considered as the must-have of each embodiment of the present application. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and are not limited to the above specific details. The above details do not limit the present application to the above specific details.

[0196] The block diagrams of the devices, apparatuses, equipment, systems involved in the present application are only illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be as shown in the block diagrams. As those skilled in the art will recognize, the devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have", and the like are open-ended words, mean "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0197] It should also be noted that in the devices, apparatuses and methods of the present application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present disclosure.

[0198] The above description is given for illustrative and descriptive purposes. In addition, the above description is not intended to limit the embodiments of the present application to the above description. Although a plurality of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

[0199] It can be understood that the various numerical numbers involved in the embodiments of the present application are only distinguished for the convenience of description, and are not used to limit the scope of the embodiments of the present application.

Claims

1. A method for determining a computing engine, characterized in that, The method comprises: acquiring a data operation statement to be processed; determining a first memory consumption of a first engine in executing the data operation statement; wherein the first engine is a memory-based engine, and the first engine is used for processing target data; determining a current memory remaining amount of a total memory allocated to the first engine; if the current memory remaining amount is greater than the first memory consumption, determining that the first engine processes the data operation statement; if the current memory remaining amount is less than or equal to the first memory consumption, determining that a second engine processes the data operation statement; wherein the second engine is a disk-based engine; the second engine is used for processing the target data, the first engine is used for an interactive scenario, and the second engine is used for an offline analysis scenario; the determining of the first memory consumption of the first engine in executing the data operation statement comprises: determining a logical execution tree of the data operation statement; wherein the logical execution tree is used for indicating a logical flow of data processing represented by the data operation statement; based on metadata of the target data and the logical execution tree, determining the first memory consumption of the first engine in executing the data operation statement; the determining of the first memory consumption of the first engine in executing the data operation statement based on the metadata of the target data and the logical execution tree comprises: for each node in the logical execution tree, based on the metadata of the target data, determining a data amount corresponding to the node; based on the data amount corresponding to the node, determining a second memory consumption corresponding to the node; wherein the second memory consumption is used for indicating a memory consumption of the first engine in executing a task corresponding to the node; based on the second memory consumption of each node in the logical execution tree, determining the first memory consumption of the first engine in executing the data operation statement; the determining of the second memory consumption corresponding to the node based on the data amount corresponding to the node comprises: taking a data amount needed to be processed by the task corresponding to the node as a memory consumption, to obtain a third memory consumption corresponding to the node; determining a data processing operation corresponding to the node; based on the data processing operation corresponding to the node, determining a correction value corresponding to the third memory consumption; based on the correction value, correcting the third memory consumption to determine the second memory consumption corresponding to the node. the determining of the first memory consumption of the first engine in executing the data operation statement based on the second memory consumption of each node in the logical execution tree comprises:

2. The method of claim 1, wherein, summing the second memory consumption of each node in the logical execution tree, and taking a result after the summing as the first memory consumption of the first engine in executing the data operation statement. the determining of the correction value corresponding to the third memory consumption based on the data processing operation corresponding to the node comprises:

3. The method according to claim 1 or 2, characterized in that, if the data processing operation is scanning, the correction value is an inverse of a data parallelism degree; or ​ If the data processing operation is a network communication-based operation, the correction value includes a first value and a second value, the first value is used to indicate a number of times of applying for memory in a process of executing a task corresponding to the node by the first engine; and the second value is used to indicate a proportion of applying for and releasing memory in the process of executing the task corresponding to the node by the first engine; or If the data processing operation is a local processing operation, the correction value is the second value.

4. The method according to claim 1 or 2, characterized in that, The obtaining of the data operation statement to be processed includes: The data operation statement of the receiving terminal is taken as the data operation statement to be processed, or the data operation statement received by the second engine is taken as the data operation statement to be processed.

5. The method according to claim 1 or 2, characterized in that, If the current memory remaining amount is greater than the first memory consumption amount, the determining of whether the first engine processes the data operation statement includes: The tasks of each node in the logical execution tree are executed by the first engine.

6. A computing engine determination device, characterized by, The device includes a processor and a memory; The memory stores program instructions; The processor is configured to execute the program instructions, so that the device executes the method in any one of claims 1 to 5. ​

Images