Calculation acceleration method and related device

Abstract

The invention provides a calculation acceleration method and a related device.The method comprises the steps that target data are calculated through a preprocessing plug-in on a calculation node, an intermediate calculation result is obtained, and the intermediate calculation result is stored in a database; the intermediate calculation result in the database is called through an interface of an on-line analytical processing (OLAP) vectorization engine on the calculation node for operation, a calculation result is obtained, and the OLAP vectorization engine can process data in a storage organization format in the database. By adopting the method provided by the embodiment of the invention, the execution efficiency of vectorization acceleration can be improved in a big data processing process.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a computing acceleration method and related devices. Background Art

[0002] Big data is the new oil. Many industries are actively processing massive amounts of data quickly to obtain valuable information. As the size of data grows, more and more server resources are needed to process this data. For example, the telecommunications industry needs hundreds of servers to analyze and calculate user data of 50 million users within an acceptable time. The higher the big data computing performance, the fewer servers are needed and the lower the cost. Therefore, higher performance in big data computing means lower costs and stronger competitiveness. High performance is a very important goal of big data computing.

[0003] The data engine usually used for big data computing is Spark, and Spark runs in the Java virtual machine (JVM). It is difficult for JVM to optimize central processing unit (CPU) instructions, and the CPU instruction-level optimizations (such as single instruction, multiple data (SIMD)) provided by JVM are far less than those of native languages ​​such as C / C++.

[0004] The current mainstream server CPUs all support SIMD vectorization instructions. If the CPU's SIMD vectorization instructions can be fully utilized for vectorization acceleration during the computing process, the parallelism can be greatly improved, thereby greatly improving the computing performance. The current vectorization acceleration process can use Spark's computing framework (foundation) + Glue (Gluten) plug-in (for bonding) + Java native interface (Java native interface, JNI) to call the vectorization engine (for acceleration). However, in this process, there are problems such as the high additional overhead caused by the repeated conversion of the vectorization engine's memory structure to the JVM's Java object, and the low execution efficiency caused by repeated calls between the JVM and the vectorization library. Summary of the invention

[0005] The embodiments of the present application provide a computing acceleration method and related devices, which can improve the execution efficiency of vectorized acceleration during big data processing.

[0006] In a first aspect, the present application provides a computing acceleration method, comprising:

[0007] The target data is calculated through the preprocessing plug-in on the computing node to obtain intermediate calculation results, and the intermediate calculation results are stored in the database; the intermediate calculation results in the database are called through the interface of the online analytical processing OLAP vectorization engine on the computing node to perform operations and obtain calculation results, wherein the OLAP vectorization engine can process data in the storage organization format of the database.

[0008] The embodiment of the present application adds a preprocessing plug-in mechanism to the OLAP vectorization engine to pre-calculate the target data, pre-process the process that is not good at distributed processing, and improve the processing efficiency. Then the intermediate calculation results obtained by the processing are stored in the intermediate table so that the OLAP vectorization engine can call the data and further execute the distributed processing process. At the same time, the redistribution mechanism is loaded to realize the redistribution of the computing tasks during the processing. In the above process, the preprocessing plug-in mechanism and the redistribution mechanism are loaded on the OLAP vectorization engine, which means that all execution processes are added to the same process, so that the entire process can process data in the database based on the same language without the need to convert data in different storage organization formats, reducing the conversion overhead and improving the execution efficiency of the vectorization process.

[0009] In a possible implementation, before the target data is calculated by the preprocessing plug-in on the computing node, the method further includes: generating the preprocessing plug-in based on the computing task by an execution plan generator, and generating a calling method of the OLAP vectorization engine; distributing the preprocessing plug-in and the calling method of the OLAP vectorization engine to the computing node by the execution plan generator.

[0010] In a possible implementation, before the target data is calculated by the preprocessing plug-in on the computing node, the method further includes: sending the distribution strategy to the distributor through the execution plan generator; and distributing the target data to the computing node based on the distribution strategy through the distributor.

[0011] In the embodiment of the present application, the target data is distributed to the computing nodes through the distributor, and the data with aggregation affinity is distributed to the same computing node, which occupies less memory and avoids frequent disk IO when obtaining data in pull mode. In addition, the aggregation calculation can achieve at most one redistribution, which greatly reduces the network IO.

[0012] In a possible implementation, before the execution plan generator sends the distribution strategy to the distributor, the method further includes:

[0013] The distribution strategy is generated or adjusted based on relevant factors by executing the plan generator, and the relevant factors include at least one of the number of nodes and the computing tasks.

[0014] It can be seen that in the embodiment of the present application, the execution plan generator can dynamically adjust the distribution strategy according to the change of the number of nodes or the computing task, or the change of other factors, and then send the adjusted distribution strategy to the distributor, so that the distributor distributes the target data according to the adjusted distribution strategy. In this way, the process of distributing the target data can be flexibly adjusted to ensure the efficiency of the computing process.

[0015] In one possible implementation, the calculation includes map and / or reduce calculation.

[0016] In a second aspect, an embodiment of the present application provides a computing acceleration system, which includes a computing node, and is used to: calculate target data through a preprocessing plug-in on the computing node to obtain intermediate calculation results, and store the intermediate calculation results in a database; call the intermediate calculation results in the database through an interface of an online analytical processing (OLAP) vectorization engine on the computing node to perform operations and obtain calculation results, wherein the OLAP vectorization engine can process data in an organizational format stored in the database.

[0017] In a possible implementation, the system further includes an execution plan generator for: generating a preprocessing plug-in based on a computing task, and generating a calling method of an OLAP vectorization engine; and distributing the preprocessing plug-in and the calling method of the OLAP vectorization engine to computing nodes.

[0018] In a possible implementation, the system further includes a distributor, wherein the execution plan generator is further configured to send the distribution strategy to the distributor; and the distributor is configured to distribute the target data to the computing nodes based on the distribution strategy.

[0019] In a possible implementation, before the execution plan generator sends the distribution strategy to the distributor, the execution plan generator is further used to: generate or adjust the distribution strategy based on relevant factors, where the relevant factors include at least one of the number of nodes and computing tasks.

[0020] In one possible implementation, the calculation includes map and / or reduce calculation.

[0021] In a third aspect, the present application also provides an electronic device, which includes: a computing module, which is used to calculate the target data through a preprocessing plug-in, obtain intermediate calculation results, and store the intermediate calculation results in a database; and is also used to call the intermediate calculation results in the database through the interface of the OLAP vectorization engine to perform operations and obtain calculation results, wherein the OLAP vectorization engine can process data in an organizational format stored in the database.

[0022] In a possible implementation, the device further includes an execution plan generation module, which is used to: generate a preprocessing plug-in based on the computing task, and generate a calling method of the OLAP vectorization engine; and distribute the preprocessing plug-in and the calling method of the OLAP vectorization engine to the computing nodes.

[0023] In a possible implementation, the device further includes a distribution module, and the execution plan generation module is further used to: send the distribution strategy to the distribution module; the distribution module is used to: distribute the target data to the calculation module based on the distribution strategy.

[0024] In a possible implementation, the execution plan generation module is further used to: generate or adjust the distribution strategy based on relevant factors, where the relevant factors include at least one of the number of nodes and computing tasks.

[0025] In one possible implementation, the calculation includes map and / or reduce calculation.

[0026] In a fourth aspect, the present application provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, and when the processor executes a computer program or instruction in the memory, the method of any implementation of the above-mentioned first aspect is executed.

[0027] Optionally, the device also includes a memory.

[0028] Optionally, the device further includes a communication interface, and the processor is coupled to the communication interface.

[0029] Optionally, there are one or more processors and one or more memories.

[0030] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0031] Optionally, the transceiver may include a transmitter (transmitter) and a receiver (receiver).

[0032] In one implementation, the electronic device is a server or a terminal device. When the electronic device is a server or a terminal device, the communication interface may be a transceiver, or an input / output interface. Optionally, the transceiver may be a transceiver circuit. Optionally, the input / output interface may be an input / output circuit.

[0033] In another implementation, the electronic device is a chip or a chip system. When the device is a chip or a chip system, the communication interface may be an input / output interface, an interface circuit, an output circuit, an input circuit, a pin or a related circuit on the chip or the chip system. The processor may also be embodied as a processing circuit or a logic circuit.

[0034] In a fifth aspect, the present application provides a computer program product, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute a method in any possible implementation of the first aspect.

[0035] In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program (also referred to as code, or instructions) which, when executed on a computer, enables the computer to execute a method in any possible implementation of the first aspect described above.

[0036] In a seventh aspect, the present application also provides a circuit, comprising: a processor and an interface, for executing a computer program or instruction stored in a memory, and executing a method in any possible implementation of the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the application of big data in various industries provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of the principle of vectorized acceleration of big data computing provided in an embodiment of the present application;

[0039] Figure 3 A logical architecture diagram of a glue component provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of a processing flow for vectorized acceleration of big data computing provided in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of a computing acceleration system provided in an embodiment of the present application;

[0042] Figure 6 A timing diagram of computing acceleration provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0045] Big data computing has a wide range of application scenarios, and is widely used in various industries in society, including telecommunications, manufacturing, Internet, finance, automobiles, catering, energy, and entertainment.

[0046] See also Figure 1 , Figure 1A schematic diagram of the application of big data in various industries provided in the embodiment of the present application is as follows: Figure 1 As shown, these include: Manufacturing, which can use the analysis and calculation of industrial big data to diagnose and predict product faults, improve process flows, and optimize energy consumption in the production process.

[0047] In the financial industry, we conduct big data analysis and calculation on massive amounts of transaction data to obtain valuable information such as credit risk analysis and social sentiment analysis to guide industry work.

[0048] The Internet industry can analyze customer behavior and launch targeted product recommendation advertising through massive big data analysis and calculations.

[0049] Big data computing plays an important role in many industries. In addition to the examples shown in the figure, it can also include the telecommunications industry. By performing big data computing on massive amounts of signaling and user plane data, network quality problems can be discovered in a timely manner to quickly improve network quality, improve user network experience, and timely grasp customer off-network tendencies, and perform necessary customer care retention. It can also include the automotive industry, energy industry, or biomedical industry, etc. The embodiments of the present application can be widely used in these scenarios where big data analysis and calculation are required or in more related scenarios. By vectorization, big data computing can be accelerated, server resources can be saved, and customers can be helped to reduce the cost of industry big data analysis and calculation.

[0050] Spark is usually used for big data computing, and Spark runs on JVM. It is difficult for JVM to optimize CPU instructions, and the CPU instruction-level optimization (such as SIMD) provided by JVM is far less than that of native languages ​​such as C / C++. Currently, mainstream server CPUs all support SIMD vectorized instructions. If the SIMD vectorized instructions of the CPU can be fully utilized for vectorized acceleration during the computing process, the degree of parallelism can be greatly improved, thereby greatly improving computing performance.

[0051] Currently, many large companies with profound technical expertise in the industry are exploring vectorized acceleration in the field of big data computing. After vectorized acceleration, computing performance has been significantly improved, and some can even reach more than three times the performance of the currently commonly used computing engine Spark.

[0052] See also Figure 2 , Figure 2 A schematic diagram of the principle of vectorized acceleration of big data computing provided in the embodiment of the present application is as follows: Figure 2As shown in the figure, the structure of vectorized acceleration of big data computing mainly consists of three parts: execution plan generator, executor, and vectorized library. When the computing task is started, the computing task is sent to the computing engine. After the execution plan generator generates the execution plan, the computing framework can do the vectorized acceleration part in the computing process. The vectorized library is called through the acceleration plug-in, and the vectorized library accelerates the SIMD instructions that call the CPU. In other words, the improvement of computing performance is achieved by calling the vectorized library during the computing process to use the vectorized instructions of the CPU to realize the parallelization of data processing, thereby achieving computing acceleration and obtaining higher computing performance.

[0053] The present application embodiment provides a commonly used method for vectorizing and accelerating big data computing based on a glue plug-in. Figure 3 , Figure 3 A logical architecture diagram of a glue component provided for an embodiment of the present application, the main components of the architecture include: Spark's computing framework, glue plug-in, and JNI call vectorization engine (native library). Spark's computing framework may include a data framework, an execution plan generator, and an executor, etc. The computing framework of Spark is also loaded with a JVM structured query language (structured query language, SQL) engine (also referred to as SparkSQL), which is used to call operation objects, perform just in time (just in time, JIT) compilation experience, and full-stage code generation. The glue plug-in is used to provide a high-level to low-level execution plan conversion, such as SQL to syntax tree conversion, syntax tree to execution pipeline (pipeline) conversion. It is also used to provide an intermediate table for storing intermediate calculation results, perform convergence calculations and redistribution (shuffle), a shim layer (or adaptation layer, shim layer) for realizing interface docking, a calculation process fallback, and a metric module (metric) for statistically analyzing and reporting indicator information during execution. In addition, the component can use an object-oriented programming language, such as Scala or Java. The distributed computing engine used is Spark, and for the convenience of development, the PySpark interface can be further adopted. In addition, the R language can be integrated for statistical computing and statistical mapping, and Koala can be integrated for graphics compilation.

[0054] In other words, the execution principle of this architecture is: intercept the query execution plan for SparkSQL through the glue plug-in on Spark, split the execution plan, and send the part that can be vectorized and accelerated to the vectorized engine for execution, skipping the inefficient execution path of native Spark. The overall execution framework still uses the existing implementation of Spark, including consumption interface, resource and execution scheduling, query plan optimization, upstream and downstream integration, etc. In other words, the computing execution framework base of big data computing is still the execution framework of Spark, relying on the mechanism of Spark plug-in, and using the glue plug-in to call the vectorized engine for vectorized acceleration.

[0055] Figure 4 A schematic diagram of a process flow for vectorized acceleration of big data computing provided in an embodiment of the present application, combined with Figure 4 The main processing flow of vectorization acceleration of big data computing based on the glue plug-in includes the following parts:

[0056] ① Execution plan splitting: Split the execution plan of SparkSQL into two parts: JVM execution + vectorization engine execution. The driver node in the entire computing framework, or the execution plan generator, splits the execution plan of SparkSQL into the above two parts. During the splitting process, preparations are made for subsequent execution calculations, including generating data distribution strategies and calling the query language used by SparkSQL.

[0057] ②JVM execution: The Spark computing framework's own JVM is used as the basic execution engine, and data is obtained from the disk by pulling. Figure 4 As shown in , steps ②, ③, and ④ are all computing tasks executed by the computing node (further, the executor on the computing node). When the computing task starts to execute, data needs to be pulled from the disk through the JVM.

[0058] ③ Vectorized engine execution: The glue plug-in calls the vectorized engine through JNI for vectorized acceleration. The part of the computing task that can be vectorized and accelerated is sent to the vectorized engine for execution. The vectorized engine can include the column storage database ClickHouse, the general execution engine Velox across different data management systems, other third-party engines, and the Apache Arrow computer engine for performing vectorized processing.

[0059] ④ Fallback during the calculation process: For the parts that the vectorization engine cannot execute, the glue plug-in returns to Spark's JVM execution path.

[0060] The main drawbacks of this technology include:

[0061] High IO: The overall execution framework is still Spark, so the data flow model is still Spark's pull model, reading data directly from the disk, resulting in high disk IO, and convergence computing cannot be achieved without redistribution, large network consumption, and high cost.

[0062] Large overhead: When the execution process falls back, the memory structure of the vectorization library is repeatedly converted to the Java object of the JVM, and the process row and column conversions are multiple times, resulting in very large overhead.

[0063] Low execution efficiency and poor maintainability: It executes a JVM (Spark itself) + a vectorization engine (C / C++ vectorization library), and repeatedly makes JNI calls during the execution process, resulting in low execution efficiency.

[0064] Based on the above problems, the present application embodiment provides a computing acceleration system. Figure 5 , the system comprises:

[0065] Computing nodes (workers) are used to perform calculations on target data and vectorize and accelerate calculation results.

[0066] The computing node includes both a preprocessing plug-in and an online analytical processing (OLAP) vectorization engine. The calculation of the target data is specifically performed by the preprocessing plug-in on the computing node, and the intermediate calculation results are obtained and stored in the database. The interface of the OLAP vectorization engine calls the intermediate calculation results in the database for calculation to obtain the calculation results, wherein the OLAP vectorization engine can call and process the data in the storage organization format in the database.

[0067] The target data is big data, specifically, for example, big data in the fields of telecommunications, manufacturing, or the Internet industry described above. The calculation of the target data can be, for example, at least one of mapping or reduction (map / reduce) operations. That is to say, the target data performs distributed operations on the computing nodes, and some intermediate calculation results can be obtained. For example, the operation process for the formula A+B+α*C is executed to obtain the calculation result D. These intermediate calculation results can then be stored in the local database (or intermediate table) on the computing node. The data format stored in the intermediate table is column storage. The computing node also includes an OLAP vectorization engine that can call vectorization instructions in a native-like language, such as C or C++. This can then achieve vectorized acceleration for further processing of the intermediate calculation results, such as array execution of mapping and reduction operations. OLAP using quasi-native languages ​​can be seamlessly connected with the preprocessing plug-in on the computing node, that is, the query statement of the OLAP vectorization engine and the pre-calculation of the preprocessing plug-in can be in the same task process, and can directly call the data stored in the intermediate table (OLAP database query statements can directly call columnar storage data) without the need to convert the storage organization format.

[0068] Specifically, the data storage organization format includes row storage format (such as TextFile, SequenceFile, Avro, etc.), column storage format (such as ORC, RC file, parquet, etc.), or can also include plain text storage format (such as text, CSV, etc.), structured storage format (such as JSON, etc.). In the existing vectorized accelerated big data computing process, the memory structure of Spark's JVM object is a row storage structure (the core data structure is a distributed dataset (RDD) and a data table DataFrame), while the vectorization engine is a column storage structure. The JVM pushes the data that can be processed by the vectorization engine to the vectorization engine for execution, and needs to perform format conversion from row storage data to column storage data. When the calculation result is returned to the JVM from the vectorization engine, it needs to be converted from column storage data to row storage data. In this way, the interaction of the calculation process needs to be repeated format conversion, resulting in a very large additional overhead. However, in the embodiment of the present application, since the OLAP vectorization engine can be seamlessly connected with the preprocessing plug-in, when the intermediate calculation results obtained after the preprocessing plug-in is processed by the OLAP vectorization engine, no data format conversion is required. The overhead in the process of vectorization acceleration of big data calculation is greatly reduced. On the other hand, in the prior art, the glue plug-in uses the form of calling the vectorization engine in JNI to directly call the local code in the Spark computing node task thread, and its calling efficiency overhead is relatively large and the execution efficiency is low. And a set of JVM and a set of C / C++ program stacks need to be maintained in one execution, and the maintainability is relatively poor. The embodiment of the present application is executed in a purely local way (OLAP uses a quasi-native language, and the preprocessing plug-in uses a local language). There is only one C / C++ program stack in the calculation process, and no JNI call is required, which further reduces the overhead in the process of vectorization acceleration of big data calculation and improves the execution efficiency.

[0069] After the OLAP vectorization engine executes the calculation task, the obtained calculation result is sent to the output (module). For the calculation tasks that have not been completed, they can be redistributed and then calculated.

[0070] It can be seen that in the embodiment of the present application, a preprocessing plug-in mechanism is added to the OLAP vectorization engine to pre-calculate the target data, pre-process the process that is not good at distributed processing, and improve the processing efficiency. Then the intermediate calculation results obtained by the processing are stored in the intermediate table so that the OLAP vectorization engine can call the data and further execute the distributed processing process. At the same time, the redistribution mechanism is loaded to realize the redistribution of the computing tasks during the processing. In the above process, the preprocessing plug-in mechanism and the redistribution mechanism are loaded on the OLAP vectorization engine, that is to say, all execution processes are added to the same process, so that the entire process can be based on the same language to process data in the database without the need to convert data in different storage organization formats, reducing the conversion overhead and improving the execution efficiency of the vectorization process.

[0071] Optionally, the system further includes an execution plan generator (planner) for: generating a preprocessing plug-in based on the computing task, and generating a calling method based on the OLAP vectorization engine; and distributing the preprocessing plug-in and the calling method based on the OLAP vectorization engine to the computing nodes.

[0072] The execution plan generator can generate a calling method for the preprocessing plug-in and the OLAP vectorization engine based on the computing task, and the calling method can specifically be a database query language. Since the embodiment of the present application can be used for the operation of the Spark framework, the execution plan generator can be used to analyze the SparkSQL computing task. The preprocessing plug-in is used to perform pre-calculation and data format conversion at the entrance. The intermediate calculation results obtained can be pushed to the intermediate table for column storage, which can obtain a higher compression ratio and thus occupy less memory. Vectorized accelerated computing is driven by the SQL of the OLAP vectorization library. Therefore, the execution plan generator can generate a specific OLAP SQL query language and distribute it to the computing nodes so that the computing nodes use the database query language to call the intermediate calculation results in the database (or can also directly call the target data that does not need to be preprocessed from the database) and perform calculations, which can specifically be map / reduce calculations to obtain the final calculation results.

[0073] Optionally, the system further includes a dispatcher, wherein the execution plan generator is further configured to send a distribution strategy to the dispatcher; and the dispatcher is configured to distribute the target data to the computing nodes based on the distribution strategy.

[0074] like Figure 5As shown in , the distributor in this system is integrated into the southbound module as a plug-in (Java SDK, C / C++, etc.), and the southbound module passes the record data to the distributor based on the memory, and the distributor pushes the target data to the preprocessing plug-in according to the distribution strategy. The distribution strategy can be generated by the execution plan generator and sent to the distributor, or it can be sent to the distributor by other modules, which is not specifically limited in the embodiments of the present application. The distribution strategy can specifically be a hash distribution strategy. The distribution strategy can also indicate that data with the same join key (having aggregation affinity) should be distributed to the preprocessing plug-in on the same computing node to further improve data calculation efficiency.

[0075] It can be seen that in the embodiment of the present application, the target data is distributed to the computing node through the distributor, and the data with aggregation affinity is distributed to the same computing node, which occupies less memory and avoids frequent disk IO when obtaining data in pull mode. And the aggregation calculation can achieve at most one redistribution, which greatly reduces the network IO.

[0076] Optionally, before the execution plan generator sends the distribution strategy to the distributor, the execution plan generator is further used to: generate or adjust the distribution strategy based on relevant factors, where the relevant factors include at least one of the number of nodes and computing tasks.

[0077] The distribution strategy is sent to the distributor by the execution plan generator, and before that, the distribution strategy is generated by the execution plan generator. Specifically, the distribution strategy can be generated according to the computing task. For example, in distributed computing, when the computing resources consumed by each computing task are similar, the generated distribution strategy is: distribute the target data related to the same computing task to the same computing node (computing node). Alternatively, the distribution strategy can also be generated according to the number of nodes, for example, the data is divided into a corresponding number of target data according to the number of nodes, and then the distribution strategy indicates that each target data is distributed to the corresponding computing node. Alternatively, the distribution strategy can be generated according to both the computing task and the number of nodes at the same time, which can be specifically combined with the above examples and will not be repeated here.

[0078] In addition, the execution plan generator can also dynamically adjust the distribution strategy according to changes in the number of nodes or computing tasks, or changes in other factors, and then send the adjusted distribution strategy to the distributor so that the distributor can distribute the target data according to the adjusted distribution strategy. This allows the process of distributing target data to be flexibly adjusted to ensure the efficiency of the computing process.

[0079] After the OLAP vectorization engine in the computing node completes the vectorization acceleration of big data computing, the final calculation result can be sent to the output (module) (sink), and then the output (module) outputs the calculation result to complete the entire big data distributed computing process.

[0080] See also Figure 6 , Figure 6 A timing diagram of a computing acceleration method provided in an embodiment of the present application, such as Figure 6 As shown, the method comprises the following steps:

[0081] 101. Calculate the target data through the preprocessing plug-in on the computing node, obtain intermediate calculation results, and store the intermediate calculation results in the database.

[0082] 102. Call the intermediate calculation results in the database through the interface of the online analytical processing (OLAP) vectorization engine on the computing node to perform calculations to obtain calculation results, wherein the OLAP vectorization engine can process data in a storage organization format in the database.

[0083] Among them, the execution plan is loaded on the computing node, mainly the preprocessing plug-in and the OLAP vectorization engine. The loaded OLAP vectorization engine can be specific to the calling method of loading the OLAP vectorization engine, such as the database query language, or the SQL query language. The preprocessing plug-in and the OLAP vectorization engine on the same computing node can be seamlessly connected. Then, after receiving the target data on the computing node, the preprocessing plug-in can be used to pre-calculate the target data, and the intermediate calculation structure obtained by the pre-calculation can be stored in the database, or the intermediate table. The OLAP vectorization engine then calls the data in the intermediate table through the SQL query language to perform map / reduce calculations to obtain the final calculation results.

[0084] Optionally, before computing the target data by the preprocessing plug-in on the computing node, the method further includes:

[0085] 103. Generate a preprocessing plug-in based on the computing task through the execution plan generator, and generate a calling method based on the OLAP vectorization engine;

[0086] 104. The preprocessing plug-in and the calling method based on the OLAP vectorization engine are distributed to the computing nodes through the execution plan generator.

[0087] The process of the execution plan generator generating the calling method of the preprocessing plug-in and the OLAP vectorization engine, and distributing the calling method of the preprocessing plug-in and the OLAP vectorization engine to the computing nodes can be specifically referred to the relevant description of the execution plan generator in the above embodiment, which will not be repeated here.

[0088] Optionally, before computing the target data by the preprocessing plug-in on the computing node, the method further includes:

[0089] 105. Send the distribution strategy to the distributor through the execution plan generator;

[0090] 106. Distribute the target data to the computing nodes through the distributor based on the distribution strategy.

[0091] The specific process of generating a distribution strategy and distributing target data to computing nodes according to the distribution strategy can be found in the relevant description of the computing acceleration system above, which will not be repeated here.

[0092] Optionally, before the execution plan generator sends the distribution strategy to the distributor, the method further includes (not shown in the figure):

[0093] 107. Generate or adjust a distribution strategy based on relevant factors by executing a plan generator, where the relevant factors include at least one of the number of nodes and computing tasks.

[0094] The specific process of generating or adjusting the distribution strategy can be found in the relevant description of the computing acceleration system, which will not be repeated here. It should be noted that when the above steps are included in the method, according to Figure 6 For steps not included in the method, they are not executed. For example, if the method only includes step 101 and step 102, then only Figure 6 The two steps are executed in the timing of . When the method further includes step 103 and step 104, the two steps are executed before step 101. In addition, since the execution plan generator can newly generate and adjust the distribution strategy, step 107 is executed before step 105. The execution plan generator can also adjust the original distribution strategy, so step 107 may also be executed after step 105. Therefore, there is no strict execution order between step 107 and step 105.

[0095] like Figure 7 The structural diagram of the electronic device shown in FIG. 1 is a schematic diagram of the electronic device shown in FIG. 1 . The embodiment of the present application further provides an electronic device 1100, which may be a server or may be used for a server. It may also be a terminal device or may be used for a terminal device. The device 1100 includes:

[0096] The calculation module 1101 is used to calculate the target data through the preprocessing plug-in, obtain the intermediate calculation results, and store the intermediate calculation results in the database;

[0097] It is also used to call the intermediate calculation results in the database through the interface of the OLAP vectorization engine to perform operations and obtain calculation results, wherein the OLAP vectorization engine can process data in the storage organization format in the database.

[0098] Optionally, the apparatus 1100 further includes an execution plan generating module 1102, which is used to:

[0099] Generate preprocessing plug-ins based on computing tasks and generate calling methods for the OLAP vectorization engine;

[0100] Distribute the calling methods of the preprocessing plug-in and OLAP vectorization engine to the computing nodes.

[0101] Optionally, the apparatus 1100 further includes a distribution module 1103, and the execution plan generation module 1102 is further used to: send the distribution strategy to the distribution module 1103;

[0102] The distribution module 1103 is used to distribute the target data to the calculation module 1101 based on the distribution strategy.

[0103] Optionally, the execution plan generation module 1102 is further used to: generate or adjust the distribution strategy based on relevant factors, where the relevant factors include at least one of the number of nodes and computing tasks.

[0104] Optionally, the computation includes map and / or reduce computation.

[0105] Optionally, the device 1100 may further include an output module (not shown in the figure), etc. For details, please refer to Figure 5 and Figure 6 The related description of the embodiments of the present application will not be described in detail.

[0106] The present application provides a computer-readable storage medium, in which computer instructions are stored, and the computer instructions instruct the server to execute the computing acceleration method provided by any of the above-mentioned embodiments.

[0107] It should also be understood that the first, second, third, fourth and various numerical numbers involved in this document are only distinctions made for the convenience of description and are not intended to limit the scope of the present application.

[0108] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0109] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0110] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0112] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0115] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0116] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0117] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0118] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A computing acceleration method, characterized in that: The method comprises: Calculate the target data through the preprocessing plug-in on the computing node to obtain intermediate calculation results, and store the intermediate calculation results in the database; The intermediate calculation results in the database are called through the interface of the online analytical processing (OLAP) vectorization engine on the computing node to perform operations and obtain calculation results, wherein the OLAP vectorization engine can process data in the storage organization format of the database.

2. The method according to claim 1, characterized in that Before calculating the target data by the preprocessing plug-in on the computing node, the method further includes: Generate the preprocessing plug-in based on the computing task through the execution plan generator, and generate a calling method of the OLAP vectorization engine; The calling methods of the preprocessing plug-in and the OLAP vectorization engine are distributed to the computing nodes through the execution plan generator.

3. The method according to claim 1 or 2, characterized in that: Before calculating the target data by the preprocessing plug-in on the computing node, the method further includes: Send the distribution strategy to the distributor through the execution plan generator; The target data is distributed to the computing nodes by the distributor based on the distribution policy.

4. The method according to claim 3, characterized in that Before the execution plan generator sends the distribution strategy to the distributor, the method further includes: The distribution strategy is generated or adjusted by the execution plan generator based on relevant factors, and the relevant factors include at least one of the number of nodes and computing tasks.

5. The method according to claim 1, characterized in that The calculation includes map and / or reduce calculation.

6. A computing acceleration system, characterized in that: The system includes a computing node for: Calculate the target data through the preprocessing plug-in on the computing node to obtain intermediate calculation results, and store the intermediate calculation results in the database; The intermediate calculation results in the database are called through the interface of the online analytical processing (OLAP) vectorization engine on the computing node to perform operations and obtain calculation results, wherein the OLAP vectorization engine can process data in the storage organization format of the database.

7. The system according to claim 6, characterized in that The system also includes an execution plan generator, which is used to: Generate the preprocessing plug-in based on the computing task, and generate a calling method of the OLAP vectorization engine; Distribute the calling methods of the preprocessing plug-in and the OLAP vectorization engine to the computing nodes.

8. The system according to claim 6 or 7, characterized in that: The system further comprises a distributor, wherein the execution plan generator is further used to send the distribution strategy to the distributor; The distributor is used to distribute the target data to the computing nodes based on the distribution strategy.

9. The system according to claim 8, characterized in that Before the execution plan generator sends the distribution strategy to the distributor, the execution plan generator is further used to: The distribution strategy is generated or adjusted based on relevant factors, where the relevant factors include at least one of the number of nodes and computing tasks.

10. The system according to claim 6, characterized in that The calculation includes map and / or reduce calculation.

11. An electronic device, characterized in that: The device comprises: A calculation module is used to calculate the target data through the preprocessing plug-in, obtain intermediate calculation results, and store the intermediate calculation results in a database; It is also used to call the intermediate calculation results in the database through the interface of the OLAP vectorization engine to perform operations to obtain calculation results, wherein the OLAP vectorization engine can process data in a storage organization format in the database.

12. The device according to claim 11, characterized in that The device also includes an execution plan generating module, which is used to: Generate the preprocessing plug-in based on the computing task, and generate a calling method of the OLAP vectorization engine; Distribute the calling methods of the preprocessing plug-in and the OLAP vectorization engine to the computing nodes.

13. The device according to claim 11 or 12, characterized in that The device further comprises a distribution module, and the execution plan generation module is further used to: send the distribution strategy to the distribution module; The distribution module is used to distribute the target data to the calculation module based on the distribution strategy.

14. The device according to claim 13, characterized in that The execution plan generation module is also used to: generate or adjust the distribution strategy based on relevant factors, and the relevant factors include at least one of the number of nodes and computing tasks.

15. The device according to claim 11, characterized in that The calculation includes map and / or reduce calculation.

16. An electronic device, characterized in that: The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory is used to store computer instructions, and the processor executes the computer instructions so that the network device executes the method according to any one of claims 1 to 5.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions instruct the communication device to execute the method according to any one of claims 1 to 5.

18. A circuit, characterized in that: include: A processor and an interface, configured to execute a computer program or instruction stored in a memory, and to execute a method according to any one of claims 1 to 5.

Images