Data vectorization acceleration method and system

Abstract

The embodiment of the invention provides a data vectorization acceleration method and system, and the method comprises the steps: writing an optimizer rule into an optimizer, and carrying out the vectorization acceleration according to the optimizer rule, respectively fusing respective original operators in the first operator rule and the second operator rule with respective matched replacement operators of the first operator rule and the second operator rule to obtain a vectorized fusion operator; and converting the vectorization fusion operator into a vectorization execution operator according to the conversion logic, so that the vectorization execution operator calls a vectorization execution engine through a local interface JNI bridge to accelerate data vectorization. Therefore, through the embodiment of the invention, the problem that the calculation efficiency is relatively low in a large-scale data processing scene due to the fact that an existing Flink framework cannot fully utilize an instruction set and vectorization hardware resources in a batch processing mode can be solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and specifically, to a data vectorization acceleration method and system. Background Art

[0002] With the rapid development of IoT technology and the rapid growth of data volumes, big data processing technology has become crucial for storing and analyzing massive amounts of data. In this context, vectorization technology leverages the SIMD (Single Instruction Multiple Data) instruction set to enable parallel data processing. However, when faced with large-scale batch processing tasks, traditional batch vectorization technologies have largely focused on the Spark framework. This is particularly true for applications requiring high throughput and low latency. These technologies rely heavily on the Spark framework during batch processing, lacking decoupling capabilities from other computing platforms.

[0003] At the same time, distributed computing frameworks (such as Apache Flink), as a powerful distributed data processing engine, are mainly used for real-time batch processing tasks due to their excellent stream processing and real-time analysis capabilities. However, Flink's existing support for vectorization technology in batch processing mode is not perfect, and it cannot fully utilize vectorization hardware resources, which limits Flink's application in high-performance computing requirements.

[0004] In summary, no effective solution has been proposed in the related art. Summary of the Invention

[0005] The embodiments of the present application provide a data vectorization acceleration method and system to at least solve the problem that the existing Flink framework fails to fully utilize the instruction set and vectorization hardware resources in batch mode, resulting in low computing efficiency in large-scale data processing scenarios.

[0006] According to one embodiment of the present application, a data vectorization acceleration method is provided, including: writing optimizer rules into an optimizer, wherein the optimizer rules include a first operator rule and a second operator rule; according to the optimizer rules, fusing the original operators in the first operator rule and the second operator rule with the replacement operators matched by the first operator rule and the second operator rule respectively to obtain a vectorized fusion operator; converting the vectorized fusion operator into a vectorized execution operator according to the conversion logic, so that the vectorized execution operator calls the vectorized execution engine through the local interface JNI bridge to perform data vectorization acceleration.

[0007] According to another embodiment of the present application, a data vectorization acceleration system is provided, including: an optimizer module, used to fuse the original operators in the first operator rule and the second operator rule with the replacement operators matched in the first operator rule and the second operator rule according to the optimizer rules, to obtain a vectorized fusion operator, wherein the optimizer rules include the first operator rule and the second operator rule; a vectorized execution operator, used to call the vectorized execution engine through the JNI bridge to perform data vectorization acceleration; a vectorized execution engine, used to receive the call of the JNI bridge and perform data vectorization acceleration; a vectorized converter, used to convert the physical plan applied for the current data processing into a local execution plan.

[0008] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0009] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in the above method embodiment.

[0010] According to another embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps in the above method embodiment when executed by a processor.

[0011] Through the above-mentioned embodiment of the present application, a data vectorization acceleration method is provided, in which the optimizer rules are written into the optimizer, wherein the optimizer rules include a first operator rule and a second operator rule; according to the optimizer rules, the original operators in the first operator rule and the second operator rule are respectively fused with the replacement operators matched by the first operator rule and the second operator rule to obtain a vectorized fusion operator; the vectorized fusion operator is converted into a vectorized execution operator according to the conversion logic, so that the vectorized execution operator calls the vectorized execution engine through the local interface JNI bridge to perform data vectorization acceleration. That is, by introducing core technologies such as optimizer rules, vectorized execution operators, and vectorized execution engines, the data batch processing performance based on Flink is optimized, and vectorized hardware resources are fully utilized to meet the efficient requirements of large-scale data processing. Therefore, the embodiment of the present application can solve the problem that the existing Flink framework fails to fully utilize the instruction set and vectorized hardware resources in batch mode, resulting in low computing efficiency in large-scale data processing scenarios, thereby achieving the effect of improving Flink computing efficiency and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 1 is a hardware structure block diagram of a computer terminal according to the data vectorization acceleration method of an embodiment of the present application;

[0013] Figure 2 1 is a system structure block diagram of a method for accelerating operation data vectorization according to an embodiment of the present application;

[0014] Figure 3 is a flowchart of a data vectorization acceleration method according to an embodiment of the present application;

[0015] Figure 4 1 is a flow chart of a data vectorization acceleration method according to an embodiment of the present application;

[0016] Figure 5 Schematic diagram of operator fusion of the data vectorization acceleration method according to an embodiment of the present application;

[0017] Figure 6 It is a structural block diagram of a data vectorization acceleration system according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] It should be noted that the terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0020] The embodiments of this application include but are not limited to supporting the following operating systems: CentOS 7, Windows 10&&11, and MacOS M1.

[0021] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal according to the data vectorization acceleration method of an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data. The computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0022] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data vectorization acceleration method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0023] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a communications provider of a computer terminal. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0024] Currently, traditional batch processing vectorization acceleration technologies are mostly focused on the Spark framework, such as MT Spark. These technologies leverage the CPU's (Central Processing Unit) SIMD (Single Instruction Multiple Data) instruction set to enable large-scale data parallel processing. However, batch processing relies heavily on the Spark framework and lacks decoupling capabilities from other computing platforms. Furthermore, for Apache Flink, vectorization acceleration technology in batch mode is still in the exploratory and early development stages, failing to fully utilize the CPU's instruction set and vectorization hardware resources, limiting Flink's potential for application in high-performance computing scenarios.

[0025] Existing Flink data processing solutions often focus on the real-time performance of streaming computing. However, in batch processing tasks, how to improve computing performance and resource utilization has become a major problem that needs to be solved urgently.

[0026] In view of the above problems, the embodiment of the present application proposes a data vectorization acceleration system based on the Apache Flink framework. The newly added optimizer rule injection technology enables Flink's optimizer to automatically identify and generate vectorized operators, ensuring the effective application of data vectorization acceleration in the execution plan; by integrating vectorized execution operators, the CPU's SIMD instruction set is fully utilized to achieve parallel processing and execution of batch data, thereby greatly improving computing efficiency and processing throughput, and improving the computing performance of the execution engine in Flink's batch mode; at the same time, the custom vectorized converter allows dynamic generation of conversion operators according to the specific needs in the batch mode, thereby improving the flexibility of the data vectorization acceleration system; the vectorized execution engine, as the core component of the data vectorization acceleration system, is independent of Flink's memory manager, and can improve the performance of vectorized computing by optimizing memory usage, ensuring the efficient operation of the system in large-scale data processing scenarios.

[0027] Each of the aforementioned components is deployed as a plug-in within the data vectorization acceleration system, integrating vectorization acceleration functionality into the Flink framework. Users can flexibly enable or disable vectorization acceleration based on actual needs, simplifying system configuration and maintenance. Furthermore, to ensure system stability and compatibility, a fallback mechanism has been designed. If an unsupported or underperforming operator is encountered during vectorization acceleration, the system automatically falls back to executing native Flink operators, ensuring the reliability of batch processing tasks.

[0028] Through the above embodiments, the data vectorization acceleration system based on the Apache Flink framework not only significantly improves Flink's computing performance and resource utilization, but also enhances its adaptability to complex data processing tasks, meeting the high-performance requirements of large-scale data processing and real-time analysis.

[0029] Figure 2 : is a system structure diagram of the method for accelerating the operation data vectorization according to an embodiment of the present application, such as Figure 2 As shown, the system of the data vectorization acceleration method specifically includes the following parts:

[0030] (1)Glue layer

[0031] The Glue layer serves as a bridge between Apache Flink and the Flaze engine. It connects Flink's Native Plan Generator and Native Runtime Framework with the Flaze engine to ensure that Flink's execution plans can be efficiently converted into a format that the Flaze engine can execute.

[0032] (2)Apache Flink

[0033] The Flink Java Runtime is a component of the Flink framework that runs in a Java environment. It is responsible for executing data stream or batch processing applications in the Java environment, including functions such as operator scheduling, task execution, state management, and network communication.

[0034] Local Single JVM indicates running Flink in local mode on a Java virtual machine. Developers can directly start a Flink cluster in the JVM on the local machine and verify data processing tasks without complex cluster configuration.

[0035] Cluster Deployment Modes refers to the deployment modes of Flink in a cluster environment, including: Standalone, Mesos, Kubernetes (K8s), Docker, and YARN;

[0036] Cloud platforms, including AWS (Amazon Web Services) and Google Cloud, provide a wealth of computing resources and services. Flink deployed on cloud platforms can leverage distributed computing capabilities to achieve large-scale data processing.

[0037] Hadoop integration is Flink's integration with the Hadoop ecosystem. Flink can directly read and write data in HDFS (Hadoop Distributed File System) and use the input and output formats of Hadoop MapReduce.

[0038] (3) Flaze Native plugin

[0039] The Flaze Native plug-in includes an optimizer module, a vectorized computation module, a data interface module, a configuration management module, and Blaze (Blaze Java Runtime, Arrow-DataFusion). The optimizer module includes optimizer rule injection. To improve Flink's performance in batch processing tasks, the data vectorization acceleration system has added an optimizer rule injection mechanism. By dynamically injecting the optimizer rules BatchPhysicalGlueSinkRule (batch glue layer physical plan sink operator rule) and BatchPhysicalGlueExchangeRule (batch glue layer physical plan exchange operator rule) into Flink's optimizer, it automatically selects the optimal execution rule when compiling the query plan based on data characteristics and the specific execution environment. Optimizer rules automatically identify and optimize the parts of the computation that require vectorization based on the needs of vectorized acceleration, significantly improving data processing efficiency. Working in conjunction with the vectorized execution operators, the optimizer not only reduces computational latency but also reduces resource consumption through parallelization and memory reuse, thereby improving overall system performance. During actual deployment, these optimizer rules can be flexibly adjusted and expanded to adapt to different workloads and data processing requirements, thus ensuring the scalability and flexibility of the data vectorization acceleration system.

[0040] The vectorized computing module includes the following parts during execution: vectorized execution operator, vectorized converter, and vectorized execution engine;

[0041] The vectorized execution operator is one of the core components in the Flaze Engine. It improves data processing efficiency by converting Flink's physical plan into an execution plan and combining it with vectorization technology. The core function of the vectorized execution operator is to realize the conversion of the Glue execution plan in three stages: BatchPhysicalPlan (batch mode physical plan) -BatchExecutionPlan (batch mode execution plan) -Transformation (data conversion operation). When processing batch data, it is not only responsible for the efficient distribution and conversion of data, but also uses vectorized acceleration technology to reduce computing time. By combining with the vectorized execution engine, it can quickly convert input data into a format suitable for calculation, ensuring that subsequent processing can be executed efficiently. When facing large-scale data, the data vectorization acceleration system can still maintain a high processing power and stability, especially in batch processing scenarios, effectively improving data throughput and reducing computing latency, while also providing a scalable solution for complex batch processing tasks;

[0042] The vectorizer plays a crucial role in accelerating Flink's batch processing. As a middleware component, the vectorizer converts traditional Flink-supported PhysicalPlans into Protobuf-supported NativePlans, suitable for vectorized execution. During the plan conversion process, the vectorizer identifies areas suitable for vectorization optimization and converts them into vectorized operations. This allows Flink to effectively leverage vectorized hardware acceleration while also improving data processing throughput. The vectorizer employs an optimization strategy based on optimizer rules, adapting plans at different stages of a query to accommodate actual data patterns and operational characteristics.

[0043] The vectorized execution engine, a core component of the data vectorization acceleration system, executes vectorized execution plans processed by the optimizer and vectorized converter. The vectorized execution engine efficiently schedules and executes vectorized operations, ensuring continuous high-performance computing support in various large-scale data scenarios. By rewriting some operators using the Datafusion and Arrow open-source vectorized execution libraries, the vectorized execution engine effectively improves task throughput and execution speed. The vectorized execution engine also supports flexible execution modes, enabling the selection of appropriate parallelization strategies based on different task requirements while ensuring data consistency and reliability during the computation process. The vectorized execution engine not only enhances Flink's processing capabilities but also provides a new technical path for batch data computation, ensuring efficient utilization of computing resources when processing large-scale data sets.

[0044] The data interface module handles data reading and writing, ensuring that data can be efficiently converted between Flink's memory model and Flaze Engine's vectorized model.

[0045] The configuration management module allows users to control the batch vectorization acceleration function through a simple configuration file, including enabling or disabling it and adjusting related parameters to ensure the operability and flexibility of the system.

[0046] Blaze includes Blaze Java Runtime and Arrow-DataFusion. Blaze Java Runtime is responsible for the conversion between data and physical plans and resource management, while Arrow-DataFusion provides vectorized data structures and an efficient operator execution environment. The two work together to implement vectorized acceleration in Flink's batch mode, significantly improving data processing performance.

[0047] In this embodiment, the vectorized computing module, optimizer module, data interface module, configuration management module, and Blaze are deployed in a plug-in manner. Flexible configuration and optimizer rules allow users to flexibly enable or disable vectorized acceleration functions according to actual needs, allowing users to dynamically control the Glue layer functions in batch mode through simple configuration switches, thereby optimizing resource usage and computing performance. In OptimizerConfigOptions (optimizer configuration options), a configuration item table.optimizer.batch-glue.enabled is added. This configuration item determines whether Flaze Glue (Flaze vectorized execution engine) is enabled. During the execution of Flink, the optimizer determines whether to trigger the optimization of the Glue layer based on the value of this configuration item. Specifically, when the user sets the value of the configuration item in flink.env to true, the optimizer will enable the BatchPhysicalGlueExchangeRule (batch mode glue layer physical plan exchange operator rule) and BatchPhysicalGlueSinkRule (batch mode glue layer physical plan sink operator rule) optimizer rules, and use these optimizer rules to further process and optimize the compute nodes in the data flow; when the user sets the value of the configuration item in flink.env to false, the optimizer will disable the BatchPhysicalGlueExchangeRule (batch mode glue layer physical plan exchange operator rule) and BatchPhysicalGlueSinkRule (batch mode glue layer physical plan sink operator rule) optimizer rules. This way, users can decide whether to enable this feature by simply setting the parameters in the configuration file without making complex code modifications. This deployment method greatly simplifies the system maintenance and upgrade process, improves the system maintainability, and enhances the user experience.

[0048] The embodiment of the present application also proposes an adaptive memory management mechanism and a fallback mechanism, which are as follows:

[0049] The adaptive memory management mechanism automatically adjusts the memory usage of Flink batch tasks to cope with varying data volumes and computational complexities. Traditional data processing frameworks often use fixed strategies for memory allocation, which can lead to insufficient memory or excessive memory waste when faced with large-scale data or complex tasks. The adaptive memory management mechanism dynamically monitors the memory requirements of tasks and flexibly adjusts the memory allocation strategy based on the real-time computational load and data scale. Specifically, the system dynamically adjusts the memory allocation of each compute node based on the execution progress and memory usage of the batch tasks, and automatically optimizes memory usage efficiency based on changes in load. This adaptive memory management mechanism can significantly improve task execution efficiency and ensure that the system can maintain stable operation even when processing large amounts of data.

[0050] In Flink's batch vectorized acceleration system, the fallback mechanism determines whether to fall back to the traditional execution path during the optimization phase, ensuring that the data vectorized acceleration system can run safely and efficiently when specific conditions are not met. This mechanism is primarily reflected in the rule optimization process. As part of the optimizer's rules, the optimizer can determine whether to abort vectorized acceleration execution based on predefined fallback conditions and revert to the traditional operator execution path, ensuring stable system operation.

[0051] Figure 3 is a flow chart of a data vectorization acceleration method according to an embodiment of the present application. Figure 3 As shown, the data vectorization acceleration process includes the following steps:

[0052] Step S302: writing optimizer rules into the optimizer, wherein the optimizer rules include a first operator rule and a second operator rule;

[0053] In this embodiment, the optimizer rules include a first operator rule and a second operator rule, wherein the first operator rule is BatchPhysicalGlueSinkRule (batch mode glue layer physical plan sink operator rule), and the second operator rule is BatchPhysicalGlueExchangeRule (batch mode glue layer physical plan exchange operator rule), which are responsible for optimizing Exchange and Sink type operators in the query execution plan.

[0054] In this embodiment, regarding Flink's batch data vectorization acceleration method, optimizer rules are injected into the optimization phase of the Flink-SQL execution process to achieve subsequent vectorized execution.

[0055] Specifically, Figure 4FIG. 1 is a flow chart of a data vectorization acceleration method according to an embodiment of the present application. Figure 4 As shown in the figure, the core of injecting optimizer rules is to modify Flink's physical optimizer. This requires defining a new optimization rule class that inherits from Flink's PhysicalRel class and implements the rule logic. An optimizer rule class may be located in the org.apache.flink.table.planner.plan.rules.physical package. Configuration items are created in the optimizer rule class to determine whether the physical plan applied to the current data processing runs the optimizer rules in the optimizer.

[0056] In some embodiments, a configuration item is created in the optimizer rule, and the configuration item is used to determine whether the physical plan applied to the current data processing runs the optimizer rule in the optimizer.

[0057] In this embodiment, for any user interface, the user only needs to define a simple configuration item to control the enabling or disabling of the batch vectorization acceleration function. By setting table.optimizer.batch-glue.enabled (customizing whether to enable vectorization configuration) to true in the configuration file, Flaze Glue's batch vectorization acceleration can be enabled without modifying any code. This configuration item is located in OptimizerConfigOptions (optional vectorization configuration) and determines whether Flink enables the optimizer rules (BatchPhysicalGlueExchangeRule (batch mode glue layer physical plan exchange operator rule) and BatchPhysicalGlueSinkRule (batch mode glue layer physical plan sink operator rule) during execution. Therefore, by creating configuration items in the optimizer rules, the operability and flexibility of the system are greatly improved.

[0058] In some embodiments, before writing the optimizer rules into the optimizer, the method further includes: configuring an adaptive memory management mechanism, and monitoring the memory usage of multiple tasks according to the adaptive memory management mechanism to adjust the memory allocation strategy.

[0059] In this embodiment, the memory usage of multiple tasks is monitored according to the adaptive memory management mechanism to adjust the memory allocation strategy, including: when the memory usage of the current task is greater than or equal to a second preset threshold, triggering a task overflow operation to overflow the data of the current task to the disk; or, when the memory usage of the current task is greater than or equal to the second preset threshold, triggering a task wait operation to store the data of the current task in the memory space released by other tasks.

[0060] In this embodiment, if Figure 4 As shown in the figure, the adaptive memory management mechanism is responsible for memory management during the execution of the vectorized execution engine. Typically, data batch processing tasks face two major issues: memory overflow and uneven allocation of memory resources. Specifically, when executing vectorized tasks, some operations require a large amount of memory, while others may only require less memory resources. To avoid wasting system memory resources and ensure efficient task execution, the adaptive memory management mechanism is configured to dynamically adjust the memory allocation strategy based on task requirements and memory usage. This adaptive memory management mechanism not only addresses the differences in memory management between Flink and native execution modes, but also automatically optimizes the memory allocation strategy based on the vectorization coverage of the task.

[0061] The core of the adaptive memory management mechanism lies in its multi-level memory management system. The MemManager class (memory manager) manages total memory allocation and consumers (i.e., tasks or modules using memory). The MemManager class maintains a memory management status (MemManagerStatus) and multiple memory consumer information (MemConsumerInfo). This information allows the system to monitor memory usage in real time and dynamically adjust memory allocation to ensure stable task execution.

[0062] In this embodiment, the MemManager class provides a global instance. The init method initializes the system's total memory (e.g., 16GB), and the get method retrieves the current memory management instance. Each memory consumer registers with the MemManager during initialization. The MemConsumerInfo class manages the consumer's memory usage. The MemManager automatically adjusts memory resource allocation based on each consumer's memory usage, performs memory overflow writes, or waits for resource release.

[0063] The essence of the adaptive memory management mechanism is adaptive memory allocation. The system automatically adjusts the allocation of memory resources based on the memory usage of each memory consumer. Specifically, the update_consumer_mem_used_with_custom_updater method (the method for updating current memory usage) can calculate whether it is necessary to overflow data or wait for resources each time the memory usage is updated. If the system is short of memory and the memory usage of a task exceeds the threshold (the second preset threshold), the system will trigger an overflow operation to write the data to disk, thereby freeing up memory space. If the overflow operation cannot be completed, the task will be required to wait for other tasks to release resources.

[0064] During memory usage, MemManager can dynamically adjust the memory usage of each consumer. When a consumer's memory usage exceeds the system threshold (the second preset threshold), MemManager can decide whether to overflow the task's data to disk or let the task wait for other tasks to release memory resources. Among them, the system implements a waiting mechanism through the Condvar object, and once enough memory is available, the task can continue to execute. This adaptive memory management mechanism ensures that the system can smoothly process tasks under high memory pressure, avoiding system crashes or task failures due to insufficient memory.

[0065] Step S304: According to the optimizer rule, the original operators in the first operator rule and the second operator rule are respectively fused with the replacement operators matched by the first operator rule and the second operator rule to obtain a vectorized fused operator;

[0066] In some embodiments, according to the optimizer rule, the original operators in the first operator rule and the second operator rule are respectively fused with the replacement operators matched by the first operator rule and the second operator rule to obtain a vectorized fusion operator, including: according to the optimizer rule, obtaining the batch type nodes in the physical plan applied to the current data processing; when the number of the batch type nodes is greater than a first preset threshold and the operator on the batch type node is a replacement operator, the original operators in the first operator rule and the second operator rule are respectively fused with the replacement operators matched by the first operator rule and the second operator rule to obtain a vectorized fusion operator.

[0067] In this embodiment, if Figure 4 As shown in the figure, the sink operator and exchange operator are fused separately. The original operator in the first operator rule is the sink operator, and the original operator in the second operator rule is the exchange operator. Starting from the Sink / Exchange operator, when each operator matches a replacement operator (for example, Sort, Join, Agg, etc.), they are fused to obtain a vectorized fusion operator.

[0068] In this embodiment, the optimizer rule recursively obtains all Batch type nodes in the physical plan applied to the current data processing, determines whether the number of Batch type nodes is greater than a first preset threshold (for example: 1), and whether each Batch node can be fused into a type supported by the vectorized fusion operator. In this embodiment, the replacement operator types supported for fusion include but are not limited to Sink, Sort, Exchange, Join, Agg, SourceScan, Calc and other types of operators.

[0069] When the matches method finds a replacement operator, the onMatch method is called to perform the fusion operation. The onMatch method first obtains the current BatchPhysicalExchange node (batch mode physical plan exchange node) and extracts necessary information from it, such as RelOptCluster (relationship optimization context), RelTraitSet (relationship feature set), and RelDistribution (data distribution characteristics). This information is a key parameter for executing the task and determines how to execute the task efficiently in a distributed system.

[0070] Specifically, Figure 5 : is a schematic diagram of operator fusion of the data vectorization acceleration method according to an embodiment of the present application, such as Figure 5 As shown, based on the current matching node, the onMatch method performs different fusion operations according to different situations. Starting from the current Sink / Exchange operator, traverse the operator tree backward and stop the replacement when encountering a specific node. The entire replacement logic is as follows Figure 5 As shown, this is divided into six cases. For example, in a simple scenario, when the current Sink operator matches a BatchPhysicalTableSourceScan node (a source node in Flink's batch mode physical plan), it can be replaced with the BatchPhysicalGlueSinkSource operator (a custom operator that connects sinks and sources, also known as a vectorized fusion operator). In this case, the new Glue operator will directly handle data source read and output operations through the native vectorization engine, thereby improving data reading efficiency. Furthermore, in some complex scenarios, a node may have multiple inputs. For example, operations such as BatchPhysicalHashJoin (a HashJoin node in Flink's batch mode physical plan) may need to process multiple input data streams. To address this situation, the onMatch method applies the BatchPhysicalGlueSinkTwoInput operator (a custom operator that connects sinks and joins) to the current execution plan. This operator can handle complex data flow operations involving two inputs (such as joins) to ensure efficient data exchange and processing.

[0071] The above implementation allows vectorized operator replacement of operators in the operator tree based on the largest replaceable segment. This is implemented in the Optimize phase of the FlinkServer layer, and is then handed over to the underlying Glue operators for physical plan generation and transformation (data conversion operations). Furthermore, the Flink Shuffle module can be reused during task execution, ensuring data consistency and timeliness in a distributed environment.

[0072] Step S306: Convert the vectorized fusion operator into a vectorized execution operator according to the conversion logic, so that the vectorized execution operator calls the vectorized execution engine through the local interface JNI bridge to perform data vectorization acceleration.

[0073] In some embodiments, the vectorized fusion operator is converted into a vectorized execution operator according to the conversion logic, including: creating a vectorized bonding layer operator corresponding to the vectorized fusion operator, wherein the bonding layer operator includes a bonding layer reading operator, a bonding layer single-input operator, and a bonding layer multi-input operator; converting the vectorized fusion operator according to the physical plan, execution plan, and conversion order of data conversion operations, and calling the vectorized bonding layer operator to obtain a vectorized execution operator.

[0074] In this embodiment, if Figure 4 As shown, after the optimizer rules are injected and the optimizer rules in the optimizer are able to run and the vectorized fusion operator is generated, the vectorized execution operator is generated. Among them, it can be implemented in LogicalOperator, Physical Operator, Exec Operator, and Transformation. The vectorized execution operator component is responsible for generating the six BatchPhysicalGlue operators in the corresponding optimizer rules, and implements the conversion logic from PhysicalPlan (physical plan)-ExecutionPlan (execution plan)-Transformation (data conversion operation) according to different types. It also creates corresponding vectorized operators supported by Flink-Table-Runtime for processing. According to different types, they are divided into GlueSourceOperator (glue layer read operator), GlueOneInputOperator (glue layer single input operator) and GlueTwoInputOperator (glue layer multiple input operator). These three types of operators are mainly responsible for calling the conversion from PhysicalPlanNode (physical plan node) supported by Flink to PhysicalPlanNode (physical plan node) supported by Protobuf, as well as the execution of the vectorized execution engine.

[0075] Specifically, taking BatchPhysicalGlueExchangeOneInput (batch glue layer ExchangeOneInput physical plan) as an example, BatchPhysicalGlueExchangeOneInput (batch glue layer ExchangeOneInput physical plan) is used to manage the distribution and processing of data streams. This class inherits from CommonPhysicalExchange and implements the BatchPhysicalRel interface. In this class, the constructor accepts multiple parameters, including RelOptCluster (relationship optimization context), RelTraitSet (relationship characteristic set), RelNode, etc., which are used to set the environment and properties of the operator. In the translateToExecNode method, the physical operator can be converted into an execution node. The converted BatchExecGlueExchangeOneInput (batch glue layer ExchangeOneInput execution plan) instance will contain all the configuration required for execution, such as input properties, output types, and specific execution descriptions.

[0076] The BatchExecGlueExchangeOneInput (batch mode glue layer ExchangeOneInput execution plan) class is an operator in the execution layer, inherited from CommonExecExchange and implements BatchExecNode <rowdata>The interface indicates that this class is responsible for executing batch data. The constructor of this class accepts multiple parameters, including ReadableConfig (readable configuration interface), InputProperty (input property descriptor), RowType (row type descriptor), etc., which set the necessary configuration of the execution node. The execution node is responsible for processing the data flow in the task, especially the calculation logic involving operations such as partitioning, exchange, and parallelism.

[0077] During the execution of BatchExecGlueExchangeOneInput (batch mode glue layer ExchangeOneInput execution plan), the translateToPlanInternal method is responsible for converting the logic into an actual execution plan. The translateToPlanInternal method obtains the output type from the input edge, constructs the appropriate input transformation (inputTransform), and then the system calls GlueOneInputOperator (glue layer single input operator) to create a vectorized execution operator. GlueOneInputOperator is the operator that processes data and is responsible for converting data into physical form and handing it over to downstream operators for processing.

[0078] GlueOneInputOperator (Glue Layer Single Input Operator) is the operator that actually performs data processing in Flink and is inherited from TableStreamOperator <rowdata>And implemented OneInputStreamOperator<RowData,RowData> Interface, which can receive data from a single input stream and process it. The main task of this operator is to convert the data passed from the upstream operator into batch data in Arrow format through the JNI (Java Native Interface) bridge call, and hand it over to the downstream operator for further processing. In the open method, the operator initializes the data collector (collector) and buffer (buffer), as well as the JNI bridge (JNIBridge). The above components ensure that the operator can efficiently process input data during execution and forward it to the downstream after the data is ready. The processElement method processes each input data and stores it in the buffer. The endInput method performs the final processing and output of the data after the input is completed. In endInput, the operator converts the data in the buffer into Arrow format, then calls the JNI bridge for local operations and outputs the results through the collector.

[0079] In some embodiments, after converting the vectorized fusion operator into a vectorized execution operator according to the conversion logic, the method further includes: converting the physical plan applied for the current data processing into a local execution plan through a vectorized converter, wherein the local execution plan is converted into a data type supported by the vectorized execution engine according to the input data type, connection specification, and data field information.

[0080] In this embodiment, if Figure 4 As shown in the figure, after the sink and exchange operators are fused separately, the Vectorized Converter is responsible for native execution plan conversion. This primarily converts specific operations in Flink's physical execution plan (such as Hash Join) into a format suitable for the vectorized execution engine, thereby accelerating batch data processing. Specifically, the Vectorized Converter constructs physical execution nodes suitable for the Protobuf format based on the input relational data type, connection specifications, and field information, and provides the necessary support for the vectorized execution engine.

[0081] When implementing a vectorizer, data type support is required. Flink's RelDataType object describes the structure of the input data, while JoinSpec defines the details of the join operation, including the fields to be joined and the join type. After receiving the input RelDataType and JoinSpec, the vectorizer constructs a PhysicalPlanNode object that conforms to the Protobuf format. This converts Flink's native data types into data types supported by Arrow. Arrow types enable more efficient batch data processing, especially in vectorized computations.

[0082] Specifically, for example, in the constructor of the BatchPhysicalHashJoinConv class, the input data type's field list (inputRowType.getFieldList()) is first converted into a set of Field objects. These Field objects contain information such as the field name, type, and whether null values ​​are allowed. This allows the vectorized converter to map Flink's data structures to the Protobuf Field format for subsequent vectorized computation. During the construction of each Field object, the convertDataTypeToArrowType method is used to convert the field type to a supported Arrow type, providing the necessary field information for the vectorized engine. In hash join operations, constructing the join condition (JoinOn) is crucial. The JoinSpec provides index information for the left and right join keys. The vectorized converter traverses the index information and creates JoinOn objects one by one. These JoinOn objects describe the left and right expressions for each join key. Each JoinOn object contains a PhysicalExprNode, representing an expression in the join condition. The PhysicalExprNode uses a PhysicalColumn to represent the specific column, which is mapped to the correct column information in Protobuf.

[0083] After converting the fields and join conditions, the vectorizer encapsulates the column information into a PhysicalPlanNode. A PhysicalPlanNode is a Protobuf object that contains node information for the entire physical execution plan. In the case of a hash join, the PhysicalPlanNode contains a BroadcastJoinExecNode, which represents a broadcast join operation. Broadcast joins are typically used when one table has a smaller amount of data than the other, accelerating the join by broadcasting the smaller table to all execution nodes.

[0084] After completing the above steps, the vectorized converter returns the constructed PhysicalPlanNode through the convertToBroadcastJoinExecNode method (which converts the BroadcastJoinExecNode into a NativeBroadcastJoinExecNode). This node can be passed to the execution engine to perform the vectorized computation. As an input node to the vectorized engine, the PhysicalPlanNode contains all the necessary execution plan information, ensuring that broadcast join operations can be executed efficiently in the vectorized execution environment.

[0085] In some embodiments, after converting the physical plan applied to the current data processing into a local execution plan through a vectorized converter, the method further includes: calling a vectorized execution engine through a JNI bridge to perform data vectorization acceleration based on the local execution plan and the vectorized execution operator, wherein the vectorized execution engine converts the original language format in the local execution plan into a target language format.

[0086] In this embodiment, after the Native execution plan (local execution plan) and the vectorized execution operator are generated, the vectorized execution engine is responsible for vectorized execution. The vectorized execution engine is based on the Rust language as the local execution code, and uses the two open source libraries DataFusion (a vectorized operator library written in the rust language) and Arrow (a vectorized data structure that supports the rust language) to complete the underlying vectorized operator design. Specifically, the Rust language provides efficient memory management and concurrent execution capabilities, enabling the vectorized execution engine to handle the massive data streams in Flink jobs. In the specific implementation process, the vectorized execution engine interacts with the Java virtual machine through JNI, receives Flink's physical execution plan, and converts it into a format executable by the Rust language. In this way, the vectorized execution engine can efficiently schedule tasks and execute various operations in the physical plan.

[0087] The implementation of the vectorized execution engine relies on the JNI bridging mechanism, which enables Rust code to collaborate efficiently with Flink in the Java environment. Through the callNative method implemented through the JNI interface, Flink can pass the physical execution plan (passed in Protobuf format) to the Rust engine and convert it into the corresponding execution plan object in Rust. Specifically, parse_glue_exec parses the incoming Protobuf data and converts it into a PhysicalPlanNode (physical plan node) structure. Then, through the ExecutionPlan conversion logic, an execution plan suitable for DataFusion is generated. The execution plan is then batch-calculated using the datafusion::physical_plan::collect method, and the calculation results are ultimately returned in Arrow format (the target language format).

[0088] The core logic of execution plan conversion is implemented through the try_into method implemented in Rust. ExecutionPlan is an execution plan structure provided by DataFusion, which abstracts specific physical execution nodes (such as scanning, filtering, projection, sorting, etc.) into a unified interface. The ExecutionPlan in Rust is compatible with Flink's task scheduling and execution process by implementing DataFusion's ExecutionPlan trait. In this part of the code, by matching each type of operation in the PhysicalPlanNode (physical plan node), the Rust code can generate corresponding execution engine operations according to different plan types. For example, when a ParquetScan node is received, the Rust engine will generate a scanner suitable for the Parquet format according to the configuration; and when a Sort node is received, a SortExec execution plan that supports external sorting will be generated.

[0089] To further improve execution efficiency during the implementation of the vectorized execution engine, some vectorized operators were rewritten based on the DataFusion ExecutionPlan trait. For example, in the Sort operation, the vectorized execution engine optimized SortExec, using an external sorting method rather than an in-memory sorting approach. This external sorting approach not only handles large amounts of data with limited memory but also efficiently utilizes disk storage, improving sorting performance.

[0090] Specifically, the SortExec operator in the vectorized execution engine manages the sorting process through ExternalSorter, which is implemented using Rust's Box::pin function. This allows the vectorized execution engine to perform sorting operations in memory and, when memory is insufficient, to overflow some data to disk, thus avoiding the risk of memory overflow. In its implementation, the SortExec operator not only provides data sorting functionality but also integrates performance statistics and resource management capabilities, tracking memory and disk usage during the sorting process through BaselineMetrics (which stores current system memory usage information) and SpillMetrics (which stores current disk usage information).

[0091] The vectorized execution engine also offers similar optimizations for other common operations. For example, the FilterExec operator, with vectorization support, can execute multiple filter conditions simultaneously when processing each batch of data, reducing operation time overhead. The ProjectionExec operator achieves more efficient data conversion by performing batch projection operations on fields.

[0092] Using Rust's asynchronous runtime, tokio::runtime::Runtime, the vectorized execution engine efficiently processes each task in parallel, scheduling and executing it according to its specific needs. When the data flow execution results are returned, they are processed using concat_batches to merge all batches of data into a complete data set. The data is then converted to a record_batch in Arrow format, interacting with the Java Virtual Machine (JVM) via FFI (Foreign Function Interface), and the processed results are transmitted back to Flink. The record_batch is converted to the FFI_ArrowSchema (provided by the Arrow library) and FFI_ArrowArray (provided by the Arrow library) formats so that Flink can read and process it through the JNI interface.

[0093] In some embodiments, a fallback condition is configured in the optimizer rule, and the fallback condition is used to determine whether to terminate the currently running data vectorization acceleration.

[0094] In some embodiments, the currently running data vectorization acceleration is terminated when at least one of the following fallback conditions is met: the replacement operator type is not satisfied; an external data table is connected; very few operators are merged; the first operator is the end point of the merge and the predicted return order of magnitude is greater than a third preset threshold; the first operator is the starting point of the merge and the input order of magnitude is greater than a fourth preset threshold.

[0095] In this embodiment, if Figure 4 As shown, a Fallback mechanism (FallbackMechanism) is configured in the data vectorization acceleration system, that is, by configuring a fallback condition in the optimizer rule, it can be determined whether to terminate the currently running data vectorization acceleration.

[0096] In this embodiment, the Fallback mechanism determines whether to fall back to the traditional execution path, ensuring that the system can run safely and efficiently when specific conditions are not met. The Fallback mechanism is mainly reflected in the optimizer rules as part of the optimizer rules. During the Flink execution process, the optimizer will determine whether it is necessary to terminate the execution of vectorized acceleration based on the configured fallback conditions and restore to the traditional operator execution path. Among them, the fallback conditions usually include specific operator types or data volume conditions to avoid enabling vectorized acceleration in inappropriate scenarios. Specifically, the currently running data vectorized acceleration is terminated when at least one of the following fallback conditions is met:

[0097] 1. Fallback when native operator replacement conditions are not met: If an operator cannot be replaced by a native operator (for example, due to computational logic complexity or data characteristics), the optimizer determines whether to enable the fallback mechanism to avoid attempting inappropriate acceleration solutions.

[0098] 2. Rollback when connecting large tables with external access: When large table joins are involved, especially outer joins, left joins, and right joins, the system may rollback to avoid performance degradation due to the large data size and the resulting intermediate results cannot be processed efficiently.

[0099] 3. Fallback when rare operator merges occur: When the system detects rare operator merges, such as single operator merges or sink-source operator merges, it triggers a fallback. This is because such merges may result in high computational overhead between operators. Fallback can avoid unnecessary computational overhead.

[0100] 4. Fallback when the Exchange operator is the merge endpoint and the predicted return volume is large: When the Exchange operator is the merge endpoint and the optimizer predicts a large volume of returned data, the fallback mechanism is triggered, taking into account the overhead of the shuffle operator during row-column conversion.

[0101] 5. When the Exchange operator is the starting point of a merge and the input is large in magnitude, the shuffle operator is backed off due to the overhead of row-column conversion.

[0102] Through the above-described embodiments, the data vectorization acceleration system improves the computational performance of the Flink batch mode execution engine by introducing vectorized execution operators and optimizer rule injection, addressing the performance bottlenecks of traditional vectorization technologies in large-scale data processing. Furthermore, the plug-in and modular design provide the system with high flexibility and scalability to meet the needs of diverse business scenarios, addressing the high reliance on specific frameworks in traditional solutions. Furthermore, independent memory management and vectorized execution engines optimize resource utilization, fully leveraging the CPU's SIMD instruction set and reducing system resource consumption. A fallback mechanism is also designed to ensure that if unsupported operators are encountered during vectorized execution or performance falls short of expectations, a fallback solution is automatically deployed, ensuring system stability and compatibility. Finally, through the configuration management module, users can flexibly enable or disable the vectorized acceleration feature and dynamically adjust related parameters, improving the system's usability and maintainability. In summary, the embodiments of the present application not only effectively improve the computational performance and resource utilization efficiency of the Flink batch mode execution engine, but also enhance the system's flexibility, stability, and usability. Therefore, the embodiments of the present application can solve the problem that the existing Flink framework fails to fully utilize the instruction set and vectorized hardware resources in batch mode, resulting in low computing efficiency in large-scale data processing scenarios, thereby achieving the effect of improving Flink's computing efficiency and resource utilization.

[0103] The present application also provides a data vectorization acceleration system. Figure 6 is a structural block diagram of a data vectorization acceleration system according to an embodiment of the present application, such as Figure 6 As shown, the data vectorization acceleration system includes: an optimizer module, a vectorized execution operator, a vectorized execution engine and a vectorized converter.

[0104] an optimizer module, configured to fuse the original operators in the first operator rule and the second operator rule with the replacement operators matched in the first operator rule and the second operator rule, respectively, according to an optimizer rule, to obtain a vectorized fused operator, wherein the optimizer rule includes the first operator rule and the second operator rule;

[0105] Vectorized execution operator, used to call the vectorized execution engine through the JNI bridge to perform data vectorization acceleration;

[0106] A vectorized execution engine, configured to receive a call from the JNI bridge and perform data vectorization acceleration;

[0107] A vectorizer that converts the physical plan applied to the current data processing into a local execution plan.

[0108] In some embodiments, the optimizer module, the vectorized execution operator, the vectorized execution engine, and the vectorized converter are deployed in the data vectorization acceleration system in the form of plug-ins.

[0109] Flink in the embodiment of the present application supports a vectorized acceleration plug-in system (vectorized acceleration system). By improving the computing performance of Flink in real-time data batch processing tasks, the vectorized acceleration plug-in system can effectively support efficient data integration and synchronization between the data lake and the data warehouse. In the lake-warehouse fusion architecture, data often needs to be stored and pre-processed on a large scale in the data lake, and then quickly migrated to the data warehouse for high-performance query analysis. The vectorized acceleration plug-in significantly reduces data processing time and improves the real-time and accuracy of data synchronization by optimizing the ETL (Extract, Transform, Load) process. At the same time, in a multi-source heterogeneous data environment, Flink achieves efficient data processing and conversion through vectorized plug-ins, which can ensure that data from different data sources can be quickly integrated and analyzed, thereby improving overall data processing capabilities.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0111] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0112] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above method embodiments when run.

[0113] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0114] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0115] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0116] According to yet another embodiment of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the method described in each embodiment of the present disclosure are implemented.

[0117] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0118] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0119] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included within the scope of protection of the present application.< / rowdata> < / rowdata>

Claims

1. A data vectorization acceleration method, characterized in that: include: Writing optimizer rules into the optimizer, wherein the optimizer rules include a first operator rule and a second operator rule; According to the optimizer rule, respectively fusing the original operators in the first operator rule and the second operator rule with the replacement operators matched by the first operator rule and the second operator rule to obtain a vectorized fused operator; The vectorized fusion operator is converted into a vectorized execution operator according to the conversion logic, so that the vectorized execution operator calls the vectorized execution engine through the local interface JNI bridge to perform data vectorization acceleration.

2. The method according to claim 1, characterized in that in, A configuration item is created in the optimizer rule, and the configuration item is used to determine whether the physical plan applied to the current data processing runs the optimizer rule in the optimizer.

3. The method according to claim 1, characterized in that According to the optimizer rule, respectively fusing the original operators in the first operator rule and the second operator rule with the replacement operators matched by the first operator rule and the second operator rule to obtain a vectorized fusion operator, including: According to the optimizer rule, obtaining a batch type node in the physical plan applied to the current data processing; When the number of the batch type nodes is greater than a first preset threshold and the operator on the batch type node is a replacement operator, the original operators in the first operator rule and the second operator rule are fused with the replacement operators matched by the first operator rule and the second operator rule respectively to obtain a vectorized fusion operator.

4. The method according to claim 1, wherein Converting the vectorized fusion operator into a vectorized execution operator according to the conversion logic includes: Creating a vectorized glue layer operator corresponding to the vectorized fusion operator, wherein the glue layer operator includes a glue layer read operator, a glue layer single input operator, and a glue layer multi-input operator; The vectorized fusion operator is converted according to the physical plan, the execution plan and the conversion order of the data conversion operation, and the vectorized glue layer operator is called to perform the conversion to obtain a vectorized execution operator.

5. The method according to claim 1, wherein After converting the vectorized fusion operator into a vectorized execution operator according to the conversion logic, the method further includes: The physical plan applied in the current data processing is converted into a local execution plan through a vectorized converter. The local execution plan is converted into a data type supported by the vectorized execution engine according to the input data type, connection specification and data field information.

6. The method according to claim 5, characterized in that After converting the physical plan applied to the current data processing into a local execution plan by the vectorized converter, the method further includes: According to the local execution plan and the vectorized execution operator, a vectorized execution engine is called through a JNI bridge to perform data vectorization acceleration, wherein the vectorized execution engine converts the original language format in the local execution plan into a target language format.

7. The method according to claim 1, characterized in that Before writing the optimizer rules into the optimizer, the method further includes: An adaptive memory management mechanism is configured, and memory usage of multiple tasks is monitored according to the adaptive memory management mechanism to adjust a memory allocation strategy.

8. The method according to claim 7, characterized in that Monitoring memory usage of multiple tasks according to the adaptive memory management mechanism to adjust memory allocation strategies includes: When the memory usage of the current task is greater than or equal to the second preset threshold, triggering a task overflow operation to overflow the data of the current task to the disk; or When the memory usage of the current task is greater than or equal to the second preset threshold, a task waiting operation is triggered to store the data of the current task in the memory space released by other tasks.

9. The method according to claim 1, characterized in that in, A fallback condition is configured in the optimizer rule, and the fallback condition is used to determine whether to terminate the currently running data vectorization acceleration.

10. The method according to claim 9, characterized in that The currently running data vectorization acceleration is terminated when at least one of the following fallback conditions is met: The replacement operator type is not satisfied; Connect to external data table; Very few operators are merged; The first operator is the merge endpoint and the predicted return magnitude is greater than a third preset threshold; The first operator is a merging starting point and the input magnitude is greater than a fourth preset threshold.

11. A data vectorization acceleration system, characterized in that: include: an optimizer module, configured to fuse the original operators in the first operator rule and the second operator rule with the replacement operators matched in the first operator rule and the second operator rule, respectively, according to an optimizer rule, to obtain a vectorized fused operator, wherein the optimizer rule includes the first operator rule and the second operator rule; Vectorized execution operator, used to call the vectorized execution engine through the JNI bridge to perform data vectorization acceleration; A vectorized execution engine, configured to receive a call from the JNI bridge and perform data vectorization acceleration; A vectorizer that converts the physical plan applied to the current data processing into a local execution plan.

12. The system according to claim 11, wherein: The optimizer module, the vectorized execution operator, the vectorized execution engine, and the vectorized converter are deployed in the data vectorized acceleration system in the form of plug-ins.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 10 are implemented.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images