Method for accelerating data and AI fusion pipeline based on native module injection

Abstract

The invention provides a data and AI fusion pipeline acceleration method based on native module injection, and the method comprises the steps: obtaining a work plan tree before a work plan is executed through PySpark, and dividing the work plan tree into a first type of subtrees and a second type of subtrees; tree root nodes and leaf nodes of the first type of sub-trees are connected to output nodes and input nodes of a Zorhu crossbow module, before the tree root nodes of the first type of sub-trees are executed, all the nodes of the first type of sub-trees are compiled through a Zorhu crossbow compiler to obtain a binary module, and based on the binary module and module functions contained in the binary module, the Zorhu crossbow compiler is used for compiling the nodes of the first type of sub-trees to obtain the Zorhu crossbow module. Leaf nodes of a first type of sub-trees are connected with tree root nodes of a second type of sub-trees, that is, the tree root nodes of the first type of sub-trees can convert a first type of data set corresponding to the first type of sub-trees into a second type of data set having the same characteristics as the second type of sub-trees after execution, that is, the work plan tree can be completely supported by PySpark. And deep integration and seamless compatibility of the two types of data sets are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of big data information processing, and in particular to an acceleration method for integrating data injected by a native module with an AI (Artificial Intelligence) pipeline. Background Art

[0002] With the development of artificial intelligence technology, the research and development of software programs for big data model training has become increasingly mature. Typical artificial intelligence software includes Scikit-learn, PyTorch, TensorFlow, etc., which can support the training and reasoning of artificial intelligence models. Typical big data engine software includes Kafka, Hadoop, Spark, Flink, etc., which solve the collection, storage, management, calculation, and use of big data throughout the life cycle. This results in the fact that artificial intelligence model training and reasoning and big data engines are supported by two separate software stacks. However, artificial intelligence model training and reasoning are inseparable from big data. Artificial intelligence model training requires high-quality sample data, and artificial intelligence reasoning needs to be integrated into the data pipeline. Artificial intelligence can enable intelligent decision-making driven by big data engines and support more accurate decisions. The use of two separate software stacks has caused difficulties in the intercommunication between artificial intelligence and data, resulting in data islands, data inconsistency, high data migration overhead between the two systems, and difficulties in operation and maintenance deployment. In order to achieve a unified artificial intelligence and data software stack, there is currently a plan to publicly disclose PySpark as an open source implementation of a unified artificial intelligence and data software stack. However, PySpark's programming interface and UDF (User Define Function) are implemented in Python, which has poor performance and is difficult to implement high-performance functional modules. If system developers want to use high-performance languages ​​such as C++ to write functional modules, there is a problem of incompatibility with PySpark interfaces and UDFs. Summary of the invention

[0003] In order to solve the problem in the prior art that PySpark's interface and UDF are difficult to be compatible with functional modules implemented in high-performance languages, the present application provides an acceleration method for data and AI fusion pipeline based on native module injection, and implements high-performance language compilation based on Zhuge Crossbow Compiler to make it compatible with PySpark interface, thereby achieving deep integration of the two.

[0004] In the first aspect, the technical solution of the present application provides a method for accelerating the data and AI fusion pipeline based on native module injection, including:

[0005] Get the work plan tree corresponding to the work plan to be executed by PySpark;

[0006] Divide the work plan tree into a first subtree and a second subtree, wherein: the nodes in the first subtree are supported by the Zhuge Nu compiler, and the nodes in the second subtree are not supported by the Zhuge Nu compiler but are supported by PySpark;

[0007] Add a Zhuge Crossbow module output node at the root node of the first type of subtree, and insert a Zhuge Crossbow module input node at the leaf node of the first type of subtree;

[0008] The Zhuge Crossbow module output node obtains the binary module corresponding to the first type of subtree contained therein obtained by compiling the Zhuge Crossbow compiler before it is executed; when the Zhuge Crossbow module output node is executed, it loads the binary module and calls the module function entry contained therein, the input of the module function includes the pointer of the second type of data set generated after the root node of the second type of subtree directly dependent on the first type of subtree is executed; the output of the module function is the pointer of the first type of data set;

[0009] Executing the output node of the Zhuge Crossbow module generates a first type of data set, which converts the first type of data set into a second type of data set when outputting; executing the input node of the Zhuge Crossbow module generates a second type of data set, which converts the second type of data set into the first type of data set when outputting.

[0010] In some schemes, the acceleration method of the data and AI fusion pipeline based on native module injection is described, wherein the work plan tree is divided into a first subtree and a second subtree, including:

[0011] Classify the nodes according to whether each node on the work plan tree is supported by the Zhuge Nu compiler, define the nodes supported by the Zhuge Nu compiler as first-class nodes, and define the nodes not supported by the Zhuge Nu compiler as second-class nodes;

[0012] A subtree formed by a continuous segment of first-category nodes is used as the first-category subtree; a subtree formed by a continuous segment of second-category nodes is used as the second-category subtree.

[0013] In some embodiments, the acceleration method of the data and AI fusion pipeline based on native module injection, wherein the work plan tree is divided into a first subtree and a second subtree, further comprising:

[0014] Colorings are assigned to the first type of nodes and the second type of nodes.

[0015] In some schemes, the acceleration method of the data injected by the native module and the AI ​​fusion pipeline is described, and before the Zhuge Crossbow module output node is executed, the binary module corresponding to the first type of subtree contained therein obtained by compiling the Zhuge Crossbow compiler is obtained, including:

[0016] The Zhuge Crossbow module output node, before being executed, translates the first type of subtree contained in the Zhuge Crossbow module output node into a program module written in a native programming language by means of code generation;

[0017] The Zhuge Crossbow compiler is called to compile the program module into the binary module.

[0018] The acceleration method of the data and AI fusion pipeline based on native module injection described in some solutions also includes:

[0019] The first type of data set and the second type of data set both have partition characteristics, distribution characteristics, and lazy evaluation characteristics, wherein:

[0020] The partition characteristic means that the first type of data set and the second type of data set are both divided into multiple partitions;

[0021] The distributed feature means that the first type of data set and the second type of data set are both stored in a distributed manner, spanning multiple physical machines;

[0022] The lazy evaluation feature means that the first type of data set and the second type of data set both perform evaluation operations corresponding to the needs when they are needed.

[0023] Some of the solutions describe an acceleration method for the data and AI fusion pipeline based on native module injection, wherein the partitioning characteristics include:

[0024] Each partition of the first type of data set is located in the native system and is stored in the Zhuge Crossbow data format;

[0025] Each partition of the second type of data set is located in a Java virtual machine system and is stored in a PySpark data format.

[0026] In some schemes, the acceleration method of the data injected by the native module and the AI ​​fusion pipeline is described, before the Zhuge Crossbow module output node is executed, the binary module corresponding to the first type of subtree contained therein obtained by compiling the Zhuge Crossbow compiler is obtained:

[0027] The binary module compiled by the Zhuge Crossbow Compiler is cached in a binary module cache until the work plan is completed.

[0028] In a second aspect, the technical solution of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the acceleration method of the data injected based on the native module and the AI ​​fusion pipeline are implemented as described in any technical solution of the first aspect.

[0029] In a third aspect, the technical solution of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the acceleration method of the data injected based on the native module and the AI ​​fusion pipeline as described in any technical solution of the first aspect.

[0030] In a fourth aspect, the technical solution of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the acceleration method of the data injected based on the native module and the AI ​​fusion pipeline as described in any technical solution of the first aspect.

[0031] Compared with the prior art, the above technical features provided by this application have at least the following effects:

[0032] The acceleration method of the data and AI fusion pipeline based on native module injection provided by the present application, before using PySpark to execute the work plan, first obtain the work plan tree, each node in the work plan tree may not be supported by the Zhuge Crossbow Compiler but only supported by PySpark, in this case, the nodes that can be supported by the Zhuge Crossbow Compiler are used as the first type of nodes, and the nodes that are only supported by PySpark are used as the second type of nodes, and the work plan tree is divided into the first type of subtree and the second type of subtree. The present application scheme establishes a seamless connection between the first type of subtree and the Zhuge Crossbow Compiler by connecting the root node and the leaf node of the first type of subtree to the output node and the input node of the Zhuge Crossbow module. Before executing the root node of the first type of subtree, each node of the first type of subtree is compiled by Zhuge Crossbow compiler to obtain a binary module. Based on the binary module and the module function contained therein, the leaf node of the first type of subtree can be connected with the root node of the second type of subtree. That is, after execution, the root node of the first type of subtree can convert the first type of data set corresponding to the first type of subtree into the second type of data set with the same characteristics as the second type of subtree, so that both the first type of data set and the second type of data set can be supported by PySpark, that is, the work plan tree corresponding to the work plan can be fully supported by PySpark. When the work plan is executed on PySpark, the deep integration and seamless compatibility of the two types of data sets are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of a method for accelerating a data and AI fusion pipeline based on native module injection according to an embodiment of the present invention;

[0034] Figure 2 A system architecture block diagram of the Zhuge Crossbow Compiler according to an embodiment of the present invention;

[0035] Figure 3 This is an example of compiling a data compatibility interface between the Zhuge Crossbow module and Python according to an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of the hardware connection relationship of an electronic device that executes the acceleration method based on native module injected data and AI fusion pipeline described in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The specific implementation of the present application is further described below with reference to the accompanying drawings.

[0038] It is easy to understand that according to the technical solution of the present application, without changing the essential spirit of the present application, a variety of structural modes and implementation modes that can be replaced by those skilled in the art can be replaced with each other. Therefore, the following specific implementation modes and drawings are only exemplary descriptions of the technical solution of the present application, and should not be regarded as the entirety of the present application or as a limitation or restriction on the technical solution of the application.

[0039] The directional terms such as up, down, left, right, front, back, front, back, top, bottom, etc. mentioned or may be mentioned in this specification are defined relative to the structures shown in the drawings. They are relative concepts and may change accordingly according to different positions and different usage conditions. Therefore, these or other directional terms should not be interpreted as restrictive terms.

[0040] The big data engine Spark implements the RDD (Resilient Distributed Datasets) programming abstraction. By reusing intermediate results in memory to avoid frequent data reading and writing, it can efficiently support iterative computing for massive data. Based on Spark RDD, Spark has developed two field support libraries, MLlib and GraphX, for fields such as machine learning and graph computing. Due to Spark's performance issues, there is still a lot of room for improvement in the performance of these two field support libraries, MLlib and GraphX. To address this problem, Chukonu (Zhuge Crossbow) was proposed in the prior art. It implements an RDD programming interface based on C++, allowing users to write data processing applications using C++API, and automatically applies compilation optimizations such as operator fusion, vectorized execution, and compact data arrangement, showing a large performance optimization space in the fields of data preprocessing, graph computing, and machine learning.

[0041] However, as DataFrame, a more advanced data structure, is widely used by data scientists using Python, the original domain support library that provides PySpark RDD API is gradually replaced by the domain support library that provides PySpark DataFrame API. For example, Spark ML is a module that Spark intends to replace SparkMLLib. GraphFrame is a version of GraphX ​​based on DataFrame API, which is also gradually used by more users. Compared with PySpark RDD API, providing PySpark DataFrame API has significant advantages. First, PySpark DataFrame API is easy to integrate with other components (Spark SQL, Structured Streaming, etc.), integrating iterative computing into complex data science pipelines such as data storage, screening, transformation, aggregation, and feature engineering. At the same time, PySpark DataFrame API can perform query optimization based on SparkCatalyst optimizer, which improves the efficiency and performance of data processing. Under the above trend, Chukonu's method of writing data analysis applications with C++RDD API has limitations and is difficult to integrate with other components based on PySparkDataFrame API. In addition, from the experience of Chukonu engineering practice, the labor cost of developing high-performance data processing pipelines based on Chukonu C++API is much higher, the development cycle is longer, and the personnel requirements are higher. On this basis, in order to solve the above limitations of Chukonu, this application adopts the Zhuge Crossbow Module (Chukonu Module), which inherits Chukonu's full big data functions, C++ acceleration, general compilation optimization and other design concepts, but based on the new modular design logic and re-implemented for PySpark data processing High-performance Native module development framework, allowing system engineers to select public modules with a high degree of reusability in the form of third-party plug-ins, based on the Chukonu RDD programming interface, develop high-performance computing kernels, and provide PySpark programming interfaces to end-user data analysts, embedded in end-to-end data processing pipelines. The Zhuge Crossbow Module solves the high threshold of Chukonu. On the one hand, end users still use the PySpark programming interface, and Chukonu has not changed the user interface. On the other hand, the Zhuge Crossbow module allows middleware providers to tackle performance issues in point-based vertical fields and develop high-performance data processing modules (such as graph computing, machine learning, etc.) for end users to choose from, saving computing resources and speeding up computing.

[0042] This embodiment provides an acceleration method for the data and AI fusion pipeline based on native module injection, which is applied to a data processor. The data processor can be set in a user computer, such as Figure 1 As shown, the method comprises the following steps:

[0043] S100: Obtain a work plan tree corresponding to the work plan to be executed by PySpark.

[0044] A work plan refers to various tasks included in the operations that need to be performed using PySpark, such as query plans, management plans, etc. Each work plan can be split into nodes according to the execution order, and the nodes are connected to each other, thus forming a work plan tree.

[0045] S200: Divide the work plan tree into a first subtree and a second subtree, wherein: the nodes in the first subtree are supported by the Zhuge Nu compiler, and the nodes in the second subtree are not supported by the Zhuge Nu compiler but are supported by PySpark.

[0046] As mentioned earlier, the Zhuge Nu compiler is used to write data analysis applications based on the C++RDD API. It can compile high-performance languages ​​into programming languages ​​supported by PySpark without changing the functionality.

[0047] S300: Adding a Zhuge Crossbow module output node at the root node of the first type of subtree, and inserting a Zhuge Crossbow module input node at the leaf node of the first type of subtree.

[0048] like Figure 2 As shown, it is a system architecture block diagram of the Chukonu module proposed in this embodiment. From the system implementation level, the Chukonu module adopts a modular architecture, including a basic runtime system, a Chukonu Module programming library (C++ API), and a PySpark domain programming interface.

[0049] One of the core components of the basic runtime system is the compute module manager, which is responsible for managing native compute modules. The compute module manager loads the functions defined in the native compute module into the system by calling the dynamic link library (DLL) loading interface provided by the operating system. In order to effectively manage these modules, the compute module manager uses reference counting to manage the life cycle of the native compute module. Reference counting can help the system automatically track the number of times each module is referenced and automatically release the module when the number of references drops to zero. This method simplifies memory management, prevents memory leaks, and ensures the effective use of system resources. The basic runtime system uses an efficient and low-overhead data object serializer, which divides the serialization process into two stages. The first stage is the "metadata serialization stage". In this stage, the serialization process is similar to traditional serialization, but is limited to writing metadata information with a small amount of data to the output stream. This step ensures that the necessary data structure and type information are accurately recorded to support subsequent data reconstruction. The second stage is the "buffer data serialization stage". This stage focuses on writing the buffer objects contained in the data object directly to the buffer output stream. The key advantage of this process is that no memory copy operation is required when transferring Buffer objects, and only pointer transfer is involved. This is due to the automatic management of the life cycle of Buffer objects by the basic runtime system, which ensures efficient memory usage and data transmission efficiency. This two-stage serialization method significantly reduces unnecessary memory copying and data movement, thereby improving the overall data processing performance. In order to support complex computing requirements in a distributed environment, the basic runtime system also needs to handle the dynamic link library loading problem during remote deserialization. The system provides two strategies to manage and synchronize dynamic link libraries. The first method is the "data lake synchronization method". In this method, the Driver node automatically uploads its loaded dynamic link library and all non-system dependencies to the central data lake when it starts. When the Executor node needs these libraries, it can pull the corresponding dependencies from the data lake to the local environment according to demand. The advantage of this method is that it can ensure that all nodes can obtain the latest dependencies on demand without manually managing the library version of each node. It is particularly suitable for environments with complex dynamic updates and dependency management. The second method is the "pre-installation method". In this method, it is assumed that all dynamic link libraries and dependencies have been pre-installed on each machine in the cluster. This method simplifies the need for runtime synchronization, reduces network transmission and dynamic loading delays, and is suitable for scenarios where the environment is relatively stable and the frequency of software updates is low. The pre-installation method is particularly suitable for Kubernetes (K8S) environments because in K8S clusters, containerized applications can pre-install necessary dynamic link libraries and dependencies.When deployed in a K8S cluster, each container is an independent running instance that can share an environment built on the same image. This means that all required libraries and dependencies can be integrated into the Docker image and deployed with the container, ensuring the consistency and isolation of the environment.

[0050] Distributed dataset executor, responsible for executing data flow graphs in a distributed environment. This executor is responsible for managing and scheduling data flow graphs constructed by dataset transformations in a distributed environment. This includes executing these data flow graphs on multiple server nodes while ensuring fault tolerance and load balancing throughout the process. To further enhance the system's functional completeness and scalability, the basic runtime system is deeply integrated with the Apache Spark ecosystem. By mapping data flow graphs to Apache Spark and using Spark's powerful computing engine to execute these graphs, the system can effectively utilize a range of mature features provided by Spark. This includes a fault tolerance mechanism based on data lineage, which can track every step in the data processing process to ensure that it can quickly recover from the most recent correct state in the event of a failure. In addition, Spark's integration with YARN (Yet Another Resource Negotiator) provides powerful support for resource management, allowing the system to allocate computing resources more efficiently. This design allows the Zhuge Crossbow module to share a basic distributed executor with PySpark. This not only simplifies the architectural design of the entire system, avoids the need to repeatedly develop and maintain different executors, but also enhances the compatibility between modules. Through this sharing mechanism, the Zhuge Crossbow module can be seamlessly connected to the existing PySpark engine, making full use of the various optimizations and functional extensions that PySpark provides for large-scale data processing and analysis. This integration not only improves the performance and reliability of the runtime system, but also provides developers with a more powerful and flexible platform for developing and deploying complex data analysis and machine learning applications.

[0051] The PySpark domain programming interface is the basic module and the basic public component. Users use the Zhuge Crossbow module programming library to write computing modules to implement various user-specific computing semantics. From a macro perspective, the Zhuge Crossbow module contains three levels: PySpark domain programming interface, Native computing module, and basic runtime system. The basic runtime system manages the Native computing module and provides the Native computing module with the necessary fault tolerance, caching, scheduling, broadcasting and other functions for distributed computing. The PySpark domain programming interface is oriented to the final data analysis users and provides a programming interface for specific data analysis fields. The Native computing module is called by the PySpark domain programming interface layer to implement the key performance part. The overall process of using the Zhuge Crossbow module in this application is as follows:

[0052] First, the system engineer develops the Native computing module through the Chukonu Module programming library (C++API), and obtains a Native computing module in the form of a dynamic link library through C++ compilation. Then, the system engineer loads the Native computing module through the Chukonu Module Python API and develops the PySpark domain programming interface for PySpark. Finally, users who use PySpark for data analysis call the programming interface developed by the system engineer to perform data analysis tasks.

[0053] Since the Zhuge Nu module is highly compatible with PySpark, this step can easily insert the output node of the Zhuge Nu module and the input node of the Zhuge Nu module into the root node and the leaf node of the first type of subtree correspondingly.

[0054] S400: Before the Zhuge Nu module output node is executed, it obtains the binary module corresponding to the first type of subtree it contains and compiled by the Zhuge Nu compiler; when the Zhuge Nu module output node is executed, it loads the binary module and calls the module function entry contained therein, and the input of the module function includes the pointer of the second type of data set generated after the root node of the second type of subtree directly dependent on the first type of subtree is executed; the output of the module function is the pointer of the first type of data set; its content is equivalent to the result generated after the first type of subtree is executed by PySpark.

[0055] The Zhuge Crossbow Compiler corresponds to the Chukonu Module programming library (C++ API), such as Figure 2 As shown, the architecture of the ChukonuModule module programming library (C++API) divides the functions into four main levels, each of which has different responsibilities and functions, jointly ensuring the high performance and flexibility of the entire system.

[0056] The data abstraction layer is the foundation of the Zhuge Cross module. It provides commonly used data abstractions in the fields of artificial intelligence and big data, including but not limited to the implementation of the Resilient Distributed Dataset (RDD) model. This layer also implements the data model of each partition in the RDD, compact data arrangement, Spark SQL data arrangement, etc., so that data can be efficiently processed and stored in a distributed system. Through this abstraction, the Zhuge Cross module can support a variety of data operations and ensure the scalability and elastic processing of data.

[0057] Based on data abstraction, the algorithm template library implements a series of commonly used algorithms for these abstract data structures, such as Join, GroupBy, Expr, and other various expression calculations. The design of this layer adopts the template programming method, which not only improves the reusability of code, but also allows type checking at compile time, thereby reducing runtime errors and improving execution efficiency. The implementation of the algorithm template layer makes full use of the strong typing and template features of C++, providing rich and powerful data operation capabilities for upper-level applications.

[0058] The responsibility of the compilation optimization layer is to optimize the programs written by users through Zhuge Crossbow at compile time, including operator fusion, vectorization, compaction, etc. Operator fusion is an optimization technology in functional programming. It effectively reduces the generation and transmission of intermediate data by merging multiple related data transformation operations into one operation, reduces the overhead of function calls and data materialization, and breaks the compilation boundary, allowing the compiler to perform more global optimization. This technology can not only reduce execution time, but also optimize the use of memory and processor resources, significantly improving the efficiency of data processing tasks. The key task of the compilation optimization layer is to identify the data pipeline in the RDD program and fuse continuous data operations into a single operation block. After a data pipeline undergoes this operator fusion, multiple operations can be merged into a for loop block for execution, greatly improving execution efficiency. Through this layer of optimization, the execution efficiency of the program can be greatly improved, and the performance potential brought by Native execution can be tapped.

[0059] The distributed computing layer provides the necessary distributed computing support for the Zhuge Nu programming library. This layer provides a lightweight encapsulation of the "distributed dataset executor" and implements key operations such as data broadcasting, serialization, packing, and dataset import and export. With these functions, the Zhuge Nu module can efficiently perform complex data processing tasks in a distributed environment while maintaining data consistency and reliability.

[0060] When users use Zhuge Crossbow for program development, they first need to implement a module entry function according to the provided template. The entry function is the core of the entire module's interaction with the outside world. It receives four main inputs. The first is the context of the underlying runtime system, which provides the module with the necessary environment information and system resources for operation, such as the computing module manager, object serializer, distributed dataset executor, etc. Then there is the dataset passed in by the PySpark caller, which is the main data source for data processing and analysis. Next is the broadcast variable passed in by the PySpark caller. Broadcast variables are used to store data shared among all working nodes, such as configuration information or model parameters, to reduce data transmission overhead. Finally, there are the boxed object parameters passed in by the PySpark caller, which usually contain additional information or control options required to execute the task. The implementation of the entry function can be written as follows:

[0061] 1 / **Entry function of the module* /

[0062] 2Arc <runtimedataset>chukonu_module_invoke(

[0063] 3Arc <runtimecontext>ctx, / **Context of the basic runtime system* /

[0064] 4Vec <Arc <runtimedataset>>inputDSs, / **Dataset passed in by the PySpark caller* /

[0065] 5Vec <Arc <runtimebroadcast>>inputBCs, / **Broadcast variables passed in by the PySpark caller* /

[0066] 6Vec <Arc <any>>inputObjs / **Boxed object parameter passed in by the PySpark caller* /

[0067] 7){

[0068] 8 / ** / **Implement user code here* /

[0069] 9}

[0070] like Figure 2 The Chukonu.Module Python API shown allows developers to take advantage of the flexibility and ease of use of the Python language while accessing the underlying high-performance computing module. The following is a detailed description of a series of Python API functions provided in the Zhuge Nu module, which involve module loading, calling, and data type processing, providing Python developers with powerful tools to perform efficient data analysis and machine learning tasks. Module loading (module_load) is responsible for loading a Chukonu module under a specified path. When a high-performance module written in C++ needs to be used in a Python environment, this API is first called to initialize and prepare the module instance. The modulePath parameter points to the compiled module binary file, and the return value is a chukonu.Module object representing the loaded module. Module unloading (module_unload) is used to unload the loaded module and release the resources associated with it. This is an important step to ensure proper memory management and stable operation of the application. The input parameter is the chukonu.Module object returned by module_load. Module calling (module_call) is the core of the interaction between the Zhuge Nu module and PySpark. It allows developers to call loaded modules in a Spark session, process Spark DataFrames, and return new DataFrames. The parameter inputDataFrames contains one or more DataFrames, which are the input data processed by the module; inputArgs provides necessary non-DataFrame parameters, such as configuration options or model parameters, which are first boxed by the object_box function; the inputBCs list contains broadcast variables that can be accessed by multiple tasks inside the module to optimize performance and resource usage; outputSchema defines the structure of the output DataFrame. The two functions object_box_long and object_unbox_long are used to box and unbox long integer data. object_box_long converts Python integers to a format recognizable by the Zhuge Crossbow framework, while object_unbox_long performs the opposite operation, which is essential for passing numerical data between Python and the Zhuge Crossbow module. The pair of functions object_box_string and object_unbox_string handles the boxing and unboxing of string data. Use object_box_string to box Python strings into objects recognized by Zhuge Crossbow, and object_unbox_string is used to convert boxed objects back to Python strings. Among them, the conversion between Zhuge Crossbow and Python data can be implemented as follows Figure 3 Compile as shown.

[0071] S500: Executing the output node of the Zhuge Crossbow module to generate a first type of data set, which converts the first type of data set into a second type of data set when outputting; executing the input node of the Zhuge Crossbow module to generate a second type of data set, which converts the second type of data set into the first type of data set when outputting.

[0072] In the above embodiment of the present application, before executing the work plan using PySpark, the work plan tree is first obtained. Each node in the work plan tree may not be supported by the Zhuge Cross compiler but only supported by PySpark. In this case, the nodes that can be supported by the Zhuge Cross compiler are used as the first type of nodes, and the nodes that are only supported by PySpark are used as the second type of nodes, and the work plan tree is divided into the first type of subtree and the second type of subtree. The solution of the present application establishes a seamless connection between the first type of subtree and the Zhuge Cross compiler by connecting the root node and the leaf node of the first type of subtree to the output node and the input node of the Zhuge Cross module. Before executing the root node of the first type of subtree, each node of the first type of subtree is compiled by Zhuge Crossbow compiler to obtain a binary module. Based on the binary module and the module function contained therein, the leaf node of the first type of subtree can be connected with the root node of the second type of subtree. That is, after execution, the root node of the first type of subtree can convert the first type of data set corresponding to the first type of subtree into the second type of data set with the same characteristics as the second type of subtree, so that both the first type of data set and the second type of data set can be supported by PySpark, that is, the work plan tree corresponding to the work plan can be fully supported by PySpark. When the work plan is executed on PySpark, the deep integration and seamless compatibility of the two types of data sets are achieved.

[0073] In some schemes, the step S200 of dividing the work plan tree into a first type of subtree and a second type of subtree includes: classifying the nodes according to whether each node on the work plan tree is supported by the Zhuge Nu compiler, defining the nodes supported by the Zhuge Nu compiler as first type nodes, and defining the nodes not supported by the Zhuge Nu compiler as second type nodes; using a subtree formed by a continuous section of first type nodes as the first type of subtree; and using a subtree formed by a continuous section of second type nodes as the second type of subtree. In specific implementation, coloring is assigned to the first type of nodes and the second type of nodes. This scheme distinguishes the first type of nodes from the second type of nodes by different coloring, thereby improving the significance of node classification. After determining the node category, directly connecting the nodes of the same category can obtain the corresponding subtree, thereby improving the convenience and accuracy of forming the first type of subtree and the second type of subtree.

[0074] Combined with the framework structure of the aforementioned Zhuge Crossbow module of the present application, in step S400 of the present application, the output node of the Zhuge Crossbow module obtains the binary module corresponding to the first type of subtree contained therein obtained by compiling the Zhuge Crossbow compiler before it is executed, including: before the output node of the Zhuge Crossbow module is executed, the first type of subtree contained in the output node of the Zhuge Crossbow module is translated into a program module written in a native programming language by code generation; and the Zhuge Crossbow compiler is called to compile the program module into the binary module. Because the Zhuge Crossbow module has the function of compiling different high-performance language programs into binary modules, and the first type of subtree itself is supported by the Zhuge Crossbow module, the binary module corresponding to the first type of subtree can be obtained by using the Zhuge Crossbow module.

[0075] Further preferably, the acceleration method of the data and AI fusion pipeline based on native module injection also includes: the first type of data set and the second type of data set both have partition characteristics, distributed characteristics and lazy evaluation characteristics, wherein: the partition characteristics refer to: the first type of data set and the second type of data set are both divided into multiple partitions; the distributed characteristics refer to: the first type of data set and the second type of data set are both stored in a distributed manner, spanning multiple physical machines; further, each partition of the first type of data set is located in the native system and stored in the Zhuge Crossbow data format; each partition of the second type of data set is located in the Java virtual machine system and stored in the PySpark data format. The lazy evaluation characteristic refers to: the first type of data set and the second type of data set both perform evaluation operations corresponding to the needs when they are needed, avoiding increasing the system processing burden. Data can be efficiently processed and stored in a distributed system, can support a variety of data operations, and ensure the scalability and elastic processing of data. This solution can efficiently perform complex data processing tasks in a distributed environment while maintaining data consistency and reliability.

[0076] In the above scheme, before the Zhuge Cross module output node is executed, it obtains the binary module corresponding to the first type of subtree contained therein obtained by compiling the Zhuge Cross compiler: the binary module compiled by the Zhuge Cross compiler is cached in the binary module cache until the work plan is completed. The binary module cache can be obtained by Figure 2 The computing module manager shown in is implemented, and this solution avoids repeated compilation optimization of the same first-category subtree.

[0077] An embodiment of the present application also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the acceleration method of the data injected based on the native module and the AI ​​fusion pipeline are implemented in any of the aforementioned embodiments.

[0078] An embodiment of the present application also provides a computer program product, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, the steps of the acceleration method of the data injected based on the native module and the AI ​​fusion pipeline are implemented in any of the above embodiments.

[0079] like Figure 4 As shown, an embodiment of the present application also provides an electronic device, the electronic device includes at least one processor 41 and at least one memory 42, at least one of the memories 42 stores program information, and at least one of the processors 41 reads the program information and executes the acceleration method of the data and AI fusion pipeline based on native module injection described in any of the schemes of the above method embodiments. The device may also include: an input device 43 and an output device 44. The processor 41, the memory 42, the input device 43 and the output device 44 can be connected in communication. The memory 42, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 41 executes various functional applications and data processing by running the non-volatile software programs, instructions and modules stored in the memory 42, that is, the acceleration method of the data and AI fusion pipeline based on native module injection provided by any of the above schemes is realized. The memory 42 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created based on the use of the acceleration method of the data and AI fusion pipeline based on native module injection, etc. In addition, the memory 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 42 may optionally include a memory remotely arranged relative to the processor 41, and these remote memories may be connected to a device for executing an acceleration method based on data injected by native modules and an AI fusion pipeline through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The input device 43 may receive input user clicks, and generate signal inputs related to user settings and function controls of the acceleration method based on data injected by native modules and an AI fusion pipeline. The output device 44 may include a display device such as a display screen. When the one or more modules are stored in the memory 42, when they are run by the one or more processors 41, the acceleration method based on data injected by native modules and an AI fusion pipeline in any of the above-mentioned method embodiments is executed.

[0080] As needed, the above technical solutions can be combined to achieve the best technical effect.

[0081] The above are only the principles and preferred embodiments of the present application. It should be noted that, for ordinary technicians in this field, on the basis of the principles of the present application, several other modifications can be made, which should also be considered as the protection scope of the present application.< / any> < / runtimebroadcast> < / runtimedataset> < / runtimecontext> < / runtimedataset>

Claims

1. A method for accelerating the data and AI fusion pipeline based on native module injection, characterized in that: include: Get the work plan tree corresponding to the work plan to be executed by PySpark; Divide the work plan tree into a first subtree and a second subtree, wherein: the nodes in the first subtree are supported by the Zhuge Nu compiler, and the nodes in the second subtree are not supported by the Zhuge Nu compiler but are supported by PySpark; Add a Zhuge Crossbow module output node at the root node of the first type of subtree, and insert a Zhuge Crossbow module input node at the leaf node of the first type of subtree; The Zhuge Crossbow module output node obtains the binary module corresponding to the first type of subtree contained therein obtained by compiling the Zhuge Crossbow compiler before it is executed; when the Zhuge Crossbow module output node is executed, it loads the binary module and calls the module function entry contained therein, the input of the module function includes the pointer of the second type of data set generated after the root node of the second type of subtree directly dependent on the first type of subtree is executed; the output of the module function is the pointer of the first type of data set; Executing the output node of the Zhuge Crossbow module generates a first type of data set, which converts the first type of data set into a second type of data set when outputting; executing the input node of the Zhuge Crossbow module generates a second type of data set, which converts the second type of data set into the first type of data set when outputting.

2. The acceleration method of the data and AI fusion pipeline based on native module injection according to claim 1 is characterized in that: The dividing the work plan tree into a first type of subtree and a second type of subtree comprises: According to whether each node on the work plan tree is supported by the Zhuge Nu compiler, the nodes are classified, the nodes supported by the Zhuge Nu compiler are defined as first-class nodes, and the nodes not supported by the Zhuge Nu compiler are defined as second-class nodes; A subtree formed by a continuous segment of first-category nodes is used as the first-category subtree; a subtree formed by a continuous segment of second-category nodes is used as the second-category subtree.

3. The acceleration method of the data and AI fusion pipeline based on native module injection according to claim 2 is characterized in that: The dividing the work plan tree into a first type of subtree and a second type of subtree also includes: Colorings are assigned to the first type of nodes and the second type of nodes.

4. The acceleration method of the data and AI fusion pipeline based on native module injection according to claim 1 is characterized in that: Before being executed, the Zhuge Crossbow module output node obtains a binary module corresponding to the first type of subtree contained therein obtained by compiling the Zhuge Crossbow compiler, including: The Zhuge Crossbow module output node, before being executed, translates the first type of subtree contained in the Zhuge Crossbow module output node into a program module written in a native programming language by means of code generation; The Zhuge Crossbow compiler is called to compile the program module into the binary module.

5. The acceleration method of the data and AI fusion pipeline based on native module injection according to any one of claims 1 to 4, characterized in that: The method further comprises: The first type of data set and the second type of data set both have partition characteristics, distribution characteristics, and lazy evaluation characteristics, wherein: The partition characteristic means that the first type of data set and the second type of data set are both divided into multiple partitions; The distributed feature means that the first type of data set and the second type of data set are both stored in a distributed manner across multiple physical machines; The lazy evaluation feature means that the first type of data set and the second type of data set both perform evaluation operations corresponding to the needs when they are needed.

6. The acceleration method of data and AI fusion pipeline based on native module injection according to claim 5 is characterized in that: Among the partition characteristics: Each partition of the first type of data set is located in the native system and is stored in the Zhuge Crossbow data format; Each partition of the second type of data set is located in a Java virtual machine system and is stored in a PySpark data format.

7. The acceleration method of the data and AI fusion pipeline based on native module injection according to any one of claims 1 to 4, characterized in that: Before being executed, the Zhuge Cross module output node obtains the binary module corresponding to the first type of subtree contained therein obtained by compiling the Zhuge Cross compiler: The binary module compiled by the Zhuge Crossbow Compiler is cached in a binary module cache until the work plan is completed.

8. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the acceleration method of the data and AI fusion pipeline based on native module injection as described in any one of claims 1-7 are implemented.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the acceleration method of the data and AI fusion pipeline based on native module injection as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the acceleration method of the data and AI fusion pipeline based on native module injection as described in any one of claims 1-7 are implemented.

Images