Data processing method and data processing device for big data

Abstract

A data processing method and a data processing device for big data. The data processing method comprises: obtaining a main program written in a native programming language; compiling the main program in a native system and generating a loadable module, wherein the loadable module comprises a first part of the program and a second part of the program; loading the loadable module by a virtual machine running in the native system, and converting the second part of the program into a virtual machine code supported by a predetermined virtual machine big data processing system; and running the main program contained in the loadable module by the virtual machine, wherein the virtual machine constructs a directed acyclic graph program corresponding to the loadable module, and then runs the directed acyclic graph program, and hands over the first part of the program to the native system for running, and hands over the converted second part of the program to the predetermined virtual machine big data processing system for running. The data processing method can build a high-performance big data processing framework on the premise of integrating into the existing virtual machine big data software ecosystem.

Description

Technical Field

[0001] The present disclosure relates to a data processing method and a data processing device for big data. Background Art

[0002] Big data refers to large and rapidly growing data sets in heterogeneous formats. Apache Hadoop is an early open source big data solution that includes a distributed file system (HDFS) for persistent storage of big data and an analytical framework based on the MapReduce abstraction. Apache Spark is a fast and general computing system designed for large-scale data processing. Spark is an open source cluster computing environment similar to Hadoop, but Spark performs better in certain workloads. Recently, Apache Spark introduced a new abstraction called Resilient Distributed Datasets (RDD) to support fault-tolerant data reuse for iterative workloads, which can achieve an order of magnitude better performance than Hadoop MapReduce. Spark provides a rich and easy-to-use application programming interface to build support libraries for graph computing, stream processing, machine learning, and SQL queries. Today, Spark is widely deployed to serve big data analysis. Summary of the invention

[0003] At least one embodiment of the present disclosure provides a data processing method for big data, comprising: obtaining a main program written in a native programming language; compiling the main program in a native system and generating a loadable module, wherein the loadable module comprises a first part program and a second part program, the first part program comprises a functional implementation of a first operator, and the second part program comprises a function pointer for calling a function corresponding to the second operator; the loadable module is loaded by a virtual machine running in the native system, and the second part program is converted into a virtual machine code supported by a predetermined virtual machine big data processing system, the virtual machine code calls the functional implementation of a function of the predetermined virtual machine big data processing system; and the main program included in the loadable module is run by the virtual machine, wherein during the running of the main program, the virtual machine constructs a directed acyclic graph program corresponding to the loadable module, and then runs the directed acyclic graph program, the first part program is handed over to the native system for running, and the converted second part program is handed over to the predetermined virtual machine big data processing system for running.

[0004] For example, in the data processing method provided by at least one embodiment of the present disclosure, the data processing method adopts distributed computing, the first operator is a local class operator, and the second operator is a global class operator.

[0005] For example, in the data processing method provided in at least one embodiment of the present disclosure, the main program is compiled in the native system and a loadable module is generated, including: optimizing the first part of the program to reduce the interaction overhead between the native system and the virtual machine.

[0006] For example, in the data processing method provided in at least one embodiment of the present disclosure, the optimization operation is performed on the first part of the program, including: merging a series of operation steps in the first part of the program into one operation step.

[0007] For example, in the data processing method provided by at least one embodiment of the present disclosure, the first part of the program is optimized, including: setting at least one operator in the first part of the program to batch process multiple data each time.

[0008] For example, in the data processing method provided in at least one embodiment of the present disclosure, the first part of the program is optimized, including: for the data to be processed by the first part of the program, data of the same data type is converted into a data arrangement continuously stored in the memory.

[0009] For example, in the data processing method provided by at least one embodiment of the present disclosure, the input of at least one operator in the first part of the program is set as a starting address pointer of the data arrangement.

[0010] For example, in the data processing method provided in at least one embodiment of the present disclosure, the predetermined virtual machine big data processing system is Apache Spark.

[0011] At least one embodiment of the present disclosure provides a data processing device for big data, including: a program acquisition unit, configured to acquire a main program written in a native programming language; a program compilation unit, configured to compile the main program in a native system and generate a loadable module, wherein the loadable module includes a first part program and a second part program, the first part program includes a functional implementation of a first operator, and the second part program includes a function pointer for calling a function corresponding to the second operator; a loading and conversion unit, configured to load the loadable module by a virtual machine running in the native system, and convert the second part program into a virtual machine code supported by a predetermined virtual machine big data processing system, the virtual machine code calls the functional implementation of a function of the predetermined virtual machine big data processing system; and an operation unit, configured to run the main program included in the loadable module by the virtual machine, wherein, during the running of the main program, the virtual machine constructs a directed acyclic graph program corresponding to the loadable module, and then runs the directed acyclic graph program, and the first part program is handed over to the native system for running, and the converted second part program is handed over to the predetermined virtual machine big data processing system for running.

[0012] The data processing device provided by at least one embodiment of the present disclosure further includes: an optimization unit configured to: perform optimization operations on the first part of the program to reduce the interaction overhead between the native system and the virtual machine.

[0013] For example, in the data processing device provided in at least one embodiment of the present disclosure, the optimization operation is performed on the first part of the program, including: merging a series of operation steps in the first part of the program into one operation step.

[0014] For example, in the data processing device provided by at least one embodiment of the present disclosure, the first part of the program is optimized, including: setting at least one operator in the first part of the program to batch process multiple data each time.

[0015] For example, in the data processing device provided by at least one embodiment of the present disclosure, the first part of the program is optimized, including: for the data to be processed by the first part of the program, data of the same data type is converted into a data arrangement continuously stored in the memory.

[0016] For example, in a data processing device provided by at least one embodiment of the present disclosure, an input of at least one operator in the first part of the program is set as a starting address pointer of the data arrangement. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0018] Figure 1 A schematic flow chart of a data processing method for big data provided by at least one embodiment of the present disclosure is shown;

[0019] Figure 2 An exemplary schematic diagram showing an optimization operation performed on a first part of a program provided by at least one embodiment of the present disclosure is shown;

[0020] Figure 3 An exemplary schematic diagram of a directed acyclic graph program provided by at least one embodiment of the present disclosure is shown;

[0021] Figure 4A An exemplary schematic diagram of a virtual machine big data processing system (Apache Spark) provided by at least one embodiment of the present disclosure is shown;

[0022] Figure 4B An exemplary schematic diagram of a native system big data processing system provided by at least one embodiment of the present disclosure is shown;

[0023] Figure 5A schematic diagram showing an example workflow of a data processing method provided by at least one embodiment of the present disclosure;

[0024] Figure 6 A schematic block diagram of a data processing device for big data provided by at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0026] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] Although Spark has advantages in memory-distributed data sets and can optimize iterative workloads in addition to providing interactive queries, recent work has shown that there is still a lot of room for improvement in its performance. Spark programs run on the basis of JVM (java virtual machine). Programs need to be translated into code that can be indirectly executed by JVM first. Big data analysis frameworks built with C++ programming language have great performance advantages over Spark, because codes written in C++ can be translated into code that can be directly executed by the machine. For example, Thrill, a big data analysis framework based on C++, achieves an average speedup of 3.26 times that of Spark on typical big data workloads. In addition, for the Java matrix multiplication kernel, switching Java to C produces a 4.4-fold speedup, and the performance of vectorization and AVX intrinsics provided by the C compiler is improved by 9.45 times. However, performance is only one aspect of big data processing. Although C++-based big data analysis frameworks have better performance than Spark, they do not have many key features for big data processing provided by Spark, such as lineage-based elasticity: Big data analysis usually uses distributed computing, that is, it is executed in a multi-tenant commodity cluster built on commodity hardware, where task failures due to machine failures, network jitters, and preemptive scheduling are very common, which makes checkpoints inefficient in handling these frequent failures. Spark's lineage-based fault tolerance mechanism allows only part of the data to be recalculated instead of all of it. Spark's elasticity also supports other features, such as load balancing, stragglers mitigation, and automatic scaling, thereby improving cluster resource utilization. In addition, Spark's ecosystem, such as performance analyzers with Web UIs and integration with various resource managers, makes it easy to deploy, monitor, and analyze applications on various private or public clouds. Thrill has a native RDD (Resilient Distributed Datasets) abstraction called DIA, but its tight coupling of data distribution with physical machines makes elasticity ineffective. Husky uses an upstream message log fault tolerance mechanism, which generates non-negligible overhead even without failures. Compared with Spark, these C++-based big data analysis frameworks lack many basic functions.

[0028] Obviously, a full-featured native big data framework needs to be designed. A direct solution is to re-implement the functions provided by Spark in a native programming language (such as C++). This solution is theoretically feasible, but it may be too expensive and unnecessary: ​​the core components of Spark 3.0.1 have 74K lines of code, of which only 9K lines are directly related to the programming framework (including RDD application interface and operator implementation), and the rest of the code is components that serve various big data functions. Therefore, a big data framework can be built that can reuse Spark's mature big data functions without re-implementing them.

[0029] However, the approach of reusing Spark's big data capabilities creates many challenges. First, existing native big data frameworks (e.g., Thrill, Husky, etc.) are incompatible with Spark's execution model, making it infeasible to integrate native big data frameworks into Spark. For example, Thrill couples each dataset partition to a specific machine, and Husky relies on stateful task execution, which violates Spark's dynamic scheduling and statelessness assumptions. Second, the fine-grained interaction between the JVM and the native world, whether through JNI (java native interface) or JNA (java native access), incurs high overhead and has the potential to become a new performance bottleneck.

[0030] At least one embodiment of the present disclosure provides a data processing method for big data, comprising: obtaining a main program written in a native programming language; compiling the main program in a native system and generating a loadable module, wherein the loadable module comprises a first part program and a second part program, the first part program comprises a functional implementation of a first operator, and the second part program comprises a function pointer for calling a function corresponding to the second operator; the loadable module is loaded by a virtual machine running in the native system, and the second part program is converted into a virtual machine code supported by a predetermined virtual machine big data processing system, the virtual machine code calls the functional implementation of a function of the predetermined virtual machine big data processing system; and the main program included in the loadable module is run by the virtual machine, wherein during the running of the main program, the virtual machine constructs a directed acyclic graph program corresponding to the loadable module, and then runs the directed acyclic graph program, the first part program is handed over to the native system for running, and the converted second part program is handed over to the predetermined virtual machine big data processing system for running.

[0031] The data processing method divides the program for processing big data into a first part program and a second part program for separate processing, which can reuse important functions of a predetermined virtual machine big data processing system (e.g., Apache Spark) and improve the performance of big data processing. The first part program includes the functional implementation of the first operator, which can be handed over to the native system for running without calling the functions of the predetermined virtual machine big data processing system, so the processing speed is faster; while the second part program calls the functions of the predetermined virtual machine big data processing system, which can easily implement the existing functions of the predetermined virtual machine big data processing system. Therefore, the data processing method according to the embodiment of the present disclosure can realize high-speed processing of big data and can make full use of the functions of the existing big data processing system to realize full-function big data processing.

[0032] A native system is a concept relative to a virtual machine, which refers to an operating system inherent in a computer, such as a Linux system or a Window system installed on the computer. A native programming language refers to a programming language used on a native system, such as a C++ programming language. For example, in some embodiments of the present disclosure, a main program is written in the C++ programming language, and the main program is compiled to form a loadable module in the form of an object code (e.g., a machine code), and then the loadable module is loaded into a virtual machine running in the native system, such as a JVM running on a Linux system.

[0033] It should be noted that although some embodiments of the present disclosure are described using Apache Spark as an example of a predetermined virtual machine big data processing system, the present disclosure is not limited to being based on Apache Spark. Those skilled in the art will understand that other virtual machine big data processing systems also have similar technical problems, and the technical solutions of the embodiments of the present disclosure are also applicable to other virtual machine big data processing systems. In the present disclosure, a virtual machine big data processing system (framework) represents a big data processing system running on a virtual machine (e.g., JVM), including an engine and a programming framework. The big data processing system provides various functions for big data processing, for example, the functions of Apache Spark described above.

[0034] At least one embodiment of the present disclosure also provides a data processing device corresponding to the above-mentioned data processing method.

[0035] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0036] Figure 1 A schematic flowchart of a data processing method for big data provided by at least one embodiment of the present disclosure is shown.

[0037] like Figure 1 As shown, the data processing method includes the following steps S101 to S104.

[0038] Step S101: Obtain a main program written in a native programming language.

[0039] The user writes the main program using a native programming language such as the C++ programming language, and the processing device obtains the main program written by the user.

[0040] Step S102: compile the main program in the native system and generate a loadable module, wherein the loadable module includes a first part program and a second part program, the first part program includes a functional implementation of the first operator, and the second part program includes a function pointer for calling a function corresponding to the second operator.

[0041] In step S102, the main program is compiled into a module in the form of target code (e.g., machine code), which can be loaded into a virtual machine and is therefore called a loadable module. The main program generally includes a call to a function function provided by the system. The main program is compiled in conjunction with a function library or data set provided by the system, and these function libraries or data sets parse the called function so as to perform corresponding processing. According to an embodiment of the present disclosure, the compiled loadable module may include a first part program and a second part program, and the first part program and the second part program perform different processing in subsequent steps. In an embodiment of the present disclosure, the "first part program" is also referred to as the "compile-time part program", and the "second part program" is referred to as the "runtime part program". The difference between the first part program and the second part program is whether the part program includes the specific functional implementation of the operator in the program. If the specific functional implementation of the operator is included, it is the first part program, and the corresponding operator is called the first operator, otherwise it is the second part program, and the corresponding operator is called the second operator. The second part program does not include the specific functional implementation of the operator, but includes a function pointer for calling the function corresponding to the operator. The specific implementation of these operators (second operators) can be achieved by setting the address of the function that implements the function to the function pointer in the subsequent process (which will be described in detail below). The division of the first operator and the second operator can be performed according to the category of the operator, for example, certain categories of operators are divided into first operators and other categories of operators are divided into second operators according to a predetermined division strategy. The division strategy can be determined according to the specific application. For example, operators with relatively simple function implementations can be divided into first operators, while operators with relatively complex function implementations can be divided into second operators, so that the functions of the virtual machine big data processing system can be reused for the second operator with complex function implementation. For another example, in a data processing method using distributed computing, local class operators can be divided into first operators, and part of global class operators can be divided into second operators. In other words, in some embodiments of the present disclosure, the data processing method uses distributed computing, the first operator is a local class operator, and the second operator is a global class operator.

[0042] Local class operators refer to operators that do not need to consider the coordination between different computing nodes in distributed computing, such as map values ​​(mapValues) operators, filter (filter) operators, transformation (flatMap) operators, hash (hash) operators (such as Hash Aggregate, Build Hash), etc.

[0043] Global class operators represent operators that need to consider the coordination between different computing nodes in distributed computing, such as partition-pruning operators, operators with more than one dependency (for example, union operators, zip operators (such as Zip Partitions), cartesian operators), shuffle operators (such as Shuffle Write, Shuffle Read), cache operators, and data-source operators.

[0044] For global class operators, since it is necessary to consider the coordination between different computing nodes, the implementation is relatively complicated, so in the embodiment of the present disclosure, it is divided into a second operator to reuse the global class operator function of the existing virtual machine big data processing system (such as Spark).

[0045] For example, partition pruning operators and operators with more-than-one dependencies can provide lineage-related information to Spark and can reuse Spark’s lineage-based fault tolerance mechanism.

[0046] For example, the shuffle operator could reuse Spark’s fault-tolerant data shuffling mechanism, which is not only difficult to implement but also cumbersome to deploy.

[0047] For example, cache operators can reuse Spark's intermediate data management mechanisms. Data source operators can provide location information to Spark and reuse Spark's location-aware task scheduling.

[0048] The local class operator part is divided into the first operator, and the global class operator is divided into the second operator, so that in the subsequent steps, the second part of the program containing the second operator can be handed over to a virtual machine big data processing system such as Spark for processing without re-implementing complex functions, thereby avoiding unnecessary manpower consumption.

[0049] In some embodiments of the present disclosure, when compiling the main program in the native system in step S102, the process may include: optimizing the first part of the program to reduce the interaction overhead between the native system and the virtual machine.

[0050] Integrating the native system's programs into the virtual machine's big data processing system will result in interaction overhead between the native system and the virtual machine. According to some embodiments of the present disclosure, these overheads can be reduced by optimizing the first part of the programs.

[0051] Figure 2An exemplary schematic diagram of optimizing the first part of the program is shown in FIG. The two mapValues ​​operators in the program "contribs.reduceByKey(_+_).mapValues(v=>0.85*v).mapValues(v=>0.15+v).join(links)" are used as an example for explanation. The native system and the virtual machine interact through JNAPointer (JNA pointer).

[0052] Figure 2 (a) is the case where no optimization is performed. Figure 2 In (a), each mapValues ​​transformation maps directly to a dataset in the VM (e.g., Spark RDD), and each value is mapped to a VM program object (e.g., Java object). This approach is simple, but it comes with a lot of overhead. Processing a simple value such as a long double pair requires serialization / deserialization in both the VM and native system, as well as the creation of a VM program object, which creates a lot of interaction overhead that may offset the performance advantages of the native system.

[0053] In some embodiments of the present disclosure, performing an optimization operation on the first part of the program may include: fusing a series of operation steps in the first part of the program into one operation step. This optimization operation may be referred to as an operator fusion operation, such as Figure 2 as shown in (b).

[0054] exist Figure 2 In (b), two consecutive mapValues ​​operation steps (.mapValues(v=>0.85*v) and .mapValues(v=>0.15+v)) are merged into one mapValues ​​operation step (.mapValues(v=>0.15+0.85*v)), which reduces the number of serialization / deserialization times and thus reduces the interaction overhead between the native system and the virtual machine.

[0055] In some embodiments of the present disclosure, performing an optimization operation on the first part of the program may include: setting at least one operator in the first part of the program to batch process multiple data at a time. This optimization operation may be referred to as a vectorization operation, such as Figure 2 as shown in (c).

[0056] exist Figure 2In (c), the operator mapValues, which was originally processed element by element in the first part of the program, is set to batch multiple data at a time. That is, instead of processing one long double-precision pair at a time, one long double-precision pair array is now processed at a time. The long-precision pair array (denoted by R) contains multiple long double-precision pairs, and each long double-precision pair is a key-value pair type ((k, v) type, k represents the key, v represents the value). This can reduce the number of calls from the virtual world to the data in the native world, reduce the number of serialization / deserialization, and thus reduce the interaction overhead between the native system and the virtual machine.

[0057] In some embodiments of the present disclosure, performing an optimization operation on the first part of the program may include: for data to be processed by the first part of the program, converting data of the same data type into a data arrangement continuously stored in the memory. This optimization operation may be referred to as a data arrangement operation.

[0058] Storing batches of data continuously through memory (as a buffer) helps to avoid memory fragmentation caused by allocating small objects and eliminates the requirement for compression when the garbage collector is running. In addition, data arrangement increases data locality and provides a regular memory access mode, thereby improving the efficiency of the central processing unit. For example, in some embodiments of the present disclosure, the input of at least one operator in the first part of the program can be set to a starting address pointer of the data arrangement. The same type of data is often processed by one or more common operators. Therefore, when the data of these operators are stored continuously in the memory, it is not necessary to provide each data or its address pointer, and only the address pointer of the starting data (i.e., the starting address pointer of the data arrangement) is provided to obtain these continuously stored data. In this case, there is no need to interact with each data to be processed between the virtual machine and the native system, but only the starting address pointer of the data arrangement is needed, which greatly reduces the interaction overhead between the two. Figure 2 (d) shows the pointer transfer between the virtual machine and the native system. Figure 2 In (d), this data arrangement is called "CompactArray", which is an array of long double-precision pairs stored in continuous memory. Only address pointers are passed between the virtual machine and the native system without data serialization / deserialization.

[0059] In some embodiments of the present disclosure, a variety of templates are provided to generate data arrangements for various data types. For example, a flat array can place fixed-length elements in a buffer. A bitmap array can place Boolean elements in a buffer. An array array and a string array can place array elements in a buffer. A nullable array can place nullable elements in a buffer. A tuple array places tuple elements in a buffer.

[0060] The above optimization operation on the first part of the program reduces the interaction overhead between the native system and the virtual machine, thereby improving the performance of big data processing. It should be noted that the present disclosure is not limited to the above optimization operation, and is also applicable to other optimization operations that can reduce the interaction overhead between the native system and the virtual machine.

[0061] Step S103: The virtual machine running in the native system loads the loadable module and converts the second part of the program into a virtual machine code supported by a predetermined virtual machine big data processing system, and the virtual machine code calls the function of the predetermined virtual machine big data processing system to implement the function.

[0062] The main program is compiled into a loadable module in the form of target code, which is loaded into a virtual machine running on a native system, such as a JVM running on a Linux system. The loading process can be performed using an interface between the virtual machine and the native system, such as JNA.

[0063] After the loading step of step S103, the second part of the program is also converted by the virtual machine into a virtual machine code supported by a predetermined virtual machine big data processing system. The second part of the program is usually a relatively complex program, so it calls the function of the predetermined virtual machine big data processing system (such as Spark) on the virtual machine without reprogramming. Therefore, the second part of the program is converted by the virtual machine into a code supported by the predetermined virtual machine big data processing system (such as Spark) during the process of loading into the virtual machine. For example, the function pointer in the second part of the program is converted into a pointer address that can actually call the function in the predetermined virtual machine big data processing system, and the called functions have the functional implementation of the second operator in the second part of the program. The virtual machine code calls the functional module of the predetermined virtual machine big data processing system (for example, the processing module of the corresponding operator) to avoid repeated programming and implementation of these functional modules. It should be noted that "predetermined" here means predetermined, that is, the processing system targeted is preselected for code conversion and calling.

[0064] Step S104: The main program included in the loadable module is run by the virtual machine, wherein, during the running of the main program, the virtual machine constructs a directed acyclic graph program corresponding to the loadable module, and then runs the directed acyclic graph program, and the first part of the program is handed over to the native system for running, and the converted second part of the program is handed over to the predetermined virtual machine big data processing system for running.

[0065] A directed acyclic graph (DAG) is a graph data structure. If a directed graph cannot be returned from any vertex through a number of edges, then the graph is a directed acyclic graph. A directed acyclic graph program is a program that renders a directed acyclic graph. A directed acyclic graph program used to process big data includes a series of operators for processing big data. Figure 3 An example of a directed acyclic graph program is shown. Figure 3 The first line in the code is a data processing code intercepted from an application (i.e., part of the main program written by the user). The directed acyclic graph program corresponding to the code is located below the code, which includes multiple operators executed sequentially: Hash Aggregate, ShuffleWrite, Shuffle Read, Map Values, Build Hash, Zip Partitions, and Probe Hash. These operators will perform corresponding processing on the data to be processed. Figure 3 As shown in the directed acyclic graph program of , the vertices represent operators, and the edges with arrows represent the dependency relationship between two operators. The directed acyclic graph program in step S104 can be constructed based on the code conversion written by the user. For example, Figure 3 The code "reduceByKey" in the example can be constructed as Hash Aggregate, Shuffle Write, Shuffle Read, and Hash Aggregate executed sequentially, and the code "join" can be constructed as Build Hash, Zip Partitions, and Probe Hash executed sequentially. These operators include the first operator and the second operator, and are respectively mapped to the first part of the program and the second part of the program.

[0066] Step S104 also illustrates different processing methods for the first part of the program and the second part of the program.

[0067] For the first part of the program, the code compiled in the native system contains the functional implementation of the operator, so it can be run by the native system using the machine code execution mechanism of the virtual machine (for example, JNI, JNA). After this part of the program is run in the native system, the functions of the operators therein will be processed. For the second part of the program, the code compiled in the native system does not contain the functional implementation of the operator, but is converted into a virtual machine code supported by a predetermined virtual machine big data processing system, so it can be executed by a predetermined virtual machine big data processing system residing on the virtual machine. When the virtual machine executes step S104, the engine of the predetermined virtual machine big data processing system can be reused to realize functions such as distributed execution, elasticity, lag mitigation, and monitoring.

[0068] As mentioned above, the predetermined virtual machine big data processing system is a big data processing system, including an execution engine and a programming framework. For example, the predetermined virtual machine big data processing system is Apache Spark. Figure 4A Introducing the scheduled virtual machine big data processing system using Apache Spark.

[0069] Figure 4A An exemplary schematic diagram of a virtual machine big data processing system (Apache Spark) provided by at least one embodiment of the present disclosure is shown.

[0070] like Figure 4A As shown, the predetermined virtual machine big data processing system includes a programming framework (RDD) and a Spark engine. The programming framework includes a Spark dataset representation (dataset representation). The Spark dataset representation can provide a Spark RDD application programming interface (API) to users.

[0071] The Spark dataset representation contains a set of partitions, a set of dependencies, a computation definition, and metadata about the dataset distribution and data placement. The Spark RDD API provided by the Spark dataset representation can be used to build DAG programs.

[0072] The Spark engine provides support and management for various Spark features, such as distributed execution, elasticity, lag mitigation, monitoring, etc. The Spark engine includes a resource manager (such as YARN, AWS), which can provide unified resource management and scheduling for upper-level applications.

[0073] Apache Spark residing on the virtual machine can parse and execute the converted code of the second part of the program (ie, the code that complies with the Spark RDD application program interface) through its programming framework and Spark engine.

[0074] According to the data processing method of the embodiment of the present disclosure, the embodiment of the present disclosure provides a native system big data processing system, whose structure is similar to the above-mentioned virtual machine big data processing system.

[0075] Figure 4B An exemplary schematic diagram of a native system big data processing system provided by at least one embodiment of the present disclosure is shown. The big data processing system of the embodiment of the present disclosure includes a programming framework and an engine, the programming framework includes a dataset representation, and the dataset representation can provide an application program interface (API).

[0076] The API of the native system big data processing system of the embodiment of the present disclosure can be designed as an RDD-style API similar to the Spark RDD API, for example, the names and semantics of its operators are the same as those of Spark operators, so as to facilitate the use of Spark users. Of course, it can also be designed to be different from the Spark RDD API, as long as it can be called by the user programmatically.

[0077] In addition to realizing the functions of a general big data processing system (such as the functions of the above-mentioned Spark data set representation), the data set representation of the native system big data processing system of the disclosed embodiment can also realize the functional implementation of the first operator, the function pointer for calling the function corresponding to the second operator, and the optimization operation of the first part of the program. The part of the data set representation related to the second part of the program can reuse the Spark data set representation, that is, it contains the function pointer for calling the function corresponding to the operator in the second part of the program. The part of the data set representation related to the first part of the program can include the functional implementation of the operator in the first part of the program. The operator in the data set representation can be in the form of a code template, so that the optimization of the first part of the program can be realized through, for example, C++ template meta-programming.

[0078] The engine of the native system big data processing system of the embodiment of the present disclosure can be an encapsulation of the engine of the virtual machine big data processing system (such as Spark) to reuse the big data functions of Spark, and at the same time include an additional functional unit for integrating the native code into the virtual machine. The engine is used to provide support and management for the various functions of the native system big data processing system. Since the engine of the native system big data processing system is an encapsulation of the existing virtual machine big data processing system engine, it does not need to reconfigure the existing cluster resource manager or recompile the existing Spark. The engine of the native system big data processing system of the present disclosure borrows most of the functions of the Spark engine, but enhances the Spark engine to support efficient Spark integration. For example, the engine of the native system big data processing system of the present disclosure includes a library in the native system, which provides Spark's C++ binding (called CppSpark). CppSpark refers to the interface of the virtual machine's functions implemented in C++ language, so the functions of the virtual machine can be called by CppSpark in the native system. When converting the second part of the program, the C++ programming interface provided by CppSpark will be called. The role of CppSpark is to forward the call operation from the native system to the virtual machine, thereby realizing the conversion of the second part of the program in the virtual machine, that is, converting the second part of the program into a virtual machine code supported by a predetermined virtual machine big data processing system. The engine of the native system big data processing system of the disclosed embodiment may include a driver in the virtual machine system, which is a virtual machine big data processing system (such as Spark) ordinary application program implemented in a virtual machine programming language (such as Java), which is used to load the compiled program code (i.e., "loadable module") from the native system, convert the second part of the program, and execute the loadable module. For example, in Spark, the driver can be submitted to a Spark compatible cluster like an ordinary Spark application. After successful submission, the driver loads the loadable module from the native system, registers the engine implementation for the loadable module, and starts the main program contained in the loadable module. In addition, in order to support remote execution, the driver can also instruct the newly created executor to prepare the environment, such as downloading the above-mentioned loadable module.

[0079] Refer to the following Figure 4B The example workflow of the data processing method provided by at least one embodiment of the present disclosure is briefly described. Figure 5 Shown in.

[0080] like Figure 5As shown, in the native system, the user writes the main program through the application program interface. The native system compiles the main program in combination with the data set representation and generates a loadable module, wherein the loadable module includes a first part program and a second part program, the first part program includes the functional implementation of the first operator, and the second part program includes a function pointer for calling the function corresponding to the second operator. The driver in the native system loads the loadable module, and in the loading process, the second part program is converted into a virtual machine code supported by a predetermined virtual machine big data processing system in combination with the library that provides the C++ binding of Spark, and the loadable module registration engine is implemented and the main program included in the loadable module is started. During the operation of the main program, the driver constructs a directed acyclic graph program corresponding to the loadable module, and then runs the directed acyclic graph program, and the first part of the program is handed over to the native system for operation. The driver can distribute the converted second part program to the cluster executor (represented by Cloud in the figure) for distributed computing through a cluster resource manager (such as YARN).

[0081] Figure 6 A schematic block diagram of a data processing device 600 for big data is shown in at least one embodiment of the present disclosure. The data processing device 600 can be used to perform Figure 1 The data processing method shown.

[0082] like Figure 6 As shown, the data processing device 600 includes a program acquisition unit 601 , a program compilation unit 602 , a loading and conversion unit 603 , and an execution unit 604 .

[0083] The program acquisition unit 601 is configured to acquire a main program written in a native programming language.

[0084] The program compilation unit 602 is configured to compile the main program in the native system and generate a loadable module, wherein the loadable module includes a first part program and a second part program, the first part program includes the functional implementation of the first operator, and the second part program includes a function pointer for calling the function corresponding to the second operator.

[0085] The loading and conversion unit 603 is configured to load the loadable module from the virtual machine running in the native system, and convert the second part of the program into a virtual machine code supported by a predetermined virtual machine big data processing system, and the virtual machine code calls the function implementation of the function of the predetermined virtual machine big data processing system.

[0086] The running unit 604 is configured to run the main program included in the loadable module by the virtual machine, wherein, during the running of the main program, the virtual machine constructs a directed acyclic graph program corresponding to the loadable module, and then runs the directed acyclic graph program, and the first part of the program is handed over to the native system for running, and the converted second part of the program is handed over to the predetermined virtual machine big data processing system for running.

[0087] For example, the data processing method of the data processing device 600 adopts distributed computing, the first operator is a local class operator, and the second operator is a global class operator.

[0088] For example, in at least one embodiment, the data processing device 600 may further include an optimization unit 605. The optimization unit 605 is configured to perform an optimization operation on the first part of the program to reduce the interaction overhead between the native system and the virtual machine.

[0089] For example, in at least one embodiment, the optimization unit 605 is further configured to merge a series of operation steps in the first part of the program into one operation step.

[0090] For example, in at least one embodiment, the optimization unit 605 is further configured to set at least one operator in the first part of the program to batch process multiple data at a time.

[0091] For example, in at least one embodiment, the optimization unit 605 is further configured to convert data of the same data type to be processed by the first part of the program into a data arrangement that is continuously stored in the memory.

[0092] For example, in at least one embodiment, the optimization unit 605 is further configured to set the input of at least one operator in the first part of the program as a starting address pointer of the data arrangement.

[0093] For example, the data processing device 600 may be implemented using hardware, software, firmware, or any feasible combination thereof, and the present disclosure is not limited thereto.

[0094] The above description of the data processing method is also applicable to the data processing device 600 and will not be repeated here.

[0095] According to an embodiment of the present disclosure, a computer program product is further provided, including a program code, and when the program code is executed by a processor, the data processing method according to the embodiment of the present disclosure is executed.

[0096] According to an embodiment of the present disclosure, there is also provided a computer-readable medium on which a program code is stored. When the program code is executed by a processor, the data processing method according to the embodiment of the present disclosure is executed.

[0097] There are a few points to note about this disclosure:

[0098] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to the general design.

[0099] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0100] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A data processing method for big data, comprising: Get the main program written in the native programming language; Compile the main program in the native system and generate a loadable module, wherein the loadable module includes a first part program and a second part program, the first part program includes a functional implementation of a first operator, and the second part program includes a function pointer for calling a function corresponding to a second operator; The loadable module is loaded by a virtual machine running in the native system, and the second part of the program is converted into a virtual machine code supported by a predetermined virtual machine big data processing system, and the virtual machine code calls the function implementation of the predetermined virtual machine big data processing system; and The main program included in the loadable module is run by the virtual machine, wherein, during the running of the main program, the virtual machine constructs a directed acyclic graph program corresponding to the loadable module, and then runs the directed acyclic graph program, the first part of the program is handed over to the native system for running, and the converted second part of the program is handed over to the predetermined virtual machine big data processing system for running.

2. The data processing method according to claim 1, wherein: The data processing method adopts distributed computing, the first operator is a local class operator, and the second operator is a global class operator.

3. The data processing method according to claim 1 or 2, wherein: Compiling the main program in the native system and generating a loadable module includes: The first part of the program is optimized to reduce the interaction overhead between the native system and the virtual machine.

4. The data processing method according to claim 3, wherein: The first part of the program is optimized, including: A series of operation steps in the first part of the program are merged into one operation step.

5. The data processing method according to claim 3, wherein: The first part of the program is optimized, including: At least one operator in the first part of the program is set to batch process a plurality of data at a time.

6. The data processing method according to claim 3, wherein: The first part of the program is optimized, including: For the data to be processed by the first part of the program, data of the same data type is converted into a data arrangement that is continuously stored in the memory.

7. The data processing method according to claim 6, wherein: An input of at least one operator in the first part of the program is set as a starting address pointer of the data arrangement.

8. The data processing method according to claim 1, wherein: The predetermined virtual machine big data processing system is Apache Spark.

9. A data processing device for big data, comprising: a program acquisition unit configured to acquire a main program written in a native programming language; A program compiling unit, configured to compile the main program in a native system and generate a loadable module, wherein the loadable module includes a first part program and a second part program, the first part program includes a functional implementation of a first operator, and the second part program includes a function pointer for calling a function corresponding to a second operator; a loading and conversion unit configured to load the loadable module by a virtual machine running in the native system, and convert the second part of the program into a virtual machine code supported by a predetermined virtual machine big data processing system, wherein the virtual machine code calls a function implementation of a function of the predetermined virtual machine big data processing system; and The running unit is configured to run the main program included in the loadable module by the virtual machine, wherein, during the running of the main program, the virtual machine constructs a directed acyclic graph program corresponding to the loadable module, and then runs the directed acyclic graph program, the first part of the program is handed over to the native system for running, and the converted second part of the program is handed over to the predetermined virtual machine big data processing system for running.

10. The data processing apparatus according to claim 9, further comprising: The optimization unit is configured to: perform optimization operations on the first part of the program to reduce the interaction overhead between the native system and the virtual machine.

11. The data processing device according to claim 10, wherein: The first part of the program is optimized, including: A series of operation steps in the first part of the program are merged into one operation step.

12. The data processing device according to claim 10, wherein: The first part of the program is optimized, including: At least one operator in the first part of the program is set to batch process a plurality of data at a time.

13. The data processing device according to claim 10, wherein: The first part of the program is optimized, including: For the data to be processed by the first part of the program, data of the same data type is converted into a data arrangement that is continuously stored in the memory.

14. The data processing apparatus according to claim 13, wherein: An input of at least one operator in the first part of the program is set as a starting address pointer of the data arrangement.

Images