Hybrid big data computing platform, computing method, equipment and storage medium

Abstract

The present application relates to a hybrid big data computing platform, computing method, device, and storage medium. The computing platform includes a computing module, a communication module, and a storage module; the computing module includes a Spark computing module and an MP1 computing module. The Spark computing module is used to receive computing tasks for double-dense application data, obtain the computing data of the computing task from the storage module through the communication module, and identify the task type of the computing task. If it is a data-intensive task, the Spark computing module generates the corresponding RDD and then calls the Spark computing paradigm to perform task calculation; if it is a computing-intensive task, the Spark computing module stores the task code and computing data of the computing task in the storage module, and calls the MP1 computing module through the communication module, and the MP1 computing module performs task calculation. The present application provides better support for computing tasks for double-dense application data, so that the computing has higher performance.

Description

Technical Field

[0001] The present application belongs to the field of big data processing technology, and in particular relates to a hybrid big data computing platform, computing method, device and storage medium. Background Art

[0002] Artificial intelligence (AI) big data applications, exemplified by machine learning, exhibit the typical dual-density characteristics of being both compute-intensive and data-intensive. These dual-density applications involve processing massive amounts of data while placing high demands on computing performance. Traditional computing frameworks have demonstrated their shortcomings when dealing with these dual-density applications. For example, Spark, a currently mainstream big data framework, possesses comprehensive big data processing mechanisms and has achieved significant performance improvements compared to other big data frameworks due to its memory-based computing mechanism and global optimization design philosophy. However, for compute-intensive applications, its performance remains far below that of MPI (Message Passing Interface), a mainstream high-performance computing framework. While MPI offers significant performance advantages for compute-intensive tasks, it cannot, like Spark, free users from the complexities of resource management, scheduling, parallelism, and fault tolerance for data-intensive tasks.

[0003] To address these shortcomings, existing technologies employ processing frameworks that integrate Spark and MPI. However, such integration approaches require addressing data communication issues during the invocation process. For example, in the existing Alchemist and Spark+MPI architectures, the Alchemist integration architecture transmits data via network communication. Before invoking the MPI library for high-performance computing, the Driver node in the Alchemist module first establishes a socket connection with the Driver node in the Spark cluster and uses this connection to exchange information about the Worker nodes in their respective clusters. Subsequently, based on the acquired information, the Worker node in the Alchemist module establishes a socket connection with the Worker node in the Spark cluster and uses this connection to exchange distributed data located on each Worker node. The Spark+MPI integration architecture transmits data via shared memory. Before invoking the MPI cluster for high-performance computing, Spark stores the computational data required for the MPI calculation in shared memory. During the MPI calculation, each node reads the data from memory separately. Spark+MPI utilizes shared memory for message passing, and the multiple copies of data generated by multiple replications can put pressure on memory. Furthermore, the multiple replications of large amounts of data can also impact performance. Compared to transmitting data to an MPI cluster through shared memory, Alchemist uses Sockets for data transmission, which is fast and does not require memory copies. However, Socket communication is affected by factors such as network bandwidth. When the amount of data to be communicated reaches a certain level, network bandwidth will become a new performance bottleneck. Summary of the Invention

[0004] This application provides a hybrid big data computing platform, computing method, device and storage medium, aiming to solve at least one of the above technical problems in the prior art to a certain extent.

[0005] In order to solve the above problems, this application provides the following technical solutions:

[0006] A hybrid big data computing platform includes a computing module, a communication module, and a storage module;

[0007] The computing module includes a Spark computing module and an MPI computing module. The Spark computing module is used to receive computing tasks of double-density application data submitted by the user end, obtain computing data of the computing task from the storage module through the communication module, and identify the task type of the computing task. The task type includes data-intensive tasks and computing-intensive tasks. If it is a data-intensive task, the Spark computing module generates the corresponding RDD according to the task requirements, and calls the Spark computing paradigm to perform task calculation; if it is a computing-intensive task, the Spark computing module stores the task code and computing data of the computing task to the storage module, and calls the MPI computing module through the communication module. The MPI computing module is used to read the task code and computing data from the storage module, and call the MPI computing paradigm to perform task calculation.

[0008] The technical solution adopted in the embodiment of the present application also includes: the storage module includes a source data storage module, an intermediate data storage module and a metadata storage module. The source data storage module is based on the HDFS big data file system and is used to store the original data required for the computing tasks of the double-density application data; the intermediate data storage module is used to store the intermediate data during the communication process between the Spark computing module and the MPI computing module; the metadata storage module is used to store metadata information of the computing data.

[0009] The technical solution adopted in the embodiment of the present application also includes: the intermediate data includes dynamic data and static data, the dynamic data includes at least the task code required to be executed by the MPI computing module, and the static data includes at least the calculation data of the Spark computing module and the MPI computing module during the communication process; the intermediate data storage module includes an intermediate code data storage sub-module and an intermediate computing data storage sub-module, the intermediate code data storage sub-module is used to store the static data in the form of a shared file, and the intermediate computing data storage sub-module is used to store the dynamic data using the organizational structure of a distributed memory file system, and the metadata storage module uses a Redis database-based method to act as the server of the intermediate computing data storage sub-module.

[0010] The technical solution adopted in the embodiment of the present application also includes: the communication module includes a high-throughput communication mode and a low-latency communication mode, and the communication process of the communication module includes:

[0011] Internal communication of the Spark computing module: During the startup phase and the execution phase of the Spark computing module, the communication module uses a high-throughput communication mode to obtain the computing data required for the current task from the source data storage module;

[0012] Internal communication of the MPI computing module: During the startup phase and the execution phase of the MPI computing paradigm of the MPI computing module, the MPI computing module accesses the intermediate data storage module through the low-latency communication mode of the communication module to read the task code, obtains metadata information of the required computing data from the metadata storage module according to the task code, establishes a low-latency communication connection with the intermediate data storage module according to the metadata information, and requests the required computing data from the intermediate data storage module;

[0013] Data transmission between the Spark computing module and the MPI computing module: The data transmitted between the Spark computing module and the MPI computing module includes distributed data and non-distributed data. The non-distributed data refers to the command and parameter data in the calling process. The communication module uses the communication method of establishing a Socket between Drivers to perform non-distributed data transmission; the distributed data refers to the computing data. The communication module uses a distributed file system combined with a high-throughput communication mode for data transmission.

[0014] Communication between the Spark computing module and the storage module: Before executing the Spark computing paradigm, the Spark computing module reads the original data required for the computing task from the source data storage module through the communication module; before calling the MPI computing module, it communicates with the intermediate data storage module through the communication module to store the task code and computing data required for the MPI computing module to calculate, and communicates with the metadata storage module to save the metadata information of the computing data; after the calculation is completed, it communicates with the source data storage module and saves the calculation results;

[0015] Communication between the MPI computing module and the storage module: After the MPI computing module is started, it accesses the intermediate data storage module through the communication module to obtain the task code to be executed, accesses the metadata storage module according to the task code to obtain metadata information of the computing data, establishes a low-latency communication connection with the intermediate data storage module according to the metadata information, and requests the required computing data from the intermediate data storage module. The intermediate data storage module sends the computing data to the MPI computing module through the low-latency communication connection.

[0016] Another technical solution adopted in the embodiment of the present application is: a hybrid big data computing method, comprising:

[0017] Receive computing tasks of dual-encrypted application data submitted by a user terminal through the Spark computing module, and obtain computing data of the computing tasks from the storage module through the communication module;

[0018] Determine whether the computing task is a data-intensive task or a computing-intensive task. If it is a data-intensive task, the Spark computing module generates a corresponding RDD according to the task requirements of the computing task, and then calls the Spark computing paradigm to perform task computing;

[0019] If it is a computationally intensive task, the task code and computational data of the computational task are stored in the storage module through the Spark computation module, and the MPI computation module is called through the communication module. The task code and computational data are read from the storage module through the MPI computation module, and the MPI computation paradigm is called to perform the task computation.

[0020] The technical solution adopted in the embodiment of the present application further includes: receiving the computing task of the dual-encrypted application data submitted by the user terminal through the Spark computing module, and obtaining the computing data of the computing task from the storage module through the communication module. Specifically,

[0021] The user terminal submits a computing task of double-encrypted application data through an application layer interface;

[0022] Reading the computing task through the Spark computing module, establishing a Spark computing environment according to the task requirements of the computing task, and generating a directed acyclic graph of the computing task;

[0023] Dividing the computing task into individual independent tasks according to the directed acyclic graph, and storing the divided tasks into a task set;

[0024] A task is taken out from the task set, and the computing data required for the current task is obtained from the source data storage module through the communication module.

[0025] The technical solution adopted in the embodiment of the present application further includes: storing the task code and calculation data of the computing task into the storage module through the Spark computing module is specifically as follows:

[0026] The obtained calculation data is partitioned through the Spark calculation module, and each partition data is stored in the memory file system of different Spark calculation nodes. The correspondence between the metadata information of the partition data and the Spark calculation node is stored in the metadata storage module. The task code required to be executed by the MPI calculation module is saved in the intermediate code data storage submodule, and the calculation data required for the task is stored in the intermediate calculation data storage submodule. The storage address for storing the calculation results is applied to the intermediate data storage module, the storage address is returned to the MPI calculation module, and the MPI calculation module is called to perform the task calculation.

[0027] The technical solution adopted in the embodiment of the present application also includes: reading the task code and calculation data from the storage module through the MPI calculation module, and calling the MPI calculation paradigm to perform task calculation specifically as follows:

[0028] The MPI computing module accesses the intermediate data storage module through the communication module to read the task code, obtains metadata information of the required computing data from the metadata storage module according to the task code, and calls the low-latency communication mode of the communication module to issue a read data command;

[0029] After receiving the command, the communication module establishes a low-latency communication connection with the intermediate data storage module according to the metadata information, and requests the required calculation data from the intermediate data storage module;

[0030] The intermediate data storage module sends calculation data to the MPI calculation module through the established low-latency communication connection. After receiving the calculation data, the MPI calculation module sends a disconnect command to the communication module and calls the MPI calculation paradigm to perform task calculation;

[0031] After the task calculation is completed, the calculation result is saved to the intermediate data storage module according to the storage address returned by the Spark calculation module.

[0032] Another technical solution adopted by the embodiment of the present application is: a device, the device comprising a processor and a memory coupled to the processor, wherein:

[0033] The memory stores program instructions for implementing the hybrid big data computing method;

[0034] The processor is used to execute the program instructions stored in the memory to control the hybrid big data computing platform.

[0035] Another technical solution adopted in the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the hybrid big data calculation method.

[0036] Compared with the prior art, the beneficial effects produced by the embodiments of the present application are as follows: the hybrid big data computing platform, computing method, device and storage medium of the embodiments of the present application provide better support for the computing tasks of double-dense application data by constructing an integrated storage and computing platform, so that the computing of double-dense application data has higher performance. The embodiments of the present application adopt a communication method combining a distributed memory file system with high-performance communication technology, which solves the memory shortage problem of the traditional communication method based on shared memory and the performance problem of socket-based communication. The embodiments of the present application combine heterogeneous storage systems to provide a storage mode tailored to the data characteristics of the computing tasks of double-dense application data, so that the storage performance can match the dual-paradigm hybrid computing and balance the relationship between big data storage and high-performance storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a diagram of the architecture of a hybrid big data computing platform according to an embodiment of the present application;

[0038] Figure 2 Schematic diagram of the computing module structure of an embodiment of the present application;

[0039] Figure 3 This is an activity diagram of the computing module and storage module of an embodiment of the present application;

[0040] Figure 4 This is a flow chart of the hybrid big data calculation method according to an embodiment of the present application;

[0041] Figure 5 This is a schematic diagram of the device structure of an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the structure of the storage medium of an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0044] The terms "first," "second," and "third" in this application are used only for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of such features. In the description of this application, "multiple" means at least two, for example, two, three, etc., unless otherwise specifically defined. All directional indications in the embodiments of this application (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship, movement, etc. between the components under a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications also change accordingly. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products, or devices.

[0045] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0046] In response to the computational complexity of double-density application data, the hybrid big data computing platform of the embodiment of the present application integrates the two computing paradigms of Spark and MPI. It can not only fully utilize the highly available user interface and mature big data processing mechanism provided by Spark, but also obtain the high performance of MPI in processing complex calculations, thereby balancing the delay problem of the high-throughput computing mode of big data, and adopts a data communication method that combines distributed shared memory with high-performance communication technology to solve the capacity problem of shared memory and the performance problem of Socket communication, so that the fusion architecture can better support the calculation of double-density application data.

[0047] Specifically, see Figure 1, is an architecture diagram of the hybrid big data computing platform of the embodiment of the present application. The hybrid big data computing platform of the embodiment of the present application includes, from top to bottom, an application layer, a computing module, a storage module, a communication module, and an infrastructure layer. The application layer is used to provide an interface for the user end, through which the user end submits computing tasks of double-dense application data to the hybrid big data computing platform and obtains computing results; the computing module is used to use the hybrid computing function of Spark and MPI dual paradigms to segment and classify computing tasks of double-dense application data, and call the Spark computing paradigm or the MPI computing paradigm to perform task computing according to the task type, thereby realizing the classification optimization scheduling of computing tasks and maximizing the computing performance of the Spark and MPI computing paradigms; the storage module is used to integrate different types of hardware facilities to construct different storage modes based on the data features generated by the computing tasks during the computing process, providing heterogeneous storage functions for data; the communication module is used to provide high-throughput and high-performance data communication modes for different computing stages of the computing tasks to balance the relationship between throughput and performance; the infrastructure layer is used to provide hardware support for the application layer, computing module, storage module, and communication module.

[0048] For details, please refer to Figure 2 , is a schematic diagram of the computing module structure of an embodiment of the present application. The computing module of the embodiment of the present application includes a Spark computing module and an MPI computing module. The Spark computing module is a computing module based on an in-memory computing model designed for large-scale data-intensive tasks, and the MPI computing module is a parallel programming module based on message passing designed for high-performance computing-intensive tasks. The combination of the two can achieve the high-performance big data analysis goals required by computing tasks for processing dual-intensive application data. After the computing module receives the computing task of the double-dense application data submitted by the user end through the application layer interface, the Spark computing module obtains the computing data from the storage module through the communication interface of the communication module and automatically identifies the task type, which includes data-intensive tasks and computing-intensive tasks. If it is a data-intensive task, the Spark computing module generates the corresponding RDD (Resilient Distributed Datasets) according to the task requirements, and then calls the Spark computing paradigm to perform task calculations; if it is a computing-intensive task, the Spark computing module stores the task code and the computing data required for the task in the storage module, and calls the MPI computing module through the communication module. After the MPI computing module reads the task code and data information from the storage module, it calls the MPI computing paradigm to perform task calculations, thereby realizing task classification optimization scheduling and reasonable allocation and sharing of resources, and maximizing the computing performance of the Spark computing paradigm and the MPI computing paradigm in processing double-dense application data. Specifically, Figure 3 , which is an activity diagram of the computing module and the storage module in an embodiment of the present application.

[0049] Furthermore, if Figure 1 As shown, the storage module includes a source data storage module, an intermediate data storage module, and a metadata storage module. These modules constitute a heterogeneous storage system, which provides a layered storage mode for different data at different stages of the calculation process of double-dense application data, so as to balance the relationship between throughput and latency and maximize the storage performance of the storage module. Among them, the source data storage module is based on the HDFS big data file system and is used to store the original data required for the calculation task of the double-dense application data submitted by the user end; the intermediate data storage module is used to store the intermediate data of the Spark calculation module and the MPI calculation module during the communication process; further, according to the different characteristics of different types of intermediate data, the embodiment of the present application divides the intermediate data into two categories: static data and dynamic data, and divides the intermediate data storage module into an intermediate code data storage submodule and an intermediate calculation data storage submodule. The intermediate code data storage submodule is used to store static data such as the task code required to be executed by the MPI calculation module in the form of a shared file, and the intermediate calculation data storage submodule is used to use the organizational structure of the distributed memory file system to store dynamic data with temporary and sudden access characteristics such as the calculation data of the Spark calculation module and the MPI calculation module during the communication process.

[0050] Based on the above, the embodiment of the present application adopts a storage method of separate storage of dynamic data and static data, which has the following advantages: independent storage of dynamic and static data can better optimize memory space utilization; the memory file system is built in the memory and disappears during power outages, which conforms to the temporary characteristics of dynamic data; the memory has higher IO access performance than the disk and can better adapt to the computing speed of the computing module; a distributed architecture is used to distribute data access across multiple nodes, and a load balancing algorithm is used to optimize data distribution, which can better cope with the characteristics of sudden access to dynamic data; data is stored in the form of files, and the tree directory of the file can be used to efficiently manage distributed data, avoiding the storage of redundant data structures and the execution of management data structure algorithms, and reducing the additional overhead of data access.

[0051] The metadata storage module is a supplement to the storage module, which is used to store metadata information of the calculation data, and adopts an implementation method based on the Redis database to act as the server of the intermediate calculation data storage submodule to improve the overall performance of the storage module. Specifically, since the intermediate calculation data storage submodule adopts the organizational structure of the distributed file system, the server is required to provide data management and record the correspondence between data and physical nodes. The embodiment of the present application uses the metadata storage module to act as the server of the intermediate calculation data storage submodule, and adopts an implementation method based on the Redis database to deal with the burst access and temporary characteristics of the intermediate calculation data storage submodule. In addition, during the fusion calculation process, the MPI calculation module needs to obtain the calculation data by establishing a communication connection with the Spark calculation node in the Spark calculation module, but due to the load balancing design, each time a task is submitted, the Spark calculation node and the MPI calculation node that executes this task are not fixed. Therefore, the MPI calculation node needs to establish a connection with the Spark calculation node in the entire Spark cluster to obtain the required calculation data, which causes a waste of resources and reduces the storage performance of the intermediate data. Based on this, the embodiment of the present application adopts a strategy of separating computing data from metadata. Before obtaining computing data, the MPI computing module first accesses the metadata storage module to obtain metadata information, and directly obtains the Spark computing node corresponding to the computing data through the metadata information and establishes a targeted connection.

[0052] Based on the above, the metadata storage module of the embodiment of the present application has the following advantages: the Redis database is a memory-based database with high read and write performance, which can match the IO speed of the intermediate data storage module and the computing speed of the computing module; by using the Redis database, the read and write performance and memory space utilization of the massive small files contained in the computing tasks of double-density application data can be optimized; the Redis database provides a variety of persistence mechanisms, which is conducive to building a safe and reliable storage system; the Redis database adopts an IO multiplexing mode and event-driven mechanism, which can well support concurrent access, thereby better coping with burst access of the intermediate computing data storage sub-module; the Redis database can be organized into a high-availability cluster, thereby avoiding the problem of single point failure of the server.

[0053] In the embodiment of the present application, the communication module includes two communication modes: high throughput and low latency (TCP / IP). When the Spark computing module and the MPI computing module call each other, different communication modes are adopted between different modules to balance the requirements of throughput and high performance. Specifically, the communication modes between the communication modules include:

[0054] Internal communication within the Spark computing module: Data communication within the Spark computing module mainly includes the Spark computing module startup phase and the Spark computing paradigm execution phase. The internal communication of the Spark computing module supports batch computing. Therefore, the communication module uses a high-throughput communication mode to obtain the computing data required for the current task from the source data storage module.

[0055] Internal communication of MPI computing module: The internal communication of MPI computing module mainly includes the MPI computing module startup phase and the MPI computing paradigm execution phase. The internal communication of MPI computing module provides support for parallel computing, so the communication module adopts a low-latency communication mode; specifically: the MPI computing module accesses the intermediate data storage module through the communication module to read the task code, obtains the metadata information of the required computing data from the metadata storage module according to the task code, and issues a read data command. After receiving the command, the communication module establishes a low-latency communication connection with the intermediate data storage module according to the metadata information, and requests the required computing data from the intermediate data storage module; the intermediate data storage module sends the computing data to the MPI computing module through the established low-latency communication connection;

[0056] Data transmission between the Spark computing module and the MPI computing module: The key to integrating the Spark computing paradigm with the MPI computing paradigm is the data transmission between the Spark computing module and the MPI computing module. The embodiment of the present application divides the transmission data between the Spark computing module and the MPI computing module into two categories: distributed data and non-distributed data. Among them, non-distributed data refers to some commands and parameter data in the calling process. For non-distributed data, the communication module adopts the communication method of establishing a Socket between Drivers for data transmission; distributed data refers to computing data. For distributed data, the communication module adopts a distributed file system combined with a high-throughput communication mode for data transmission.

[0057] Communication between the Spark computing module and the storage module: Before executing the Spark computing paradigm, the original data required for the computing task is read from the source data storage module through the communication module; before calling the MPI computing module, the communication module communicates with the intermediate data storage module to store the task code and computing data required for the MPI computing module, and at the same time communicates with the metadata storage module to save the metadata information of the computing data; after the calculation is completed, it communicates with the source data storage module again and saves the calculation results.

[0058] Communication between the MPI computing module and the storage module: After the MPI computing module is started, it needs to access the intermediate data storage module through the communication module to obtain the task code to be executed, and access the metadata storage module according to the task code to obtain the metadata information of the computing data, and call the low-latency communication mode to issue a read data command. After receiving the command, the communication module establishes a low-latency communication connection with the intermediate data storage module according to the metadata information, and requests the required computing data from the intermediate data storage module. The intermediate data storage module sends the computing data to the MPI computing module through the established low-latency communication connection. After receiving the computing data, the MPI computing module sends a disconnect command to the communication module, and after the connection is disconnected, calls the MPI computing paradigm to perform task computing.

[0059] Based on the above, the hybrid big data computing platform of the embodiment of the present application provides better support for the computing tasks of dual-density application data by constructing an integrated storage and computing platform, so that the computing of dual-density application data has higher performance. The embodiment of the present application adopts a communication method combining a distributed memory file system with high-performance communication technology, which solves the memory shortage problem of the traditional shared memory-based communication method and the performance problem of socket-based communication. The embodiment of the present application combines a heterogeneous storage system to provide a storage mode tailored to the data characteristics of the computing tasks of dual-density application data, so that the storage performance can match the dual-paradigm hybrid computing and balance the relationship between big data storage and high-performance storage.

[0060] See also Figure 4 , is a flow chart of the hybrid big data calculation method of the embodiment of the present application. The hybrid big data calculation method of the embodiment of the present application includes the following steps:

[0061] S1: The user terminal submits the computing task of dual-density application data to the computing module through the application layer interface;

[0062] S2: Read the computing task through the Spark computing module, establish the Spark computing environment according to the task requirements of the computing task, and generate a directed acyclic graph of the computing task;

[0063] S3: The Spark computing module divides the computing tasks of dual-density application data into individual independent tasks based on a directed acyclic graph and stores the divided tasks in a task set.

[0064] S4: The Spark computing module takes a task from the task set, obtains the computing data required for the current task from the source data storage module through the communication module, and determines whether the task is data-intensive or computing-intensive. If it is a data-intensive task, execute S5; if it is a computing-intensive task, execute S6;

[0065] S5: The Spark computing module generates the corresponding RDD (Resilient Distributed Datasets) according to the task requirements, calls the Spark computing paradigm to perform the task calculation, and returns the calculation results;

[0066] S6: The Spark computing module partitions the acquired computing data, stores each partition data in the memory file system of a different Spark computing node, stores the corresponding relationship between the metadata information of the partition data and the Spark computing node in the metadata storage module, saves the task code required for execution by the MPI computing module in the intermediate code data storage submodule, stores the computing data required for the task in the intermediate computing data storage submodule, applies to the intermediate data storage module for a storage address for storing the computing results, returns the storage address to the MPI computing module, and calls the MPI computing module to execute the task computing;

[0067] Among them, since the Spark computing paradigm runs on the JVM (Java Virtual Machine), its main data structure is RDD, while the MPI computing paradigm runs on the bottom layer and its main programming language is C++, the Spark computing module needs to perform data serialization operations before calling the MPI computing module, that is, converting the data structure into a data format that the MPI computing module can recognize, in order to solve the problem of data format mismatch between the two paradigms. When performing data serialization operations, the embodiment of the present application uses a template design pattern to serialize data. Users can customize appropriate serialization methods for different applications and pass them to the system in the form of parameters, thereby solving the problem of heterogeneous multivariate data serialization while purposefully optimizing sequence performance and reducing performance losses caused by serialization operations. At the same time, the embodiment of the present application divides the data serialization operation into two stages. In the first stage, the Spark data structure is serialized into a format that is compatible with the intermediate data storage module. In the second stage, the data format of the intermediate data storage module is deserialized into a data format that can be recognized by the MMPI calculation module, thereby deconstructing the Spark data structure and the MPI data set format, reducing the types of serialization methods that need to be defined, increasing code reconstruction, and optimizing the storage format of the storage module, solving the problem of incomplete compatibility between the storage format of multi-heterogeneous data and the intermediate data storage module.

[0068] S7: The MPI computing module accesses the intermediate data storage module to read the task code, obtains metadata information of the required computing data from the metadata storage module according to the task code, and calls the low-latency communication mode of the communication module to issue a read data command;

[0069] S8: After receiving the command, the communication module establishes a low-latency communication connection with the intermediate data storage module according to the metadata information, and requests the required computing data from the intermediate data storage module;

[0070] S9: The intermediate data storage module sends the calculation data to the MPI calculation module through the established low-latency communication connection;

[0071] S10: After receiving the calculation data, the MPI calculation module sends a disconnection command to the communication module, and after the connection is disconnected, calls the MPI calculation paradigm to perform task calculation;

[0072] S11: After the task calculation is completed, the calculation result is saved to the intermediate data storage module according to the storage address returned by the Spark calculation module;

[0073] S12: Exit the MPI computing environment and re-execute S4 to determine the type and schedule the next task;

[0074] Among them, before the Spark computing module determines the type and schedules the next task, it needs to clear the data involved in this call in the intermediate computing data storage module and update the metadata information.

[0075] Based on the above, the calculation method of the hybrid big data computing platform of the embodiment of the present application can fully utilize the highly available user interface and mature big data processing mechanism provided by Spark by integrating the two computing paradigms of Spark and MPI, and can obtain the high performance of MPI in processing complex calculations, thereby balancing the delay problem of the high-throughput computing mode of big data, and adopting a data communication method that combines distributed shared memory with high-performance communication technology to solve the capacity problem of shared memory and the performance problem of Socket communication, so that the fusion architecture can better support the calculation of dual-density application data.

[0076] See also Figure 5 , is a schematic diagram of the device structure of an embodiment of the present application. The device 50 includes:

[0077] A memory 51 storing executable program instructions;

[0078] a processor 52 connected to the memory 51;

[0079] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: receiving the computing task of double-encrypted application data submitted by the user end through the Spark computing module, and obtaining the computing data of the computing task from the storage module through the communication module; judging whether the computing task is a data-intensive task or a computing-intensive task. If it is a data-intensive task, the Spark computing module generates the corresponding RDD according to the task requirements of the computing task, and then calls the Spark computing paradigm to perform task calculation; if it is a computing-intensive task, the task code and computing data of the computing task are stored in the storage module through the Spark computing module, and the MPI computing module is called through the communication module, the task code and computing data are read from the storage module through the MPI computing module, and the MPI computing paradigm is called to perform task calculation.

[0080] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0081] See also Figure 6 , Figure 6Schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 61 that can implement the following steps: receiving the computing task of the double-dense application data submitted by the user end through the Spark computing module, and obtaining the computing data of the computing task from the storage module through the communication module; judging whether the computing task is a data-intensive task or a computing-intensive task, if it is a data-intensive task, the Spark computing module generates the corresponding RDD according to the task requirements of the computing task, and then calls the Spark computing paradigm to perform task calculation; if it is a computing-intensive task, the task code and computing data of the computing task are stored in the storage module through the Spark computing module, and the MPI computing module is called through the communication module, and the task code and computing data are read from the storage module through the MPI computing module, and the MPI computing paradigm is called to perform task calculation. Among them, the program instructions 61 can be stored in the above-mentioned storage medium in the form of a software product, including several instructions for making a device (which can be a personal computer, server, or network device, etc.) or a processor (processor) perform all or part of the steps of the various embodiments of the present application. The aforementioned storage media include: various media that can store program instructions, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0083] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of this application, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A hybrid big data computing platform, characterized by: Includes a computing module, a communication module and a storage module; The computing module includes a Spark computing module and an MPI computing module. The Spark computing module is used to receive computing tasks of dual-density application data submitted by the user end, obtain computing data of the computing tasks from the storage module through the communication module, and identify the task type of the computing tasks, which includes data-intensive tasks and computing-intensive tasks. If it is a data-intensive task, the Spark computing module generates the corresponding RDD according to the task requirements and calls the Spark computing paradigm to perform task calculation; If it is a computationally intensive task, the Spark computing module stores the task code and computational data of the computational task in the storage module, and calls the MPI computing module through the communication module. The MPI computing module is used to read the task code and computational data from the storage module and call the MPI computing paradigm to perform task computation; wherein: The storage module includes a source data storage module, an intermediate data storage module and a metadata storage module. The source data storage module is based on the HDFS big data file system and is used to store the original data required for the computing task of the double-density application data; the intermediate data storage module is used to store the intermediate data during the communication process between the Spark computing module and the MPI computing module; and the metadata storage module is used to store metadata information of the computing data. The intermediate data includes dynamic data and static data, the dynamic data includes at least the task code required to be executed by the MPI computing module, and the static data includes at least the calculation data of the Spark computing module and the MPI computing module during the communication process; the intermediate data storage module includes an intermediate code data storage submodule and an intermediate computing data storage submodule, the intermediate code data storage submodule is used to store the static data in the form of a shared file, and the intermediate computing data storage submodule is used to store the dynamic data using the organizational structure of a distributed memory file system, and the metadata storage module acts as the server of the intermediate computing data storage submodule in a manner based on the Redis database.

2. The hybrid big data computing platform according to claim 1, characterized in that: The communication module includes a high-throughput communication mode and a low-latency communication mode. The communication process of the communication module includes: Internal communication of the Spark computing module: During the startup phase and the execution phase of the Spark computing module, the communication module uses a high-throughput communication mode to obtain the computing data required for the current task from the source data storage module; Internal communication of the MPI computing module: During the startup phase and the execution phase of the MPI computing paradigm of the MPI computing module, the MPI computing module accesses the intermediate data storage module through the low-latency communication mode of the communication module to read the task code, obtains metadata information of the required computing data from the metadata storage module according to the task code, establishes a low-latency communication connection with the intermediate data storage module according to the metadata information, and requests the required computing data from the intermediate data storage module; Data transmission between the Spark computing module and the MPI computing module: The data transmitted between the Spark computing module and the MPI computing module includes distributed data and non-distributed data. The non-distributed data refers to the command and parameter data in the calling process. The communication module uses the communication method of establishing a socket between drivers to perform non-distributed data transmission; the distributed data refers to the calculation data. The communication module uses a distributed file system combined with a high-throughput communication mode for data transmission; Communication between the Spark computing module and the storage module: Before executing the Spark computing paradigm, the Spark computing module reads the original data required for the computing task from the source data storage module through the communication module; before calling the MPI computing module, it communicates with the intermediate data storage module through the communication module to store the task code and computing data required for the MPI computing module to calculate, and communicates with the metadata storage module to save the metadata information of the computing data; after the calculation is completed, it communicates with the source data storage module and saves the calculation results; Communication between the MPI computing module and the storage module: After the MPI computing module is started, it accesses the intermediate data storage module through the communication module to obtain the task code to be executed, accesses the metadata storage module according to the task code to obtain metadata information of the computing data, establishes a low-latency communication connection with the intermediate data storage module according to the metadata information, and requests the required computing data from the intermediate data storage module. The intermediate data storage module sends the computing data to the MPI computing module through the low-latency communication connection.

3. A hybrid big data computing method, which is implemented by the hybrid big data computing platform according to any one of claims 1-2, characterized in that: include: Receive computing tasks of dual-encrypted application data submitted by a user terminal through the Spark computing module, and obtain computing data of the computing tasks from the storage module through the communication module; Determine whether the computing task is a data-intensive task or a computing-intensive task. If it is a data-intensive task, the Spark computing module generates a corresponding RDD according to the task requirements of the computing task, and then calls the Spark computing paradigm to perform task computing; If it is a computationally intensive task, the task code and computational data of the computational task are stored in the storage module through the Spark computation module, and the MPI computation module is called through the communication module. The task code and computational data are read from the storage module through the MPI computation module, and the MPI computation paradigm is called to perform the task computation.

4. The hybrid big data calculation method according to claim 3, characterized in that: The Spark computing module receives the computing task of the double-encrypted application data submitted by the user, and obtains the computing data of the computing task from the storage module through the communication module, specifically: The user terminal submits a computing task of double-encrypted application data through an application layer interface; Reading the computing task through the Spark computing module, establishing a Spark computing environment according to the task requirements of the computing task, and generating a directed acyclic graph of the computing task; Dividing the computing task into individual independent tasks according to the directed acyclic graph, and storing the divided tasks into a task set; A task is taken out from the task set, and the computing data required for the current task is obtained from the source data storage module through the communication module.

5. The hybrid big data calculation method according to claim 4, characterized in that: The specific steps of storing the task code and calculation data of the computing task into the storage module through the Spark computing module are as follows: The obtained calculation data is partitioned through the Spark calculation module, and each partition data is stored in the memory file system of different Spark calculation nodes. The correspondence between the metadata information of the partition data and the Spark calculation node is stored in the metadata storage module. The task code required to be executed by the MPI calculation module is saved in the intermediate code data storage submodule, and the calculation data required for the task is stored in the intermediate calculation data storage submodule. The storage address for storing the calculation results is applied to the intermediate data storage module, the storage address is returned to the MPI calculation module, and the MPI calculation module is called to perform the task calculation.

6. The hybrid big data calculation method according to claim 5, characterized in that: The MPI calculation module reads the task code and calculation data from the storage module, and calls the MPI calculation paradigm to perform task calculation as follows: The MPI computing module accesses the intermediate data storage module through the communication module to read the task code, obtains metadata information of the required computing data from the metadata storage module according to the task code, and calls the low-latency communication mode of the communication module to issue a read data command; After receiving the command, the communication module establishes a low-latency communication connection with the intermediate data storage module according to the metadata information, and requests the required calculation data from the intermediate data storage module; The intermediate data storage module sends calculation data to the MPI calculation module through the established low-latency communication connection. After receiving the calculation data, the MPI calculation module sends a disconnect command to the communication module and calls the MPI calculation paradigm to perform task calculation; After the task calculation is completed, the calculation result is saved to the intermediate data storage module according to the storage address returned by the Spark calculation module.

7. A device, characterized in that The device includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the hybrid big data computing method according to any one of claims 3 to 6; The processor is used to execute the program instructions stored in the memory to control the hybrid big data computing platform.

8. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the hybrid big data computing method described in any one of claims 3 to 6.

Images