Multi-framework collaborative database analysis system

Abstract

The invention discloses a multi-framework collaborative database analysis system, which is provided with a storage framework, a computing framework and a middle layer, and data transmission is carried out between the middle layer and the storage framework through a predefined communication protocol. Data transmission is carried out between the intermediate layer and the computing framework through a predefined interaction protocol, and the multi-framework collaborative database analysis system executes the following steps S1 to S6 when working: S1: creating a computing graph; s2, data request and preparation; s3, data type conversion; s4, data transmission and calculation processing; s5, receiving, converting and storing a result; and S6, finally outputting a result.

Description

Technical Field

[0001] The present invention relates to the field of multi-framework collaboration, and in particular to a multi-framework collaborative database analysis system. Background Art

[0002] With the rapid development of information technology, database analysis systems have become an indispensable tool in the fields of enterprise decision-making, scientific research, etc. Traditional database analysis systems usually rely on a single storage framework and computing framework, which often faces performance bottlenecks and insufficient flexibility when processing large-scale and complex data.

[0003] In recent years, with the rise of cloud computing, big data, artificial intelligence and other technologies, database analysis systems that work in collaboration with multiple frameworks have gradually become a research hotspot. By integrating the advantages of different frameworks, these systems can break through the performance limitations of a single framework and provide more efficient and flexible data processing and analysis capabilities. Summary of the invention

[0004] The present invention provides a multi-framework collaborative database analysis system, which comprises a storage framework, a computing framework and an intermediate layer. The intermediate layer and the storage framework perform data transmission through a predefined communication protocol, and the intermediate layer and the computing framework perform data transmission through a predefined interactive protocol.

[0005] The multi-framework collaborative database analysis system performs the following steps S1 to S6 when working:

[0006] S1: Computational graph creation

[0007] The middle layer creates a calculation graph based on the instructions of the master control system or the user's request. The calculation graph defines multiple calculation steps to be executed, data conversion logic, and the calculation framework used.

[0008] S2: Data Request and Preparation

[0009] S21: The middle layer obtains the data required for the first calculation step in the calculation graph from the first storage framework;

[0010] S22: The first storage framework responds to the request, prepares data and sends it to the middle layer;

[0011] S3: Data type conversion

[0012] S31: The middle layer receives data from the first storage framework;

[0013] S32: The middle layer converts the data from the format of the first storage framework to the format required by the first computing framework;

[0014] S4.:Data transmission and computing processing

[0015] S41: The middle layer sends the converted data to the first computing framework;

[0016] S42: The first computing framework executes the first computing step in the computing graph and returns the result to the middle layer;

[0017] S5: Result reception, conversion and storage

[0018] S51: The middle layer receives result data of the first computing framework;

[0019] S52: If necessary, the middle layer converts the result data into a format required by the second storage framework;

[0020] S53: The middle layer sends the result data or the location information of the result data to the second storage framework for storage;

[0021] S6: Final output results

[0022] When all the calculation steps in the calculation graph are executed, the middle layer sends the final result data or the location information of the result data to the master control system or the user.

[0023] In one embodiment of the present invention, a computational graph defines the entire workflow of a multi-framework collaborative database analysis system. The computational graph is a directed acyclic graph. There is no path in the computational graph that starts from a vertex, passes through a series of edges, and then returns to the vertex. Each edge of the computational graph points from one vertex to another, indicating a unidirectional relationship or dependency. The computational graph does not allow cycles, that is, any node in the computational graph can only be passed through once.

[0024] In one embodiment of the present invention, each node in the computation graph corresponds to a basic unit, and each basic unit includes the following definition items: index, task type, execution framework, and task details. The meaning of each definition item is as follows:

[0025] Index: The index of the current node in the entire computational graph, which increases sequentially starting from 1;

[0026] Task type: the type of task that the current node needs to complete, including downloading, analyzing, and uploading;

[0027] Execution framework: the working framework of the current node. Each execution framework has a unique identifier in the multi-framework collaborative database analysis system;

[0028] Task details: Detailed configuration of the current node task.

[0029] In one embodiment of the present invention, the middle layer customizes an intermediate data type and performs data type conversion by creating a mapping relationship.

[0030] The steps for the middle layer to upload and distribute data to the storage framework or computing framework are as follows s1~s3:

[0031] s1: The middle layer maps the existing data and converts it into data types supported by the storage framework or computing framework;

[0032] s2: The middle layer sends a message to the storage framework or computing framework to inform the storage framework or computing framework of the location of the data;

[0033] s3: The storage framework or computing framework retrieves data from the middle layer through network transmission;

[0034] The steps of the storage framework or computing framework downloading and collecting data to the middle layer are as follows s4 to s7:

[0035] s4: The storage framework or computing framework sends a message to the middle layer to inform the storage framework or computing framework of the location of the data;

[0036] s5: The middle layer retrieves data through network transmission;

[0037] s6: The middle layer maps the retrieved data and converts it into an intermediate data type;

[0038] s7: Transmit to the next node according to the computational graph requirements;

[0039] After receiving the completion message of the previous node, the middle layer sends a task request to the execution frame where the next node is located according to the information of the current node. The task request must contain the index of the node that needs to be executed now;

[0040] After receiving the task request from the middle layer, the working framework executes the current task according to the node index. After completing the task, it sends a completion message to the middle layer.

[0041] In one embodiment of the present invention, the computing framework integrates at least one acceleration library, the acceleration library includes OpenBLAS and ViennaCL, the computing framework is customizable and extensible and supports distributed deployment and load balancing.

[0042] The multi-framework collaborative database analysis system provided by the present invention achieves the following beneficial technical effects:

[0043] 1. Adaptability of the middle layer framework: The present invention provides a solution based on the middle layer, which can flexibly customize and add the required computing framework (or storage framework) through adaptation work, thereby enhancing the scalability and adaptability of the system.

[0044] 2. Decoupling of computing and storage: The present invention integrates the storage database and computing framework, breaking the limitation of computing and storage being bound to each other in traditional systems. By leveraging their respective advantages, computing and storage are no longer restricted to a single framework, thus improving the flexibility and efficiency of the system.

[0045] 3. Wide support for different analysis tasks: The present invention achieves wide support for a variety of frameworks suitable for completing different tasks (such as traditional machine learning, deep learning neural networks, database query and screening, data analysis, etc.) by defining intermediate data types and unified interfaces. This design enables the present invention to serve as an efficient and flexible platform to provide an integrated solution for various analysis tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 is a schematic diagram of a calculation graph according to an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of a basic unit;

[0049] Figure 3 It is a schematic diagram of the execution of the computation graph;

[0050] Figure 4 This is a schematic diagram of the data flow between the computing framework, storage framework, and middle layer. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The main technologies used in the present invention are as follows:

[0053] (1) General large-scale computing framework

[0054] In modern database analysis systems, general-purpose large-scale computing frameworks play a vital role. These frameworks achieve rapid processing and analysis of large-scale data by optimizing data processing processes, utilizing distributed computing resources, and in-memory computing technologies. These frameworks usually support multiple programming interfaces and computing models to adapt to different application scenarios and requirements.

[0055] They are able to handle a variety of tasks from simple queries to complex data analysis while ensuring high reliability and fault tolerance of data. By leveraging the computing power of the cluster, these frameworks are able to process large amounts of data in parallel, increase processing speed, and reduce the load on a single node.

[0056] (2) Database storage system

[0057] In database systems, distributed storage technology is the key to efficient data processing and analysis. By adopting a distributed storage system, data can be stored in different nodes, thereby improving the scalability and fault tolerance of data.

[0058] These database systems usually have the characteristics of high performance, high reliability and easy scalability. They can support the storage and access of massive data, while providing efficient data management and query functions. By utilizing distributed storage technology, database analysis systems can achieve rapid access and analysis of large-scale data, providing users with better services.

[0059] (3) Multi-frame collaboration technology

[0060] Multi-framework collaboration technology aims to achieve efficient data processing and storage by integrating different computing frameworks (such as Spark, TensorFlow, etc.) and storage frameworks (such as Hadoop, Hive, S3, etc.). This technology can make full use of the strengths of different frameworks to improve the efficiency and flexibility of data processing and analysis. For example, use Spark to download data from Hive for calculation and store the calculation results in Hadoop.

[0061] (4) Middle layer technology

[0062] In the database analysis system, the middle layer is a crucial component. It connects the client, computing framework (such as Spark, etc.) and storage framework (such as Hadoop, etc.), and plays the role of a bridge and link.

[0063] 1. The middle layer supports efficient and reliable data transmission between the storage framework and the computing framework. By implementing interfaces and data exchange protocols between various frameworks, the middle layer can ensure seamless data transmission between different frameworks and reduce delays and errors in the data transmission process;

[0064] 2. The middle layer provides the ability to convert data types, which can convert specific data types of different frameworks into unified middle layer data types. This conversion not only solves the compatibility problem of data between different frameworks, but also simplifies the data processing process and improves the efficiency and accuracy of data processing.

[0065] 3. The middle layer can perform certain pre-processing and post-processing on the transmitted data, such as data cleaning, data conversion, data aggregation, etc. These processing operations can be customized according to business needs to meet the needs of different analysis scenarios.

[0066] 4. The middle layer has load balancing and fault tolerance capabilities, and can dynamically adjust the distribution of data transmission and processing tasks according to the system load conditions to ensure the stability and reliability of the system.

[0067] 5. The middle layer adopts modular and component-based design concepts. New functions and components can be easily added to the middle layer to meet the continuous development and changes of the system. At the same time, the middle layer provides complete monitoring and management functions to facilitate administrators to manage and maintain the system.

[0068] The present invention provides a multi-framework collaborative database analysis system, which comprises a storage framework, a computing framework and an intermediate layer. The intermediate layer and the storage framework perform data transmission through a predefined communication protocol, and the intermediate layer and the computing framework perform data transmission through a predefined interactive protocol.

[0069] The multi-framework collaborative database analysis system performs the following steps S1 to S6 when working:

[0070] S1: Computational graph creation

[0071] The middle layer creates a calculation graph based on the instructions of the master control system or the user's request. The calculation graph defines multiple calculation steps to be executed, data conversion logic, and the calculation framework used.

[0072] S2: Data Request and Preparation

[0073] S21: The middle layer obtains the data required for the first calculation step in the calculation graph from the first storage framework;

[0074] S22: The first storage framework responds to the request, prepares data and sends it to the middle layer;

[0075] S3: Data type conversion

[0076] S31: The middle layer receives data from the first storage framework;

[0077] S32: The middle layer converts the data from the format of the first storage framework to the format required by the first computing framework;

[0078] S4.:Data transmission and computing processing

[0079] S41: The middle layer sends the converted data to the first computing framework;

[0080] S42: The first computing framework executes the first computing step in the computing graph and returns the result to the middle layer;

[0081] S5: Result reception, conversion and storage

[0082] S51: The middle layer receives result data of the first computing framework;

[0083] S52: If necessary, the middle layer converts the result data into a format required by the second storage framework;

[0084] S53: The middle layer sends the result data or the location information of the result data to the second storage framework for storage;

[0085] S6: Final output results

[0086] When all the calculation steps in the calculation graph are executed, the middle layer sends the final result data or the location information of the result data to the master control system or the user.

[0087] In the present invention, the storage framework is mainly responsible for the persistent storage and management of data. Generally speaking, the present invention can directly use mature traditional database frameworks (such as Hive, MySQL, etc.) to achieve this function. These database frameworks provide efficient and stable data storage and query mechanisms to ensure the reliability and security of data. In actual use, users can perform preliminary screening and query operations on stored data by writing SQL statements. The required information is extracted from the data through SQL statements. Subsequently, the system will transfer the results to the middle layer for subsequent calculation and analysis tasks. The storage framework is also responsible for storing the analysis results. When the calculation and analysis tasks are completed, the middle layer will transfer the result data back to the storage framework for persistent storage. In this way, users can access and query these analysis results at any time for subsequent data analysis and decision support.

[0088] The computing framework undertakes the core computing and analysis tasks in the present invention, and is an important basis for realizing data processing and intelligent analysis. The present invention generally adopts ready-made mature and efficient computing frameworks, such as Spark, Eigen, Pytorch, etc., to support diverse analysis tasks from clustering algorithms, neural networks to support vector machines. In order to further improve the computing performance, the present invention can further integrate a variety of acceleration libraries in the computing framework, such as OpenBLAS, ViennaCL, etc. These acceleration libraries enable the data in the database to be processed using acceleration devices such as GPU, or to be accelerated by optimization algorithms such as vectorization and parallel computing, thereby significantly improving the computing efficiency in most cases. These computing frameworks have powerful computing power and excellent scalability, can easily cope with the processing requirements of large-scale data sets, and perform complex computing and analysis tasks. They provide a rich algorithm library and tool set, support various machine learning, deep learning, data mining and other tasks, and ensure that the system can efficiently and accurately complete various analysis requirements.

[0089] In addition to supporting a variety of analytical tasks, the computing framework is also highly customizable and scalable. Users can customize computing processes and algorithms according to actual business needs to meet the requirements of different business scenarios. In addition, the computing framework also supports distributed deployment and load balancing, which can fully utilize computing resources, improve the overall performance and throughput of the system, and ensure efficient and stable operation under high concurrency and large data volumes.

[0090] The middle layer module is the key bridge connecting the computing framework and the storage framework. Its core role is to solve the problem of data incompatibility between different frameworks (especially between the computing framework and the storage framework). The middle layer module involves two main protocols: one is the communication protocol with the storage framework, and the other is the interaction protocol with the computing framework.

[0091] 1. Protocol with the storage framework: When the middle layer needs to access data in the storage framework, it sends a request to the storage framework. Once the storage framework prepares the data, it returns the data (or the location information of the data) to the middle layer. This mechanism (location information of the data) is particularly important when processing large amounts of data because it can effectively improve system performance.

[0092] 2. Protocol with the computing framework: When the middle layer needs to call the computing framework for calculation, it sends the computing request and the required data (or data location) to the computing framework. After the computing framework completes the calculation, it returns the result data (or result data location) to the middle layer. Then, the middle layer can directly output the result or store it in the storage framework as needed.

[0093] The main functions of the middle layer module are reflected in the following two aspects:

[0094] 1. Data type conversion: The middle layer defines a universal data type and can realize the conversion between this data type and the data types between different frameworks. This conversion capability ensures the seamless flow of data between different frameworks, thereby improving data reusability and system flexibility.

[0095] 2. Calls between frameworks: After receiving the command from the master control system, the middle layer will build a calculation graph. This calculation graph plans in detail the data transmission path and the frameworks that should be called in each link, ensuring the efficiency and order of the entire calculation process.

[0096] Figure 1 is a schematic diagram of a calculation graph of an embodiment of the present invention, such as Figure 1 As shown in the figure, the computational graph defines the entire workflow of the database analysis system with multi-framework collaboration. The computational graph is a directed acyclic graph. There is no path in the computational graph that starts from a vertex, passes through a series of edges, and then returns to the vertex. Each edge of the computational graph points from one vertex to another, indicating a unidirectional relationship or dependency. The computational graph does not allow loops, that is, any node in the computational graph can only be passed once.

[0097] In one embodiment of the present invention, each node in the computation graph corresponds to a basic unit. Figure 2 is a schematic diagram of the basic unit, such as Figure 2 As shown, each basic unit includes the following definition items: index, task type, execution framework and task details. The meaning of each definition item is as follows:

[0098] Index: The index of the current node in the entire computational graph, which increases sequentially starting from 1;

[0099] Task type: the type of task that the current node needs to complete, including downloading, analyzing, and uploading;

[0100] Execution framework: the working framework of the current node. Each execution framework has a unique identifier in the multi-framework collaborative database analysis system;

[0101] Task details: detailed configuration of the current node task. The task details have a high degree of customization and can be defined according to actual needs. Those skilled in the art can define them independently according to their needs, and the present invention will not go into details about this.

[0102] Figure 3 It is a schematic diagram of the execution of the computation graph. Figure 3 In the figure, from top to bottom, the indexes are 1 to 3, which correspond to nodes 1 to 3 respectively. Figure 3 The computational graph execution flow shown is as follows:

[0103] Step 1: The middle layer sends the computational graph node message of the corresponding node to each execution framework according to the execution framework;

[0104] Step 2: Execute node 1. The middle layer sends a task request for node 1 to execution framework A. At this time, execution framework A starts to execute the task. After the execution is completed, execution framework A sends a message to the middle layer, indicating that the task has been completed. The middle layer then gets the result back and performs data conversion.

[0105] Step 3: After the middle layer completes data preparation (data conversion in the second step), it sends a task request for node 2 to execution framework B. At this time, execution framework B starts to execute the task. After the execution is completed, execution framework B sends a message to the middle layer, indicating that the task has been completed. The middle layer then gets the result back and performs data conversion.

[0106] Step 4: After the middle layer completes data preparation (data conversion in the third step), it sends a task request for node 3 to execution framework A. At this time, execution framework A starts to execute the task. After the execution is completed, execution framework A sends a message to the middle layer. Because the next node is the end, the middle layer does not take back the data and the task ends.

[0107] Figure 4 This is a schematic diagram of the data flow between the computing framework, storage framework, and middle layer, such as Figure 4 As shown, the middle layer can download data from the storage framework or upload data to the storage framework; the middle layer can distribute data to the computing framework or collect data (computation results) from the computing framework.

[0108] In one embodiment of the present invention, the middle layer customizes an intermediate data type and performs data type conversion by creating a mapping relationship.

[0109] The steps for the middle layer to upload and distribute data to the storage framework or computing framework are as follows s1~s3:

[0110] s1: The middle layer maps the existing data and converts it into data types supported by the storage framework or computing framework;

[0111] s2: The middle layer sends a message to the storage framework or computing framework to inform the storage framework or computing framework of the location of the data;

[0112] s3: The storage framework or computing framework retrieves data from the middle layer through network transmission;

[0113] The steps of the storage framework or computing framework downloading and collecting data to the middle layer are as follows s4 to s7:

[0114] s4: The storage framework or computing framework sends a message to the middle layer to inform the storage framework or computing framework of the location of the data;

[0115] s5: The middle layer retrieves data through network transmission;

[0116] s6: The middle layer maps the retrieved data and converts it into an intermediate data type;

[0117] s7: Transmit to the next node according to the computational graph requirements;

[0118] After receiving the completion message of the previous node, the middle layer sends a task request to the execution frame where the next node is located according to the information of the current node. The task request must contain the index of the node that needs to be executed now;

[0119] After receiving the task request from the middle layer, the working framework executes the current task according to the node index. After completing the task, it sends a completion message to the middle layer.

[0120] In one embodiment of the present invention, the computing framework integrates at least one acceleration library, the acceleration library includes OpenBLAS and ViennaCL, the computing framework is customizable and extensible and supports distributed deployment and load balancing.

[0121] The key technical points of the present invention are as follows:

[0122] 1. Framework integration and storage and computing decoupling: This invention innovatively solves the problem of decoupling storage and computing by designing multiple framework integration methods. This design enables storage and computing functions to run and optimize independently of each other, improving the flexibility and scalability of the system.

[0123] 2. Design of intermediate data types: In order to solve the problem of data format differences between different frameworks, the present invention designs intermediate data types. This data type serves as a bridge for data interaction between different frameworks, ensuring the universality and consistency of data, so that data can be efficiently transmitted and processed between different frameworks.

[0124] 3. Middle layer integration framework: The present invention introduces the design of the middle layer, through which different storage frameworks and computing frameworks can be more easily integrated into the analysis system. The middle layer provides a unified interface and data protocol, simplifies the interaction process between frameworks, and reduces the complexity of system development and maintenance.

[0125] The multi-framework collaborative database analysis system provided by the present invention achieves the following beneficial technical effects:

[0126] 1. Adaptability of the middle layer framework: The present invention provides a solution based on the middle layer, which can flexibly customize and add the required computing framework (or storage framework) through adaptation work, thereby enhancing the scalability and adaptability of the system.

[0127] 2. Decoupling of computing and storage: The present invention integrates the storage database and computing framework, breaking the limitation of computing and storage being bound to each other in traditional systems. By leveraging their respective advantages, computing and storage are no longer restricted to a single framework, thus improving the flexibility and efficiency of the system.

[0128] 3. Wide support for different analysis tasks: The present invention achieves wide support for a variety of frameworks suitable for completing different tasks (such as traditional machine learning, deep learning neural networks, database query and screening, data analysis, etc.) by defining intermediate data types and unified interfaces. This design enables the present invention to serve as an efficient and flexible platform to provide an integrated solution for various analysis tasks.

[0129] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0130] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-framework collaborative database analysis system, characterized in that: The multi-framework collaborative database analysis system has a storage framework, a computing framework, and an intermediate layer. The intermediate layer and the storage framework transmit data through a pre-defined communication protocol, and the intermediate layer and the computing framework transmit data through a pre-defined interaction protocol. The multi-framework collaborative database analysis system performs the following steps S1 to S6 when working: S1: Computational graph creation The middle layer creates a calculation graph based on the instructions of the master control system or the user's request. The calculation graph defines multiple calculation steps to be executed, data conversion logic, and the calculation framework used. S2: Data Request and Preparation S21: The middle layer obtains the data required for the first calculation step in the calculation graph from the first storage framework; S22: The first storage framework responds to the request, prepares data and sends it to the middle layer; S3: Data type conversion S31: The middle layer receives data from the first storage framework; S32: The middle layer converts the data from the format of the first storage framework to the format required by the first computing framework; S4.:Data transmission and computing processing S41: The middle layer sends the converted data to the first computing framework; S42: The first computing framework executes the first computing step in the computing graph and returns the result to the middle layer; S5: Result reception, conversion and storage S51: The middle layer receives result data of the first computing framework; S52: If necessary, the middle layer converts the result data into a format required by the second storage framework; S53: The middle layer sends the result data or the location information of the result data to the second storage framework for storage; S6: Final output results When all the calculation steps in the calculation graph are executed, the middle layer sends the final result data or the location information of the result data to the master control system or the user.

2. The multi-framework collaborative database analysis system according to claim 1, characterized in that: The computational graph defines the entire workflow of a multi-framework collaborative database analysis system. The computational graph is a directed acyclic graph. There is no path in the computational graph that starts from a vertex, passes through a series of edges, and then returns to the vertex. Each edge of the computational graph points from one vertex to another, indicating a unidirectional relationship or dependency. The computational graph does not allow cycles, that is, any node in the computational graph can only be passed through once.

3. The multi-framework collaborative database analysis system according to claim 2, characterized in that: Each node in the computational graph corresponds to a basic unit. Each basic unit includes the following definition items: index, task type, execution framework, and task details. The meaning of each definition item is as follows: Index: The index of the current node in the entire computational graph, which increases sequentially starting from 1; Task type: the type of task that the current node needs to complete, including downloading, analyzing, and uploading; Execution framework: the working framework of the current node. Each execution framework has a unique identifier in the multi-framework collaborative database analysis system; Task details: Detailed configuration of the current node task.

4. The multi-framework collaborative database analysis system according to claim 3, characterized in that: The middle layer customizes an intermediate data type and converts the data type by creating a mapping relationship. The steps for the middle layer to upload and distribute data to the storage framework or computing framework are as follows s1~s3: s1: The middle layer maps the existing data and converts it into data types supported by the storage framework or computing framework; s2: The middle layer sends a message to the storage framework or computing framework to inform the storage framework or computing framework of the location of the data; s3: The storage framework or computing framework retrieves data from the middle layer through network transmission; The steps of the storage framework or computing framework downloading and collecting data to the middle layer are as follows s4 to s7: s4: The storage framework or computing framework sends a message to the middle layer to inform the storage framework or computing framework of the location of the data; s5: The middle layer retrieves data through network transmission; s6: The middle layer maps the retrieved data and converts it into an intermediate data type; s7: Transmit to the next node according to the computational graph requirements; After receiving the completion message of the previous node, the middle layer sends a task request to the execution frame where the next node is located according to the information of the current node. The task request must contain the index of the node that needs to be executed now; After receiving the task request from the middle layer, the working framework executes the current task according to the node index. After completing the task, it sends a completion message to the middle layer.

5. The multi-framework collaborative database analysis system according to any one of claims 1 to 4, characterized in that: The computing framework integrates at least one acceleration library, including OpenBLAS and ViennaCL. The computing framework is customizable and extensible and supports distributed deployment and load balancing.

Images