Graph computing platform based on cloud-native technology

Abstract

Disclosed in the present invention is a graph computing platform based on cloud-native technology. The platform comprises a software and hardware collaborative system, a graph storage system, a graph computing engine and a graph development workshop system, which are designed by using a cloud-native architecture, wherein the software and hardware collaborative system, the graph storage system and the graph computing engine run in the backend, and a graph development workshop runs in the frontend. The software and hardware collaborative system provides hardware computing resources and a software and hardware adaptation environment; the graph storage system comprises a graph division module, a distributed persistent storage module and a distributed shared memory pool module; the graph computing engine comprises a graph query engine, a graph analysis engine and a graph learning engine; and the graph development workshop system is a visual operating system which is constructed on the basis of the graph computing engine and is for the field of graph computing, and the graph development workshop system is deployed on the basis of K8s containerization technology, and is used for modularizing, streamlining and visualizing the whole development process of a graph computing algorithm. The platform in the present invention has clear architecture, strong expandability, is easy to learn to use, and has high computing efficiency.

Description

A graph computing platform based on cloud native technology Technical Field

[0001] The present invention relates to the field of graph computing, and in particular to a graph computing platform based on cloud native technology. Background Art

[0002] The development of graph theory can be traced back to the 18th century, when Euler proposed the famous "Seven Bridges of Königsberg Problem," sparking a wave of research in graph theory that has gradually become an independent discipline over time. Euclidean data has a regular distribution and a fixed structure, making it inflexible in representing complex relationships between objects. Graph structures in non-Euclidean space can represent the complex relationships between all things and possess powerful data representation capabilities. In recent years, graph computing research has been one of the hottest areas in artificial intelligence. Graph computing is considered a core element in driving artificial intelligence from the "perceptual intelligence" stage to the "cognitive intelligence" stage, with diverse application scenarios including social network analysis, bioinformatics, road planning, financial risk control, and recommendation systems. As the infrastructure for graph computing, graph computing platforms play a crucial role in facilitating the efficient development and implementation of graph algorithms.

[0003] Graph computing platforms generally include graph data storage and query, distributed graph analysis algorithms, graph neural network models and training. The computing modes are diverse and involve the coordination of multiple complex systems. The threshold for users to learn and use is high. At the same time, because the amount of graph data is generally large, the scale of the graph will affect the efficiency of the computing framework. To reduce the difficulty of implementation, the general practice is to design according to the needs and characteristics of specific scenarios.

[0004] The mixed execution behavior of the graph aggregation and graph update phases leads to differences in memory access patterns, data reuse rates, computing patterns, computing intensity, and execution constraints. This makes efficient graph computing extremely complex, and its challenges can be mainly divided into two aspects: computing and memory access. 1) Computing: It needs to be able to efficiently handle both irregular and dense regular computing. Because the nodes in the graph are extremely irregular and follow a power-law distribution, traversing neighbor nodes during the graph aggregation phase will lead to serious load imbalance. 2) Memory access: It needs to be able to efficiently handle both irregular and regular coarse-grained memory access and high bandwidth requirements. These two problems limit the computational scale and efficiency of graph computing.

[0005] Graph learning algorithms enable machine learning to be applied to graph structures in non-Euclidean spaces, enabling the ability to learn from graphs. However, the graph computing platforms currently developed by the industry are limited by their respective business scenarios, with each subsystem being fragmented. Some platforms focus on the storage and query analysis of graph data, while others excel at graph analysis algorithms. Most existing graph learning platforms are suitable for supervised and centralized business scenarios. They all address some of the issues in graph computing and lack streamlined, visual system support for the overall graph computing R&D cycle. Furthermore, current graph computing platforms offer limited support for data and algorithms in scientific computing, and their support for domestically produced hardware in bottleneck areas is incomplete, creating a high barrier to entry for non-professionals to conduct R&D in graph computing.

[0006] Summary of the Invention

[0007] Based on the common problems of complex systems, high usage barriers and low computing efficiency in current graph computing systems, the present invention proposes a graph computing platform with cloud native technology. The platform can use hardware virtualization technology to support hardware resources and graph computing chips to provide heterogeneous computing capabilities coordinated by software and hardware. It solves the I / O bottleneck in graph computing based on a shared memory pool, provides the main computing engine system and storage engine system in the graph computing platform, and performs task scheduling and resource management of graph computing through containerized deployment of various subsystems. It provides a one-stop visual graph development workshop, which allows both personnel with no graph algorithm development background and senior graph algorithm personnel to complete graph data modeling, query analysis, graph algorithm training, model deployment and other tasks through componentized and process-based operations, supporting the research and development of field applications.

[0008] The purpose of the present invention is achieved through the following technical solutions:

[0009] A graph computing platform based on cloud-native technology, comprising a software and hardware collaborative system, a graph storage system, a graph computing engine, and a graph development workshop system designed using a cloud-native architecture; wherein the software and hardware collaborative system, the graph storage system, and the graph computing engine run on the backend, and the graph development workshop runs on the frontend;

[0010] The software and hardware collaborative system provides hardware computing resources and a software and hardware adaptation environment. The hardware computing resources include various central processors, data processors, FPGA accelerators, and graph computing chips. The software and hardware adaptation environment includes a data flow acceleration library and an operator acceleration library that accelerate graph computing algorithms based on heterogeneous hardware resources.

[0011] The graph storage system includes a graph partitioning module, a distributed persistent storage module, and a distributed shared memory pool module. The graph partitioning module has multiple built-in graph partitioning algorithms. The distributed persistent storage module is used to store graph data on disk through a distributed file system and provides interfaces and services for reading and writing graph data. The distributed shared memory pool module is used to manage the global memory space of the host memory and GPU display memory on multiple computing nodes based on remote direct data access technology, thereby realizing distributed memory sharing in the cluster.

[0012] The graph computing engine includes a graph query engine, a graph analysis engine, and a graph learning engine, which are respectively used to implement direct query of graph-related data, distributed graph analysis, and training of parallelized graph learning models;

[0013] The graph development workshop system is a visual operating system for the graph computing field built based on the graph computing engine. It is deployed with K8s containerization technology and is used to componentize, streamline and visualize the entire development process of the graph computing algorithm.

[0014] Furthermore, the hardware and software collaborative system supports scalable hardware devices including GPU processors, CPU processors, FPGA acceleration cards, and storage-computing integrated chips, develops corresponding compilation environments and software tools, provides storage-computing integrated operators for graph data, and operator acceleration modules adapted to specific hardware, providing heterogeneous computing capabilities for the graph computing engine.

[0015] Furthermore, the distributed persistent storage module is implemented based on NVMe SSD technology; the distributed shared memory pool module is implemented based on RDMA technology, and data synchronization is performed based on the distributed persistent storage module and the distributed file system.

[0016] Furthermore, direct queries of graph association data in the graph query engine include multi-order neighbor queries of a single point, association path queries between two points, and subgraph queries for obtaining associations between multiple points.

[0017] Furthermore, the graph query engine includes a graph query parser and a graph query executor. The graph query parser is used to parse the query statement input by the user, build a query syntax tree and generate an execution plan; the graph query executor is used to optimize the syntax tree produced by the graph query parser, and then perform resource scheduling and plan execution according to resource quotas, using distributed parallel computing technology to process large-scale graph data in the graph storage system in a manner that minimizes running time and resource consumption.

[0018] Furthermore, the graph analysis engine includes a graph analysis algorithm library and a graph analysis computing module. The graph analysis algorithm library contains efficient data structures for graph data processing as well as various general functions and class libraries, providing users with a programmable tool set that allows users to implement and test graph analysis algorithms, as well as change and adjust the parameters of graph analysis algorithms, thereby optimizing and improving the performance of graph analysis algorithms.

[0019] The graph analysis computing module runs on a distributed system and includes a series of underlying operators and data processing logic that can process graph structures, supports communication and collaboration between multiple nodes, and combines the computing power provided by hardware computing resources to realize distributed computing and parallel computing of graph analysis algorithms.

[0020] Furthermore, the graph learning engine includes a graph learning algorithm library and a graph learning training module. The graph learning algorithm library is a software toolkit for building and training graph neural network models, including various different graph data sets and pre-trained graph neural network models; the graph learning training module is used to sequentially implement feature extraction and representation, subgraph sampling, model construction and selection, training strategy and parameter adjustment, and model evaluation and tuning.

[0021] Furthermore, the graph development workshop system includes a graph data modeling module, a graph data visualization module, a graph algorithm component module, a graph training process construction module and a graph model deployment module;

[0022] The graph data modeling module is used to define a standard storage structure for graph data, abstract real-world objects into nodes and edges in a graph, and use the structure and properties of the graph to describe the connections and features between objects;

[0023] The graph data visualization module is used to visualize the result data after graph query, graph analysis and graph learning;

[0024] The graph algorithm component module is used for the implementation of model code and related algorithms for model training and reasoning, including graph data conversion components, feature engineering components, graph algorithm operator components, GNN model training and evaluation components, parameter optimization components, and architecture search components;

[0025] The graph training process construction module uses automated tools to construct and manage the processing steps of data preprocessing, feature engineering, model training, and model evaluation in a pipeline manner, forming a complete and repeatable data processing process;

[0026] The graph model deployment module is used to deploy the trained graph algorithm model to a remote server to implement model deployment, model monitoring, configuration management and model reasoning services, and provide support for business systems or end users.

[0027] Furthermore, visualization methods include: adjusting node coloring and size, edge drawing and coloring, clustering and subgraphs, and interactive control.

[0028] Furthermore, visualization is performed through clustering and subgraphs, specifically including dividing nodes into different groups based on the similarity between nodes and presenting them as subgraphs.

[0029] The beneficial effects of the present invention are as follows:

[0030] The multiple subsystems of existing graph computing platforms often lack effective collaboration. The graph computing platform provided by the present invention based on cloud-native technology can achieve efficient collaboration between the various subsystems of the platform, perform reasonable resource management according to needs, and realize dynamic expansion / contraction; the software and hardware collaborative system uses cloud-native technology to virtualize and manage heterogeneous computing resources, and based on the provided graph algorithm operator library and software and hardware adaptation, realizes the transparency of computing resources and computing logic.

[0031] Based on the distributed shared memory pool module, the GPU memory and host memory of each node in the cluster are interconnected. Relying on the bandwidth of the high-speed network, it can achieve extremely high communication efficiency, solve the I / O bottleneck that is widely present in graph computing, and accelerate the distributed computing of large-scale graph algorithms.

[0032] The graph development workshop designed in the present invention is a visual, one-stop, multi-user collaborative graph computing R&D front-end system. It is a machine learning operating system in the field of graph computing. It can lower the threshold of graph computing R&D through component-based, process-based, and easy-to-operate usage, and greatly improve the efficiency of algorithm development and online iteration. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0036] FIG1 is a schematic diagram of the architecture of a graph computing platform based on cloud native technology according to an embodiment of the present invention.

[0037] FIG2 is a schematic diagram of the architecture of a software and hardware collaboration system according to an embodiment of the present invention.

[0038] FIG3 is a schematic diagram of the architecture of a graph storage system according to an embodiment of the present invention.

[0039] FIG4 is a schematic diagram of the architecture of a graph computing engine according to an embodiment of the present invention.

[0040] FIG5 is a schematic diagram of the architecture of a graph development workshop system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0042] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used in this application refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0043] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0044] CPU: Central Processing Unit, central processing unit;

[0045] GPU: Graphics Processing Unit, graphics processing unit;

[0046] DPU: Data Processing Unit, data processing unit;

[0047] UI: User Interface, user interface;

[0048] RDMA: Remote Direct Memory Access, remote direct data access;

[0049] MLOps: Machine Learning Operations, machine learning operating system;

[0050] NVMe: Non-Volatile Memory Express, non-volatile memory;

[0051] SSD: Solid State Disk, solid-state memory;

[0052] HDD: Hard Disk Drive, mechanical hard disk;

[0053] DPU: Data Processing Unit, data processor;

[0054] ROM: Read-Only Memory, read-only memory;

[0055] RAM: Random Access Memory, random access memory;

[0056] FPGA: Field Programmable Gate Array, field programmable gate array;

[0057] K8s: Kubernetes, a distributed architecture solution based on container technology;

[0058] API: Application Programming Interface, application program interface;

[0059] GNN: Graph Neural Network, graph neural network.

[0060] The cloud-native technology-based graph computing platform of the present invention can be implemented in any computer language such as C, C++, Java, Go, Python, etc. The implementation method and system can be run on one or more computer nodes, each of which is implemented in the following hardware environment, which can include one or more of the following components: CPU processor, GPU processor, FPGA accelerator card, memory, and network card. The memory can include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, codes, code sets, or instructions.

[0061] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. In addition, those skilled in the art will understand that the above-mentioned hardware does not constitute a limitation on the type of hardware. The hardware may include more or fewer components, or a combination of certain components, or different component arrangements. For example, the hardware also includes components such as radio frequency circuits, input units, sensors, audio circuits, power supplies, etc., which will not be repeated here.

[0062] As shown in Figure 1, the cloud-native graph computing platform of this embodiment includes subsystems such as a hardware and software collaboration system, a graph storage system, a graph computing engine, and a graph development workshop system designed using a cloud-native architecture. Each subsystem uses containerization technology for resource management and task scheduling, and the systems interact through standardized interfaces. The hardware and software collaboration system, graph storage system, and graph computing engine run on the backend, while the graph development workshop includes backend and frontend modules, providing a frontend UI functionality.

[0063] 1. Software and Hardware Collaborative System

[0064] The software and hardware collaborative system provides mainstream hardware computing resources and software and hardware adaptation environments at home and abroad, as shown in Figure 2.

[0065] 1. Hardware resources: These include mainstream CPUs, GPUs, and DPUs from both domestic and international manufacturers, including NVIDIA, AMD, Intel, Huawei, and Shenwei. Each computing node is equipped with one or more CPUs and one or more GPUs. Graph computing chips are custom-developed specifically for the graph computing field. Storage-computing chips include near-storage computing chips and storage-computing chips. FPGA accelerator cards are FPGA programmable array computing units customized for the graph computing field. Storage resources include RAM, ROM, HDD, and SSD. Network resources include network cards, routers, network communication protocols, and other communication facilities. The overall hardware resources belong to a heterogeneous computing architecture.

[0066] 2. Software and Hardware Adaptation: To adapt to heterogeneous computing and storage resources, this paper virtualizes the software and hardware adaptation environment based on cloud native technology, achieving transparency at the hardware resource and software implementation levels, and abstracting the following submodules:

[0067] (1) Hardware environment adaptation: underlying driver and programming interface, module adaptation;

[0068] (2) Compilation environment and tools: including hardware drivers, compilers, and development toolkits;

[0069] (3) Algorithm operator module, which provides a data flow acceleration library and operator acceleration library for accelerating graph computing algorithms based on heterogeneous hardware resources. It proposes a programming model suitable for heterogeneous architectures, adapts computing resources such as CPUs, GPUs, FPGAs, and specialized chips for graph computing, and provides a universal and portable development interface.

[0070] (4) Interactive intelligent module: optimizes parameter automatic learning, provides intelligent programming interface, and automatically recommends parameters and models.

[0071] 2. Graph Storage System

[0072] The graph storage system layer provides large-scale graph data partitioning algorithms, distributed persistent storage, and distributed shared memory pools based on the collaborative capabilities of software and hardware. This provides the graph computing engine with the ability to read and write massive graph data. As shown in Figure 3, it includes the following modules:

[0073] The graph storage system is based on the built-in standardized format specification of graph data to build graph data benchmarks and graph modeling methods, abstract the general graph data loading / upload toolkit, and define the read and write interaction specifications of the graph storage system.

[0074] 1. Graph Partitioning Module

[0075] It provides large graph partitioning methods, including common graph partitioning algorithms such as random partitioning, node partitioning, edge partitioning, and time series partitioning. Different partitioning methods are selected according to the size, type, and business scenario of the graph data. The partitioning algorithm can be applied to two storage methods: graph persistent storage (such as graph database) and distributed shared memory pool (graph in-memory database).

[0076] 2. Distributed persistent storage module

[0077] Based on HDD and SSD hard drives, and based on the partitioning algorithm provided by the graph partitioning module, graph data is segmented and stored on disk through distributed file systems such as HDFS, UnionFS, and JuiceFS. This implements a distributed persistent storage solution for large-scale graph data and provides interfaces and services for reading and writing graph data.

[0078] 3. Distributed shared memory pool module

[0079] The distributed shared memory pool module, based on RDMA technology, can manage the global memory space of the host memory and GPU memory on multiple computing nodes, and realize distributed memory / memory sharing in the cluster. Based on the partitioning algorithm provided by the graph partitioning module, the graph data is segmented, and the graph data is loaded from the distributed persistent storage module to the distributed shared memory pool module, which can effectively solve the I / O performance bottleneck problem in graph computing. The present invention uses a shared memory pool to realize the sharing of distributed data in memory in an extremely low-latency manner, reduce additional I / O storage overhead, and improve real-time performance; at the same time, it performs high-level abstraction on the graph data in memory, uses hierarchical metadata to describe objects, and eliminates the cost of data format conversion. This solution solves the following problems in graph computing to a great extent:

[0080] Solve the problems of high data reading and writing waiting and communication overhead caused by frequent iterations of graph computing.

[0081] Solve the problem of graph algorithms' computational dependence on neighbor information of nodes and edges.

[0082] Solve the problem that the complex structure of graph data makes it difficult for graph algorithms to implement parallel computing on unevenly distributed blocks.

[0083] At the same time, the present invention provides multiple ways to load graph data into the memory pool:

[0084] Graph structure data is stored in video memory, and feature data fragments are stored in video memory;

[0085] Graph structure data is stored in video memory, and feature data fragments are stored in main memory;

[0086] Graph structure data is stored in memory, and feature data fragments are stored in video memory;

[0087] Graph structure data is stored in memory, and feature data shards are stored in memory.

[0088] The above loading methods can be selected according to the size of the graph data and the business scenario. The present invention provides relevant toolkits and parameter selections.

[0089] Graph Computing Engine

[0090] The graph computing engine is designed based on a cloud-native architecture and consists of three core computing engines, as shown in Figure 4:

[0091] 1. Graph Query Engine

[0092] Based on the distributed read / write interface of the graph storage layer, the graph query engine optimizes the storage model, data structure, and query algorithm of graph data. It provides direct query capabilities for graph-related data using Cypher and Gremlin query languages, including multi-order neighbor queries for a single point, querying the associated path between two points, and subgraph queries to obtain the associations between multiple points. It supports the efficient insertion of graph data such as knowledge graphs, especially the rapid insertion of data volumes exceeding 100 million. It further includes the following two submodules:

[0093] (1) Graph Query Parser

[0094] It allows users to write and execute graph query statements in a SQL-like manner, retrieving relevant information from graph data in graph storage systems. Using a graph query parser, users can precisely filter required data based on their needs and perform customized data analysis. This process typically involves parsing the user-entered query, building a query syntax tree, and generating an execution plan. Common graph query languages ​​include Cypher and Gremlin.

[0095] (2) Graph Query Executor

[0096] As the underlying runtime engine of the graph query engine system, it aims to improve query efficiency and scalability, enabling users to quickly complete complex query operations on large graphs. Its primary function is to optimize the syntax tree produced by the query parser, then schedule resources and plan execution according to resource quotas. Using distributed parallel computing technology, it processes large-scale graph data in the graph storage system in a manner that minimizes runtime and resource consumption.

[0097] 2. Graph Analysis Engine

[0098] The graph analysis engine has a rich library of graph analysis algorithms, such as graph clustering, segmentation, spanning trees, and PageRank calculations. It also provides distributed computing capabilities for graph analysis algorithms, including offline graph analysis capabilities that support large-scale graph data and online graph analysis capabilities for streaming graph data. The graph analysis engine interacts with the graph storage system to read and write data. The graph analysis engine includes the following submodules:

[0099] (1) Graph Analysis Algorithm Library

[0100] It typically includes efficient data structures for graph data processing, as well as various general functions and class libraries, providing users with a programmable, fully customizable toolset that allows them to easily implement and test graph analysis algorithms, as well as change and adjust algorithm parameters to optimize and improve algorithm performance. Graph analysis algorithm libraries are also well-suited for searching and identifying large-scale combinatorial patterns and pattern similarities, helping users discover hidden data correlations and patterns. As an efficient and fast graph analysis tool provided to users, it implements various graph analysis algorithms, such as community discovery, node centrality calculation, graph matching, and path calculation. It can quickly and easily process large amounts of complex graph data and is applied to various graph-related research, including social networks, network security, and financial transaction analysis.

[0101] (2) Graph analysis and calculation module

[0102] Based on the algorithm package provided by the graph analysis algorithm library, it provides parallel computing capabilities for graph analysis algorithms. It includes a series of underlying operators and data processing logic that can process graph structures, and supports communication and collaboration between multiple nodes. Combined with the computing power provided by hardware resources, it can well implement distributed computing and parallel computing of graph analysis algorithms through the coordinated acceleration of software and hardware. At the same time, it has the characteristics of high scalability, high reliability and high efficiency, and can quickly process large amounts of complex graph structure data.

[0103] 3. Graph Learning Engine

[0104] The graph learning engine is used to implement parallel graph learning model training, including:

[0105] (1) Graph Learning Algorithm Library

[0106] This toolkit primarily provides scalable implementations of common graph neural network algorithms, supporting a variety of graph models, dataset loading, and data preprocessing. It is a software toolkit for building and training graph neural network models. It typically includes implementations of a range of graph neural network-based algorithms, such as graph convolutional networks (GCNs), graph attention networks (GATs), graph autoencoders, and graph generative networks. Unlike traditional deep neural networks, graph neural networks are specifically suited for processing graph and network data. They treat both nodes and edges in graphs and networks as trainable data, enabling better description of the relationships between nodes and edges. The algorithm library is designed to provide efficient implementations and typically offers a variety of graph datasets and pretrained models as a foundation for experimentation, significantly simplifying the implementation and training of graph neural network models. In addition to a variety of built-in graph neural networks, it also provides a user-friendly parameter configuration interface, allowing users to build customized and specialized graph neural network architectures based on their specific needs within the graph algorithm framework.

[0107] (2) Graph Learning Training Module

[0108] The algorithm component based on the graph learning algorithm library provides an efficient distributed computing engine and the ability to optimize various parameters during graph neural network model training, enabling the model to efficiently learn features that represent the relationships between nodes and edges in a given graph data. This includes the following functions: feature extraction and representation, which involves vectorizing nodes, embedding edges, or other forms of feature abstraction; subgraph sampling, which utilizes distributed storage of graph topology and attribute information and provides an efficient graph sampling query interface. The algorithm model obtains subgraphs and performs calculations by calling graph sampling and graph query interfaces; model construction and selection, which requires selecting an appropriate graph neural network model for training based on the characteristics of the problem during training; training strategy and parameter adjustment, which specifies the corresponding training strategy and parameter adjustment method, and sets the corresponding training stop conditions and adjustment strategies to ensure the convergence and performance stability of the model during training; model evaluation and tuning, which ensures that the actual effect of the model meets expectations through validation set evaluation and test set evaluation, combined with parameter optimization and other means.

[0109] 4. Graphics Development Workshop System

[0110] The Graph Development Workshop system, based on three core graph computing engines and a graph storage system, constructs an MLOPS visualization operating system for the graph computing field. This system provides continuous delivery and automated pipeline capabilities for the graph development lifecycle, componentizing, streamlining, and visualizing the entire graph computing algorithm development process to facilitate team collaboration and iterative algorithm maintenance. The Graph Development Workshop system's underlying layer is compatible with the engine layer, enabling rapid call access to graph algorithm operator libraries by setting parameter configurations. Based on the graph algorithm operator library interface, a pipeline execution model is constructed. Deployed using K8s containerized technology, the Graph Development Workshop supports multi-user collaborative R&D, as shown in Figure 5. It primarily includes the following submodules:

[0111] 1. Graph Data Modeling Module

[0112] The goal of graph data modeling is to abstract real-world objects into nodes and edges in a graph, and to use the graph's structure and attributes to describe the connections and characteristics between objects. This module defines a standard storage structure for graph data, designing semantic specifications for graph structure and feature data by defining a schema. It also provides easy-to-use development tools and a visual design interface, enabling standardized modeling of domain data, facilitating the generation and loading of graph data, and providing a unified read and write interface for other modules.

[0113] 2. Graph Data Visualization Module

[0114] Most of the result data from graph query, graph analysis, and graph learning can be presented in the form of graph data visualization to help people better understand and analyze the data. Visualization methods include: adjusting node coloring and size. The size and color of nodes can be adjusted according to node attributes, such as degree centrality and clustering coefficient. This can more clearly express the importance of the node and the characteristics of the group it belongs to. Edge drawing and coloring can adjust the color, line width, dashed line style, etc. of the edge to reflect different edge attributes. Clustering and subgraphs can divide nodes into different groups based on the similarity between nodes and present them as subgraphs. Interactive control can add interactive control elements such as zooming, dragging, and selection, allowing users to freely browse the data as needed, and also making it easier to analyze and mine the data.

[0115] 3. Graph Algorithm Component Module

[0116] The model code and related algorithm implementations used for model training and inference are used to handle tasks such as graph data conversion, feature engineering, GNN model training and evaluation, parameter optimization, and architecture search. Therefore, the graph algorithm component module includes graph data conversion, feature engineering, graph algorithm operator, GNN model training and evaluation, parameter optimization, and architecture search. Each component is designed and implemented according to specific logic, encompassing implementations of numerous graph machine learning frameworks and libraries, such as TensorFlow and PyTorch (dgl, pyg), as well as graph data loading, graph data processing, graph feature extraction, graph sampling, graph convolution, and graph analysis algorithm components. These components help developers quickly build and train machine learning models and, during deployment, package the trained models into executable files for subsequent deployment.

[0117] 4. Graph Training Process Building Module

[0118] This module manages a series of processing steps in the graph machine learning model training process in a process-oriented manner, including data preprocessing, feature engineering, model training, model evaluation, etc. These steps consist of multiple parallel and serial processing steps, which are built and managed using automated tools. These processing steps are organized in a pipeline manner to form a complete and repeatable data processing process, thereby improving the efficiency and accuracy of model training.

[0119] 5.Graph Model Deployment Module

[0120] This module is used to deploy trained graph algorithm models to remote servers, providing online reasoning and large-scale parallel request processing capabilities for graph algorithm models. Specifically, it deploys trained graph algorithm models to production environments to provide services for business systems or end users. It provides model deployment, model monitoring, configuration management, and model reasoning services, effectively reducing the operation and maintenance costs of developers and enterprises and improving their business efficiency. Model services are usually implemented using Web APIs or other network services, and can provide real-time prediction or recommendation capabilities to support enterprise production and commercial applications.

[0121] The cloud-native technology-based graph computing platform of the present invention can also include various field applications, using the platform's graph computing algorithm library, models, algorithm components, computing engine and visualization development workshop to incubate and build various application systems for industry fields, such as intelligent computing products developed using graph algorithms for data in the fields of computational pharmaceuticals, computational biology, computational astronomy, etc. Each field application is deployed with K8s containerized technology.

[0122] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A graph computing platform based on cloud native technology, characterized in that: The platform includes a software and hardware collaborative system, a graph storage system, a graph computing engine, and a graph development workshop system designed using a cloud native architecture; wherein the software and hardware collaborative system, the graph storage system, and the graph computing engine run on the backend, and the graph development workshop runs on the frontend; The software-hardware collaborative system provides hardware computing resources and a software-hardware adaptation environment. The hardware computing resources include various central processors, data processors, FPGA accelerators, and graph computing chips. The software-hardware adaptation environment includes a data flow acceleration library and an operator acceleration library for accelerating graph computing algorithms based on heterogeneous hardware resources. The graph storage system includes a graph partitioning module, a distributed persistent storage module and a distributed shared memory pool module. The graph partitioning module has multiple graph partitioning algorithms built in. The distributed persistent storage module is used to store graph data on disk through a distributed file system and provide interfaces and services for reading and writing graph data. The distributed shared memory pool module is used to manage the global memory space of the host memory and GPU display memory on multiple computing nodes based on remote direct data access technology to achieve distributed memory sharing in the cluster. The graph computing engine includes a graph query engine, a graph analysis engine and a graph learning engine, which are respectively used to realize direct query of graph-related data, distributed graph analysis and training of parallel graph learning models; The graph development workshop system is a visual operating system for the graph computing field built based on the graph computing engine. It is deployed with K8s containerization technology and is used to componentize, streamline and visualize the entire development process of the graph computing algorithm.

2. The cloud-native technology-based graph computing platform according to claim 1, characterized in that: The hardware-software collaborative system supports scalable hardware devices including GPU processors, CPU processors, FPGA acceleration cards, and storage-computing integrated chips, develops corresponding compilation environments and software tools, provides storage-computing integrated operators for graph data, and operator acceleration modules adapted to specific hardware, providing heterogeneous computing capabilities for graph computing engines.

3. The cloud-native technology-based graph computing platform according to claim 1, characterized in that: The distributed persistent storage module is implemented based on NVMe SSD technology; the distributed shared memory pool module is implemented based on RDMA technology, and data synchronization is performed based on the distributed persistent storage module and the distributed file system.

4. The cloud-native technology-based graph computing platform according to claim 1, characterized in that: The direct query of graph association data in the graph query engine includes multi-order neighbor query of a single point, association path query between two points, and subgraph query for obtaining associations between multiple points.

5. The cloud-native technology-based graph computing platform according to claim 1, characterized in that: The graph query engine includes a graph query parser and a graph query executor. The graph query parser is used to parse the query statements input by the user, build a query syntax tree and generate an execution plan; the graph query executor is used to optimize the syntax tree produced by the graph query parser, and then perform resource scheduling and plan execution according to resource quotas, using distributed parallel computing technology to process large-scale graph data in the graph storage system in a way that minimizes running time and resource consumption.

6. The cloud-native technology-based graph computing platform according to claim 1, characterized in that: The graph analysis engine includes a graph analysis algorithm library and a graph analysis computing module. The graph analysis algorithm library contains efficient data structures for graph data processing and various general functions and class libraries, providing users with a programmable tool set, allowing users to implement and test graph analysis algorithms, as well as change and adjust the parameters of graph analysis algorithms, thereby optimizing and improving the performance of graph analysis algorithms; The graph analysis computing module runs on a distributed system, includes a series of underlying operators and data processing logic that can process graph structures, supports communication and collaboration between multiple nodes, and combines the computing power provided by hardware computing resources to realize distributed computing and parallel computing of graph analysis algorithms.

7. The cloud-native technology-based graph computing platform according to claim 1, characterized in that: The graph learning engine includes a graph learning algorithm library and a graph learning training module. The graph learning algorithm library is a software toolkit for building and training graph neural network models, including various different graph data sets and pre-trained graph neural network models; the graph learning training module is used to sequentially implement feature extraction and representation, subgraph sampling, model construction and selection, training strategies and parameter adjustment, and model evaluation and tuning.

8. The cloud-native technology-based graph computing platform according to claim 1, characterized in that: The graph development workshop system includes a graph data modeling module, a graph data visualization module, a graph algorithm component module, a graph training process construction module and a graph model deployment module; The graph data modeling module is used to define a standard storage structure for graph data, abstract objects in the real world into nodes and edges in a graph, and use the structure and attributes of the graph to describe the connections and features between objects; The graph data visualization module is used to visualize the result data after graph query, graph analysis and graph learning; The graph algorithm component module is used for the implementation of model code and related algorithms for model training and reasoning, including graph data conversion components, feature engineering components, graph algorithm operator components, GNN model training and evaluation components, parameter optimization components, and architecture search components; The graph training process construction module uses automated tools to construct and manage the processing steps of data preprocessing, feature engineering, model training, and model evaluation in a pipeline manner to form a complete and repeatable data processing process; The graph model deployment module is used to deploy the trained graph algorithm model to a remote server to implement model deployment, model monitoring, configuration management and model reasoning services, and to provide support for business systems or end users.

9. The cloud-native technology-based graph computing platform according to claim 8, characterized in that: Visualization methods include: adjusting node coloring and size, edge drawing and coloring, clustering and subgraphs, and interactive control.

10. The cloud-native technology-based graph computing platform according to claim 9, characterized in that: Visualization through clustering and subgraphs, which involves dividing nodes into different groups based on their similarities and presenting them as subgraphs.

Images