Graph database management method and device
Through the Kubernetes container orchestration engine and DAG workflow management graph database instances and clusters, the problem of insufficient complexity and fault tolerance of distributed graph database management is solved, and automated management and efficient resource expansion are achieved.
Patent Information
- Application Number
- CN202210434796.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-24
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-04-24
AI Technical Summary
The prior art cannot effectively reduce the complexity of distributed graph database management and improve fault tolerance, especially in terms of elastic scaling and resource expansion of database instance nodes.
Using a Kubernetes-based container orchestration engine, the graph database instance is managed through the workflow operator and the graph operator, and the directed acyclic graph DAG is used to process instances and cluster resource change events to realize automatic expansion and state management of instances and clusters.
It effectively reduces the complexity of distributed graph database management, improves fault tolerance, realizes automated management of instances and clusters, adapts to multiple cloud environments and blocks infrastructure differences.
Smart Images

Figure CN114925242B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of Internet technology applications, and in particular to a method and device for managing a graph database. Background Art
[0002] Database-as-a-Service (DBaaS) is a form of Software-as-a-Service (SaaS) in which IT organizations can order databases from cloud service providers without having to purchase hardware and software and deploy their own databases. SP (Software Provisioner) will manage server hardware, storage hardware, software, and licenses, and provide secure network connections for database clients. IT organizations only need to specify service level objectives for DBaaS to the service provider, and SP will provide services to customers. There is no need to worry about hardware, virtualization, software, operating systems, networks, security, and database tuning. SP will also ensure that all infrastructure components are upgraded when appropriate. Customers will pay for usage on the database in a subscription model.
[0003] There are many ways to build the infrastructure layer of a DBaaS platform. It can be based on IaaS (Infrastructure as a Service) or PaaS (Platform as a Service), and can run on a private cloud or a public cloud according to business needs. The different technical systems chosen to build the infrastructure will ultimately make a big difference in the time efficiency and operation and maintenance costs of managing the database.
[0004] In related technologies, enterprises or scientific research institutions generally adopt traditional operation and maintenance management methods, using shell scripts or manual intervention for operations from computing and storage resource preparation to database startup, shutdown, expansion and reduction, which cannot achieve real-time resource expansion and elastic scaling of database instance nodes.
[0005] Currently, no effective solution has been proposed to address the above-mentioned problem that the existing technology cannot effectively reduce the complexity of distributed graph database management and improve the fault tolerance of distributed graph databases. Summary of the invention
[0006] In order to solve the above technical problems, the embodiments of the present invention hope to provide a graph database management method and device, so as to at least solve the problem that the existing technology cannot effectively reduce the complexity of distributed graph database management and improve the fault tolerance of distributed graph database.
[0007] The technical solution of the present invention is achieved in this way:
[0008] In the first aspect, an embodiment of the present invention provides a management method for a graph database, including: a business module responds to an instance management API request; the business module calls the instance application program interface Instance-API of the workflow operator workflow-operator to operate the instance; wherein the workflow operator workflow-operator is deployed in the container orchestration engine Kubernetes; the container orchestration engine Kubernetes verifies the instance parameters in the instance management API request, and submits the management API request to the database when the verification passes; the instance controller Instance-Controller listens to the resource change event generated in the instance custom resource InstanceCR, and uses a workflow based on a directed acyclic graph DAG to process the instance Instance; the instance controller Instance-Controller calls the graph application program interface Graph-API of the graph operator graph-operator to manage the graph custom resource GraphCR; the business module listens to the resource change event of the instance custom resource InstanceCR, and when the state of the instance custom resource Instance CR is changed to a specified state, it is determined that the state change is completed.
[0009] Optionally, before the business module responds to the instance management API request, the method also includes: using the container orchestration engine Kubernetes as a PaaS layer service, and classifying the container orchestration engine Kubernetes to obtain a Manage class container orchestration engine and a Business class container orchestration engine; writing an infrastructure template for the container orchestration engine Kubernetes, and uploading the written infrastructure template to the application container engine Docker image, and adding a running environment identifier to the application container engine Docker image; implementing a graph operator graph-operator based on the kubebuilder framework, wherein the graph operator graph-operator is used to manage graph custom resource definition Graph-CRD resource objects; implementing a workflow operator workflow-operator based on the kubebuilder framework, wherein the workflow operator workflow-operator is used to manage graph database instances on the DBaaS platform; deploying the workflow operator to a cluster of the Manage class container orchestration engine; and establishing a connection between the cluster of the Manage class container orchestration engine and the cluster of the Business class container orchestration engine.
[0010] Further, optionally, the graph operator graph-operator includes: a graph cluster controller Graph-Cluster-Controller, a graph composition controller Graph-Component-Controller, a graph custom resource definition Graph-CRD and a graph application program interface Graph-API, wherein the graph cluster controller Graph-Cluster-Controller is used to globally coordinate the graph database state and drive the Component-Controller to manage each graph database component; the graph composition controller Graph-Component-Controller is used to operate the graph database components; the graph custom resource definition Graph-CRD is used to declaratively define the resource characteristics of the graph database; the graph application program interface Graph-API is used to provide an interface call to create a graph database.
[0011] Optionally, the workflow operator workflow-operator includes: cluster controller Cluster-Controller, cluster custom resource definition Cluster-CRD, cluster application program interface Cluster-API, instance controller Instance-Controller, instance custom resource definition Instance-CRD and instance application program interface Instance-API, wherein the cluster controller Cluster-Controller is used for elastic expansion of the container orchestration engine Kubernetes cluster; the cluster custom resource definition Cluster-CRD is used to declaratively define the resource characteristics of the container orchestration engine Kubernetes cluster; the cluster application program interface Cluster-API is used to provide an interface to the outside world to create a cluster of the container orchestration engine Kubernetes; the instance controller Instance-Controller is used for lifecycle management of graph database instances to shield environmental differences in the infrastructure layer; the instance custom resource definition Instance-CRD is used to declaratively define the business attributes and resource characteristics of the graph database in the DBaaS platform; the instance application program interface Instance-API is used to provide an interface to the outside world to manage graph database instances.
[0012] Optionally, the method also includes: when the monitoring module detects that the cluster resources of the Business class container orchestration engine are insufficient, calling the cluster expansion interface of the business module; the business module responds to the cluster expansion request, calling the group Cluster-API of the workflow operator workflow-operator to create a custom resource CR; the cluster controller Cluster-Controller monitors that a resource change event of the group custom resource ClusterCR is generated, and uses a workflow based on a directed acyclic graph DAG to expand the cluster of the Business class container orchestration engine; the business module monitors the resource change event of the group custom resource ClusterCR; when the state of the group custom resource ClusterCR changes, determining that the expanded cluster is available, and adding the expanded cluster to the set of schedulable clusters; the monitoring module cancels the resource shortage alarm.
[0013] Further, optionally, the business module responds to the cluster expansion request and calls the group Cluster-API of the workflow operator workflow-operator to create a custom resource CR, including: the monitoring module starts the timer to call the List interface of the group Cluster-API of the workflow operator workflow-operator every preset period; traverses each Cluster resource object in turn, and accumulates the number of available instances available-instance in the status Status; calculates the threshold value according to the number of available instances; when the threshold value is lower than the configured threshold, uses gRPC to call the cluster expansion interface of the business module; after receiving the cluster module, the business module calls the Create interface of the group Cluster-API of the workflow operator workflow-operator, and submits the Cluster CR resource object to the kube application programming interface server kube-apiserver of the container orchestration engine Kubernetes.
[0014] Optionally, using a directed acyclic graph (DAG)-based workflow to expand the cluster of a Business class container orchestration engine includes: calling each child node number of the directed acyclic graph (DAG); defining and creating a container orchestration engine Kubernetes, instance characteristics, and defining a network function orchestration process to obtain an executable DAG graph.
[0015] Furthermore, optionally, create a container orchestration engine Kubernetes, use the CreateK8S node to create a K8S Job, inject environment variables into the Job container, including the cloud environment, authentication key, and cluster metadata, and use terraform in the container to perform K8S cluster initialization; the instance feature InstallCharts node is a collection of several dependent application components, which are installed in sequence by calling helm after the K8S cluster is initialized; define the network function ConfigNetwork node, which is determined based on the running cloud environment.
[0016] Optionally, when the DAG graph is completed, the StatusPhase field of the Cluster resource object is updated to Ready.
[0017] In a second aspect, an embodiment of the present invention provides a management device for a graph database, comprising: a business module for responding to an instance management API request; a business module for calling the instance application program interface Instance-API of a workflow operator workflow-operator to operate an instance; wherein the workflow operator workflow-operator is deployed in a container orchestration engine Kubernetes; the container orchestration engine Kubernetes is used to verify the instance parameters in the instance management API request, and submit the management API request to the database when the verification passes; an instance controller Instance-Controller is used to monitor the resource change event generated in the instance custom resource InstanceCR, and use a workflow based on a directed acyclic graph DAG to process the instance Instance; the instance controller Instance-Controller is used to call the graph application program interface Graph-API of a graph operator graph-operator to manage a graph custom resource GraphCR; the business module is used to monitor the resource change event of the instance custom resource InstanceCR, and when the state of the instance custom resource Instance CR is changed to a specified state, determine that the state change is completed.
[0018] The embodiment of the present invention responds to the instance management API request through the business module; the business module calls the instance application program interface Instance-API of the workflow operator workflow-operator to operate the instance; wherein the workflow operator workflow-operator is deployed in the container orchestration engine Kubernetes; the container orchestration engine Kubernetes verifies the instance parameters in the instance management API request, and submits the management API request to the database when the verification passes; the instance controller Instance-Controller listens to the resource change event generated in the instance custom resource InstanceCR, and uses the workflow based on the directed acyclic graph DAG to process the instance Instance; the instance controller Instance-Controller calls the graph application program interface Graph-API of the graph operator graph-operator to manage the graph custom resource GraphCR; the business module listens to the resource change event of the instance custom resource InstanceCR, and when the state of the instance custom resource InstanceCR is changed to the specified state, it is determined that the state change is completed, thereby effectively reducing the complexity of distributed graph database management and improving the fault tolerance of the distributed graph database. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0020] Figure 1 A schematic diagram of a flow chart of a graph database management method provided in Embodiment 1 of the present invention;
[0021] Figure 2 A schematic diagram of a calling sequence in a method for identifying multi-tag behaviors provided in the first embodiment of the present invention;
[0022] Figure 3 This is a schematic diagram of a DAG process in a graph database management method provided in Example 1 of the present invention. DETAILED DESCRIPTION
[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0024] It should be noted that the terms "first", "second" and the like in the specification, claims and drawings of the present invention are used to distinguish different objects rather than to limit a specific order.
[0025] It should also be noted that the following embodiments of the present invention may be executed separately, or in combination with each other, and the embodiments of the present invention do not impose any specific limitations on this.
[0026] Embodiment 1
[0027] In a first aspect, an embodiment of the present invention provides a method for managing a graph database. Figure 1 A flowchart of a method for identifying multi-label behavior provided in Embodiment 1 of the present invention; Figure 1 As shown, the management method of the graph database provided in the embodiment of the present application includes:
[0028] Step S101, the business module responds to the instance management API request;
[0029] Step S103, the business module calls the instance application program interface Instance-API of the workflow operator workflow-operator to operate the instance; wherein the workflow operator workflow-operator is deployed in the container orchestration engine Kubernetes;
[0030] Step S105, the container orchestration engine Kubernetes verifies the instance parameters in the instance management API request, and submits the management API request to the database if the verification passes;
[0031] Step S107, the instance controller Instance-Controller monitors that a resource change event is generated in the instance custom resource InstanceCR, and uses a workflow based on a directed acyclic graph DAG to process the instance Instance;
[0032] Step S109, the instance controller Instance-Controller calls the graph application program interface Graph-API of the graph operator graph-operator to manage the graph custom resource GraphCR;
[0033] Step S111 : The business module monitors the resource change event of the instance custom resource InstanceCR, and when the state of the instance custom resource InstanceCR is changed to a specified state, it is determined that the state change is completed.
[0034] Optionally, before the business module responds to the instance management API request in step S101, the graph database management method provided in the embodiment of the present application also includes: using the container orchestration engine Kubernetes as a PaaS layer service, and classifying the container orchestration engine Kubernetes to obtain a Manage class container orchestration engine and a Business class container orchestration engine; writing the infrastructure template of the container orchestration engine Kubernetes, and uploading the written infrastructure template to the application container engine Docker image, and adding a running environment identifier to the application container engine Docker image; implementing a graph operator graph-operator based on the kubebuilder framework, wherein the graph operator graph-operator is used to manage graph custom resource definition Graph-CRD resource objects; implementing a workflow operator workflow-operator based on the kubebuilder framework, wherein the workflow operator workflow-operator is used to manage graph database instances on the DBaaS platform; deploying the workflow operator to a cluster of the Manage class container orchestration engine; and establishing a connection between the cluster of the Manage class container orchestration engine and the cluster of the Business class container orchestration engine.
[0035] Further, optionally, the graph operator graph-operator includes: a graph cluster controller Graph-Cluster-Controller, a graph composition controller Graph-Component-Controller, a graph custom resource definition Graph-CRD and a graph application program interface Graph-API, wherein the graph cluster controller Graph-Cluster-Controller is used to globally coordinate the graph database state and drive the Component-Controller to manage each graph database component; the graph composition controller Graph-Component-Controller is used to operate the graph database components; the graph custom resource definition Graph-CRD is used to declaratively define the resource characteristics of the graph database; the graph application program interface Graph-API is used to provide an interface call to create a graph database.
[0036] Optionally, the workflow operator workflow-operator includes: cluster controller Cluster-Controller, cluster custom resource definition Cluster-CRD, cluster application program interface Cluster-API, instance controller Instance-Controller, instance custom resource definition Instance-CRD and instance application program interface Instance-API, wherein the cluster controller Cluster-Controller is used for elastic expansion of the container orchestration engine Kubernetes cluster; the cluster custom resource definition Cluster-CRD is used to declaratively define the resource characteristics of the container orchestration engine Kubernetes cluster; the cluster application program interface Cluster-API is used to provide an interface to the outside world to create a cluster of the container orchestration engine Kubernetes; the instance controller Instance-Controller is used for lifecycle management of graph database instances to shield environmental differences in the infrastructure layer; the instance custom resource definition Instance-CRD is used to declaratively define the business attributes and resource characteristics of the graph database in the DBaaS platform; the instance application program interface Instance-API is used to provide an interface to the outside world to manage graph database instances.
[0037] Optionally, the graph database management method provided by the embodiment of the present application also includes: when the monitoring module detects that the cluster resources of the Business class container orchestration engine are insufficient, calling the cluster expansion interface of the business module; the business module responds to the cluster expansion request, calling the group Cluster-API of the workflow operator workflow-operator to create a custom resource CR; the cluster controller Cluster-Controller listens to the resource change event of the group custom resource ClusterCR, and uses the directed acyclic graph DAG-based workflow to expand the cluster of the Business class container orchestration engine; the business module listens to the resource change event of the group custom resource ClusterCR; when the state of the group custom resource ClusterCR changes, it is determined that the expanded cluster is available, and the expanded cluster is added to the set of schedulable clusters; the monitoring module cancels the resource shortage alarm.
[0038] Further, optionally, the business module responds to the cluster expansion request and calls the group Cluster-API of the workflow operator workflow-operator to create a custom resource CR, including: the monitoring module starts the timer to call the List interface of the group Cluster-API of the workflow operator workflow-operator every preset period; traverses each Cluster resource object in turn, and accumulates the number of available instances available-instance in the status Status; calculates the threshold value according to the number of available instances; when the threshold value is lower than the configured threshold, uses gRPC to call the cluster expansion interface of the business module; after receiving the cluster module, the business module calls the Create interface of the group Cluster-API of the workflow operator workflow-operator, and submits the Cluster CR resource object to the kube application programming interface server kube-apiserver of the container orchestration engine Kubernetes.
[0039] Optionally, using a directed acyclic graph (DAG)-based workflow to expand the cluster of a Business class container orchestration engine includes: calling each child node number of the directed acyclic graph (DAG); defining and creating a container orchestration engine Kubernetes, instance characteristics, and defining a network function orchestration process to obtain an executable DAG graph.
[0040] Furthermore, optionally, create a container orchestration engine Kubernetes, use the CreateK8S node to create a K8S Job, inject environment variables into the Job container, including the cloud environment, authentication key, and cluster metadata, and use terraform in the container to perform K8S cluster initialization; the instance feature InstallCharts node is a collection of several dependent application components, which are installed in sequence by calling helm after the K8S cluster is initialized; define the network function ConfigNetwork node, which is determined based on the running cloud environment.
[0041] Optionally, when the DAG graph is completed, the StatusPhase field of the Cluster resource object is updated to Ready.
[0042] In summary, in the graph database management method provided in the embodiment of the present application, kubernetes, referred to as K8s, is an abbreviation formed by replacing the eight characters "ubernete" in the middle of the name with 8. It is an open source containerized application for managing multiple hosts in a cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and powerful. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance; Figure 2 As shown, Figure 2 This is a schematic diagram of the call sequence in a graph database management method provided in Example 1 of the present invention. The graph database management method provided in the embodiment of this application based on Kubernetes is implemented as follows:
[0043] S1, abstractly describe the infrastructure PaaS layer, use Kubernetes (hereinafter referred to as K8S) as the PaaS layer service, and divide it into two roles: Manage and Business (that is, in the embodiment of the present application, the container orchestration engine Kubernetes is used as the PaaS layer service, and the container orchestration engine Kubernetes is classified to obtain the Manage class container orchestration engine and the Business class container orchestration engine);
[0044] S2, using Terraform HCL language to write infrastructure templates, and making differential adjustments to the templates according to the deployment environment (i.e., writing infrastructure templates for the container orchestration engine Kubernetes in the embodiment of the present application);
[0045] Among them, define the Kubernetes cluster; optionally, define the network service VPC (Virtual Private Cloud), VPC-Peering, DNS-Zone (Domain Name System Zone);
[0046] S3, packaging the template into a Docker image and marking it with an operating environment identifier tag (i.e., in the embodiment of the present application, uploading the written infrastructure template to the application container engine Docker image, and adding an operating environment identifier to the application container engine Docker image);
[0047] S4, implementing graph-operator based on kubebuilder framework for managing Graph-CRD resource objects (Custom Resources Definition, CRD for short); (i.e., implementing graph-operator based on kubebuilder framework in the embodiment of the present application)
[0048] Among them, graph-operator is composed of Graph-Cluster-Controller, Graph-Component-Controller, Graph-CRD and Graph-API;
[0049] Graph-Cluster-Controller, which is used to globally coordinate the graph database status and drive the Component-Controller to manage each graph database component;
[0050] Graph-Component-Controller, used to operate graph database components, such as create, delete, start, and stop;
[0051] Graph-CRD declaratively defines the resource characteristics of the graph database, such as the number of replicas of each component and the image version used;
[0052] Graph-API is used to provide external interface calls to create a graph database.
[0053] S5, based on the kubebuilder framework, implement workflow-operator for the DBaaS platform to manage graph database instances (i.e., the workflow operator workflow-operator is implemented based on the kubebuilder framework in the embodiment of the present application);
[0054] Among them, workflow-operator is composed of Cluster-Controller, Cluster-CRD, Cluster-API, Instance-Controller, Instance-CRD, Instance-API;
[0055] Cluster-Controller is used for elastic expansion of K8S clusters;
[0056] Cluster-CRD declaratively defines the resource characteristics of the K8S cluster;
[0057] Cluster-API is used to provide an interface to create a K8S cluster;
[0058] Instance-Controller is used for lifecycle management of graph database instances, shielding environmental differences at the infrastructure layer;
[0059] Instance-CRD declaratively defines the business attributes and resource characteristics of the graph database in the DBaaS platform;
[0060] Instance-API is used to provide an interface to manage graph database instances.
[0061] S6, using helm charts to deploy workflow-operator to the K8S cluster of the Manage role (that is, deploying the workflow operator to the cluster of the Manage class container orchestration engine in the embodiment of the present application);
[0062] S7, open up the network connection between the ManageK8S cluster and the BusinessK8S cluster (that is, establish a connection between the cluster of the Manage class container orchestration engine and the cluster of the Business class container orchestration engine in the embodiment of the present application);
[0063] S8, the business module responds to the instance management API request and calls the Instance-API operation instance of the workflow-operator (ie, S101 and 103 in the embodiment of the present application);
[0064] S9, Kubernetes verifies the Instance parameters and submits them to the Etcd database (etcd is a highly consistent distributed key-value storage) if they are legal; (i.e., step S105 in the embodiment of the present application)
[0065] S10, Instance-Controller monitors that a resource change Event is generated by InstanceCR, and uses a DAG-based workflow to orchestrate the Instance, where DAG (Directed acyclic graph) (i.e., S107 in the embodiment of the present application);
[0066] S11, Instance-Controller calls the Graph-API of graph-operator to manage GraphCR resources (ie, S109 in the embodiment of the present application);
[0067] S12, the business module monitors the resource change event of InstanceCR. When the Phase of the Status field of InstanceCR is Running or Stopped, the state change is considered to be completed; (ie, S111 in the embodiment of the present application)
[0068] S13, when the monitoring module senses that the Business K8S cluster resources are insufficient, the business module cluster expansion interface is called; (that is, in the embodiment of the present application, when the monitoring module detects that the cluster resources of the Business class container orchestration engine are insufficient, the cluster expansion interface of the business module is called)
[0069] S14, the business module responds to the cluster expansion request and calls the Cluster-API of the workflow-operator to create a CR resource; (that is, in the embodiment of the present application, the business module responds to the cluster expansion request and calls the Cluster-API of the workflow operator workflow-operator to create a custom resource CR)
[0070] S15, Cluster-Controller monitors that ClusterCR has a resource change event, and uses a DAG-based workflow to expand the BusinessK8S cluster; (that is, in the embodiment of the present application, the cluster controller Cluster-Controller monitors that the group-defined resource ClusterCR has a resource change event, and uses a directed acyclic graph DAG-based workflow to expand the cluster of the Business class container orchestration engine)
[0071] S16, the business module monitors the resource change event of ClusterCR. When the Phase of the Status field of ClusterCR is Ready, the newly added cluster can be considered available and added to the set of schedulable clusters; (that is, in the embodiment of the present application, the business module monitors the resource change event of the group-defined resource ClusterCR. When the state of the group-defined resource ClusterCR changes, it is determined that the expanded cluster is available, and the expanded cluster is added to the set of schedulable clusters)
[0072] S17, the monitoring module cancels the resource shortage alarm in the latest round of inspection. (That is, the monitoring module cancels the resource shortage alarm in the embodiment of the present application).
[0073] The details of K8S cluster elastic expansion are as follows:
[0074] S1, the monitoring module starts the timer and calls the List interface of the workflow-operator Cluster-API every 1 minute;
[0075] S2, traverse each Cluster resource object in turn and accumulate the number of available-instances in Status;
[0076] S3, Sum(available-instance) / Sum(quota-instance), get threshold;
[0077] S4, when the threshold is lower than the configured threshold, use gRPC to call the business module cluster expansion interface;
[0078] S5: After receiving the cluster module, the business module calls the Create interface of the workflow-operator Cluster-API and submits the Cluster CR resource object to the kube-apiserver of K8S.
[0079] S6, Cluster-Controller listens to the Create Event of ClusterCR and executes the coordination process of Cluster expansion;
[0080] The DAG framework is called to number each child node, and the CreateK8S->InstallCharts->ConfigNetwork orchestration process is defined to obtain an executable DAG graph. The CreateK8S node creates a K8S Job, injects environment variables into the Jobcontainer, including the cloud environment, authentication key, and cluster metadata, and the terraform in the container executes the K8S cluster initialization.
[0081] The InstallCharts node is a collection of several dependent application components, which are installed in sequence by calling helm after the K8S cluster is initialized;
[0082] The ConfigNetwork node depends on the running cloud environment. The public cloud will perform VPC creation and VPC-Peering docking, and the private cloud subnets can be accessed;
[0083] S7: Run the DAG graph until the total progress reaches 100%, and update the StatusPhase field of the Cluster resource object to Ready.
[0084] That is, the monitoring module in the embodiment of the present application starts the timer to call the List interface of the group Cluster-API of the workflow operator workflow-operator every preset period; traverses each Cluster resource object in turn, and accumulates the number of available instances available-instance in the status Status; calculates the threshold value according to the number of available instances; when the threshold value is lower than the configured threshold, uses gRPC to call the business module cluster expansion interface; after receiving the cluster module, the business module calls the Create interface of the group Cluster-API of the workflow operator workflow-operator, and submits the Cluster CR resource object to the kube application programming interface server kube-apiserver of the container orchestration engine Kubernetes;
[0085] Among them, using the directed acyclic graph (DAG)-based workflow to expand the cluster of the Business class container orchestration engine includes: calling each child node number of the directed acyclic graph (DAG); defining and creating the container orchestration engine Kubernetes, instance characteristics and defining the network function orchestration process to obtain an executable DAG graph.
[0086] Create a container orchestration engine Kubernetes, use the CreateK8S node to create a K8S Job, inject environment variables into the Jobcontainer, including the cloud environment, authentication key, and cluster metadata, and use terraform in the container to initialize the K8S cluster; the instance feature InstallCharts node is a collection of several dependent application components, which are installed in sequence by calling helm after the K8S cluster is initialized; define the network function ConfigNetwork node, which is determined based on the running cloud environment.
[0087] When the DAG graph is completed, the StatusPhase field of the Cluster resource object is updated to Ready.
[0088] In this embodiment of the present application, Cluster-CRD can be defined as:
[0089]
[0090]
[0091] Operator cascade management in the embodiment of this application
[0092] A. Graph-CRD Definition
[0093]
[0094]
[0095] B. Instance-CRD Definition
[0096]
[0097]
[0098] In addition, the DAG process in the embodiment of the present application is as follows Figure 3 As shown, Figure 3 This is a schematic diagram of a DAG process in a graph database management method provided in Example 1 of the present invention. The DAG process will be encapsulated in the coordination logic of the Instance-Controller, and each box represents an execution node.
[0099] Among them, DAG workflow
[0100] S1, define TaskId as string type, TaskIds as map type, key is TaskId, value is empty, implement List() method, which can retrieve all keys of TaskIds, and support sorting and non-sorting strategies;
[0101] S2, define a type Task to represent a task node, including three fields: Name, Fn, and Dependencies;
[0102] S3, define the Add() method to add several Tasks to the DAG nodes;
[0103] S4, define the Graph() method to construct the relationship between tasks into DAG;
[0104] S5, embed the task flow represented by DAG into the Operator controller;
[0105] S6: When a node fails to execute, the error TaskId is recorded in the Status of CR. In the next coordination process, the error environment is cleared according to TaskId and the retry is continued.
[0106] The graph database management method provided in the embodiment of the present application builds a graph DBaaS platform based on Kubernetes. Compared with the prior art, the effects produced are: effectively reducing the complexity of distributed graph database management, improving the fault tolerance of distributed graph databases, and saving significant time in the scenario of batch management of multiple graph database clusters. At the same time, it can be adapted to a variety of cloud environments, and the DBaaS platform can shield the differences in infrastructure.
[0107] The embodiment of the present invention responds to the instance management API request through the business module; the business module calls the instance application program interface Instance-API of the workflow operator workflow-operator to operate the instance; wherein the workflow operator workflow-operator is deployed in the container orchestration engine Kubernetes; the container orchestration engine Kubernetes verifies the instance parameters in the instance management API request, and submits the management API request to the database when the verification passes; the instance controller Instance-Controller listens to the resource change event generated in the instance custom resource InstanceCR, and uses the workflow based on the directed acyclic graph DAG to process the instance Instance; the instance controller Instance-Controller calls the graph application program interface Graph-API of the graph operator graph-operator to manage the graph custom resource GraphCR; the business module listens to the resource change event of the instance custom resource InstanceCR, and when the state of the instance custom resource InstanceCR is changed to the specified state, it is determined that the state change is completed, thereby effectively reducing the complexity of distributed graph database management and improving the fault tolerance of the distributed graph database.
[0108] Embodiment 2
[0109] In a second aspect, an embodiment of the present invention provides a management device for a graph database, comprising: a business module for responding to an instance management API request; a business module for calling the instance application program interface Instance-API of a workflow operator workflow-operator to operate an instance; wherein the workflow operator workflow-operator is deployed in a container orchestration engine Kubernetes; the container orchestration engine Kubernetes is used to verify the instance parameters in the instance management API request, and submit the management API request to the database when the verification passes; an instance controller Instance-Controller is used to monitor the resource change event generated in the instance custom resource InstanceCR, and use a workflow based on a directed acyclic graph DAG to process the instance Instance; the instance controller Instance-Controller is used to call the graph application program interface Graph-API of a graph operator graph-operator to manage a graph custom resource GraphCR; the business module is used to monitor the resource change event of the instance custom resource InstanceCR, and when the state of the instance custom resource Instance CR is changed to a specified state, determine that the state change is completed.
[0110] The embodiment of the present invention responds to the instance management API request through the business module; the business module calls the instance application program interface Instance-API of the workflow operator workflow-operator to operate the instance; wherein the workflow operator workflow-operator is deployed in the container orchestration engine Kubernetes; the container orchestration engine Kubernetes verifies the instance parameters in the instance management API request, and submits the management API request to the database when the verification passes; the instance controller Instance-Controller listens to the resource change event generated in the instance custom resource InstanceCR, and uses the workflow based on the directed acyclic graph DAG to process the instance Instance; the instance controller Instance-Controller calls the graph application program interface Graph-API of the graph operator graph-operator to manage the graph custom resource GraphCR; the business module listens to the resource change event of the instance custom resource InstanceCR, and when the state of the instance custom resource InstanceCR is changed to the specified state, it is determined that the state change is completed, thereby effectively reducing the complexity of distributed graph database management and improving the fault tolerance of the distributed graph database.
[0111] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.
Claims
1. A method for managing a graph database, characterized in that: include: The business module responds to instance management API requests; The business module calls the instance application program interface Instance-API of the workflow operator workflow-operator to operate the instance; wherein the workflow operator workflow-operator is deployed in the container orchestration engine Kubernetes; The container orchestration engine Kubernetes verifies the instance parameters in the instance management API request, and submits the management API request to the database if the verification passes; The instance controller Instance-Controller listens to the resource change event generated in the instance custom resource InstanceCR, and uses the directed acyclic graph (DAG)-based workflow to process the instance Instance; The instance controller Instance-Controller calls the graph application program interface Graph-API of the graph operator graph-operator to manage the graph custom resource GraphCR; The business module monitors the resource change event of the instance custom resource InstanceCR, and determines that the state change is completed when the state of the instance custom resource InstanceCR is changed to a specified state.
2. The method according to claim 1, characterized in that Before the business module responds to the instance management API request, the method further includes: Using the container orchestration engine Kubernetes as a PaaS layer service, and classifying the container orchestration engine Kubernetes to obtain a Manage class container orchestration engine and a Business class container orchestration engine; Compile an infrastructure template for the container orchestration engine Kubernetes, upload the compiled infrastructure template to the application container engine Docker image, and add a running environment identifier to the application container engine Docker image; Implementing a graph operator graph-operator according to the kubebuilder framework, wherein the graph operator graph-operator is used to manage graph custom resource definition Graph-CRD resource objects; Implementing the workflow operator workflow-operator according to the kubebuilder framework, wherein the workflow operator workflow-operator is used for managing graph database instances on a DBaaS platform; Deploy the workflow operator to a cluster of the Manage class container orchestration engine; A connection is established between the cluster of the Manage container orchestration engine and the cluster of the Business container orchestration engine.
3. The method according to claim 2, characterized in that The graph operator graph-operator includes: a graph cluster controller Graph-Cluster-Controller, a graph component controller Graph-Component-Controller, a graph custom resource definition Graph-CRD and a graph application program interface Graph-API, wherein: The graph cluster controller Graph-Cluster-Controller is used to globally coordinate the graph database status and drive the Component-Controller to manage each graph database component; The graph-component-controller is used to operate the graph database component; The graph custom resource definition Graph-CRD is used to declaratively define resource characteristics of the graph database; The graph application program interface Graph-API is used to provide an interface call to create a graph database.
4. The method according to claim 2, characterized in that: The workflow operator workflow-operator includes: a cluster controller Cluster-Controller, a cluster custom resource definition Cluster-CRD, a cluster application program interface Cluster-API, an instance controller Instance-Controller, an instance custom resource definition Instance-CRD and an instance application program interface Instance-API, wherein: The cluster controller Cluster-Controller is used for elastic expansion of the container orchestration engine Kubernetes cluster; The cluster custom resource definition Cluster-CRD is used to declaratively define the resource characteristics of the container orchestration engine Kubernetes cluster; The cluster application program interface Cluster-API is used to provide an interface to create a cluster of the container orchestration engine Kubernetes; The instance controller Instance-Controller is used for lifecycle management of graph database instances and shielding environmental differences at the infrastructure layer; The instance custom resource definition Instance-CRD is used to declaratively define the business attributes and resource characteristics of the graph database in the DBaaS platform; The instance application program interface Instance-API is used to provide an interface to manage graph database instances externally.
5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When the monitoring module detects that the cluster resources of the Business class container orchestration engine are insufficient, the cluster expansion interface of the business module is called; The business module responds to the cluster expansion request and calls the cluster-API of the workflow operator workflow-operator to create a custom resource CR; The cluster controller Cluster-Controller monitors that a resource change event of the cluster custom resource ClusterCR is generated, and uses the workflow based on the directed acyclic graph DAG to expand the cluster of the Business class container orchestration engine; The business module monitors the resource change event of the group custom resource ClusterCR; When the state of the cluster custom resource ClusterCR changes, it is determined that the expanded cluster is available, and the expanded cluster is added to the set of schedulable clusters; The monitoring module cancels the resource shortage alarm.
6. The method according to claim 5, characterized in that The business module responds to the cluster expansion request and calls the cluster-API of the workflow operator workflow-operator to create a custom resource CR, including: The monitoring module starts a timer to call the List interface of the cluster-API of the workflow operator workflow-operator at a preset period; Traverse each Cluster resource object in turn and accumulate the number of available instances in the Status column. Calculating a threshold value according to the number of available instances; When the threshold value is lower than the configured threshold, gRPC is used to call the business module cluster expansion interface; After receiving the cluster module, the business module calls the Create interface of the cluster-API of the workflow operator workflow-operator, and submits the Cluster CR resource object to the kube application programming interface server kube-apiserver of the container orchestration engine Kubernetes.
7. The method according to claim 6, characterized in that The method of expanding the cluster of the Business class container orchestration engine using the workflow based on the directed acyclic graph DAG includes: Call each child node number of the directed acyclic graph DAG; Define and create the container orchestration engine Kubernetes, instance characteristics, and define the network function orchestration process to obtain an executable DAG graph.
8. The method according to claim 7, characterized in that Create the container orchestration engine Kubernetes, use the CreateK8S node to create a K8S Job, inject environment variables into the Jobcontainer, including the cloud environment, authentication key, cluster metadata, and execute the K8S cluster initialization with terraform in the container; The instance feature InstallCharts node is a collection of several dependent application components, which are installed in sequence by calling helm after the K8S cluster is initialized; The definition of the network function ConfigNetwork node is determined according to the operating cloud environment.
9. The method according to claim 7, characterized in that: When the DAG graph is completed, the status StatusPhase field of the Cluster resource object is updated to Ready.
10. A graph database management device, characterized in that: include: Business module, used to respond to instance management API requests; The business module is used to call the instance application program interface Instance-API of the workflow operator workflow-operator to operate the instance; wherein the workflow operator workflow-operator is deployed in the container orchestration engine Kubernetes; The container orchestration engine Kubernetes is used to verify the instance parameters in the instance management API request, and submit the management API request to the database if the verification passes; The instance controller Instance-Controller is used to monitor resource change events generated in the instance custom resource InstanceCR and use a directed acyclic graph (DAG)-based workflow to process the instance Instance; The instance controller Instance-Controller is used to call the graph application program interface Graph-API of the graph operator graph-operator to manage the graph custom resource GraphCR; The business module is used to monitor the resource change event of the instance custom resource InstanceCR, and when the state of the instance custom resource InstanceCR is changed to a specified state, determine that the state change is completed.
