Graph database-based copy self-adaptive adjustment method and device

Abstract

The invention discloses a method and a device for adaptively adjusting a copy based on a graph database. According to the method, in a cluster creation stage, a corresponding expected copy value is set for each predefined Zone; detecting a copy state reported by a storage layer in real time to obtain a PartView of current data copy distribution and a StoreView of a physical resource use condition; the number of copies in the PartView is compared with an expected copy value; when the number of the copies is different from an expected copy value, generating a copy change instance, selecting a target storage node through a filter and a speed limiter according to StoreView, and executing capacity expansion or capacity reduction operation of the copies; and when the number of the copies is the same as the expected copy value, ending the copy adjustment process. The technical effect of improving the availability of the database can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of graph database, and particularly relates to a method and device for adaptive adjustment of replicas based on a graph database. BACKGROUND

[0002] Traditional graph databases have some limitations in data storage and management, for example: Regarding the static number of replicas, if the number of replicas is fixed when the cluster is deployed, the replica threshold cannot be reduced in time to ensure writeable services when the primary zone is sorted out of failure; Regarding manual intervention, manual operation is required for primary / backup switching, replica completion, and the like, and the RTO is high; Regarding disk failure, when a disk failure of a storage node causes data loss, the replica is not automatically supplemented, and the number of degraded replicas is long-term, increasing the risk of data loss; Currently, no effective solution has been proposed to solve the problem of low running efficiency of a graph database due to the fixed number of replicas of a traditional graph database and the need for manual intervention. SUMMARY

[0003] To solve the above technical problems, the embodiments of the present application aim to provide a method and device for adaptive adjustment of replicas based on a graph database, to at least solve the problem of low running efficiency of a graph database due to the fixed number of replicas of a traditional graph database and the need for manual intervention.

[0004] The technical solution of the present application is as follows: The embodiments of the present application provide a method for adaptive adjustment of replicas based on a graph database, comprising: setting a corresponding expected replica value for each Zone in a predefinition in a cluster creation stage; detecting a replica state reported by a storage layer in real time to obtain a PartView of a current data replica distribution and a StoreView of a physical resource usage condition; comparing the number of replicas in the PartView with the expected replica value; when the number of replicas is different from the expected replica value, generating a replica change instance, and selecting a target storage node according to the StoreView through a filter and a speed limiter to perform an expansion or contraction operation of the replica; and when the number of replicas is the same as the expected replica value, the replica adjustment process ends.

[0005] Optionally, the method further comprises: when the primary region fails, sending a metadata replica downsizing request to the raft group of the metadata service metad; performing a member change operation on the raft group of the metadata service metad according to the metadata replica downsizing request, so that the metadata service metad of the backup region becomes a leader node; after the metadata service metad of the backup region becomes the leader node, sending a data replica downsizing request to the leader node; performing a downsizing operation on the data replica of the leader node according to the data replica downsizing request, so that the leaders of all data shards are concentrated in the backup region, and the backup region provides write service; when the primary region recovers, sending a metadata replica upsizing request to the metadata service metad, the metadata replica upsizing request being used to restore the number of replicas of the metadata service metad; after the number of replicas of the metadata service metad is restored, sending a data replica upsizing request to the metadata service metad of the backup region of the leader node; performing a replica upsizing operation on the data shard according to the data replica upsizing request and an expected replica value; when the number of replicas of all data shards reaches the expected replica value, performing a balance leader operation to make the leader of the data shard appear in the primary region, and the primary region provides write service again.

[0006] Optionally, selecting a target storage node through a filter according to the StoreView comprises: selecting a target storage node for carrying a new replica according to at least one factor filter, the at least one factor comprising: isolation, node capacity and state, snapshot quantity, and replica filling efficiency.

[0007] Optionally, each storage node is maintained by a speed limiter, and during the replica upsizing process, the allocation rate of selecting storage nodes for the plurality of data shards is controlled, so that the selection of storage nodes for the plurality of data shards is sequentially staggered to uniformly allocate resources.

[0008] Optionally, performing the upsizing or downsizing operation of the replica comprises: sending a metadata replica adjustment instruction to the metadata service metad through an ngctl tool; sending a data replica adjustment instruction to the metadata service metad through an ngql tool; and performing replica adjustment processing by the leader node.

[0009] Optionally, the method further comprises: when it is detected that a disk failure of a storage node causes data loss, automatically downsizing the replicas of the affected shards; when it is identified in subsequent detection that the number of replicas of the shards is lower than an expected replica value, automatically triggering a replica upsizing process, selecting a new storage node, and issuing a raft group to the new storage node to complete automatic recovery of data.

[0010] Optionally, the method further comprises: in a data import scenario, constructing a single-copy cluster; wherein the replica expansion is initiated by specifying a desired number of replicas of a single Zone or multiple Zones, and the distribution adjustment of the replicas is completed by the metadata service metad according to the filter and the speed limiter.

[0011] Optionally, the method further comprises: when the metadata service metad receives a replica change request, persisting the change information locally; periodically reading the global configuration by a background scheduling thread of the metadata service metad, comparing the shard information with the desired replica value, and triggering the replica adjustment process when they are inconsistent.

[0012] Optionally, the method runs in a graph database system composed of a metadata service metad, a storage service storaged and a query service graphd, wherein the query service graphd obtains a shard distribution view from the metadata service metad at a time, and adjusts the access strategy according to the available state of the Zone.

[0013] The embodiment of the application provides a device for adaptive adjustment of replicas based on a graph database, comprising: a configuration module, configured to set a corresponding desired replica value for each Zone in a pre-defined cluster creation stage; a detection module, configured to detect the replica state reported by a storage layer in real time, to obtain a PartView of the current data replica distribution and a StoreView of the physical resource usage; a comparison module, configured to compare the number of replicas in the PartView with the desired replica value; a first adjustment module, configured to generate a replica change instance when the number of replicas is different from the desired replica value, and to select a target storage node according to the StoreView through a filter and a speed limiter, to perform an expansion or contraction operation of the replica; and a second adjustment module, configured to end the replica adjustment process when the number of replicas is the same as the desired replica value.

[0014] The embodiment of the application provides a method and device for adaptive adjustment of replicas based on a graph database, by setting a corresponding desired replica value for each Zone in a pre-defined cluster creation stage; detecting the replica state reported by a storage layer in real time, to obtain a PartView of the current data replica distribution and a StoreView of the physical resource usage; comparing the number of replicas in the PartView with the desired replica value; generating a replica change instance when the number of replicas is different from the desired replica value, and selecting a target storage node according to the StoreView through a filter and a speed limiter, to perform an expansion or contraction operation of the replica; and ending the replica adjustment process when the number of replicas is the same as the desired replica value, thereby achieving the technical effect of improving the availability of the database. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings: Figure 1 A flowchart of a method for adaptive adjustment of replicas based on a graph database according to an embodiment of the present application is shown in FIG. 2. Figure 2 A flowchart of another method for adaptive adjustment of replicas based on a graph database according to an embodiment of the present application is shown in FIG. 3. Figure 3 A schematic diagram of a device for adaptive adjustment of replicas based on a graph database according to an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0016] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0017] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to limit a specific order.

[0018] It should also be noted that the various embodiments described below can be executed independently, and the various embodiments can also be executed in combination with each other, and the embodiments of the present application do not specifically limit this.

[0019] Technical terms related to the embodiments of the present application: Graph Database: A database that stores data in a graph structure, capable of efficiently processing complex relational data.

[0020] Primary zone: A geographic region or availability zone (AZ) that provides read and write services in normal state.

[0021] Backup zone: An availability zone that is geographically isolated from the primary zone and is used for disaster recovery, and will assume a read-only or warm backup role in normal times.

[0022] High Availability: The ability of a system to continue to provide services in the event of partial component failure.

[0023] Replica: Multiple copies of data used to improve data reliability and availability.

[0024] Desired replica value: the target number of replicas set by the user for the data.

[0025] Partition failure: the primary zone network is unreachable, power failure or node downtime, resulting in the loss of service capability of the partition.

[0026] Disk failure: single node disk damage or data loss, but node process / OS survives.

[0027] metad: metadata service.

[0028] graphd: stateless query service.

[0029] store / storaged: storage service.

[0030] leader: master node in multiple copies.

[0031] ngctl: nebula graph built-in tool for interacting with metad to adjust resources.

[0032] ngql: nebula graph built-in tool for interacting with graphd to execute gql statements.

[0033] raft group: multiple copies of data, using raft synchronization protocol between copies to ensure strong consistency of data.

[0034] reset / member change: the former forces to modify the number of replicas of the raft group when the majority is unavailable; the latter ensures that the majority is available to change the member normally.

[0035] balance leader: that is, perform a balance adjustment of the leader distribution to ensure that the number of leaders on each node is roughly uniform.

[0036] The embodiment of the application provides a kind of based on graph database's copy adaptive adjustment method, Figure 1 For the process schematic diagram of the copy adaptive adjustment method based on graph database provided by the embodiment of the application; As Figure 1 The copy adaptive adjustment method based on graph database provided by the embodiment of the present application comprises: Step S100, in the cluster creation phase, set the corresponding desired replica value for each Zone predefined; Step S102, real-time detection of the replica state reported by the storage layer, to obtain the current data replica distribution PartView and the physical resource usage StoreView; Step S104, comparing the number of copies in the PartView with the expected copy value; Step S106, when the number of copies is different from the expected copy value, generating a copy change instance, and selecting a target storage node through a filter and a speed limiter according to the StoreView, and performing a capacity expansion or capacity reduction operation of the copy; Step S108, when the number of copies is the same as the expected copy value, the copy adjustment process ends.

[0037] Optionally, the method for adaptive adjustment of copies based on a graph database provided by the embodiment of the application further comprises: when the primary area fails, sending a metadata copy capacity reduction request to a raft group of a metadata service metad; performing a member change operation on the raft group of the metadata service metad according to the metadata copy capacity reduction request, so that the metadata service metad of the backup area becomes a leader node; after the metadata service metad of the backup area becomes the leader node, sending a data copy capacity reduction request to the leader node; and performing a data copy capacity reduction operation on the leader node according to the data copy capacity reduction request, so that the leaders of all data shards are concentrated in the backup area and the backup area provides write services.

[0038] Further, optionally, when the primary area recovers, a metadata copy capacity expansion request is sent to the metadata service metad, the metadata copy capacity expansion request is used to restore the number of copies of the metadata service metad; after the number of copies of the metadata service metad is restored, a data copy capacity expansion request is sent to the metadata service metad of the backup area of the leader node; a data shard copy capacity expansion operation is performed according to the data copy capacity expansion request and an expected copy value; and when the number of copies of all data shards reaches the expected copy value, a balance leader operation is performed, so that the leader of the data shard appears in the primary area and the primary area provides write services again.

[0039] Optionally, selecting a target storage node through a filter according to the StoreView comprises: selecting a target storage node for carrying a new copy according to at least one factor filter, the at least one factor comprising: isolation, node capacity and state, snapshot quantity, and copy filling efficiency.

[0040] Optionally, each storage node is maintained through a speed limiter, wherein, in the copy capacity expansion process, the allocation rate of selecting storage nodes for multiple data shards is controlled, so that the selection of storage nodes for multiple data shards is sequentially staggered to uniformly allocate resources.

[0041] Optionally, the method for adaptive adjustment of replicas based on a graph database provided in the embodiments of the present application further includes: when it is detected that a disk failure occurs in a storage node, causing data loss, automatically reducing the number of replicas of the affected shard; and when it is identified in subsequent detection that the number of replicas of the shard is lower than the expected replica value, automatically triggering a replica expansion process, selecting a new storage node, and issuing a raft group to the new storage node, thereby completing automatic recovery of the data.

[0042] Optionally, the method for adaptive adjustment of replicas based on a graph database provided in the embodiments of the present application further includes: in a data import scenario, constructing a single-replica cluster; and initiating replica expansion by specifying the expected number of replicas of a single Zone or multiple Zones, and completing distribution adjustment of the replicas according to a filter and a limiter by a metadata service metad.

[0043] Optionally, the method for adaptive adjustment of replicas based on a graph database provided in the embodiments of the present application further includes: initiating replica expansion or reduction by sending a metadata replica adjustment instruction to the metadata service metad through an ngctl tool, sending a data replica adjustment instruction to the metadata service metad through an ngql tool, and performing replica adjustment processing by a leader node.

[0044] Optionally, the method for adaptive adjustment of replicas based on a graph database provided in the embodiments of the present application further includes: when the metadata service metad receives a replica change request, persisting the change information locally; periodically reading a global configuration by a background scheduling thread of the metadata service metad, comparing shard information with an expected replica value, and triggering a replica adjustment process when they are inconsistent.

[0045] Optionally, the method for adaptive adjustment of replicas based on a graph database provided in the embodiments of the present application is run in a graph database system composed of a metadata service metad, a storage service storaged, and a query service graphd, wherein the query service graphd obtains a shard distribution view from the metadata service metad at a timing, and adjusts an access strategy according to an available state of a Zone.

[0046] In summary, Figure 2 Another flowchart of a method for adaptive adjustment of replicas based on a graph database provided in the embodiments of the present application is shown in FIG. 2. Figure 2 The method for adaptive adjustment of replicas based on a graph database provided in the embodiments of the present application is specifically as follows: The method for adaptive adjustment of replicas based on a graph database provided in the embodiments of the present application mainly includes the following functions: Function 1: High availability support: allows users to customize the number of Zone replicas, and when the main Zone fails, the standby Zone has read and write capabilities by adjusting the number of replicas; when the main Zone recovers, the number of replicas is adjusted again, and the main Zone regains write capability; during this process, only the number of replicas needs to be adjusted by the user, and the operations of master-slave switching and replica completion are completed by the service itself; Function 2: Data loss: when the upper layer detects that a node has data loss, the corresponding shard replica will be scaled down immediately, and then adaptive replica scaling is performed to ensure the number of replicas; As shown in Figure 2 , at the entrance, the operations of adjusting metad and adjusting storaged are performed through ngctl and ngql tools, respectively.

[0047] For adjusting metad, the specific change member needs to be specified in the command, and whether a forced reset operation is required.

[0048] For adjusting storaged, the request is submitted to the metad leader node, which performs specific replica adjustment processing.

[0049] Regarding the complete replica scaling flowchart, it has been marked in Figure 2 : After the user initiates a replica change request, the metad writes the scaling information to the local and ensures persistence; The background scheduling thread of metad periodically reads the global configuration to confirm whether a modification has occurred; After confirming the modification, the shard information is traversed and compared to determine whether there is inconsistency; After inconsistency is found, scaling is considered, and the optimal storaged needs to be found before scaling, because a large number of shard scaling is involved, so the present application embodiment needs a filter and a speed limiter to control the success rate and rate of shard scaling; After the optimal one is found, the replica creation and synchronization request instructions are issued; Finally, metad waits for the heartbeat information reported by storaged to confirm whether the scaling is completed, and compares the latest view information with the expected value; After the change is completed, the next configuration change request is waited for, and thus a complete shard change process is completed.

[0050] In a preferred example, in a disaster recovery scenario, when a primary zone fails, the built-in tool ngctl inputs the command scalemetad = {…} to send a request to the metad of the standby zone to scale down the replica, and the built-in tool ngql inputs the command scale replica ={…} to send a request to the metad to scale down the data replica after the metad becomes the leader, and all the leaders of the data shards are concentrated in the standby zone to provide write services. When the primary zone recovers, the number of metads and data replicas is adjusted again through the ngctl command, the number of replicas is adjusted adaptively within the service, and the primary zone reacquires write services.

[0051] In a preferred example, the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the present application is as follows in the initialization process: In the cluster creation phase, the default expected value of each Zone is given in the process of creating the Zone, wherein the corresponding Zone information is created, including the corresponding priority and the number of replicas, and is represented as follows: ngctl svcgrp add-zone <cluster><zone_name> -p <priority> -r <replica-factor> After the resource is created, the system internally records all zones created under the cluster through the sub-table record, zone table, including the corresponding number of replicas and priority. All creation processes are determined based on the user's use scenario, so the user customizes the size of the zone, the division of the master and slave zones, and the separate allocation of resources.

[0052] In a preferred example, the method for adaptive adjustment of replicas based on a graph database provided by the embodiment of the application in the process of failure of the master zone is as follows: ngctl scale meta {h1:p1} [force] to perform the scale-in operation of the meta, as follows: The entire metad replica is a raft group, and here a member change is performed. Since the member may be unavailable, a forced reset is performed instead of a member change; ngql > scale replica {zone1=0, zone2=1} [force] to perform the scale-in operation of the data replica, as follows: The metad internally maintains a global configuration to record the expected value. For example, in the initialization phase, the configuration data is set to 3 replicas. Here, after the initialization is completed, the expected value is recorded as 3. After the storaged reports the complete replica information, it is confirmed to be consistent with the expected value, and the initialization is completed; PartView reflects the current latest data view of the storage layer, and StoreView reflects the physical layer information, such as the resource usage of a single machine, snapshot information, etc. The scheduler periodically compares the replica number in the view with the expected value, and when it is confirmed that the replica number is inconsistent, a change instance is created to handle the inconsistency problem. For the scale-in scenario, a forced replica scale-in is required, so here only a reset request is needed for all shards; When it is confirmed that PartView is consistent with the expected value, it is considered that the expected value has been reached, and no change instance is generated, and the scale-in is completed.

[0053] In a preferred example, the method for adaptive adjustment of replicas based on a graph database provided by the embodiment of the application in the running state process is as follows: In terms of access form, graphd will periodically obtain the latest shard distribution view information from the metad node. When it is found that only the standby zone is available, the zone-based access strategy will be invalid, which is equivalent to having no concept of zone.

[0054] In a preferred example, the method for replica adaptive adjustment based on a graph database provided by the embodiments of the application in the process of master zone recovery is as follows: ngctl scale meta {h1:p1, h2:p2, h3:p3} to perform the expansion operation of meta; ngql > scale replica {zone1=2, zone2=1} to perform the expansion operation of data replica; Wherein, when the data replica is confirmed to be inconsistent, the corresponding replica expansion is performed according to the number of replicas specified for each Zone; Because the change behavior involves all shards, a filter + rate limiter is provided here to limit the content and rate of generating change instances to avoid unstable impact on the cluster, and the specific execution process is as follows: Suppose there is initially 1 storage node, 3 shard replica number is 1, now 3 machines are added, and the replica number is adjusted to 3, the scheduler in metad confirms that the replica number is inconsistent, and a change instance is created for each shard to perform the change; When multiple replicas are selected here, isolation (the same storaged cannot store multiple replicas of the same shard), node capacity / state, snapshot number, replica filling efficiency, etc. are considered, each factor point is sorted into a filter, which is used when selecting effective storaged nodes, and will affect the final selection of storaged; Because all change instances are executed in series in the actual execution phase, and in the case that all storaged nodes are in normal state, the information seen is consistent, such as 2, 3, and 4 effective nodes when selecting storaged for shard 1, 2, 3, and 4, shard 2 also sees 2, 3, and 4, and shard 3 also sees 2, 3, and 4, because the new machines are empty, so the rules based on the sorting are consistent, which will cause a problem, assuming that shard 1 selects 2 and 3, and shard 2 and 3 also select 2 and 3 according to the information seen, which results in a machine being empty, causing resource waste. The essential problem here is that after the shard is selected, it does not have some impact on the other shards. The solution is to maintain a rate limiter for each storaged, which can be controlled at the second level, and select one storaged at a time until the desired value is met. Still in the above scenario, shard 1 selects 2, shard 2 selects 3, and shard 3 selects 4; again, the Ns limit is required, and then shard 1 selects 3, shard 2 selects 4, and shard 3 selects 1, and thus the new replica position is successfully confirmed, and the replica creation and synchronization change process begins.

[0055] Similarly, when the PartView is consistent with the expected value, it is considered that the expectation has been reached, and no Event is generated, and the expansion is completed.

[0056] Because the underlying uses raft, a balance leader needs to be executed to ensure that the leader appears in the primary zone, and the primary zone provides write services.

[0057] In a preferred example, the method for adaptive adjustment of replicas based on a graph database provided by the embodiment of the application in other scenarios is as follows: Scenario 1, derivative: Build a single replica cluster (here, set the number of replicas to 1 when creating through ngctl; other operations remain unchanged; derivative operations are generally executed through a derivative tool inside nebula graph); Start expanding replicas (here, the parameters for replica expansion, single Zone or multi-zone, specify the expected number of replicas), if there is a new Zone requirement, a new Zone will be created at the same time, and the number of replicas is specified as 0; After all Zones are created, start executing the scale replace instruction.

[0058] Scenario 2, disk damage, data loss (3 replicas): When reporting on the storage layer, it will be aware that there is a shard loss, at which time a replica shrink operation from 3 replicas to 2 replicas will be performed on the current shard; The metad will be aware of the number of shard replicas that is less than expected during the next detection, and will trigger the behavior of expanding replicas, and will select a new storage node and issue a raft group; At this point, no manual intervention is required, and the automatic expansion of replicas is completed.

[0059] In a preferred example, the method for adaptive adjustment of replicas based on a graph database provided by the embodiment of the application in the cluster creation stage: Through the setting of Zone, real-time sensing, and the coordination of rate control based on multi-factor filtering, closed-loop support is provided for subsequent adaptive adjustment.

[0060] First, perform the Zone configuration operation through the command ngctl svcgrp add-zone <cluster><zone_name> -p <priority> -r <replica-factor>Create Zone information, where -p <priority>Business priority of the specified Zone (primary zone priority is higher than secondary zone), -r <replica-factor>Set the expected replica value of the corresponding Zone. Through the sub-table "zone table", record the number of replicas, priority and resource allocation information of all Zones under the cluster. The expected replica value will be used as the baseline threshold for all subsequent adjustment operations, directly related to the trigger conditions of real-time perception, the screening direction of multi-factor filtering and the execution scale of rate control.

[0061] For example, create primary zone Zone1 (priority P1=3) and backup zone Zone2 (priority P2=2), and set the expected replica value to Zone1=2 and Zone2=1, with a total replica number of 3. This configuration not only clearly defines the replica distribution target in the normal state, but also presets the priority rules for adaptive switching in fault scenarios, i.e., when the primary zone fails, the backup zone can quickly take over the service based on its own priority and expected replica value; after the primary zone recovers, the system can restore the write service authority of the primary zone according to the priority.

[0062] At the same time, during cluster initialization, the detection module synchronously starts the real-time monitoring mechanism, regularly acquires the replica state reported by the storage layer, and preliminarily constructs PartView (shard-replica mapping view) and StoreView (node capacity, snapshot number, survival state, load situation, and other physical resource association views); the multi-factor filter completes the initial parameter configuration, and presets the screening weights of isolation, node capacity and state, snapshot number, and replica filling efficiency (isolation 0.3, node capacity and state 0.25, snapshot number 0.15, and replica filling efficiency 0.1); the speed limiter initializes and allocates a counter for each storaged node, sets the zero-clearing period to 1 second and the maximum number of replica allocation requests per second to 2, laying a foundation for subsequent resource balanced allocation of expansion and contraction operations.

[0063] In a preferred example, the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the present application in the scenario of primary zone failure: When the primary zone experiences partition failure such as network interruption, power failure, or node downtime, the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the present application performs multi-factor filtering through real-time perception, and then performs coordinated linkage of rate control, achieving rapid degradation and backup zone takeover without manual intervention. The specific coordination process is as follows: Step1, real-time perception trigger: by timing monitoring the heartbeat signal of the storaged node, it is found that the heartbeat of all nodes in the primary zone Zone1 is lost, the PartView and StoreView are synchronously obtained, the PartView shows that the number of each shard replica is 3 (Zone1=2, Zone2=1), the StoreView confirms that the 2 storaged nodes in Zone2 are in normal state (capacity usage rate 60%, snapshot number 2, load lower than threshold), and then a replica adjustment signal is triggered, and the state information is synchronized to the raft group of metad.

[0064] Step2, metadata replica cooperative downsizing: the user inputs the command ngctl scale meta{h1:p1} [force] through the ngctl tool to send a metadata replica downsizing request to the raft group of the metadata service metad. Since the primary zone node is unavailable, the metad performs a regular member change by a forced reset operation to perform a raft group member change, and filters the Zone2 node with the best state to upgrade to a leader, that is, the filtering process has integrated the core logic of multi-factor filtering, and the node with sufficient capacity, few snapshots, and fast response rate is preferentially selected to ensure the rapid recovery of the metadata service.

[0065] Step3, data replica cooperative downsizing: after the standby zone metad becomes a leader node, the command ngql > scale replica {zone1=0, zone2=1} [force] is input through the ngql tool to send a data replica downsizing request to the leader node. The scheduler of metad filters the available nodes in Zone2 based on the physical resource information in StoreView: first, the isolation is the primary condition, and the nodes in the same shard replica set are excluded; then, the weight scoring algorithm is used to comprehensively score the nodes in Zone2 (example node score 0.9), to determine the optimal candidate node; finally, the speed limiter controls the downsizing rate, and only 1 shard downsizing operation per second is processed to avoid consistency protocol conflicts caused by simultaneous operations.

[0066] Step4, seamless service takeover: after the scaling-in operation is completed, the leaders of all data shards are concentrated in Zone2, and the PartView displays the number of shard replicas consistent with the adjusted expected replica value (Zone1=0, Zone2=1), and the standby zone officially provides write services. The entire collaborative process takes no more than 30 seconds, which is significantly lower than the RTO of more than 30 minutes of existing technologies, and the core benefits from real-time perception and rapid triggering, filtering and screening optimal nodes, rate control to ensure stable closed-loop collaboration.

[0067] In a preferred example, the method for replica self-adaptive adjustment based on a graph database provided by the embodiments of the application in a master zone recovery scenario: When the master zone is recovered and the nodes are normal, the scaling-out operation is performed through multi-link collaboration to restore the write service capability of the master zone. The specific collaborative process is as follows: Step1, metadata replica collaborative scaling: send a metadata replica scaling request to metad through the command ngctl scale meta {h1:p1, h2:p2, h3:p3}, and metad detects the node state (capacity, load, snapshot quantity, etc.) of the recovered master zone based on StoreView, selects the nodes that meet the conditions through a multi-factor filter, and re-adds them to the raftgroup, restores the number of metadata service replicas, and ensures the redundancy and reliability of the control plane.

[0068] Step2, data replica collaborative scaling: after the number of metadata replicas is restored, send a data replica scaling request through the command ngql > scale replica {zone1=2, zone2=1}. The scheduler of metad compares the current PartView (replica number 1) with the expected replica value (Zone1=2, Zone2=1), confirms that there is a difference, and generates a replica change instance, and starts the collaborative execution mechanism of filtering and rate limiting: Multi-factor filtering selection: for each shard, first, ensure that the new replica of the same shard is not distributed in the same Zone or adjacent node as the existing replica; then, combine the capacity utilization (which needs to be less than 80%) of the recovered master zone node, the state (which needs to be normal), the snapshot quantity (≤3), and the replica filling efficiency (≥100MB / s) to score the weight, and select the optimal candidate node (master zone node first, because it has a higher priority).

[0069] Rate control allocation: the rate controller controls the allocation rate based on the counter, and each node receives at most 2 shard copy requests per second. For example, the cluster contains 3 shards, and the primary zone recovers 3 nodes. When scaling, shard 1 selects primary zone node A first, shard 2 selects primary zone node B, and shard 3 selects primary zone node C. After waiting for 1 second for the counter to clear, shard 1 selects primary zone node B again, shard 2 selects primary zone node C, and shard 3 selects primary zone node A, ensuring uniform resource allocation and avoiding node overload.

[0070] Step 3, Leader balance cooperative switching: when the number of replicas of all data shards reaches the expected replica value, the balance leader operation is performed. This operation also links the multi-factor filter, and based on factors such as node load and leader number, the leader node of the data shard is migrated to the primary zone node. During the screening process, the node with fewer leaders and lower load in the primary zone is preferentially selected to ensure the load balancing of the primary zone node. Finally, the primary zone provides write service again, the standby zone resumes the warm standby role, and the entire cooperative scaling and switching process is smooth without service interruption.

[0071] In a preferred example, the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the application in a disk failure scenario: When a disk failure occurs on a storage node, causing data loss (disk failure), the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the application realizes 3-2-3 automatic self-healing through real-time sensing, and then through the closed-loop cooperation of shrinkage, filtering, and rate limiting to expansion, the specific process is as follows: Step 1, real-time sensing of failure and emergency shrinkage: through the state information reported by the storage layer, the disk failure signal of a certain storaged node is captured in real time, confirming that the replicas of 10 shards on the node are lost, and the PartView displays that the number of replicas of these shards decreases from 3 to 2. To maintain the normal operation of the consistency protocol of the remaining replicas, the cooperative adjustment module automatically triggers the shrinkage operation to officially shrink the number of replicas of the affected shards to 2, updates the PartView, and synchronizes the failure information to the metad.

[0072] Step 2, difference detection and expansion triggering: the background scheduling thread of the metad periodically reads the global configuration, compares the shard information with the expected replica value, finds that the number of replicas of the affected shards (2) is lower than the expected replica value (3), and immediately triggers the replica expansion process to generate a replica change instance containing the target number of replicas (3) and the identifier of the shard to be adjusted.

[0073] Step3, filtering and rate limiting for expansion: metad, based on StoreView, filters new storage nodes through a multi-factor filter, prioritizing nodes that are different from the failed node, have a capacity usage rate below 70%, have a small number of snapshots, and have a high fill efficiency (example score 0.85). The rate limiter controls the expansion rate by allocating 1 shard replica request per second for each target node, avoiding resource contention. Subsequently, the raft group is issued to the selected nodes, and data synchronization is completed through the raft synchronization protocol.

[0074] Step4, verification of self-healing completion: the detection module monitors the replica synchronization progress in real time and periodically obtains PartView. When the number of replicas for all affected shards is restored to 3 and consistent with the expected replica value, the self-healing process is complete. The entire process does not require human intervention and takes no more than 10 minutes, with a data loss risk reduced to below 0.01%. The core benefit is the multi-link closed-loop coordination through emergency handling based on fault awareness and subsequent synchronization recovery through screening and distribution.

[0075] In a preferred example, the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the present application is used in a data import scenario: In a data import scenario, to improve import speed and reduce failure probability, a single replica cluster is built and efficiently expanded through multi-link coordination, and the specific process is as follows: Step1, single replica cluster construction: when creating a cluster through the ngctl tool, set the expected replica value of all Zones to 1 to construct a single replica cluster, avoiding the impact of multi-replica synchronization on import speed. At this time, the detection module continuously monitors the storage layer state, PartView records the shard distribution of single replicas, and StoreView updates the capacity occupation and load of nodes in real time.

[0076] Step2, coordinated expansion execution: after data import is complete, initiate an expansion request by specifying the expected replica value of a single Zone or multiple Zones (such as ngql > scale replica {zone1=2, zone2=1}). metad generates a replica change instance, and then links multi-factor filters and rate limiters to complete replica distribution adjustment: The filter prioritizes nodes with sufficient capacity, low load, and a small number of snapshots to ensure replica synchronization efficiency; The rate limiter controls the expansion rate to avoid instability caused by a large number of shards synchronizing simultaneously, ensuring that the imported data is safe and quickly completes the redundancy configuration.

[0077] In a preferred example, the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the application provides a detailed workflow of the multi-link collaborative adaptive adjustment mechanism, as follows: The core of the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the application is to construct and run an adaptive system closely coordinated by multiple links such as target setting, state sensing, intelligent decision-making and balanced control, closed-loop execution, etc. The following explains how each link specifically cooperates to achieve intelligent management of graph database replicas when the system is running.

[0078] Step 1, definition and data interface of each collaborative link Target setting link: Zone and expected replica value configuration A global configuration table is maintained to record the unique identifier, priority and corresponding expected replica value of each predefined available zone (Zone). The value is set through a management command (such as ADD ZONE) when the cluster is created, and can be dynamically updated at runtime. For example, the expected replica value of the primary zone (Zone_A) can be set to 2, and the backup zone (Zone_B) to 1, which constitutes the target baseline of the entire adaptive adjustment process.

[0079] The output of this link is a mapping relationship Map<Zone_ID, Expected_Replica_Count>, which is the input of the decision comparison link.

[0080] State sensing link (double view: PartView and StoreView): Specific generation of PartView (logical view): Each storage service (storaged) periodically reports (for example, every second) the replica state of all data partitions (Partition) it is responsible for to the metadata service (metad), including replica ID, node, Raft group role (Leader / Follower), log synchronization progress, etc. The metad aggregates all node reporting information to generate a global PartView, which is essentially a data structure reflecting the actual number and distribution of each data partition in the cluster at the current time.

[0081] Specific generation of StoreView (physical view): The storage service also reports the physical resource state of the node it is in, including disk capacity usage, memory usage, CPU load, network bandwidth, number of snapshot tasks being executed, etc. The metad aggregates to generate StoreView, which is a data structure reflecting the health and load capacity of each storage node in the cluster at the current time.

[0082] The update cycle of PartView and StoreView can be dynamically adjusted according to system load. When a large difference in the number of replicas or a fault event is detected, the reporting and aggregation frequency can be temporarily increased to achieve adaptive sensing accuracy.

[0083] Intelligent decision-making and balanced control (filter and speed limiter collaboration): Set decision triggers, and the background scheduler compares the actual number of replicas in PartView for each shard with the expected value of the corresponding Zone provided by the target setting link every period (for example, every 5 seconds). Once a discrepancy (too many or too few) is found, a replica change instance is generated for the shard.

[0084] Multi-factor filter (intelligent screening) For a shard that needs to increase replicas, the filter scores all candidate storage nodes using StoreView as input. The scoring function considers multiple factors, such as: Isolation factor: Prefer nodes that are not in the same physical rack, switch, or Zone as the existing replicas of the shard to prevent cascading failures.

[0085] Capacity and status factor: Select nodes with sufficient disk space (e.g., usage <80%), low load (normal CPU / memory usage), and status "healthy".

[0086] Efficiency factor: Avoid selecting nodes that are performing a large number of snapshots or data compression to reduce interference with replica synchronization performance.

[0087] The filter outputs an ordered list of candidate nodes in descending order of comprehensive score.

[0088] Rate controller (speed limiter) maintains a lightweight token bucket rate limiter for each storage node. Each token represents a "permission" to add a new replica task on the node. Tokens are replenished at a fixed rate (e.g., 1-2 per second) to control the upper limit of new replica creation tasks that a single node can handle per unit of time.

[0089] The collaborative decision-making process includes: when selecting a target node for a certain shard, it is not directly selected as the first in the filter list. Instead, it checks whether the nodes in the list have available tokens. The first node with available tokens is selected and one token is consumed. This mechanism ensures: Spatial balance: High-scoring nodes have tokens quickly consumed due to frequent selection, and subsequent shards' filter outputs may be the same, but the node has no tokens, so the passive balanced distribution of replicas among high-quality nodes is achieved.

[0090] Time smoothing: Token replenishment rate limits the instantaneous pressure of change operations on a single node, making cluster load changes more smooth and avoiding performance jitter caused by concentrated operations.

[0091] Closed-loop execution and self-healing: During normal scaling execution, according to the target node selected by the above-mentioned coordination link, the metadata service issues a Raft member change instruction to the corresponding storage node. After execution is completed, the new state will be captured through the next cycle of the state awareness link, and will enter the comparison again until the PartView and the expected value are consistent, forming a closed loop of execution, feedback and verification.

[0092] Among them, the coordination of the disk failure closed-loop self-healing (3-2-3) is embodied: this scenario is a high embodiment of the above-mentioned multi-link coordination capability.

[0093] First stage: rapid perception and response (safe degradation). When StoreView perceives that a node disk fails (such as an IO error), the system immediately identifies the affected data shard.

[0094] At this time, skip the complex filter scoring process, directly target to ensure the availability of the remaining replica consistency protocol, and initiate a forced scaling down (member removal) request to the Raft group of the shard, reducing the number of replicas from N (such as 3) to N-1 (2), which prioritizes the immediate availability of the service.

[0095] Second stage: steady-state coordinated recovery (intelligent reconstruction), in the subsequent periodic scheduling, the scheduler discovers through PartView that the number of replicas (2) of the shard is lower than the expected value (3), and then generates a standard replica scaling change instance.

[0096] The instance will go through the intelligent decision-making and balanced control link to select the optimal new node through the filter, obtain operation permission through the speed limiter, and then smoothly execute the scaling operation to restore the number of replicas from 2 to 3 and complete data synchronization.

[0097] Through the mode switching from rapid response to steady-state recovery, the self-adaptive decision-making capability of the replica self-adaptive adjustment method based on a graph database provided by the embodiments of the present application for different fault severity is exhibited, and a real closed-loop self-healing is realized.

[0098] Step 2, driving and scheduling of the coordinated workflow The entire multi-link coordinated workflow is driven by a background scheduling engine. The work of the background scheduling engine is not a simple polling, but has a state machine logic: Listen to configuration changes: receive user modifications to the expected replica value.

[0099] Trigger global comparison: compare new expectation with current PartView, generate a queue of pending shard change instances.

[0100] Schedule execution: take instances from queue, execute above-mentioned intelligent decision and balance control and closed-loop execution process in turn. Governor mechanism ensures that the execution of instances in the queue is staggered and smooth.

[0101] State synchronization and cycle: wait for the next sensing period, verify the execution result using the updated PartView. If the expectation has been achieved, mark the instance as complete; if not (for example, the target node operation fails), requeue the instance or trigger an exception handling process. This process is repeated, so that the system always converges towards the desired state.

[0102] As can be seen from the above detailed workflow, the adaptive adjustment of the replica adaptive adjustment method based on a graph database provided by the embodiments of the present application is not a simple stacking of independent functions, but a deep coupling of multiple links through the carefully designed data flow interface, i.e., the dual view of PartView and StoreView; the decision logic, i.e., the filter and governor; and the feedback loop, i.e., the execution verification, forming an intelligent collaborative system that can cope with various complex scenarios and has efficiency and stability.

[0103] The core of the replica adaptive adjustment method based on a graph database provided by the embodiments of the present application lies in the construction of a five-link collaborative closed loop of Zone setting (benchmark layer), real-time sensing (trigger layer), multi-factor filtering (screening layer), rate control (execution layer) and disk failure closed-loop self-healing (bottom layer), each link is not independent operation, but mutual support, dynamic linkage: The benchmark layer (Zone setting) provides target orientation and priority rules for the entire collaboration, ensuring that the adjustment operation meets business requirements; The trigger layer (real-time sensing) captures state changes through dual views, providing accurate trigger signals for collaborative adjustment; The screening layer (multi-factor filtering) selects the optimal node based on multi-dimensional parameters, solving the core problem of "who to assign"; The execution layer (rate control) ensures uniform distribution of resources, avoiding failure of the screened node due to overload; The bottom layer (disk failure self-healing) integrates fault handling into the collaborative closed loop, achieving automatic recovery in abnormal scenarios.

[0104] The cooperative closed loop effectively solves the pain points of static rigidity, manual intervention and resource imbalance of traditional graph database replica management, and the cooperative effect can be verified by quantifying data: through the cooperative rapid scaling and switching mechanism of the metadata and the data replica, the service recovery of the primary zone failure can be realized within minutes or even faster, compared with the traditional scheme relying on manual intervention, the recovery time objective (RTO) is significantly reduced, that is, the RTO of the primary zone failure is shortened from more than 30 minutes to within 30 seconds; through the closed-loop self-healing process of stable preservation by scaling down first and intelligent expansion later, the data replica redundancy can be efficiently and automatically recovered, greatly reducing the risk of permanent data loss caused by disk failure, for example, the success rate of disk failure self-healing is 99.9%, the scaling and scaling failure rate is reduced from 30% to 0.5% or less, and the storage node load balancing degree is improved by 60%, which significantly improves the availability, automation level and resource utilization efficiency of the graph database.

[0105] It should be noted that the specific numerical values involved in the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the present application are only described as optimal examples, and the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the present application is used as a reference, and is not limited. In the method for adaptive adjustment of replicas based on a graph database provided by the embodiments of the present application, the replicas can be adaptively adjusted, the read-write capability can be quickly recovered after the primary zone failure, manual intervention is not required, the availability and data import performance are improved; when the shard data is lost, 3-2-3 data recovery can be performed internally to realize disk-level failure self-healing; during the scaling and scaling of the replicas, the next replica creation position is confirmed according to the information of the physical view, the filter and the limiter are used to balance the resources, and resource use balancing is realized.

[0106] The embodiments of the present application provide a method for adaptive adjustment of replicas based on a graph database, which sets a corresponding expected replica value for each Zone in the cluster creation stage; detects the replica state reported by the storage layer in real time to obtain a PartView of the current data replica distribution and a StoreView of the physical resource use condition; compares the number of replicas in the PartView with the expected replica value; when the number of replicas is different from the expected replica value, a replica change instance is generated, and a target storage node is selected according to the StoreView through a filter and a limiter to perform a scaling or scaling operation of the replica; when the number of replicas is the same as the expected replica value, the replica adjustment process ends, thereby achieving the technical effect of improving the availability of the database.

[0107] The embodiments of the present application provide a device for adaptive adjustment of replicas based on a graph database, Figure 3 A schematic diagram of a device for adaptive adjustment of replicas based on a graph database provided by the embodiments of the present application is shown in FIG. Figure 3 As shown, the device for adaptive adjustment of replicas based on a graph database provided by the embodiment of the application comprises: The configuration module 30 is configured to set a corresponding expected replica value for each Zone in a predefinition in a cluster creation stage; the detection module 32 is configured to detect a replica state reported by a storage layer in real time to obtain a PartView of a current data replica distribution and a StoreView of a physical resource usage condition; the comparison module 34 is configured to compare the number of replicas in the PartView with the expected replica value; the first adjustment module 36 is configured to generate a replica change instance when the number of replicas is different from the expected replica value, and select a target storage node according to the StoreView through a filter and a speed limiter to perform an expansion or contraction operation of the replica; and the second adjustment module 38 is configured to end the replica adjustment process when the number of replicas is the same as the expected replica value.

[0108] The embodiment of the application provides a device for adaptive adjustment of replicas based on a graph database, which sets a corresponding expected replica value for each Zone in a predefinition in a cluster creation stage; detects a replica state reported by a storage layer in real time to obtain a PartView of a current data replica distribution and a StoreView of a physical resource usage condition; compares the number of replicas in the PartView with the expected replica value; generates a replica change instance when the number of replicas is different from the expected replica value, and selects a target storage node according to the StoreView through a filter and a speed limiter to perform an expansion or contraction operation of the replica; and ends the replica adjustment process when the number of replicas is the same as the expected replica value, so that the technical effect of improving the availability of a database can be achieved.

[0109] The above is only a preferred embodiment of the application, and is not used to limit the protection scope of the application. < / priority> < / priority> < / cluster> < / priority> < / cluster>

Claims

1. A method for replica self-adaptation based on a graph database, characterized in that, The method comprises: In the cluster creation phase, a corresponding expected replica value is set for each predefined Zone; Real-time detection of the replica state reported by the storage layer obtains the PartView of the current data replica distribution and the StoreView of the physical resource usage status; The number of replicas in the PartView is compared with the expected replica value; When the number of replicas is different from the expected replica value, a replica change instance is generated, and target storage nodes are selected through filters and limiters according to the StoreView to perform expansion or contraction operations of the replicas; When the number of replicas is the same as the expected replica value, the replica adjustment process ends.

2. The method for graph database based replica adaptation according to claim 1, c h a r a c t e r i z e d b y, The method further comprises: When the primary zone fails, a metadata replica contraction request is sent to the raft group of the metadata service metad; According to the metadata replica contraction request, a member change operation is performed on the raft group of the metadata service metad, so that the metadata service metad of the backup zone becomes a leader node; After the metadata service metad of the backup zone becomes a leader node, a data replica contraction request is sent to the leader node; According to the data replica contraction request, a contraction operation of the data replica is performed on the leader node, so that all leaders of the data shards are concentrated in the backup zone, and the backup zone provides write services; When the primary zone recovers, a metadata replica expansion request is sent to the metadata service metad, which is used to restore the number of replicas of the metadata service metad; After the number of replicas of the metadata service metad is restored, a data replica expansion request is sent to the metadata service metad of the backup zone of the leader node; According to the data replica expansion request and the expected replica value, a replica expansion operation of the data shard is performed; When the number of replicas of all data shards reaches the expected replica value, a balance leader operation is performed to make the leaders of the data shards appear in the primary zone, and the primary zone provides write services again.

3. The method for graph database based replica adaptation according to claim 1, c h a r a c t e r i z e d b y, The selection of the target storage node through the filter according to the StoreView comprises: According to at least one factor, the filter selects a target storage node for carrying a new replica, and the at least one factor includes isolation, node capacity and state, snapshot quantity, and replica filling efficiency.

4. The method for graph database based replica adaptation according to claim 3, c h a r a c t e r i z e d b y, The execution of the expansion or contraction operation of the replica comprises: Sending a metadata replica adjustment instruction to the metadata service metad through an ngctl tool; Sending a data replica adjustment instruction to the metadata service metad through an ngql tool; Performing a replica adjustment process by the leader node.

5. The method for adaptive adjustment of replicas based on a graph database according to claim 1, characterized in that, The speed limiter is used to maintain each storage node, wherein during the replica expansion process, the allocation rate of selecting storage nodes for multiple data shards is controlled, so that the resource is evenly distributed by sequentially staggering the selection of storage nodes for the multiple data shards.

6. The method for graph database based replica adaptation according to claim 1, c h a r a c t e r i z e d b y, The method further comprises: When a disk failure of a storage node is detected, causing data loss, the replicas of the affected shards are automatically scaled down; When the number of replicas of the shards is identified to be lower than the expected replica value in subsequent detection, the replica expansion process is automatically triggered, new storage nodes are selected, and the raft group is issued to the new storage nodes, completing the automatic recovery of data.

7. The method for graph database based replica adaptation according to claim 1, c h a r a c t e r i z e d b y, The method further comprises: In the data import scenario, a single replica cluster is constructed; wherein the replica expansion is initiated by specifying the expected number of replicas of single Zone or multiple Zones, and the metadata service metad adjusts the distribution of replicas according to the filter and the speed limiter.

8. The method for graph database based replica adaptation according to claim 1, c h a r a c t e r i z e d b y, The method further comprises: After the metadata service metad receives the replica change request, the change information is persisted to the local; The background scheduling thread of the metadata service metad periodically reads the global configuration, compares the shard information with the expected replica value, and triggers the replica adjustment process when they are inconsistent.

9. The method for graph database based replica adaptation according to claim 1, c h a r a c t e r i z e d b y, The method runs in a graph database system composed of a metadata service metad, a storage service storaged, and a query service graphd, wherein the query service graphd obtains the shard distribution view from the metadata service metad at regular intervals, and adjusts the access strategy according to the available state of Zone.

10. An apparatus for replica adaptation based on a graph database, the apparatus comprising: It includes: A configuration module is configured to set a corresponding expected replica value for each pre-defined Zone during the cluster creation stage; A detection module is configured to detect the replica state reported by the storage layer in real time to obtain the PartView of the current data replica distribution and the StoreView of the physical resource usage; A comparison module is configured to compare the number of replicas in the PartView with the expected replica value; A first adjustment module is configured to generate a replica change instance when the number of replicas is different from the expected replica value, and select target storage nodes according to the StoreView through a filter and a speed limiter to perform the expansion or scaling operation of the replica; A second adjustment module is configured to end the replica adjustment process when the number of replicas is the same as the expected replica value.

Images