A graph database load balancing method and device based on request type

Abstract

The present application discloses a graph database load balancing method and device based on request type, which relates to the field of graph database query technology, and solves the problem in the prior art that several complex queries with long time consumption and heavy load may all fall on the same query service, causing the query service to be under great pressure and slow response, because only focusing on the resource utilization of the server and ignoring the characteristics of the query request itself. The method includes: constructing a mapping table, and according to the complexity of the query request, independently polling and routing the requests for complex queries according to the type, and independently polling and routing the requests for non-complex queries. Since the complex queries are routed to different query services, the effective dispersion of complex query requests of the same type is achieved, and the pressure on each query service to process data is balanced and small, thereby effectively improving the overall query efficiency and reducing the query time.

Description

Technical Field

[0001] The present application relates to the field of graph database query technology, and in particular to a graph database load balancing method and device based on request type. Background Art

[0002] With the rapid development of big data and social network analysis, graph databases have become increasingly important due to their efficiency in processing complex relational data. However, executing complex query operations, especially those large-scale queries defined in the LDBC benchmark, often causes high load, resulting in longer query response times and lower overall system performance.

[0003] Traditional load balancing strategies usually distribute requests based on server resource usage (such as CPU and memory load), which may not achieve the best results when processing insufficient query diversity. For example, traditional load balancing strategies have limitations: Traditional load balancing strategies focus on server resource utilization, while ignoring the characteristics of the request itself (such as complexity, processing time, etc.). If the diversity of requests is insufficient, that is, most requests have similar characteristics, then these strategies cannot distinguish the processing requirements of different requests. If the characteristic type of the request is not considered when performing request load balancing, it may cause several complex queries that take a long time and have a heavy load to fall on the same query service, causing the service to be under great pressure and respond slowly, which will cause subsequent simple queries that fall on the service to take a long time due to queuing. Summary of the invention

[0004] The purpose of this application is to overcome the problem in the prior art that several complex queries that are time-consuming and heavy-loaded may all fall on the same query service, causing the query service to be under great pressure and slow to respond, which is caused by only focusing on the resource utilization of the server and ignoring the characteristics of the query request itself. A method and device for graph database load balancing based on request type is provided.

[0005] In a first aspect, a graph database load balancing method based on request type is provided, including:

[0006] Build a mapping table from query request type to graph database client connection pool list index;

[0007] Obtaining a query request and determining whether the query request is a complex query;

[0008] If the judgment result is yes, the query request type is obtained, and different query request types independently select a routing target graph database client connection pool from the mapping table in a polling manner;

[0009] If the judgment result is no, a polling method is used to select the routing target graph database client connection pool;

[0010] The query request is routed to the routing target graph database client connection pool to serve the corresponding target graph database.

[0011] In some possible implementations, the graph database client connection pool list includes at least two graph database client connection pools, each of the graph database client connection pools corresponds to a graph database service address, and an initial index is assigned to the query type.

[0012] In some possible implementations, determining whether the query request is a complex query includes:

[0013] Determine whether the amount of data involved in the query request exceeds a preset first threshold, and / or

[0014] Determine whether the processing time of the query request exceeds a preset second threshold.

[0015] In some possible implementations, different query request types independently select a routing target graph database client connection pool from the mapping table in a polling manner, including:

[0016] Initialize the mapping table from query request type to graph database client connection pool list index;

[0017] According to the query request type, obtain from the mapping table the graph database client connection pool list index to which the query type was last routed;

[0018] The index of the graph database client connection pool list routed last time is increased by 1 as the target index. If the query request corresponding to the query request type is the first query, the initial index corresponding to the query request type is used as the target index. The corresponding routing target graph database client connection pool is obtained from the graph database client connection pool list according to the target index.

[0019] In some possible implementations, the method further includes: receiving a response result from the target graph database service.

[0020] In some possible implementations, it also includes: performing logging, wherein the content of the logging includes at least one of the following: the routing target graph database client connection pool, the target graph database service, the response result, the query delay and the query time of each query request.

[0021] In a second aspect, a graph database load balancing device based on request type is provided, including:

[0022] A mapping table building module is used to build a mapping table from query request type to graph database client connection pool list index;

[0023] A processing module, used for obtaining a query request and determining whether the query request is a complex query;

[0024] If the judgment result is yes, the query request type is obtained, and different query request types independently select a routing target graph database client connection pool from the mapping table in a polling manner;

[0025] If the judgment result is no, a polling method is used to select the routing target graph database client connection pool;

[0026] The routing module is used to route the query request to the routing target graph database client connection pool to the corresponding target graph database service.

[0027] In a third aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0028] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes steps for executing the method in any one of the implementations of the first aspect described above.

[0029] In a fifth aspect, an electronic device is provided, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements a method as in any one of the implementations in the first aspect above.

[0030] The present application has the following beneficial effects: the present application distributes query load to different query service nodes according to the type of query request rather than the server status, and improves the overall business performance by dispersing the pressure of similar complex queries on query services by loading the same type of complex queries to different query request services. Since complex queries are loaded on different query services, the same type of complex query requests are effectively dispersed, and the pressure of each query service to process data is balanced and small, thereby effectively improving the overall query efficiency and reducing the query time. For simple query requests (i.e., non-complex queries) that take less time and occupy less CPU and memory, a polling method is used to select the service pool and is independent of the polling of complex queries, thereby ensuring that all query service nodes are evenly utilized, and further optimizing the fairness and efficiency of resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 It is a flowchart of the graph database load balancing method based on request type of the present application;

[0034] Figure 2 It is a flowchart of a graph database load balancing method based on request type according to Embodiment 1 of the present application;

[0035] Figure 3 It is a structural block diagram of a graph database load balancing device based on request type according to Embodiment 2 of the present application;

[0036] Figure 4 It is a schematic diagram of the internal structure of the electronic device of Example 5 of the present application.

[0037] Reference numerals:

[0038] 100, mapping table building module; 200, processing module; 300, routing module. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Example 1

[0041] like Figure 1 and Figure 2 As shown, a graph database load balancing method based on request type involved in Embodiment 1 of the present application includes:

[0042] S100, constructing a mapping table from query request type to graph database client connection pool list index;

[0043] In this embodiment, a class named database connection state management is implemented, which maintains a graph database query and multiple graph database client connection pools, where multiple refers to the number of distributed graph database query services. The key is to introduce a mapping table from query request type to graph database client connection pool, each graph database client connection pool corresponds to a graph database service address, and an initial index is assigned to the query type. For example: Map{query1->0,query2->1,query3->2} means that query request type 1 is mapped to the 0th graph database client connection pool, and the initial index of query request type 1 is 0, query request type 2 is mapped to the 1st graph database client connection pool, and the initial index of query request type 2 is 1, query request type 3 is mapped to the 2nd graph database client connection pool, and the initial index of query request type 3 is 2. Of course, the initial indexes of different types of complex query requests can be different or the same, and they are assigned to the specified graph database client connection pool according to the type of query request to ensure that the same type of queries are dispersed to different graph database services for execution, avoiding the situation where a single type of complex query is concentrated on a certain graph database service, causing excessive pressure on the service.

[0044] In this embodiment, taking the LDBC Benchmark test as an example, the LDBC Benchmark test has 14 different types of complex queries with heavy loads, and multiple simple queries and update operations with light loads. There are two different routing mechanisms for complex queries 1 to 14 for the LDBC Benchmark test and other requests such as simple queries and update operations. For complex queries 1 to 14 for the LDBC Benchmark test, the query request type will be separately recorded to which graph database client connection pool it is currently routed. When the same query request type is initiated, it will be routed to the graph database client connection pool in a polling manner to avoid a single type of complex query being concentrated on a certain graph database service, causing excessive pressure on the service; for complex query requests not for the LDBC Benchmark test, the query request type is not judged, and only the graph database client connection pool is polled and routed.

[0045] Normally, initialization is required first, that is, by reading configuration properties, creating multiple client connection pool lists connected to the graph database server. Each pool in the graph database client connection pool list corresponds to a graph database service address. At the same time, the initial client connection pool index is pre-assigned to LDBC query types 1 to 14. For example, if a distributed graph database has three graph database services: A, B, and C, then three graph database client connection pools for graph database service A, graph database service B, and graph database service C will be created during initialization. The initial indexes pre-assigned to complex query types 1 to 14 are 0, respectively, which means that each query is initially routed to the first graph database client connection pool. The index is used to record which graph database client connection pool the last query of a certain type was routed to, so that the current complex query of this type can be routed to the next graph database client connection pool according to the index + 1 of the last complex query of this type, wherein each type of complex query is polled independently, for example: complex query type 1 is routed to the initial index, i.e., the first graph database client connection pool, complex query type 1 is routed to the initial index + 1, i.e., the second graph database client connection pool, complex query type 2 is routed to the initial index, i.e., the first graph database client connection pool, complex query type 2 is routed to the initial index + 1, i.e., the second graph database client connection pool, complex query type 2 is routed to the routing index + 1 of the last complex query type 2, i.e., the third graph database client connection pool, complex query type 1 is routed to the routing index + 1 of the last complex query type 1, i.e., the third graph database client connection pool, and so on, and each type of complex query performs its own independent polling strategy without interfering with each other.

[0046] The following is an implementation code example of constructing a client connection pool collection that connects to multiple graph database services and adding it to the connection pool list:

[0047] for (String addr : graphAddresses) {

[0048] NebulaPool pool = NebulaPool.builder(addr, userName,password)

[0049] .withMaxClientSize(maxClientSize)

[0050] .withMinClientSize(maxClientSize)

[0051] .withConnectTimeoutMills(connectTimeout)

[0052] .withRequestTimeoutMills(requestTimeout)

[0053] .withBlockWhenExhausted(true)

[0054] .withMaxWaitMills(maxSessionWaitTime)

[0055] .withHealthCheckTimeMills(-1)

[0056] .build();

[0057] pools.add(new NewNebulaPool(pool, addr));

[0058] }

[0059] } catch (Exception e) {

[0060] throw new RuntimeException(e);

[0061] }.

[0062] S200, obtaining a query request and determining whether the query request is a complex query;

[0063] It should be noted that how to judge whether it is a complex query request depends on whether the amount of data involved in the query and the time taken to process the request exceed the preset threshold. For example, in business, queries involving more than 100,000 edge data points and taking more than 1 second can be considered complex queries, while queries involving less than 100,000 edge data points or taking less than 1 second can be considered non-complex queries. Of course, queries involving more than 100,000 edge data points or taking more than 1 second can also be considered complex queries. The specific selection can be made based on user needs, and the threshold can also be customized according to user needs or business types.

[0064] If the judgment result is yes, the query request type is obtained, and different query request types independently select the routing target graph database client connection pool from the mapping table in a polling manner. The implementation code example is as follows:

[0065] int currentClientPoolIndex = requestTypeAndClientPoolIndex.get(requestType);

[0066] int nextClientPoolIndex = (currentClientPoolIndex + 1) %graphServerSize;

[0067] requestTypeAndClientPoolIndex.put(requestType,nextClientPoolIndex);

[0068] return pools.get(currentClientPoolIndex);

[0069] };

[0070] If the result is no, the routing target graph database client connection pool is selected by polling. The implementation code example is as follows:

[0071] / / if operation is not query, then just balance the graphd through the requests.

[0072] return pools.get(poolClientIndex.getAndIncrement() %pools.size());

[0073] };

[0074] In this embodiment, taking the LDBC Benchmark test as an example, complex queries 1 to 14 of the LDBC Benchmark test are routed separately, and the graph database client connection pool to which each LDBC complex query type is currently routed is recorded in the mapping table of query request type to graph database client connection pool list index. If complex query 1 is currently routed to graph database client connection pool 1, the next complex query 1 will be routed to graph database client connection pool 2, and the next complex query 1 will be routed to graph database client connection pool 3, and so on, and independent polling is performed. The strategies for complex queries 2 to 14 are consistent with the strategy for complex query 1 above, and are performed independently from the polling of other complex queries and the polling of simple queries.

[0075] It is worth noting that in the LDBC Benchmark test, each type of query request is encapsulated into an operation. Each operation is identified by the request type, represented by type. Type values ​​1 to 14 represent complex queries. When determining the query request type, you only need to determine the value of operation.getType to know the type of complex query.

[0076] For other non-complex queries tested by the LDBC Benchmark, only the graph database client connection pool is polled. If a non-complex query request is currently routed to graph database client connection pool 1, the next non-complex query request is directly routed to graph database client connection pool 2, and the next non-complex query request is directly routed to graph database client connection pool 3, and so on, and independent polling is performed on non-complex queries.

[0077] S300, routing the query request to the routing target graph database client connection pool to provide services to the corresponding target graph database.

[0078] In step S200, the target graph database client connection pool for routing has been determined. It is only necessary to route the query request to the routing target graph database client connection pool to the corresponding target graph database service to achieve directional distribution of the query request. For non-complex query requests, the connection pool is selected by polling to maintain load balancing, and the response result is received after the target graph database service completes the processing.

[0079] In this embodiment, it also has configuration flexibility and monitoring functions: in addition to the load balancing strategy, it also integrates configuration parameters (such as connection timeout, request timeout, etc.), as well as query information logging functions, which facilitate system tuning and performance monitoring, and ensure the stable operation and maintainability of the system. The query information logging function configuration switch, if the logging is turned on, will record the routing target graph database client connection pool, target graph database service, response results, query delay and query time of each query request, where the query delay refers to the time interval from the query request being issued to the start of processing the request, and the query time refers to the total time taken from the start of processing to the completion of processing and returning the results, and the log records can be viewed, printed, and other operations, the implementation code example is as follows:

[0080] / / Close the connection resource

[0081] public boolean isEnableQueryInfoLog() {

[0082] return enableQueryInfoLog;

[0083] }.

[0084] This embodiment is specifically designed to optimize load balancing for various graph database query requests with different complexities and large load differences. The core of this embodiment is to distribute query loads to different query service nodes based on the type of query request rather than the server status, and to improve overall business performance by distributing the pressure of similar large queries (i.e., complex queries) on query services by distributing the pressure of similar large queries (i.e., complex queries) on query services. Since large queries are loaded onto different query services, effective dispersion of similar large query requests is achieved, and the pressure of processing data by each query service is balanced and small, so that the overall query and write operation performance can be improved. This method is particularly suitable for business scenarios with specific queries with different loads. Compared with traditional distribution strategies based on server load, this method more efficiently copes with the high concurrency challenges brought by specific types of queries.

[0085] Example 2

[0086] like Figure 3 As shown, a graph database load balancing device based on request type involved in Embodiment 2 of the present application includes:

[0087] A mapping table construction module 100 is used to construct a mapping table from a query request type to a graph database client connection pool list index;

[0088] The processing module 200 is used to obtain a query request and determine whether the query request is a complex query;

[0089] If the judgment result is yes, the query request type is obtained, and different query request types independently select a routing target graph database client connection pool from the mapping table in a polling manner;

[0090] If the judgment result is no, a polling method is used to select the routing target graph database client connection pool;

[0091] The routing module 300 is used to route the query request to the routing target graph database client connection pool to the corresponding target graph database service.

[0092] It should be noted that other specific implementations of the graph database load balancing device based on request type in this embodiment can refer to the specific implementations of the graph database load balancing method based on request type mentioned above. In order to avoid redundancy, they will not be repeated here.

[0093] Example 3

[0094] A computer program product involved in Example 3 of the present application is used to store a computer program. When the computer program runs on a computer, the method in any one of the implementation modes in Example 1 of the present application is implemented.

[0095] Example 4

[0096] A computer-readable storage medium according to Embodiment 4 of the present application, wherein the computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing a method in any one of the implementations in Embodiment 1 of the present application;

[0097] Among them, the computer-readable storage medium can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM); the computer-readable storage medium can store program code, and when the program stored in the computer-readable storage medium is executed by the processor, the processor is used to execute the steps of the method in any one of the implementation methods in Example 1 of the present application.

[0098] Example 5

[0099] like Figure 4 As shown, an electronic device involved in Embodiment 5 of the present application includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, a method in any one of the implementations in Embodiment 1 of the present application is implemented;

[0100] Among them, the processor can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the method in any one of the implementation methods in Example 1 of the present application.

[0101] The processor may also be an integrated circuit electronic device with signal processing capability. In the implementation process, each step of the method in any implementation of Embodiment 1 of the present application may be completed by an integrated logic circuit of hardware in the processor or by instructions in software form.

[0102] The above-mentioned processor may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the functions required to be performed by the unit included in the data processing device of the embodiment of the present application in combination with its hardware, or executes the method in any one of the implementation modes in the embodiment 1 of the present application.

[0103] The above are only preferred specific implementations of the present application; however, the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and its improved ideas of the present application within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A graph database load balancing method based on request type, characterized in that: include: Construct a mapping table from query request type to graph database client connection pool list index, wherein the graph database client connection pool list includes at least two graph database client connection pools, each of which corresponds to a graph database service address, and assigns an initial index to the query type; Obtaining a query request and determining whether the query request is a complex query; If the judgment result is yes, then the query request type is obtained, and different query request types independently select the routing target graph database client connection pool from the mapping table in a polling manner, specifically including: initializing the mapping table from the query request type to the graph database client connection pool list index; obtaining the graph database client connection pool list index that the query type was last routed to from the mapping table according to the query request type; adding 1 to the graph database client connection pool list index that was last routed to as the target index, and if the query request corresponding to the query request type is the first query, then the initial index corresponding to the query request type is used as the target index, and the corresponding routing target graph database client connection pool is obtained from the graph database client connection pool list according to the target index; If the judgment result is no, a polling method is used to select the routing target graph database client connection pool; The query request is routed to the routing target graph database client connection pool to serve the corresponding target graph database.

2. The graph database load balancing method based on request type according to claim 1 is characterized in that: Determining whether the query request is a complex query includes: Determine whether the amount of data involved in the query request exceeds a preset first threshold, and / or Determine whether the processing time of the query request exceeds a preset second threshold.

3. The graph database load balancing method based on request type according to claim 1 or 2, characterized in that: Also includes: Receive a response result from the target graph database service.

4. The graph database load balancing method based on request type according to claim 3 is characterized in that: Also includes: Perform log recording, wherein the content of the log recording includes at least one of the following: the routing target graph database client connection pool, the target graph database service, the response result, the query delay and the query time of each query request.

5. A graph database load balancing device based on request type, characterized in that: include: A mapping table construction module is used to construct a mapping table from a query request type to a graph database client connection pool list index, wherein the graph database client connection pool list includes at least two graph database client connection pools, each of which corresponds to a graph database service address, and allocates an initial index to the query type; A processing module, used for obtaining a query request and determining whether the query request is a complex query; If the judgment result is yes, then the query request type is obtained, and different query request types independently select the routing target graph database client connection pool from the mapping table in a polling manner, specifically including: initializing the mapping table from the query request type to the graph database client connection pool list index; obtaining the graph database client connection pool list index that the query type was last routed to from the mapping table according to the query request type; adding 1 to the graph database client connection pool list index that was last routed to as the target index, and if the query request corresponding to the query request type is the first query, then the initial index corresponding to the query request type is used as the target index, and the corresponding routing target graph database client connection pool is obtained from the graph database client connection pool list according to the target index; If the judgment result is no, a polling method is used to select the routing target graph database client connection pool; The routing module is used to route the query request to the routing target graph database client connection pool to the corresponding target graph database service.

6. A computer program product, characterized in that The computer program product stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing the method according to any one of claims 1 to 4.

8. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction implements the method as claimed in any one of claims 1 to 4 when executed by the processor.

Images