A graph storage method for handling mixed query loads of transactional and analytical types.

Abstract

This invention discloses a graph storage method for mixed query loads of transactional and analytical queries, belonging to the field of graph data storage. The method includes: storing graph data in a row-oriented storage structure before reaching the maximum capacity allocated in the LSM tree memory; when the written graph data reaches the maximum capacity, allocating new empty memory in the LSM tree memory to store the new graph data in a row-oriented storage structure, and converting the graph data in the old memory that has reached the maximum capacity into a column-oriented storage structure and migrating it to the LSM tree's external storage; in response to receiving mixed query concurrency requests, transactional query requests preferentially retrieve the first target graph data from the row-oriented storage structure of the LSM tree memory, while analytical query requests preferentially retrieve the second target graph data from the column-oriented storage structure of the LSM tree's external storage. This solution can reduce the response latency and computational resource consumption when handling mixed transactional / analytic queries.

Description

Technical Field

[0001] This invention relates to the field of graph data storage technology, and in particular to a graph storage method for mixed query loads of transactional and analytical queries. Background Technology

[0002] In modern data-driven applications, the importance of graph data is increasingly prominent, and graph computing has become the foundation for many data analysis scenarios, such as e-commerce recommendations, social media network analysis, fraud detection, and graph-based deep learning. Based on query characteristics, graph queries are categorized into transactional queries (GTP) and analytical queries (GAP). GTP queries typically access only a single vertex, edge, or their neighborhood (i.e., the vertices or edges adjacent to them) and require atomicity guarantees when concurrently inserting, deleting, or updating the attributes of multiple vertices and edges. GAP queries typically involve exploring a large portion or the entire graph in a static graph snapshot (such as PageRank and Depth-First Search), which often involves complex multi-hop graph traversal and pattern matching.

[0003] Currently, different storage methods are typically used when processing GTP and GAP query requests separately. Graph storage methods for GTP queries usually use flexible and loose row storage data structures such as adjacency lists or edge lists, while graph storage methods for GAP queries usually use compact and efficient column storage data structures such as compressed arrays of adjacency matrices (CSR).

[0004] As business grows, it often becomes necessary to initiate both GTP and GAP requests simultaneously (i.e., mixed transactional / analytic query loads) in actual use. To handle this, enterprises typically need to deploy two graph storage systems with different storage methods. These two systems not only need to store the same graph data separately, but also require independent read / write links, high availability guarantees, and operational maintenance links, further increasing deployment and usage costs and operational complexity. Furthermore, to ensure data consistency and timeliness for both GAP and GTP storage systems, additional data verification and synchronization links are required, further increasing costs and operational complexity. Therefore, when handling mixed transactional and analytical query loads, this storage method results in significant query latency and consumes substantial computing resources, making it difficult to meet the efficiency requirements of existing business operations for handling mixed transactional / analytic query loads.

[0005] Therefore, there is an urgent need for a new graph storage method that addresses both transactional and analytical query loads. Summary of the Invention

[0006] This invention provides a graph storage method for mixed query loads of transactional and analytical queries, which solves the problems of significant query latency and excessive computational resource consumption in existing graph storage methods for mixed query loads. The technical solution is as follows:

[0007] On the one hand, a graph storage method for mixed query loads of transactional and analytical queries is provided, the method comprising:

[0008] Before reaching the maximum capacity allocated in the LSM tree memory, the graph data is stored in a row-oriented storage structure for transactional queries.

[0009] When the written graph data reaches the capacity limit, the LSM tree memory allocates new empty memory to store the new graph data in a row-oriented storage structure, and migrates the graph data in the old memory that has reached the capacity limit to the LSM tree's external memory; wherein, the upper limit of the new empty memory storing the new graph data in a row-oriented storage structure is the capacity limit.

[0010] The graph data in the old memory is converted into a columnar storage structure oriented towards analytical queries and stored in the LSM tree external storage;

[0011] When a mixed query concurrency request is received, transactional query requests first retrieve the first target graph data from the row-oriented storage structure of the LSM tree memory, while analytical query requests first retrieve the second target graph data from the column-oriented storage structure of the LSM tree external memory.

[0012] On the other hand, a graph storage device is provided for mixed query loads of transactional and analytical queries, the device comprising:

[0013] The row storage module is used to store graph data in a row-oriented storage structure for transactional queries until the maximum capacity allocated in the LSM tree memory is reached.

[0014] The migration module is used to allocate new empty memory in the LSM tree memory to store the new graph data in a row-oriented storage structure when the written graph data reaches the capacity limit, and to migrate the graph data in the old memory that has reached the capacity limit to the LSM tree's external memory; wherein, the upper limit of the new empty memory storing the new graph data in a row-oriented storage structure is the capacity limit.

[0015] The columnar storage module is used to convert graph data in old memory into a columnar storage structure oriented towards analytical queries and store it in LSM tree external storage;

[0016] The query module is used to respond to mixed concurrent query requests. Transactional query requests prioritize retrieving the first target graph data from the row-oriented storage structure of the LSM tree memory, while analytical query requests prioritize retrieving the second target graph data from the column-oriented storage structure of the LSM tree external memory.

[0017] On the other hand, a computer device is provided, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for executing the computer program stored in the memory to implement the steps of the method described above.

[0018] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the method described above.

[0019] On the other hand, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0020] The technical solution provided by this invention can bring at least the following beneficial effects:

[0021] Abandoning the traditional architecture of using two separate storage systems to handle transactional and analytical requests, this invention connects a transactional row-based storage structure and an analytical column-based storage structure based on an LSM-Tree hierarchical structure, enabling a single system to efficiently handle mixed transactional / analytic query requests. Furthermore, more recent graph data is stored in memory in a columnar structure, while the in-memory row-based storage structure prioritizes responding to transactional query requests, adapting to the requirement of transactional requests to retrieve the latest target graph data. Therefore, the storage method proposed in this invention can significantly reduce system deployment and maintenance costs, eliminating the computational overhead of data retrieval, transmission, and synchronization between the two traditional systems, thereby effectively reducing response latency and computational resource consumption when handling mixed transactional / analytic query loads. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a graph storage method for a hybrid query load of transactional and analytical queries, provided by an embodiment of the present invention.

[0024] Figure 2 This is a structural diagram of a graph storage device for a hybrid query load that is oriented towards transactional and analytical queries, according to an embodiment of the present invention.

[0025] Figure 3This is a hardware architecture diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0027] The following describes the specific implementation of the above concept.

[0028] Please refer to Figure 1 This invention provides a graph storage method for hybrid query loads of transactional and analytical queries, the method comprising:

[0029] Step 100: Before reaching the maximum capacity allocated in the LSM tree memory, store the graph data in a row-oriented storage structure for transactional queries.

[0030] Step 102: When the written graph data reaches the capacity limit, the LSM tree memory allocates new empty memory to store the new graph data in a row-oriented storage structure, and migrates the graph data in the old memory that has reached the capacity limit to the LSM tree's external memory; wherein, the upper limit of the new empty memory storing the new graph data in a row-oriented storage structure is the capacity limit.

[0031] Step 104: Convert the graph data in the old memory into a columnar storage structure oriented towards analytical queries and store it in the LSM tree external storage;

[0032] Step 106: In response to receiving a mixed query concurrency request, transactional query requests first retrieve the first target graph data from the row-oriented storage structure of the LSM tree memory, and analytical query requests first retrieve the second target graph data from the column-oriented storage structure of the LSM tree external memory.

[0033] In this invention, the traditional architecture of using two separate storage systems to handle transactional and analytical requests is abandoned. Instead, this invention connects a transactional row-based storage structure and an analytical column-based storage structure based on an LSM-Tree hierarchical structure, using a single system to efficiently handle mixed transactional / analytic query requests. Furthermore, more recent graph data is stored in memory in a columnar structure, while the in-memory row-based storage structure prioritizes responding to transactional query requests, adapting to the requirement of transactional requests to retrieve the latest target graph data. Therefore, the storage method proposed in this invention can significantly reduce system deployment and maintenance costs, eliminating the computational overhead of data retrieval, transmission, and synchronization in traditional two-system architectures, thereby effectively reducing response latency and computational resource consumption when handling mixed transactional / analytic query loads.

[0034] The following description Figure 1 The execution method of each step is shown.

[0035] For steps 100 and 102:

[0036] In some implementations, the row-based storage structure is an adjacency list structure;

[0037] LSM tree memory stores graph data in a row-oriented storage structure in the following way:

[0038] When storing the first graph data, the inserted edges are located in the corresponding adjacency list memory block based on the LSM tree memory sorting structure;

[0039] For each graph data entry stored thereafter, execute the following:

[0040] Allocate new edge storage space in memory for the adjacency list, and link the pointers of the previous edges associated with the current graph data to the newly inserted edges that represent the current graph data, thus completing the insertion of the current graph data.

[0041] In some implementations, the columnar storage structure is a compressed array structure of an adjacency matrix.

[0042] In this embodiment, the LSM tree is divided into memory and external storage. Graph data is initially stored in memory using an adjacency list structure. When the space allocated to the adjacency list in memory reaches the set capacity limit, the graph data is merged and converted from memory to external storage. This is because the adjacency list structure supports edge insertion and modification, while inserting and modifying graph data under a compressed array structure requires reconstructing the entire compressed array. Therefore, the memory used to store the graph data must first use a row-based storage structure, not a column-based storage structure.

[0043] Regarding step 104:

[0044] In some implementations, step 104 may include:

[0045] Divide the external memory of the LSM tree into several levels;

[0046] The graph data migrated from the old memory is stored starting from the topmost level zero of the LSM tree external storage, and then compressed layer by layer from the upper-level columnar array to the lower-level into a global compressed array.

[0047] In this embodiment, each time the capacity limit is reached, a new empty memory is allocated to migrate the graph data in the old memory to the external memory of the LSM tree. The data is stored starting from the topmost level zero of the LSM tree external memory, and is gradually compressed from the relatively loose columnar array at the upper level to a compact global CSR array at the lower level.

[0048] Regarding step 106:

[0049] In some implementations, in step 106, the transactional query request preferentially retrieves the first target graph data from the row-oriented storage structure of the LSM tree memory, including:

[0050] A transactional query request first retrieves the first target graph data that meets the query conditions from the row-oriented storage structure of the LSM tree in memory;

[0051] If not found, the LSM tree external storage is searched layer by layer from top to bottom until the first target graph data is obtained.

[0052] For example, in a banking system, a transactional query might retrieve customer A's current balance. Transactional queries typically access only a single vertex, edge, or their neighborhood (i.e., the vertices or edges adjacent to them), and atomicity guarantees are required when concurrently inserting, deleting, or updating the attributes of multiple vertices and edges. Since the most up-to-date target graph data is usually needed, transactional query requests first retrieve the first target graph data from the row-oriented storage structure of the LSM tree in memory. Row-oriented storage is more suitable for transactional queries, allowing for faster retrieval. If the data is not found, the query then proceeds to the LSM tree's external storage, retrieving it layer by layer from top to bottom until the first target graph data is obtained. This ensures the response efficiency of transactional query requests.

[0053] In some implementations, the analytical query request in step 106 preferentially retrieves the second target graph data from the columnar storage structure of the LSM tree external storage, including:

[0054] Analytical query requests first traverse the graph data in the LSM tree external storage layer by layer from the bottom to the top, and retrieve the second target graph data that meets the query conditions;

[0055] If not found, continue traversing the LSM tree memory until the second target graph data is obtained.

[0056] For example, analytical queries in a banking system, such as querying customer A's income and expenditure details for a specified date period, typically require traversing most or all of the graph data.

[0057] Since the columnar storage structure of the LSM tree external storage is more suitable for analytical query requests, analytical query requests are preferentially traversed in the LSM tree external storage. Furthermore, if a transactional query request cannot find the data in memory, it is retrieved layer by layer from top to bottom in the LSM tree external storage. To avoid mutual blocking between transactional and analytical queries and reduce the probability of queuing, in this embodiment, analytical query requests preferentially traverse the graph data layer by layer from bottom to top in the LSM tree external storage to more quickly retrieve the second target graph data that meets the analytical query conditions.

[0058] In summary, this invention utilizes an LSM-Tree (Log-Structed-Merge-Tree) structure to organically connect the graph storage data structure oriented towards GTP and the graph storage data structure oriented towards GAP within a single system service. By converting the in-memory row-based storage structure to the external storage column-based storage structure, and designing an innovative graph data read / write link, it efficiently handles concurrent requests for graph transaction queries and graph analysis queries. This significantly reduces the deployment and maintenance costs of the system, avoids the computational overhead of data retrieval, transmission, and synchronization in traditional two-system architectures, and effectively reduces response latency and computational resource consumption when handling mixed transaction / analysis query loads.

[0059] Please refer to Figure 2 This invention provides a graph storage device for mixed query loads of transactional and analytical queries, the device comprising:

[0060] The row storage module 201 is used to store graph data in a row-oriented storage structure for transactional queries before reaching the maximum capacity allocated in the LSM tree memory.

[0061] The migration module 202 is used to allocate new empty memory in the LSM tree memory to store the new graph data in a row-oriented storage structure when the written graph data reaches the capacity limit, and to migrate the graph data in the old memory that has reached the capacity limit to the LSM tree's external memory; wherein, the upper limit of the new empty memory to store the new graph data in a row-oriented storage structure is the capacity limit.

[0062] The column storage module 203 is used to convert graph data in old memory into a columnar storage structure oriented towards analytical queries and store it in LSM tree external storage;

[0063] The query module 204 is used to respond to the receipt of mixed query concurrency requests by retrieving the first target graph data from the row-oriented storage structure of the LSM tree memory for transactional query requests and the second target graph data from the column-oriented storage structure of the LSM tree external storage for analytical query requests.

[0064] In one embodiment of the present invention, the row storage structure in the row storage module 201 and the migration module 202 is an adjacency list structure;

[0065] LSM tree memory stores graph data in a row-oriented storage structure in the following way:

[0066] When storing the first graph data, the inserted edges are located in the corresponding adjacency list memory block based on the LSM tree memory sorting structure;

[0067] For each graph data entry stored thereafter, execute the following:

[0068] Allocate new edge storage space in memory for the adjacency list, and link the pointers of the previous edges associated with the current graph data to the newly inserted edges that represent the current graph data, thus completing the insertion of the current graph data.

[0069] In one embodiment of the present invention, the columnar storage structure in the migration module 202 is a compressed array structure of an adjacency matrix.

[0070] In one embodiment of the present invention, the columnar storage module 203 is used to perform:

[0071] Divide the external memory of the LSM tree into several levels;

[0072] The graph data migrated from the old memory is stored starting from the topmost level zero of the LSM tree external storage, and then compressed layer by layer from the upper-level columnar array to the lower-level into a global compressed array.

[0073] In one embodiment of the present invention, when the query module 204 performs a transactional query request to retrieve the first target graph data from the row-oriented storage structure of the LSM tree memory, it is used to:

[0074] A transactional query request first retrieves the first target graph data that meets the query conditions from the row-oriented storage structure of the LSM tree in memory;

[0075] If not found, the LSM tree external storage is searched layer by layer from top to bottom until the first target graph data is obtained.

[0076] In one embodiment of the present invention, when the query module 204 performs an analytical query request to retrieve the second target graph data from the columnar storage structure of the LSM tree external storage, it is configured to:

[0077] Analytical query requests first traverse the graph data in the LSM tree external storage layer by layer from the bottom to the top, and retrieve the second target graph data that meets the query conditions;

[0078] If not found, continue traversing the LSM tree memory until the second target graph data is obtained.

[0079] It should be noted that the graph storage device for mixed query loads of transactional and analytical queries provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the above device embodiments and method embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0080] Embodiments of this application also provide a computer device, please refer to... Figure 3 The computer device includes a processor and a memory, in which at least one instruction, at least one program, code set, or instruction set is stored. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the graph storage method for mixed query loads of transactional and analytical types provided in the above-described method embodiments.

[0081] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the graph storage method for mixed query loads of transactional and analytical types provided in the above-described method embodiments.

[0082] Embodiments of this application also provide a computer program product comprising a computer program, wherein a processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the graph storage method for mixed query loads of transactional and analytical queries as described in any of the above embodiments.

[0083] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0084] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0085] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0086] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A graph storage method for mixed query load of transactional and analytical types, characterized by, The method includes: Before reaching the maximum capacity allocated in the LSM tree memory, the graph data is stored in a row-oriented storage structure for transactional queries. When the written graph data reaches the capacity limit, the LSM tree memory allocates new empty memory to store the new graph data in a row-oriented storage structure, and migrates the graph data in the old memory that has reached the capacity limit to the LSM tree's external memory; wherein, the upper limit of the new empty memory storing the new graph data in a row-oriented storage structure is the capacity limit. The graph data in the old memory is converted into a columnar storage structure oriented towards analytical queries and stored in the LSM tree external storage; When a mixed query concurrency request is received, transactional query requests first retrieve the first target graph data from the row-oriented storage structure of the LSM tree memory, while analytical query requests first retrieve the second target graph data from the column-oriented storage structure of the LSM tree external memory. The row-based storage structure is an adjacency list structure; LSM tree memory stores graph data in a row-oriented storage structure in the following way: When storing the first graph data, the inserted edges are located in the corresponding adjacency list memory block based on the LSM tree memory sorting structure; For each graph data entry stored thereafter, execute the following: Allocate new edge storage space in memory for the adjacency list, and link the pointers of the previous edges associated with the current graph data to the newly inserted edges that represent the current graph data, so as to complete the insertion of the current graph data; The columnar storage structure is a compressed array structure of an adjacency matrix; The process of converting graph data in old memory into a columnar storage structure oriented towards analytical queries and storing it in LSM tree external storage includes: Divide the external memory of the LSM tree into several levels; Each time the capacity limit is reached, a new empty memory is allocated to store the graph data migrated from the old memory starting from the top level zero of the LSM tree external storage, compressing it layer by layer from the upper column array to the lower level into a global compressed array; The transactional query request preferentially retrieves the first target graph data from the row-oriented storage structure of the LSM tree memory, including: The transactional query request first retrieves the first target graph data that meets the query conditions from the row-oriented storage structure of the LSM tree memory; If not found, the LSM tree external storage is searched layer by layer from top to bottom until the first target graph data is obtained; The analytical query request preferentially retrieves the second target graph data from the columnar storage structure of the LSM tree external storage, including: The analytical query request first traverses the graph data in the LSM tree external storage layer by layer from the bottom to the top, and retrieves the second target graph data that meets the query conditions. If not found, continue traversing the LSM tree memory until the second target graph data is obtained.

2. A graph storage device for hybrid query loads of transactional and analytical queries, used to implement the steps of the method described in claim 1, characterized in that, The device includes: The row storage module is used to store graph data in a row-oriented storage structure for transactional queries until the maximum capacity allocated in the LSM tree memory is reached. The migration module is used to allocate new empty memory in the LSM tree memory to store the new graph data in a row-oriented storage structure when the written graph data reaches the capacity limit, and to migrate the graph data in the old memory that has reached the capacity limit to the LSM tree's external memory; wherein, the upper limit of the new empty memory storing the new graph data in a row-oriented storage structure is the capacity limit. The columnar storage module is used to convert graph data in old memory into a columnar storage structure oriented towards analytical queries and store it in LSM tree external storage; The query module is used to respond to mixed concurrent query requests. Transactional query requests prioritize retrieving the first target graph data from the row-oriented storage structure of the LSM tree memory, while analytical query requests prioritize retrieving the second target graph data from the column-oriented storage structure of the LSM tree external memory.

3. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the method described in claim 1.

4. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in claim 1.

5. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method of claim 1.

Images