Dynamic graph data storage system, retrieval system and corresponding methods

Abstract

This application discloses a dynamic graph data storage system, a reading system, and a corresponding method. Applied to the field of computer technology, the system is located at the execution layer of a storage device. A data acquisition component obtains vertex and edge data to be processed from the graph data. A vertex storage component uses the vertex data to update vertex block regions, and updates the vertex block address array using the update result. The vertex block region contains the vertex blocks of each vertex, and the vertex block address array stores the vertex block address information corresponding to each vertex. An edge storage component determines a segment space, which includes edge regions. The edge data is used to update the edge regions corresponding to the starting vertices of the edge data. This application can effectively improve the storage efficiency of dynamic graph data.

Description

[0001] This application is a divisional application of Chinese patent application No. 202211118844.7, filed on September 15, 2022, entitled "Dynamic Graph Data Storage System, Reading System and Corresponding Method". Technical Field

[0002] This application relates to the field of computer technology, and in particular to a dynamic graph data storage system, a reading system, and a corresponding method. Background Technology

[0003] Large-scale graph computing is playing an increasingly important role in various fields. In practical scenarios such as information recommendation, user profiling, device clustering, and risk identification, various graph computation methods are involved, including connected component discovery, graph pattern matching, local subgraph traversal, graph neural networks, and real-time page ranking. However, the graph data used in graph computing is often not static but constantly changes with the service, presenting a dynamic graph data that changes in real time.

[0004] In dynamic graph computing scenarios, attribute graphs are commonly used as data models to solve analytical problems involving relationships between entities. An attribute graph is a directed graph where each vertex and edge has a type label, and each type of vertex and edge can have several attributes. Because attribute graph data is constantly updated, there is a pressing need for an efficient method for storing dynamic graph data. Summary of the Invention

[0005] This application provides a dynamic graph data storage system, a retrieval system, and a corresponding method to improve the storage efficiency of dynamic graph data.

[0006] This application provides the following solution:

[0007] In a first aspect, a dynamic graph data storage system is provided, disposed at the execution layer of a storage device, the system comprising:

[0008] The data acquisition component is configured to acquire edge data to be processed, the edge data including the starting vertex information, ending vertex information and attribute information of the edge;

[0009] An edge storage component is configured to determine a segment space, the segment space including a segment header and an edge region; determine the edge region corresponding to the starting vertex in the segment space based on the edge pointers of each vertex contained in the segment header; and update the edge region corresponding to the starting vertex using the ending vertex information and the attribute information.

[0010] According to one achievable method in an embodiment of this application, the edge data further includes label information;

[0011] When determining the segment space, the edge storage component specifically performs the following steps: using the tag information to query the segment tag array to determine the segment address information corresponding to the tag information; and determining the corresponding segment space based on the segment address information.

[0012] According to one achievable method in an embodiment of this application, the data acquisition component is further configured to acquire vertex data to be processed;

[0013] The system further includes: a vertex storage component, configured to update vertex blocks in a vertex block region using the vertex data, and to update the vertex block address array using the update result of the vertex block region;

[0014] The vertex block region contains the vertex blocks of each vertex, and the vertex block address array is used to store the vertex block address information corresponding to the vertex.

[0015] According to one achievable method in an embodiment of this application, the part to be processed is the part to be added, and the update is the addition process; or...

[0016] The term "to be processed" refers to "to be deleted," and the term "update" refers to either deletion processing or marking an invalid label.

[0017] According to one achievable method in the embodiments of this application, if the edge data to be processed is edge data to be added, then when the edge storage component uses the tag information to query the segment tag array and determines the segment address information corresponding to the tag information, it specifically performs the following:

[0018] Use the aforementioned tag information to query the segment tag array;

[0019] If the segment address information corresponding to the tag information is not found, then a storage space of the first size is allocated as the segment space corresponding to the tag information, metadata is added to the segment header of the segment space, and the segment address information of the segment space corresponding to the tag information is added to the segment tag array.

[0020] According to one achievable method in the embodiments of this application, the metadata includes segment space identification information, the tag information, and the size information of the segment space.

[0021] According to one achievable method in an embodiment of this application, if the edge data to be processed is edge data to be added, then when the edge storage component determines the edge region corresponding to the starting vertex in the segment space based on the edge pointers of each vertex contained in the segment header, it specifically performs the following:

[0022] Query the edge pointers of each vertex contained in the segment header;

[0023] If the edge pointer of the starting vertex is not found, a storage area of ​​the second size is allocated in the segment space as an edge region for the starting vertex; if the edge pointer of the starting vertex is found, the edge region corresponding to the starting vertex is determined in the segment space.

[0024] According to one possible implementation in an embodiment of this application, the system further includes: a segment migration component;

[0025] When the edge storage component allocates a storage region of the second size as an edge region for the starting vertex in the segment space, it specifically performs the following:

[0026] If there is a second-sized free space in the segment space, then a storage area of ​​the second size is allocated from the free space as an edge region for the starting vertex; otherwise, after the segment migration component is triggered to migrate the segment space to another larger segment space, a storage area of ​​the second size is allocated as an edge region for the starting vertex in the other larger segment space.

[0027] According to one possible implementation in an embodiment of this application, the system further includes: a region migration component;

[0028] If the edge data to be processed is edge data to be added, then when the edge storage component updates the edge region corresponding to the starting vertex using the ending vertex information and the attribute information, it specifically performs the following:

[0029] If there is an empty slot in the edge region corresponding to the starting vertex, the ending vertex information and the attribute information are added to the slot; otherwise, the region migration component is triggered to migrate the edge region corresponding to the starting vertex to a larger edge region, and then the ending vertex information and the attribute information are added to the larger edge region.

[0030] According to one possible implementation in an embodiment of this application, the system further includes: a segment migration component;

[0031] The region migration component is specifically configured as follows:

[0032] If there is a free space of size four in the segment space, then allocate a storage area of ​​size four from the free space as a new edge area corresponding to the starting vertex, copy the contents of the original edge area corresponding to the starting vertex to the new edge area, and update the edge pointer of the starting vertex contained in the segment header according to the new edge area.

[0033] If there is no free space of the fourth size in the segment space, the segment migration component is triggered to migrate the segment space to another larger segment space. Then, a storage area of ​​the fourth size is allocated from the free space of the migrated segment space as the new edge region corresponding to the starting vertex. The contents of the original edge region corresponding to the starting vertex are copied to the new edge region. The edge pointer of the starting vertex contained in the segment header is updated according to the new edge region.

[0034] According to one achievable method in an embodiment of this application, the segment migration component is specifically configured as follows:

[0035] Allocate a third-sized storage space as a new segment space corresponding to the tag information;

[0036] Copy the contents of the original segment space corresponding to the tag information, excluding the segment header, to the new segment space. Add the segment header of the new segment space based on the copied content, and update the segment address information corresponding to the tag information in the segment tag array to the new segment space address using the new segment space.

[0037] According to one achievable method in an embodiment of this application, the segment header further includes a pointer to the epoch table of time segments for each vertex, the epoch table including: the offset of the edge data added in the corresponding epoch in the edge region corresponding to the starting vertex;

[0038] The edge storage component is further configured to: after updating the edge region corresponding to the starting vertex using the ending vertex information and the attribute information, determine whether to update the epoch table of the starting vertex based on the current time.

[0039] According to one achievable method in an embodiment of this application, the edge region includes an edge block region and an edge attribute region;

[0040] When the edge storage component updates the edge region corresponding to the starting vertex using the ending vertex information and the attribute information, it specifically performs the following:

[0041] The edge block region corresponding to the starting vertex is updated using the ending vertex information, and the edge attribute region corresponding to the starting vertex is updated using the attribute information.

[0042] According to one achievable method in the embodiments of this application, in the segment space excluding the segment head portion, the edge block regions of each vertex are sequentially allocated according to a first direction, the end vertex information is sequentially added according to the first direction, the attribute regions of each vertex are sequentially allocated according to a second direction, and the attribute information is sequentially added according to the second direction, wherein the first direction and the second direction are opposite.

[0043] Secondly, a dynamic graph data reading system is provided, configured at the execution layer of a storage device, the system comprising:

[0044] The transaction acquisition component is configured to acquire the first read transaction information of the edge data, and the first read transaction information includes at least the starting vertex information to be queried;

[0045] An edge reading component is configured to determine a segment space, the segment space including a segment header and an edge region; determine the edge region corresponding to the starting vertex in the segment space based on the edge pointers of each vertex contained in the segment header; and read edge data from the edge region corresponding to the starting vertex.

[0046] According to one achievable method in an embodiment of this application, the first read transaction information further includes tag information;

[0047] The process of determining the segment space includes: querying the segment tag array using the tag information to determine the segment address information corresponding to the tag information; and determining the corresponding segment space based on the segment address information.

[0048] According to one achievable method in an embodiment of this application, the transaction acquisition component is further configured to acquire second read transaction information of vertex data, the second read transaction information including vertex information to be queried;

[0049] The system further includes: a vertex reading component, configured to query a vertex block address array using the vertex information to be queried, to obtain vertex block address information corresponding to the vertex information to be queried; and to read vertex data from the vertex block pointed to by the vertex block address information.

[0050] According to one achievable method in an embodiment of this application, the segment header further includes a pointer to the epoch table of time segments for each vertex, the epoch table including: the offset of the edge data added in the corresponding epoch in the edge region corresponding to the starting vertex;

[0051] When the edge reading component reads edge data from the edge region corresponding to the starting vertex, it specifically performs the following steps: based on the starting vertex information to be queried, it queries the pointers to the epoch tables of each vertex in the segment header to determine the epoch table corresponding to the starting vertex information to be queried; based on the epoch table, it determines the edge region in the edge region corresponding to the starting vertex whose corresponding epoch is less than or equal to the time of the read transaction; and it reads the edge data in the determined edge region.

[0052] According to one achievable method in an embodiment of this application, the edge region includes an edge block region and an edge attribute region;

[0053] When the edge reading component reads edge data from the edge region corresponding to the starting vertex, it specifically performs the following:

[0054] Read the end vertex information from the edge block region corresponding to the starting vertex and / or read the edge attribute information from the edge attribute region corresponding to the starting vertex.

[0055] Thirdly, a dynamic graph data storage method is provided, which includes:

[0056] Obtain the edge data to be processed, the edge data including the starting vertex information, ending vertex information and attribute information of the edge;

[0057] Define a segment space, which includes segment headers and edge regions;

[0058] Based on the edge pointers of each vertex contained in the segment header, determine the edge region corresponding to the starting vertex in the segment space;

[0059] The edge region corresponding to the starting vertex is updated using the ending vertex information and the attribute information.

[0060] Fourthly, a method for reading dynamic graph data is provided, which includes:

[0061] Obtain the first read transaction information of the edge data, which includes at least the starting vertex information to be queried;

[0062] Define a segment space, which includes segment headers and edge regions;

[0063] Based on the edge pointers of each vertex contained in the segment header, determine the edge region corresponding to the starting vertex in the segment space;

[0064] Read edge data from the edge region corresponding to the starting vertex.

[0065] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the third or fourth aspect above.

[0066] According to a sixth aspect, an electronic device is provided, characterized in that it comprises:

[0067] One or more processors; and

[0068] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first aspects above.

[0069] According to the specific embodiments provided in this application, this application can achieve the following technical effects:

[0070] 1) This application proposes a novel dynamic view data storage system based on segment space. In the segment space, each vertex has a corresponding edge region to store the information of the ending vertex and the attribute information of the edges. Therefore, when updating edge data, such as adding or deleting edge data, the entire process only needs to update the edge region corresponding to the initial vertex of the edge in the segment space, without affecting the edge data of other vertices that have already been stored. This can shorten the dynamic update time of graph data and improve the storage efficiency of dynamic graph data.

[0071] 2) In this application, the segment space for storing edge data corresponds to the label information of the edge, so that edge data with the same label information are stored in the same segment space, which greatly facilitates the reading of edge data based on labels and is suitable for improving the efficiency of edge data reading in most graph computing scenarios.

[0072] 3) In this application, vertex data is stored in the vertex block of each vertex, and the vertex block address information of each vertex block is maintained by the vertex block address array. Edge data is stored in the segment space. This method of storing vertex data and edge data independently can facilitate the updating of vertex data and edge data, minimize the mutual influence, and improve the efficiency of dynamic graph data storage.

[0073] 4) In this application, edge data corresponding to the same vertex are stored in the same edge region in the segment space, so that the edge data corresponding to the same vertex are adjacent; and the edge data corresponding to each vertex is stored in the edge region in the segment space according to the vertex ID, so that the edge data corresponding to adjacent vertex IDs are also stored adjacently, thereby improving data locality and significantly improving the system performance and reading efficiency of the dynamic graph data storage system when reading edge data vertex by vertex.

[0074] 5) In this application, when there are insufficient free slots in the edge region, the edge region is migrated; when there are insufficient free spaces in the segment space, the segment space is migrated. This ensures that the edge data of the newly added vertex is still adjacent, and the edge data corresponding to the adjacent vertex IDs is also adjacent. This better ensures data locality and further improves the system performance and reading efficiency of the dynamic graph data storage system when reading data vertex-by-vertex and edge-by-edge.

[0075] 6) This application further distinguishes between edge block regions and edge attribute regions, which are used to store the end vertex information and edge attribute information of the edges, respectively, thereby accelerating the performance of reading edge structure and attribute information. Furthermore, the allocation direction of the edge block region and edge attribute region for each vertex is opposite, i.e., allocated sequentially from both ends towards the middle. When adding end vertices and attribute information, the addition direction is also opposite, i.e., added sequentially from both ends towards the middle. This approach can more fully and effectively utilize segment space, thus improving storage space utilization and reading performance.

[0076] 7) In this embodiment of the application, an independent epoch table is used to store the update time of the edge block. Only the offset of the edge block that is updated in the epoch needs to be maintained in the epoch table. There is no need to maintain the timestamp of each version for each edge block, which reduces the system overhead. At the same time, it ensures the consistency between the read edge block data and the read transaction time, and ensures the normal operation of concurrency control.

[0077] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is a schematic diagram of the compression format in the prior art.

[0080] Figure 2 A schematic diagram of the system architecture provided for embodiments of this application;

[0081] Figure 3a A schematic diagram of the main structure of a segment space provided in an embodiment of this application;

[0082] Figure 3b Another main structural diagram of the segment space provided in the embodiments of this application;

[0083] Figure 4 This is a flowchart illustrating a method for storing dynamic graph data according to an embodiment of this application.

[0084] Figure 5 A flowchart illustrating a method for storing graph data provided in an embodiment of this application;

[0085] Figure 6 A schematic diagram of a vertex block address array provided in an embodiment of this application;

[0086] Figure 7 A schematic diagram of the segment label table address array provided in an embodiment of this application;

[0087] Figure 8 A flowchart of a dynamic graph data reading method provided in an embodiment of this application;

[0088] Figure 9 A schematic diagram of an epoch table provided in an embodiment of this application;

[0089] Figure 10This is a structural diagram of the dynamic graph data storage system provided in the embodiments of this application;

[0090] Figure 11 This is a structural diagram of the dynamic graph data reading system provided in the embodiments of this application;

[0091] Figure 12 This is a structural framework diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0092] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0093] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0094] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0095] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0096] The storage method of dynamic graph data largely determines the read and write performance of the graph data, thus affecting the performance of graph computation workloads. Currently, there are several graph data storage methods for graph computation, such as CSR (Compressed Sparse Row) and CSC (Compressed Sparse Column). Taking CSR as an example, CSR is a compact graph storage format used for static graph computations, such as... Figure 1As shown, it uses a vertex array and an edge array to store the outgoing edges of each vertex. The edge array stores edges in order of their origin IDs, and the vertex array uses vertex IDs as indices to store the offsets of the outgoing edges of each vertex within the edge array. Although CSR can guarantee efficient contiguous memory access, it does not support dynamic real-time updates of graph data. For example, when adding an edge based on CSR, both the edge array and the vertex array need to be updated. Furthermore, updating the edge array will affect the edge data of other vertices, resulting in very low storage efficiency and making it unsuitable for dynamic graph data update requirements.

[0097] In view of this, this application proposes a new method for storing dynamic graph data. Figure 2 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 2 As shown, the system mainly includes storage devices, upstream devices for the storage devices, and downstream devices.

[0098] The upstream devices of the storage device can be those that provide graph data to the storage device. For example, an upstream device could be a log processing device that can convert real-time generated log data (such as logs of various online transaction behaviors, online dating behaviors, etc.) into graph data. This involves extracting the actors and objects of the behavior as vertices, and the behavioral relationships between the actors and objects as edges. The generated or updated graph data is then provided to the storage device. Another example is an RDBMS (Relational Database Management System), which can convert relational data in a relational database into graph data or update graph data and provide it to the storage device.

[0099] Storage devices are responsible for storing graph data and performing operations such as adding, deleting, modifying, and querying the stored graph data. Storage devices are mainly divided into an execution layer and a storage layer. The storage layer stores the graph data, while the execution layer of the storage device handles the specific storage methods and operations such as adding, deleting, modifying, and querying the stored graph data. In some scenarios, the execution layer is also referred to as the storage engine or engine layer.

[0100] Storage devices can be single devices, groups of multiple devices, or devices on cloud servers. Cloud servers, also known as cloud computing servers or cloud hosts, are a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPs) services, such as high management difficulty and weak service scalability.

[0101] Downstream devices are typically those that initiate read transactions to storage devices; these can be user devices or graph analysis processing devices. In specific applications, graph analysis processing devices need to query vertex and / or edge data to obtain vertex and edge features for decision-making.

[0102] It should be understood that Figure 2 The number of upstream devices, downstream devices, and storage devices shown is merely illustrative. Depending on implementation needs, any number of upstream devices and storage devices can be included.

[0103] To facilitate understanding of this application, the novel dynamic graph data storage format provided in this application is first described. This application uses a segment space to manage a certain amount of edge data. This segment space mainly includes a segment header and edge regions. The segment header may include metadata and edge pointers for each vertex, and may further include pointers to the epoch (time segment) table for each vertex.

[0104] Metadata mainly includes the identifier of the segment space and the size of the segment space, and may also include edge label information, etc.

[0105] In a segment space, each vertex, as the starting vertex of an edge, corresponds to an edge region. The edge pointers of each vertex in the segment header point to the edge region corresponding to that vertex. Each edge region initially has multiple free slots to fill with the edge data corresponding to the starting vertex.

[0106] As one possible way, such as Figure 3a As shown, the end vertex information and edge attribute information can be stored in the edge region corresponding to the starting vertex. An edge's attribute information and end vertex information can be filled into an empty slot in the edge region; both the attribute information and end vertex information have a fixed length. For example, in the edge region of vertex 0, if an edge's end vertex is v1 and its attribute is p1, it can be filled into an empty slot. If an edge's end vertex is v2 and its attribute is p2, it can also be filled into an empty slot.

[0107] However, in many application scenarios, the requirements for reading edge attribute information and end vertex information are independent. For example, in some graph computing scenarios, only the edge attribute information needs to be read; in others, only the end vertex information needs to be read; and in still others, both the end vertex information and attribute information of each edge need to be scanned vertex by vertex. To improve reading efficiency in such scenarios, this application provides a preferred implementation method that divides the edge region into an edge block region and an edge attribute region. The following description will use this preferred implementation method as an example.

[0108] like Figure 3bAs shown, in a segment space, each vertex, as the starting vertex of an edge, corresponds to an edge block region, which contains the edge block corresponding to that vertex. The edge block may store the information of the ending vertex of the edge. Each edge block region initially has multiple empty slots, each slot being used to fill the information of the ending vertex of an edge. For example... Figure 3b As shown, the edge block region corresponding to the vertex with ID 0 stores the ending vertex v1 of one edge and the ending vertex v2 of another edge. There are also some empty slots for information on other edges to be inserted later from this starting vertex. Additionally, each vertex's edge block region can contain metadata called block metadata, which includes information such as the size of the edge block region. The edge block region of a vertex can be determined based on the edge pointers of each vertex in the segment header.

[0109] The edge attribute region stores the attribute information of the edges. In a segment space, edge blocks (which store the graph's structural data) and edge attribute information are stored separately. In the segment space, excluding the space at the segment head, edge block regions for each vertex are allocated sequentially according to the first direction, and information for the ending vertex is added sequentially according to the first direction. Attribute regions for each vertex are allocated sequentially according to the second direction, and attribute information is added sequentially according to the second direction, with the first and second directions being opposite. That is, edge blocks and attribute information are filled from both ends into the empty space in the middle. Figure 3b As shown, the edge block area expands continuously from left to right, and the edge attribute area expands continuously from right to left, until the empty area in the middle is completely consumed.

[0110] Similar to the edge block region, each vertex, as the starting vertex of an edge, corresponds to an edge attribute region. This region contains the attribute information of the edges originating from that starting vertex. Each edge attribute region initially has multiple empty slots, each for filling with the attribute information of one edge. For example... Figure 3b As shown, the edge attribute region corresponding to the vertex with ID 0 stores the attribute information p1 of one edge (p1 may contain more than one attribute of that edge) and the attribute information p2 of another edge (p2 may contain more than one attribute of that other edge). There are also some empty slots for inserting the attribute information of other edges from this starting vertex. Furthermore, since the attribute information of edges is usually of a fixed length, metadata may not be present in the edge attribute region. The edge attribute region of a vertex can also be determined based on the edge pointers of each vertex in the segment header.

[0111] Since the edge block region and the edge attribute region are distributed in a similar mirror image starting from both ends, the edge pointer in the segment header can contain the overall offset within the segment for the edge block region and the edge attribute region corresponding to each vertex ID. However, the edge block region is addressed from left to right based on the overall offset within the segment, finding the starting position of the edge block region of the vertex; while the edge attribute region is addressed from right to left based on the overall offset within the segment, finding the starting position of the edge attribute region of the vertex.

[0112] The above describes the main storage architecture for edge data. Other aspects of this architecture will be further detailed in subsequent embodiments. Based on this main storage architecture, the processing of edge data can proceed as follows: Figure 4 As shown, the steps include:

[0113] Step 402: Obtain the edge data to be processed. The edge data includes the starting vertex information, ending vertex information, and attribute information of the edge.

[0114] Step 404: Determine the segment space, which includes the segment header and edge regions.

[0115] Step 406: Determine the edge region corresponding to the starting vertex in the segment space based on the edge pointers of each vertex contained in the segment header.

[0116] Step 408: Update the edge region corresponding to the starting vertex using the above-mentioned ending vertex information and attribute information.

[0117] Step 402 described above can be performed by the data acquisition component in the dynamic graph data storage system, and the storage processing of edge data in steps 404 to 408 can be performed by the edge storage component in the dynamic graph data storage system. The composition of the dynamic graph data storage system will be described in detail in later embodiments.

[0118] If adopted Figure 3b In the preferred storage structure shown, the aforementioned edge region includes an edge block region and an edge attribute region. Based on the edge pointers of each vertex contained in the segment header, the edge block region and the edge attribute region corresponding to the starting vertex are determined in the segment space. The edge block region corresponding to the starting vertex is updated using the ending vertex information, and the edge attribute region is updated using the attribute information.

[0119] As can be seen from the above process, this application proposes a novel storage architecture based on segment space. In the segment space, each vertex has a corresponding edge region to store the information of the ending vertex and the attribute information of the edges. Therefore, when updating edge data, such as adding or deleting, the entire process only needs to update the edge region corresponding to the initial vertex of the edge in the segment space, without affecting the edge data of other vertices that have already been stored, making it more efficient for dynamic graph data storage.

[0120] The edge data to be processed mentioned above can be either edge data to be added or edge data to be deleted. It should be noted that modifying edge data can be considered as deleting and then adding edge data.

[0121] When it is necessary to store graph data of a graph structure, the above storage architecture needs to be established to add vertex data and edge data one by one. The process is described in detail below with reference to the embodiments.

[0122] Figure 5 A flowchart illustrating the method for storing graph data provided in this application embodiment, such as... Figure 5 As shown, the method may include the following steps:

[0123] Step 502: Storage device initialization.

[0124] Storage device initialization is performed after the storage device starts up, or when graph data needs to be stored. During storage device initialization, a block of memory is allocated for the vertex block address array and the segment label table address array, and a virtual address space (which can be a preset size, such as 10G) is allocated for subsequent use by the segment label table, segment space, vertex blocks, and epoch table.

[0125] One possible approach is to use functions such as mmap to allocate the aforementioned virtual address space. Mmap (memory map) is a method of memory-mapped files that maps a file or other object into the process's address space, establishing a one-to-one correspondence between the file's disk address and a segment of virtual addresses in the process's virtual address space.

[0126] In this embodiment, the top layer of the storage structure may include a vertex block address array. This vertex block address array is used to maintain the vertex block address of each vertex. One vertex block is used to store the attribute data of a vertex, and the vertex block address array may store the address of the latest version of the vertex block for each vertex. Figure 6 As shown, the vertex block address array contains the vertex block address of each vertex ID, which points to the latest version of the vertex block for each vertex. Each vertex block contains metadata and attribute data. The metadata may include vertex block size information, vertex block addition time information, etc. A vertex block may also have a prev (forward) pointer, pointing to the previous version of the vertex block, thus forming a linked list structure of vertex blocks. Figure 6 Take, for example, the two versions of the vertex block with vertex ID 0 forming a linked list structure.

[0127] Alternatively, as one possible approach, edge data for each vertex can be placed in a single segment space without distinguishing between edge categories.

[0128] However, edges are usually categorized, and in many graph query and graph computation scenarios, it is necessary to distinguish the categories of edges. For example, in the graph data obtained in the scenario of identifying online transaction risks, vertices can be financially related entities and objects such as accounts, bank cards, and red envelope IDs, and the edges between vertices can be various categories of financially related behavioral relationships such as payment behavior, deposit and withdrawal behavior, contract binding behavior, and sending and receiving red envelopes. As another example, in the graph data obtained in the scenario of identifying online dating fraud, vertices can be social network accounts, instant messaging tool accounts, financial accounts, host addresses, client identifiers, etc., and the edges between vertices can be various categories of behaviors such as sending friend requests, adding friends, chatting, transferring money, sending and receiving red envelopes, and sending links.

[0129] Therefore, edges can be distinguished based on their label information, and edge data corresponding to the same label information can be stored in the same space, which facilitates the maintenance and querying of edge data.

[0130] A segment space can be represented by (x, y), which indicates that the segment space maintains the edge data of all vertices with IDs in the range x and labeled y. For example, the segment space (0, a) maintains the edge data of all vertices with IDs from 0 to 4095 and labeled 'a', while the segment space (1, b) maintains the edge data of all vertices with IDs from 4096 to 8091 and labeled 'b'.

[0131] Therefore, the top layer of the storage structure can also include an array of segment label table addresses. This array is used to maintain the segment label table corresponding to each segment ID. For example... Figure 7 As shown, each segment ID corresponds to a segment label table, and each entry in the segment label table is a pointer to the segment space corresponding to the label, that is, a mapping from edge label to segment space address.

[0132] Step 504: Add vertex data.

[0133] For each vertex data to be added, vertex blocks are added to the vertex block region one by one to store the vertex's attribute information. Then, the addresses of the added vertex blocks are added to the vertex block address array, thus forming a mapping relationship between vertex IDs and vertex block addresses.

[0134] During subsequent additions, if the vertex data of a certain vertex is updated, and a vertex block corresponding to that vertex already exists in the vertex block address array, then the vertex block for that vertex is added to store the vertex's attribute data. A `prev` pointer is then added to this vertex block, pointing to the previous version of the vertex block. The metadata of each vertex block can store the update time information for each vertex block.

[0135] The storage and processing of vertex data in this step can be performed by the vertex storage component in the dynamic graph data storage system.

[0136] Step 506: Obtain the edge data to be added. The edge data includes the starting vertex u information, the ending vertex v information, the label information L, and the attribute information p.

[0137] Step 508: Use the tag information L to query the segment tag array. If the corresponding segment space address is not found, it means that the segment space corresponding to the tag information has not been established. Execute step 510. If the corresponding segment space address is found, execute step 518.

[0138] Step 510: Allocate a storage space of the first size as the segment space corresponding to the tag information, add metadata to the segment header of the segment space, and add the segment address information of the segment space corresponding to the tag information to the segment tag array.

[0139] In this step, the first size can be selected based on a preset value, for example, allocating 1M to the segment space corresponding to the tag information L. Then, the segment space ID, segment size information, tag information L, etc., are added to the metadata in the segment header of the segment space.

[0140] Step 512: Determine if there is a free area of ​​the second size in the segment space. If so, proceed to step 514; otherwise, proceed to step 530.

[0141] It should be noted that if the segment space is initially allocated in step 510, the judgment in step 512 can be omitted, assuming that there is always enough free area to serve as the edge region during the initial allocation of the segment space. Alternatively, this judgment can be performed every time the edge region is initially allocated.

[0142] Step 514: Allocate a storage area of ​​the second size for vertex u in the segment space as an edge region, and add the information of the ending vertex v and the attribute information p to the free slots of the edge region of vertex u.

[0143] by Figure 3bTaking the storage structure shown as an example, in this step, a storage area needs to be allocated as an edge block area for vertex u, and another storage area needs to be allocated as an edge attribute area. The edge block area is allocated from left to right, and the edge attribute area is allocated from right to left. The size of the edge block area allocated to vertex u can be a preset value, or it can be flexibly selected according to actual needs, such as based on the storage area required for the end vertex information. The size of the edge attribute area allocated to vertex u can be a preset value. As one possible implementation, the size of the edge attribute area allocated to vertex u corresponds to the size of the edge block area. For example, if the edge block area corresponding to vertex u has 8 free slots, then the edge attribute area corresponding to vertex u also has 8 free slots.

[0144] In addition to containing information about the ending vertex v, the edge region can also contain metadata about the edge. Therefore, this step can further add metadata about the edge, such as the size of the edge and the ID of the edge.

[0145] Step 516: Allocate an epoch table for vertex u, add the current epoch to the epoch table, and add the offset of the edge added in step 514 within the edge region. Update the edge pointer and epoch table pointer of vertex u in the segment header. Then proceed to step 506 to add the next edge data to be added.

[0146] The epoch table involved in this step is a preferred method for concurrent read and write operations during dynamic graph data storage, and its details will be described in subsequent embodiments. It should be noted that if concurrent read and write operations are not required, the epoch table processing in this procedure can be omitted. Alternatively, if other alternative methods are used to control concurrent read and write operations, the epoch table processing in this procedure can also be omitted.

[0147] Step 518: Determine the segment space corresponding to the label information L.

[0148] This step can use the tag information L to query the segment tag array, determine the segment address information corresponding to the tag information L, and determine the corresponding segment space based on the segment address information.

[0149] Step 520: Query the edge pointer corresponding to vertex u from the segment head of the segment space. If the edge pointer of vertex u is found, proceed to step 522; otherwise, it means that there is no edge block region of vertex u yet, and proceed to step 512.

[0150] Step 522: Determine whether there is an empty slot in the edge region of vertex u pointed to by the edge pointer of vertex u. If yes, proceed to step 524; otherwise, proceed to step 526.

[0151] Step 524: Add the information of the end vertex v and the attribute information p to the free slots of the edge region of vertex u, update the epoch table pointer of vertex u as needed, and then go to step 506 to add the next edge data to be added.

[0152] If adopted Figure 3b In the storage structure shown, the determination of whether there is a free slot in the edge region in step 522 includes whether there is a free slot in the edge block region of vertex u and whether there is a free slot in the edge attribute region. If there are free slots in both, the information of the ending vertex v is added to the free slot in the edge block region of vertex u, and the attribute information p of the edge is added to the edge attribute region corresponding to vertex u in the segment space.

[0153] The on-demand update of the epoch table pointer for vertex u includes: if the current epoch already exists in the epoch table of vertex u in the segment header, no update is needed; if the current epoch does not exist in the epoch table of vertex u in the segment header, the current epoch is added to the epoch table of vertex u, and the offset of the edge data in the edge region is added to the epoch table. Details will be provided in subsequent embodiments.

[0154] Step 526: Determine whether there is a free space of size 4 in the segment space. If yes, proceed to step 528; otherwise, proceed to step 530.

[0155] Step 528: Allocate a storage area of ​​size 4 from the free space as the new edge area corresponding to vertex u, copy the contents of the original edge area corresponding to vertex u to the new edge area, update the edge pointer of vertex u contained in the segment header according to the new edge area, and proceed to step 524 in the new edge area.

[0156] The fourth size is larger than the second size; for example, the fourth size can be twice the second size. This step essentially involves migrating the edge region of vertex u to another, larger edge region when the edge region of vertex u is exhausted.

[0157] If adopted Figure 3b In the storage structure shown, if there is no free slot in the edge block region of vertex u, then in this step, a new, larger storage region is allocated from the free space as the new edge block region corresponding to vertex u. The contents of the original edge block region corresponding to vertex u are copied to the new edge block region. The edge pointer of vertex u contained in the segment header is updated using the overall offset of the new edge block region.

[0158] If there are no free slots in the edge attribute region of vertex u, then in this step, a new, larger storage region is allocated from the free space as the new edge attribute region corresponding to vertex u. The contents of the original edge attribute region corresponding to vertex u are copied to the new edge attribute region, and the edge pointer of vertex u contained in the segment header is updated using the overall offset of the new edge attribute region. Since the edge attribute region and the edge block region are mirror images, the free slots in the edge attribute region are usually exhausted when the free slots in the edge block region are exhausted. Therefore, the overall offset of the same edge data in the edge block and the edge attribute in the segment is consistent.

[0159] The original edge block region and original edge attribute region corresponding to the starting vertex u can be deleted or marked as invalid. However, since the migration process may involve reading the original edge block region and original edge attribute region, deletion or marking as invalid can be performed after all reading processing of the original edge block region has been completed, and deletion or marking as invalid can be performed after all reading processing of the original edge attribute region has been completed.

[0160] Step 530: Allocate a storage space of the third size as a new segment space corresponding to the tag information L. Copy the contents of the original segment space corresponding to the tag information L, except for the segment header, to the new segment space. Add a segment header to the new segment space based on the copied contents. Update the segment address information corresponding to the tag information L in the segment tag array with the address of the new segment space using the new segment space. Proceed to step 524 in the new segment space.

[0161] The third size is larger than the first size; for example, the third size can be twice the first size. This step essentially involves migrating the segment space to a larger segment space when the segment space corresponding to the label information L is exhausted.

[0162] As can be seen from the above process, the embodiments of this application store the edge data corresponding to the same vertex in the same edge region in the segment space, making the edge data corresponding to the same vertex adjacent; and the edge data corresponding to each vertex is stored in the edge region of the segment space according to the vertex ID, making the edge data corresponding to adjacent vertex IDs also stored adjacently, thereby improving data locality and significantly improving the performance of edge data reading vertex by vertex. Here, "data locality" is a characteristic exhibited when a program accesses data, and the optimization goal is to make the data to be read during the processing as close as possible.

[0163] The migration of edge regions involved in step 528 of the above process can be performed by the region migration component in the dynamic graph data storage system. The migration of segment space involved in step 530 can be performed by the segment migration component in the dynamic graph data storage system. The other steps concerning the storage processing of edge data can be performed by the edge storage component in the dynamic graph data storage system.

[0164] Figure 5 The process shown involves the migration of edge regions and the migration of segment space. The migration of segment space in the above process illustrates two scenarios: the first is when there is insufficient free space in the segment space when allocating an edge region for a new vertex; the second is when there is insufficient free space in the segment space to perform the edge region migration for a given vertex.

[0165] As one possible approach, during segment migration, all write operations (including addition, deletion, etc.) are blocked, while read operations can be processed normally.

[0166] One feasible approach involves blocking write operations on the migrating edge region during the migration process, while read operations can proceed normally. Read and write operations on other edge regions can also be processed normally.

[0167] Regarding the above process, it should be noted that step 504 involves adding vertex data, while steps starting from step 506 involve adding edge data. When storing graph data, vertex data can be added first, followed by edge data. Alternatively, the data of a single vertex can be stored, and then all edge data originating from that vertex can be stored for that vertex. Vertex and edge data can also be added in parallel.

[0168] Figure 5 The illustrated process describes the entire process of storing graph data. However, the graph structure changes over time, resulting in dynamic updates to the graph data. For example, in the scenario of online transaction risk identification, user behavior unfolds gradually over time. New accounts may be created and closed, new accounts may interact with bank cards, and new types of behavior may occur between accounts. This may involve adding vertex data, adding edge data, deleting vertex data, and deleting edge data. Changes to vertex or edge data can be viewed as deleting first and then adding. Specifically, if one or more vertex data need to be added to the existing graph data, the process is directly executed... Figure 5 Step 504 in the illustrated process. If you want to add one or more edge data to the existing stored graph data, you can directly retrieve the edge data from the graph data. Figure 5 Step 506 in the process shown will begin execution.

[0169] When deleting vertex data, the vertex block corresponding to the vertex is determined from the vertex block address array. This vertex block can be deleted or marked as invalid. Simultaneously, the edge data corresponding to the vertex is also deleted. When deleting edge data, the segment address is obtained by querying the segment label array using the edge data's label information. Then, the edge pointer corresponding to the starting vertex of the edge is queried from the segment header of the segment space to determine the edge region pointed to by the edge pointer, including the edge block region and the edge attribute region. The ending vertex information of the edge is deleted from the corresponding edge block region or marked as invalid, and the attribute information of the edge is deleted from the corresponding attribute region or marked as invalid.

[0170] Based on the above storage structure, the method for reading dynamic graph data will be described in detail below with reference to the embodiments.

[0171] Figure 8 The flowchart of the dynamic graph data reading method provided in the embodiments of this application is as follows: Figure 8 As shown, the method may include the following steps:

[0172] Step 802: Obtain the first read transaction information of the edge data. The first read transaction information includes at least the starting vertex information to be queried.

[0173] Graph data is mainly used in OLTP (online transaction processing), which performs ACID (Acceptable and Distributed) transactions on data. A transaction can be considered as a process of adding, deleting, modifying, and querying data within a data model.

[0174] Read operations on edge data are typically performed after a read transaction is acquired. Read transactions can be triggered by various scenarios, but most are based on a starting vertex. For example, in some graph computations, when calculating the feature vector of a specific vertex, it's necessary to query both the attribute data of that specific vertex and the attribute data of all edges originating from that specific vertex. For the latter, the read transaction information will include the information of that specific vertex, which serves as the starting vertex information to be queried.

[0175] For example, in some graph computations, it is necessary to scan edge data vertex by vertex. In this case, the resulting read transaction reads the edge data of each vertex separately, and each vertex in the graph structure is used as the starting vertex to be queried.

[0176] For example, in some graph computations, it is necessary to locate a specific edge. The resulting read transaction reads the data of that specific edge, and the read transaction information will include the information of the starting vertex and the ending vertex.

[0177] Step 804: Determine the segment space, which includes the segment header and edge regions.

[0178] For the storage structure of the segment space, please refer to the relevant description in the previous embodiments. The edge region may include the edge block region and the edge attribute region, which will not be elaborated here.

[0179] As one possible approach, segment spaces are established separately for edge label information. Therefore, the first read transaction information can also include label information, i.e., edge data with specific labels needs to be read. In this step, the label information can be used to query the segment label array to determine the segment address information corresponding to the label information; the corresponding segment space is then determined based on the segment address information.

[0180] Step 806: Determine the edge region corresponding to the starting vertex in the segment space based on the edge pointers of each vertex contained in the segment header.

[0181] Step 808: Read edge data from the edge region corresponding to the starting vertex.

[0182] If adopted Figure 3b Given the storage structure shown, in step 806, the edge block region and edge attribute region corresponding to the starting vertex can be determined, and in step 808, the end vertex information can be read from the edge block region corresponding to the starting vertex, and / or the attribute information of the edge can be read from the edge attribute region corresponding to the starting vertex.

[0183] In this step, depending on the type of read transaction, all edge data corresponding to the starting vertex can be read from the edge block region and edge attribute region corresponding to the starting vertex. Alternatively, edge data matching the query conditions can be read from the edge block region and edge attribute region corresponding to the starting vertex, depending on the more granular query conditions of the read transaction. For example, the query conditions can be based on the ending vertex, the update time, the quantity, and so on.

[0184] Step 802 in the above process can be executed by the transaction acquisition component in the dynamic graph data reading system. Steps 804 to 808 can be executed by the edge reading component in the dynamic graph data reading system. The composition of the dynamic graph data reading system will be described in detail in later embodiments.

[0185] exist Figure 5The illustrated embodiment involves the creation and updating of an epoch table. The epoch table used in this embodiment is a preferred method for concurrent read / write operations during dynamic graph data storage, employing an epoch-based MVCC (Multi-Version Concurrency Control) mechanism. The MVCC mechanism creates a new version of a value when it is updated, ensuring concurrent read operations access a stable version. However, multi-version mechanisms introduce additional overhead to read operations; the finer the granularity of the MVCC, the greater the overhead. Therefore, this embodiment employs a coarse-grained MVCC mechanism to reduce the overhead of multi-version operations.

[0186] Figure 5 As mentioned in the illustrated embodiment, the segment header of the segment space can further include pointers to the epoch tables of each vertex, pointing to the epoch tables of each vertex. The epoch table stores the offset of the edge data added in the corresponding epoch within the edge region. An epoch refers to a time segment, dividing time into continuous and non-overlapping epochs. The duration of each epoch can be set according to actual needs, for example, to tens of microseconds. Each epoch can have a number, and each vertex in the segment space is associated with an epoch table, which maintains the epoch information of all edges stored in the latest edge region of each vertex.

[0187] After updating the edge region corresponding to the starting vertex, determine whether to update the epoch table of the starting vertex based on the current time. If the epoch table corresponding to the current time already exists in the epoch table of the starting vertex, there is no need to update the epoch table of the starting vertex; otherwise, create a new epoch corresponding to the current time in the epoch table of the starting vertex. This epoch stores the offset of the aforementioned edge data in the edge region, and this offset is the internal offset of the edge region.

[0188] Because the edge block region and the edge attribute region are mirrored, therefore in Figure 9 Let's take an edge region as an example to explain the principle of the epoch table. Assume vertex ID 0 is the starting vertex. At time t1, an ending vertex v1 is added to the edge region corresponding to this vertex. Then, in the epoch table corresponding to vertex ID 0, the epoch number 0 of t1 is added. This epoch number 0 stores the offset of the edge region containing v1. Assuming an epoch lasts 50 microseconds, at time t2, an ending vertex v2 is added to the edge region corresponding to vertex ID 0. If t1 and t2 belong to the same epoch, then there is no need to add a new epoch to the epoch table corresponding to vertex ID 0.

[0189] If a terminal vertex v3 is added to the edge region corresponding to the vertex with ID 0 at time t3, and time t3 already belongs to the 4th epoch, then the epoch number 3 to which t3 belongs is added to the epoch table corresponding to the vertex with ID 0. This epoch number 3 stores the offset of the edge region where v3 is located. If terminal vertices v7 and v4 are added at times t4 and t5 respectively, but times t4 and t5 belong to the same epoch as time t3, then there is no need to update the epoch table.

[0190] If the ending vertex v5 is added to the edge block region corresponding to the vertex with ID 0 at time t6, and time t6 already belongs to the 5th epoch, then add the epoch number 4 to the epoch table corresponding to the vertex with ID 0. The epoch number 4 stores the offset of the edge block where v5 is located.

[0191] The above-described addition process is actually a write operation. Based on the concurrent read control generated by the MVCC mechanism, in step 808, the pointers to the epoch tables of each vertex in the segment header are queried according to the starting vertex information to be queried, thus determining the epoch table corresponding to the starting vertex information to be queried. Then, based on the epoch table, edge data with an epoch less than or equal to the time of the read transaction is searched in the edge region corresponding to the starting vertex to be queried. For example, edge blocks with an epoch less than or equal to the time of the read transaction are searched in the edge block region corresponding to the starting vertex to be queried. Another example is attribute information with an epoch less than or equal to the time of the read transaction in the edge attribute region corresponding to the starting vertex to be queried. The time of the read transaction can be the start time of the read transaction or a user-specified time, such as specifying one or more specific epochs. If it is one or more specific epochs specified by the user, these epochs are all less than the epoch of the current time, and edge data with an epoch not exceeding this specified time is searched in the edge region corresponding to the starting vertex to be queried.

[0192] Since the MVCC mechanism does not prohibit write operations, to ensure the normal operation of read operations and avoid read and write operations conflicting and causing read operation anomalies, read operations are only allowed to read edge data whose data update time is before the time of the read transaction, that is, edge data whose corresponding epoch is less than or equal to the time of the read transaction.

[0193] Still with Figure 9For example, assuming read transaction 1's time is t6 and the ID of the starting vertex to be queried is 0, after querying the epoch table for vertex ID 0, it is determined that epoch numbers less than or equal to t6 are 0 and 3. Therefore, read transaction 1 searches from the edge pieces containing v1, v2, v3, v7, and v4. The current time must be greater than or equal to t6, so any write operation occurring at the current time must have occurred after epoch number 5. Therefore, it will not conflict with the read operation corresponding to read transaction 1 and thus will not affect read transaction 1. It is understandable that when multiple read transactions occur concurrently, this processing logic can guarantee the normal read processing of all read transactions.

[0194] Furthermore, in some scenarios, the number of concurrent read transactions is often much greater than that of write transactions. Therefore, the requests corresponding to read transactions may be delayed in the queue. To avoid read data errors caused by read operation delays that originally occurred before the write transaction but occurred after the write transaction, it is also necessary to use the epoch table to search for edge data in the edge region corresponding to the starting vertex to be queried, where the corresponding epoch is less than or equal to the time of the read transaction. This ensures the consistency between the read edge data and the read transaction time.

[0195] Still with Figure 9 For example, suppose the time for read transaction 1 is t3 and the time for write transaction 2 is t4. Due to the large number of concurrent read transactions, write transaction 2 is processed first, meaning edge block v7 is written at t4. Therefore, when processing read transaction 1, if the data is read according to its processing time, an error will occur, and edge block v7 will also be included. However, if the method in this embodiment is used, reading edge blocks with an epoch less than or equal to t3 (i.e., read transaction time) will not include v7, thus ensuring the correctness of the read data.

[0196] As can be seen, in this embodiment, by using an independent epoch table to store the update time of the edge block, it is only necessary to maintain the offset of the edge data that was updated in the epoch table. Compared with the traditional MVCC, which requires maintaining the timestamp of each version of each edge data and performing version checks when reading, the overhead is greatly reduced. At the same time, it also ensures the consistency between the read edge data and the read transaction time, and ensures the normal operation of concurrency control.

[0197] Besides the MVCC mechanism for the epoch table mentioned above, other methods can also be used. For example, locking mechanisms can be used to implement concurrent read and write control, and so on.

[0198] As one possible approach, a locking mechanism can be used in this application embodiment to avoid conflicts caused by concurrent writes, such as preventing concurrent writes from still occurring in the original segment space during segment migration.

[0199] If a locking mechanism is used to control concurrent read and write operations, read-write locks can be applied to the segment space. Normal edge addition operations require attempting to acquire a shared lock on the segment space. If a transaction T acquires a shared lock on the segment space, other transactions can concurrently read and write data, but cannot perform segment migrations.

[0200] If the edge addition operation triggers a segment migration, a mutex lock on the segment space is acquired. When the segment space is locked by a transaction's mutex lock, the mutex lock stores the transaction's identifier. Only this transaction can unlock the mutex lock thereafter. Before the lock is unlocked, no transactions, including this transaction itself, can write to this segment space, but they can read from it.

[0201] While locking mechanisms can control concurrent operations, they incur significant system overhead. Therefore, the MVCC mechanism based on an epoch table provided in this embodiment is preferred. Alternatively, a partial locking mechanism can be combined with the MVCC mechanism based on an epoch table.

[0202] In addition, when reading edge data in the edge block area and edge attribute area, data that has been marked with an invalid label will not be read.

[0203] If a downstream device of the storage device needs to read vertex data, the execution layer of the storage device will obtain the second read transaction information of the vertex data. This second read transaction information contains the vertex information to be queried. The vertex reading component in the dynamic graph data reading system can use the vertex information to query the vertex block address data to obtain the vertex block address information corresponding to the vertex information to be queried; and read the vertex data from the vertex block pointed to by the vertex block address information.

[0204] As one possible approach, if the second read transaction information contains time information, which can be a query condition, such as querying vertex data corresponding to a certain time point; or the time information can also be the occurrence time of the read transaction; then when executing the above-mentioned reading of vertex data from the vertex block pointed to by the vertex block address information, the vertex block pointed to by the vertex block address information can be determined first, and the version with an update time less than or equal to the above-mentioned time information can be determined from the various versions of the vertex block, and the vertex data can be read from the vertex block of the determined version.

[0205] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0206] According to another embodiment, a dynamic graph data storage system is provided. Figure 10 This diagram illustrates a schematic block diagram of a dynamic graph data storage system according to one embodiment, the system being configured in... Figure 1 The execution layer of the storage device in the architecture shown. For example... Figure 10 As shown, the system 1000 includes: a data acquisition component 1001 and an edge storage component 1002, and may further include a vertex storage component 1003, a segment migration component 1004, and a region migration component 1005. The main functions of each component are as follows:

[0207] The data acquisition component 1001 is configured to acquire edge data to be processed. The edge data includes the starting vertex information, ending vertex information, and attribute information of the edge.

[0208] The edge storage component 1002 is configured to determine a segment space, which includes a segment header and an edge region; determine the edge region corresponding to the starting vertex in the segment space based on the edge pointers of each vertex contained in the segment header; and update the edge region corresponding to the starting vertex using the ending vertex information and attribute information.

[0209] As one possible approach, the edge data can also include tag information. Accordingly, the edge storage component 1002 can be specifically configured to: query the segment tag array using the tag information to determine the segment address information corresponding to the tag information; and determine the corresponding segment space based on the segment address information.

[0210] As one possible implementation method, the data acquisition component 1001 can also be configured to acquire vertex data to be processed.

[0211] The vertex storage component 1003 can be configured to: update the vertex blocks in the vertex block region using vertex data, and update the vertex block address array using the update result of the vertex block region; wherein the vertex block region contains the vertex blocks of each vertex, and the vertex block address array is used to store the vertex block address information corresponding to the vertex.

[0212] As one possible approach, the above-mentioned pending processing refers to pending addition, and the above-mentioned update refers to addition processing.

[0213] As another possible approach, the above-mentioned pending processing is to be deleted, and the above-mentioned update is to delete or mark as invalid.

[0214] As one possible implementation method, if the edge data to be processed is edge data to be added, the edge storage component 1002 can be specifically configured to: query the segment tag array using the tag information; if the segment address information corresponding to the tag information is not found, allocate a storage space of the first size as the segment space corresponding to the tag information, add metadata to the segment header of the segment space, and add the segment address information of the segment space corresponding to the tag information to the segment tag array.

[0215] The metadata includes segment space identification information, tag information, and segment space size information.

[0216] As one possible implementation, if the edge data to be processed is edge data to be added, the edge storage component 1002 can be specifically configured to: query the edge pointers of each vertex contained in the segment header; if the edge pointer of the starting vertex is not found, allocate a storage area of ​​the second size in the segment space as the edge region for the starting vertex; if the edge pointer of the starting vertex is found, determine the edge region corresponding to the starting vertex in the segment space.

[0217] As one possible implementation, when the edge storage component 1002 allocates a storage area of ​​the second size as an edge region for the starting vertex in the segment space, if there is a free space of the second size in the segment space, then the storage area of ​​the second size is allocated as an edge region from the free space; otherwise, after the segment migration component 1004 is triggered to migrate the segment space to another larger segment space, the edge storage component 1002 allocates a storage area of ​​the second size as an edge region for the starting vertex in the other larger segment space.

[0218] As one possible implementation, if the edge data to be processed is edge data to be added, then when the edge storage component 1002 updates the edge region corresponding to the starting vertex using the end vertex information and attribute information, it can be specifically configured as follows: if there is a free slot in the edge region corresponding to the starting vertex, the end vertex information and edge attribute information are added to the slot; otherwise, after the region migration component 1005 is triggered to migrate the edge region corresponding to the starting vertex to a larger edge region, the edge storage component 1002 adds the edge data in the larger edge region.

[0219] As one possible implementation method, when migrating the edge region corresponding to the starting vertex to a larger edge region, the region migration component 1005 specifically performs the following:

[0220] If there is a free space of size 4 in the segment space, then allocate a storage area of ​​size 4 from the free space as the new edge area corresponding to the starting vertex, copy the contents of the original edge block area corresponding to the starting vertex to the new edge area, and update the edge pointer of the starting vertex contained in the segment header according to the new edge area.

[0221] If there is no free space of the fourth size in the segment space, the segment migration component 1004 is triggered to migrate the segment space to another larger segment space. Then, the region migration component 1005 allocates a storage region of the fourth size from the free space of the migrated segment space as the new edge region corresponding to the starting vertex, copies the contents of the original edge region corresponding to the starting vertex to the new edge region, and updates the edge pointer of the starting vertex contained in the segment header according to the new edge region.

[0222] The original edge block region and original edge attribute region corresponding to the starting vertex u can be deleted or marked as invalid. However, since the migration process may involve reading the original edge block region and original edge attribute region, deletion or marking as invalid can be performed after all reading processing of the original edge block region has been completed, and deletion or marking as invalid can be performed after all reading processing of the original edge attribute region has been completed.

[0223] As one possible implementation, the segment migration component 1004 performs the following when migrating a segment space to another larger segment space:

[0224] Allocate a third-sized storage space as a new segment space corresponding to the tag information;

[0225] Copy the contents of the original segment space corresponding to the tag information, excluding the segment header, to the new segment space. Add the segment header of the new segment space based on the copied content, and use the new segment space to update the segment address information corresponding to the tag information in the segment tag array to the new segment space address.

[0226] As one possible implementation, the segment header also includes pointers to the epoch tables of time segments for each vertex. The epoch tables contain the offset of the edge data added in the corresponding epoch within the edge region corresponding to the starting vertex. The edge storage component 1002 is further configured to determine whether to update the epoch table of the starting vertex based on the current time after updating the edge region corresponding to the starting vertex using the end vertex information and attribute information.

[0227] In one preferred embodiment, the aforementioned edge region includes an edge block region and an edge attribute region.

[0228] When the edge storage component 1002 updates the edge region corresponding to the starting vertex using the end vertex information and attribute information, it is specifically configured to update the edge block region corresponding to the starting vertex using the end vertex information and update the edge attribute region corresponding to the starting vertex using the attribute information.

[0229] As a more preferred implementation, in the segment space excluding the segment header, the edge block regions of each vertex are allocated sequentially according to a first direction, the end vertex information is added sequentially according to the first direction, the attribute regions of each vertex are allocated sequentially according to a second direction, and the attribute information is added sequentially according to the second direction, wherein the first direction and the second direction are opposite.

[0230] According to another embodiment, a dynamic graph data reading system is provided. Figure 11 This diagram illustrates a schematic block diagram of a dynamic graph data reading system according to one embodiment, the system being configured in... Figure 1 The execution layer of the storage device in the architecture shown. For example... Figure 11 As shown, the system 1100 includes: a transaction acquisition component 1101 and an edge reading component 1102, and may further include a vertex reading component 1103. The main functions of each component are as follows:

[0231] The transaction acquisition component 1101 is configured to acquire the first read transaction information of edge data. The first read transaction information includes at least the starting vertex information to be queried.

[0232] The edge reading component 1102 is configured to determine a segment space, which includes a segment header and an edge region; determine the edge region corresponding to the starting vertex in the segment space based on the edge pointers of each vertex contained in the segment header; and read edge data from the edge region corresponding to the starting vertex.

[0233] As one possible implementation method, the first read transaction information also includes tag information; accordingly, when determining the segment space, the edge reading component 1102 can be specifically configured to: use the tag information to query the segment tag array to determine the segment address information corresponding to the tag information; and determine the corresponding segment space based on the segment address information.

[0234] As one possible implementation, the transaction acquisition component 1101 is further configured to acquire second read transaction information of vertex data, the second read transaction information including vertex information to be queried.

[0235] Vertex reading component 1103 is configured to query the vertex block address array using the vertex information to be queried, obtain the vertex block address information corresponding to the vertex information to be queried, and read vertex data from the vertex block pointed to by the vertex block address information.

[0236] As one possible implementation, the segment header also contains pointers to the epoch tables of time segments for each vertex, which contain the offset of the edge data added in the corresponding epoch within the edge region corresponding to the starting vertex.

[0237] When the edge reading component 1102 reads edge data from the edge region corresponding to the starting vertex, it can be configured to query the pointers to the epoch tables of each vertex in the segment header based on the starting vertex information to be queried, and determine the epoch table corresponding to the starting vertex information to be queried; based on the epoch table, determine the edge region in the edge region corresponding to the starting vertex whose corresponding epoch is less than or equal to the time of the read transaction; and read the edge data in the determined edge region.

[0238] In one preferred embodiment, the edge region includes an edge block region and an edge attribute region.

[0239] When the edge reading component 1102 reads edge data from the edge region corresponding to the starting vertex, it can be specifically configured to read the ending vertex information from the edge block region corresponding to the starting vertex and / or read the edge attribute information from the edge attribute region corresponding to the starting vertex.

[0240] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0241] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0242] And an electronic device, comprising:

[0243] One or more processors; and

[0244] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0245] in, Figure 12An exemplary architecture of an electronic device is shown, which may include a processor 1210, a video display adapter 1211, a disk drive 1212, an input / output interface 1213, a network interface 1214, and a memory 1220. The processor 1210, video display adapter 1211, disk drive 1212, input / output interface 1213, network interface 1214, and memory 1220 can communicate with each other via a communication bus 1230.

[0246] The processor 1210 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0247] The memory 1220 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1220 can store the operating system 1221 for controlling the operation of the electronic device 1200, and the basic input / output system (BIOS) 1222 for controlling the low-level operations of the electronic device 1200. Additionally, it can store a web browser 1223, a data storage management system 1224, and a dynamic image data storage / retrieval system 1225, etc. The aforementioned dynamic image data storage / retrieval system 1225 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 1220 and executed by the processor 1210.

[0248] Input / output interface 1213 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0249] Network interface 1214 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0250] Bus 1230 includes a pathway for transmitting information between various components of the device, such as processor 1210, video display adapter 1211, disk drive 1212, input / output interface 1213, network interface 1214, and memory 1220.

[0251] It should be noted that although the above-described device only shows the processor 1210, video display adapter 1211, disk drive 1212, input / output interface 1213, network interface 1214, memory 1220, bus 1230, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0252] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0253] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0254] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A dynamic graph data storage system, disposed at an execution layer of a storage device, characterized in that, The system includes: The data acquisition component is configured to acquire edge data corresponding to each vertex to be processed. The edge data includes the starting vertex information, ending vertex information, and attribute information of the edge. Each vertex includes: the starting vertex. An edge storage component is configured to determine a segment space, the segment space including a segment header and an edge region; determine the edge region corresponding to the starting vertex in the segment space based on the edge pointers of each vertex contained in the segment header; update the edge region corresponding to the starting vertex using the ending vertex information and the attribute information, wherein the edge data corresponding to the same vertex is stored in the same edge region, the edge region including an edge block region and an edge attribute region corresponding to each vertex, the edge block region including: the edge block corresponding to the vertex, the edge block being used to store the ending vertex information.

2. The system according to claim 1, characterized in that, The edge data also includes label information; When determining the segment space, the edge storage component performs the following steps: using the tag information to query the segment tag array to determine the segment address information corresponding to the tag information; and determining the corresponding segment space based on the segment address information.

3. The system according to claim 1, characterized in that, The data acquisition component is further configured to acquire vertex data to be processed; The system further includes: a vertex storage component, configured to update vertex blocks in a vertex block region using the vertex data, and to update the vertex block address array using the update result of the vertex block region; The vertex block region contains the vertex blocks of each vertex, and the vertex block address array is used to store the vertex block address information corresponding to the vertex.

4. The system according to any one of claims 1 to 3, characterized in that, The term "pending processing" refers to "pending addition," and the term "update" refers to "addition processing"; or... The term "to be processed" refers to "to be deleted," and the term "update" refers to either deletion processing or marking an invalid label.

5. The system according to claim 2, characterized in that, If the edge data to be processed is edge data to be added, then when the edge storage component uses the tag information to query the segment tag array and determine the segment address information corresponding to the tag information, it executes: Use the tag information to query the segment tag array; If the segment address information corresponding to the tag information is not found, then a storage space of the first size is allocated as the segment space corresponding to the tag information, metadata is added to the segment header of the segment space, and the segment address information of the segment space corresponding to the tag information is added to the segment tag array.

6. The system according to claim 5, characterized in that, The metadata includes the segment space's identification information, the tag information, and the segment space's size information.

7. The system according to claim 1, characterized in that, If the edge data to be processed is edge data to be added, then when the edge storage component determines the edge region corresponding to the starting vertex in the segment space based on the edge pointers of each vertex contained in the segment header, it performs the following: Query the edge pointers of each vertex contained in the segment header; If the edge pointer of the starting vertex is not found, then a storage area of ​​the second size is allocated in the segment space for the starting vertex as an edge area; If the edge pointer of the starting vertex is found, then the edge region corresponding to the starting vertex is determined in the segment space.

8. The system according to claim 7, characterized in that, The system also includes: a segment migration component; When the edge storage component allocates a storage region of the second size as an edge region for the starting vertex in the segment space, it executes: If there is a free space of the second size in the segment space, then a storage area of ​​the second size is allocated from the free space for the starting vertex as an edge region; otherwise, after the segment migration component is triggered to migrate the segment space to another larger segment space, a storage area of ​​the second size is allocated from the other larger segment space for the starting vertex as the edge region.

9. The system according to claim 1, characterized in that, The system also includes: a region migration component; If the edge data to be processed is edge data to be added, then when the edge storage component updates the edge region corresponding to the starting vertex using the ending vertex information and the attribute information, it performs the following: If there is an empty slot in the edge region corresponding to the starting vertex, the ending vertex information and the attribute information are added to the empty slot; otherwise, the region migration component is triggered to migrate the edge region corresponding to the starting vertex to a larger edge region, and then the ending vertex information and the attribute information are added to the larger edge region.

10. The system according to claim 9, characterized in that, The system also includes: a segment migration component; The region migration component is configured as follows: If there is a free space of size four in the segment space, then allocate a storage area of ​​size four from the free space as a new edge area corresponding to the starting vertex, copy the contents of the original edge area corresponding to the starting vertex to the new edge area, and update the edge pointer of the starting vertex contained in the segment header according to the new edge area. If there is no free space of the fourth size in the segment space, the segment migration component is triggered to migrate the segment space to another larger segment space. Then, the storage area of ​​the fourth size is allocated from the free space of the migrated segment space as the new edge region corresponding to the starting vertex. The contents of the original edge region corresponding to the starting vertex are copied to the new edge region. The edge pointer of the starting vertex contained in the segment header is updated according to the new edge region.

11. The system according to claim 8, characterized in that, The edge data also includes label information; the segment migration component is configured as follows: Allocate a third-sized storage space as a new segment space corresponding to the tag information; Copy the contents of the original segment space corresponding to the tag information, excluding the segment header, to the new segment space. Add the segment header of the new segment space based on the copied content, and update the segment address information corresponding to the tag information in the segment tag array to the new segment space address using the new segment space.

12. The system according to claim 1, characterized in that, The header also includes a pointer to the epoch table of time segments for each vertex, the epoch table containing the offset of the edge data added in the corresponding epoch within the edge region corresponding to the starting vertex; The edge storage component is further configured to: after updating the edge region corresponding to the starting vertex using the ending vertex information and the attribute information, determine whether to update the epoch table of the starting vertex based on the current time.

13. The system according to any one of claims 1, 7 to 10, and 12, characterized in that, When the edge storage component updates the edge region corresponding to the starting vertex using the ending vertex information and the attribute information, it performs the following: The edge block region corresponding to the starting vertex is updated using the ending vertex information, and the edge attribute region corresponding to the starting vertex is updated using the attribute information.

14. The system according to claim 13, characterized in that, In the segment space excluding the segment header, the edge block regions of each vertex are allocated sequentially according to a first direction, the end vertex information is added sequentially according to the first direction, the attribute regions of each vertex are allocated sequentially according to a second direction, and the attribute information is added sequentially according to the second direction, wherein the first direction and the second direction are opposite.

15. A dynamic graph data reading system, configured at the execution layer of a storage device, characterized in that, The system includes: The transaction acquisition component is configured to acquire the first read transaction information of the edge data, and the first read transaction information includes at least the starting vertex information to be queried; An edge reading component is configured to determine a segment space, the segment space including a segment header and an edge region; based on the edge pointers of each vertex contained in the segment header, determine the edge region corresponding to the starting vertex in the segment space, wherein each vertex includes the starting vertex, and the edge data corresponding to the same vertex is stored in the same edge region, the edge region including the edge block region and the edge attribute region corresponding to each vertex, the edge block region including the edge block corresponding to the vertex, the edge block being used to store the end vertex information of the edge; and read edge data from the edge region corresponding to the starting vertex.

16. The system according to claim 15, characterized in that, The first read transaction information also includes tag information; When determining the segment space, the edge reading component performs the following steps: using the tag information to query the segment tag array to determine the segment address information corresponding to the tag information; and determining the corresponding segment space based on the segment address information.

17. The system according to claim 15, characterized in that, The transaction acquisition component is further configured to acquire second read transaction information of vertex data, the second read transaction information including vertex information to be queried; The system further includes: a vertex reading component, configured to query a vertex block address array using the vertex information to be queried, to obtain vertex block address information corresponding to the vertex information to be queried; and to read vertex data from the vertex block pointed to by the vertex block address information.

18. The system according to claim 15, characterized in that, The header also includes a pointer to the epoch table of time segments for each vertex, the epoch table containing the offset of the edge data added in the corresponding epoch within the edge region corresponding to the starting vertex; When the edge reading component reads edge data from the edge region corresponding to the starting vertex, it performs the following steps: based on the starting vertex information to be queried, it queries the pointers to the epoch tables of each vertex in the segment header to determine the epoch table corresponding to the starting vertex information to be queried; based on the epoch table, it determines the edge region in the edge region corresponding to the starting vertex whose corresponding epoch is less than or equal to the time of the read transaction; and it reads the edge data in the determined edge region.

19. The system according to any one of claims 15 to 18, characterized in that, The edge region includes the edge block region and the edge attribute region; When the edge reading component reads edge data from the edge region corresponding to the starting vertex, it performs the following: Read the end vertex information from the edge block region corresponding to the starting vertex and / or read the attribute information from the edge attribute region corresponding to the starting vertex.

20. A method for storing dynamic graph data, characterized in that, The method includes: Obtain the edge data corresponding to each vertex to be processed. The edge data includes the starting vertex information, ending vertex information, and attribute information of the edge. Each vertex includes: the starting vertex. Define a segment space, which includes segment headers and edge regions; Based on the edge pointers of each vertex contained in the segment header, determine the edge region corresponding to the starting vertex in the segment space; The edge region corresponding to the starting vertex is updated using the ending vertex information and the attribute information. The edge data corresponding to the same vertex is stored in the same edge region. The edge region includes an edge block region and an edge attribute region corresponding to each vertex. The edge block region includes an edge block corresponding to the vertex. The edge block is used to store the ending vertex information.

21. A method for reading dynamic graph data, characterized in that, The method includes: Obtain the first read transaction information of the edge data, which includes at least the starting vertex information to be queried; Define a segment space, which includes segment headers and edge regions; Based on the edge pointers of each vertex contained in the segment header, the edge region corresponding to the starting vertex is determined in the segment space. Each vertex includes the starting vertex. The edge data corresponding to the same vertex is stored in the same edge region. The edge region includes the edge block region and the edge attribute region corresponding to each vertex. The edge block region includes the edge block corresponding to the vertex. The edge block is used to store the end vertex information of the edge. Read edge data from the edge region corresponding to the starting vertex.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method of claim 20 or 21.

23. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method of claim 20 or 21.

Images