Large-scale time-varying graph storage method and system based on snapshot similarity

Abstract

The present invention relates to a large-scale time-varying graph storage method and system based on snapshot similarity, the method comprising the following steps: step S1, representing the time-varying graph based on snapshot union and snapshot bitmap; step S2, establishing a time-varying graph database; step S3, updating the time-varying graph database; step S4, loading multiple snapshots into memory based on snapshot union deletion. The method and system are not only conducive to reducing the space required for storing the time-varying graph and improving the storage efficiency of the time-varying graph, but also conducive to reducing the memory space required for loading the time-varying graph and improving the utilization rate of the memory space.

Description

Technical Field

[0001] The present invention belongs to the technical field of time-varying graph processing, and in particular relates to a large-scale time-varying graph storage method and system based on snapshot similarity. Background Art

[0002] With the widespread popularity of e-commerce and social networks, the application demand for time-varying graph computing and storage is constantly developing. A time-varying graph consists of multiple snapshots, each of which represents the state of a certain point in time during the evolution of the graph data structure. By analyzing the internal relationship between the snapshots of the time-varying graph, we can predict the future development trend of the real world and provide decision support for various fields such as national security, finance, new retail, social networking, government, and enterprises.

[0003] Due to the complexity of graph data, the use of in-memory graph computing mode can achieve higher algorithm execution efficiency. The existing method is to load multiple snapshots into memory at the same time, allocate independent storage space for each snapshot, and then optimize the execution efficiency of the algorithm for multiple snapshots to shorten the execution time of the time-varying graph algorithm. This method stores snapshots independently, so the snapshot access speed is fast, but duplicate vertices and edges are stored, resulting in low memory utilization; one method is to use the "baseline snapshot + incremental snapshot" storage method, the first snapshot is stored in the memory first, that is, all vertices and edges of the baseline snapshot, and the remaining incremental snapshots only store vertices and edges that are different from the baseline snapshot; and the other method uses the "baseline snapshot + log" storage method, the first snapshot is stored in the memory first, that is, all vertices and edges of the baseline snapshot, and each snapshot log records the vertices and edges added or deleted in the process of the snapshot evolving from the previous snapshot to the current snapshot.

[0004] However, since the multiple snapshots that make up the time-varying graph are highly similar, that is, there are a large number of identical vertices and edges between adjacent snapshots, vertices and edges are stored repeatedly. Therefore, whether storing the time-varying graph in external memory or loading some snapshots of the time-varying graph into memory for calculation, a lot of unnecessary storage space is wasted, greatly reducing storage utilization. When using the "baseline snapshot + incremental snapshot" or "baseline snapshot + log" method, when an external business system needs to access a snapshot, it is necessary to traverse all incrementally stored vertices and edges (or logs) between the snapshot and the baseline snapshot to obtain the required snapshot, and the snapshot reconstruction speed is slow.

[0005] In view of the above problems of low storage utilization and slow snapshot reconstruction speed in storing and loading multiple snapshots of time-varying graphs, there is an urgent need for an efficient storage method and system for large-scale time-varying graphs based on snapshot similarity. Summary of the invention

[0006] The purpose of the present invention is to provide a large-scale time-varying graph storage method and system based on snapshot similarity, which method and system are not only conducive to reducing the space required for storing time-varying graphs and improving the storage efficiency of time-varying graphs, but also conducive to reducing the memory space required for loading time-varying graphs and improving the utilization of memory space.

[0007] To achieve the above object, the technical solution adopted by the present invention is: a large-scale time-varying graph storage method based on snapshot similarity, comprising the following steps:

[0008] Step S1, representing a time-varying graph based on a snapshot union and a snapshot bitmap;

[0009] Step S2, establishing a time-varying graph database;

[0010] Step S3, updating the time-varying graph database;

[0011] Step S4: Loading multiple snapshots into memory based on snapshot union reduction.

[0012] Furthermore, the implementation method of step S1 is:

[0013] A time-varying graph is composed of multiple snapshots, each of which represents a state of the time-varying graph at a certain point in time. The method first performs a union operation on multiple snapshots in the time-varying graph to obtain a snapshot union of the time-varying graph, which includes all vertices and edges that have appeared in multiple snapshots of the time-varying graph, and each object, that is, each vertex or edge is stored only once and will not be stored repeatedly; in the process of creating the snapshot union, a corresponding bitmap consisting of N bits is created for each object, including creating a point bitmap for vertices and an edge bitmap for edges, where the number of bits N in the bitmap is equal to the total number of snapshots, and each bit of the object bitmap occupies one bit of storage space; the existence bit method is used to indicate whether an object belongs to a certain snapshot, and for the i-th bit of an object bitmap, 1≤i≤N, "0" indicates that the object does not belong to the i-th snapshot, and "1" indicates that the object belongs to the i-th snapshot.

[0014] Furthermore, the step S2 specifically includes the following steps:

[0015] Step A1: Obtain baseline snapshot and incremental snapshot data;

[0016] Step A2, creating a corresponding time-varying graph directory;

[0017] Step A3, creating vertices and their bitmap files according to the baseline snapshot;

[0018] Step A4: Create edges and their bitmap files according to the baseline snapshot.

[0019] Furthermore, the implementation method of step A1 is:

[0020] In the data set provided by the external business system, the first snapshot is stored in the external memory in the form of an edge list, and the remaining snapshots are stored in the form of snapshot incremental data; the first snapshot and the snapshot incremental data are sorted; to reduce the sorting cost, the snapshot degree file is used as an auxiliary structure. The snapshot degree file records the degree of each vertex in the current snapshot. The source vertices in the edge list of the first snapshot are first sorted, and the edge list space is pre-allocated for each source vertex in the snapshot according to the vertex degree. Then, the target vertices with the same source vertex are sorted, so as to obtain a snapshot edge list file with ordered source vertices and ordered target vertices, which is selected as the baseline snapshot;

[0021] The snapshot incremental files include snapshot vertex incremental files and snapshot edge incremental files, which are used to describe the addition and deletion of vertices or edges in the new snapshot edge list with reference to the previous snapshot edge list. The snapshot vertex incremental files directly sort the vertices, while the snapshot edge incremental files also use the snapshot degree file as an auxiliary structure for sorting.

[0022] After sorting is completed, the baseline snapshot and snapshot incremental files are stored in external memory;

[0023] The implementation method of step A3 is:

[0024] Create vertex and bitmap files in the corresponding time-varying graph directory, access the baseline snapshot edge list file sequentially, create a vertex data for each vertex in the baseline snapshot, and write the vertex data into the vertex and bitmap files. The vertex data contains a unique ID assigned to each vertex and a bitmap. The number of bits in the vertex bitmap is 1, and the first bit is set to "1", indicating that the vertex belongs to the first snapshot.

[0025] Repeat the above operation until the end of the baseline snapshot edge list file is accessed, and then the operation ends. The vertices and bitmap files of the baseline snapshot are successfully created and stored in the external memory.

[0026] The implementation method of step A4 is:

[0027] After creating the vertices and their bitmaps, create edges and their bitmap files in the corresponding time-varying graph directory, access the baseline snapshot edge list file sequentially, create an edge data for each edge in the baseline snapshot, and write the edge data into the edge and its bitmap file. The edge data contains the source vertex ID, target vertex ID, edge weight, and edge bitmap of the edge. The number of bits in the edge bitmap is 1, and the first bit is set to "1", indicating that the edge belongs to the first snapshot.

[0028] Repeat the above operation until the end of the baseline snapshot edge list file is accessed, and then end the operation. The baseline snapshot edge and its bitmap file are successfully created and stored in the external memory.

[0029] Furthermore, the step S3 specifically includes the following steps:

[0030] Step B1, update the vertex and its bitmap;

[0031] Step B2: Update the edge and its bitmap.

[0032] Furthermore, the implementation method of step B1 is:

[0033] Each row of data in the snapshot vertex increment file required to update the vertex and its bitmap consists of the vertex ID and the change type. When the change type is "0", it means deleting the vertex, and when it is "1", it means restoring or adding the vertex.

[0034] Assume that N snapshots are stored in the current system. Due to the addition of the N+1th new snapshot, the system opens the vertex increment file of the N+1th snapshot and the vertex and bitmap files of the snapshot union respectively, and adds the N+1th bit to the point bitmap of each vertex in the file; before performing the update operation on the vertex and bitmap files of the snapshot union, the legality of the operation is judged according to the vertices and their bitmaps; the vertex increment files are accessed sequentially. If the change type in the current data row is "0", the operation is legal only if the vertices in the data row belong to the previous snapshot, access the bitmap corresponding to the corresponding vertex, and set the N+1th bit of the bitmap to "0", indicating that the vertex is deleted in the N+1th snapshot; if the change type in the current data row is "1" and the vertex ID of the data row is If the corresponding vertex does not belong to the previous snapshot but belongs to the vertices and bitmap files of the snapshot union, the operation is legal. The bitmap corresponding to the corresponding vertex is accessed, and the N+1th bit of the bitmap is set to "1", indicating that the vertex is restored in the N+1th snapshot. If the change type in the current data row is "1" and the vertex corresponding to the vertex ID of the row of data does not belong to the vertices and bitmap files of the snapshot union, a suitable position is found in the vertex and bitmap files of the snapshot union, and a row of vertex data is inserted. The vertex ID is the vertex ID, the first N bits of the vertex bitmap are all set to "0", and the N+1th bit is set to "1", indicating that a new vertex that does not belong to the snapshot union is added to the N+1th snapshot. For vertices that have not changed, the N+1th bit of their vertex bitmap is the same as the Nth bit.

[0035] Repeat the above operation until the end of the vertex increment file is accessed, and then end the operation to obtain the vertices and bitmap files of the updated snapshot union;

[0036] The implementation method of step B2 is:

[0037] Each row of data in the snapshot edge incremental file required to update the edge and its bitmap consists of the source vertex ID, the target vertex ID, the edge weight, and the change type. When the change type is "0", it means deleting the edge, "1" means adding the edge, and "2" means restoring the edge.

[0038] Assuming that N snapshots are currently stored, due to the addition of the N+1th new snapshot, the edge incremental file of the N+1th snapshot and the edge and bitmap files of the snapshot union are opened respectively, and the N+1th bit is added to the edge bitmap of each edge in the file; before performing the update operation on the edge and bitmap files of the snapshot union, the legality of the operation is judged according to the edges and their bitmaps and the updated vertices and their bitmaps, and each row of data in the edge incremental file is accessed sequentially. Different operations are taken according to the change type of the edge. If the current change type is "0", the operation is legal only if the edge in the data row belongs to the previous snapshot. The N+1th bit of the edge bitmap of the edge is set to "0", indicating that an edge is deleted. The edge belongs to the previous snapshot; if the change type is "1" and the edge does not belong to the edge and its bitmap of the snapshot union, but the two vertices of the edge belong to the updated vertices and their bitmap, it means that the operation is legal, and the first N bits of the edge bitmap of the edge are set to "0", and the N+1th bit is set to "1", indicating that an edge that does not exist in the snapshot union is added; if the change type is "2" and the edge does not belong to the previous snapshot but belongs to the edge and its bitmap of the snapshot union, it means that the operation is legal, and the N+1th bit of the edge bitmap of the edge is set to "1", indicating that an edge that belongs to the snapshot union but does not belong to the previous snapshot is restored; for edges that have not changed, the N+1th bit of its edge bitmap is the same as the Nth bit;

[0039] Repeat the above operation until the end of the edge incremental file is accessed, end the operation, and obtain the edges and their bitmap files of the updated snapshot union.

[0040] Furthermore, the step S4 specifically includes the following steps:

[0041] Step C1, the external business system selects the required snapshot group;

[0042] Step C2: Generate a snapshot group union according to the requirements of the external business system and load it into the memory.

[0043] Furthermore, the step C2 specifically includes the following steps:

[0044] Step C21, creating a vertex data file based on snapshot union deletion;

[0045] Step C22, creating an edge data file based on snapshot union deletion;

[0046] Step C23: Merge the vertex data file and the edge data file.

[0047] Furthermore, the implementation method of step C21 is:

[0048] According to the needs of the external business system, determine the snapshot group range that needs to be accessed, select the vertex and the corresponding bit in its bitmap, and determine whether to insert the vertex data into the vertex data file based on the bitmap composed of the corresponding bits. The vertex data includes the vertex ID, vertex bitmap, vertex new value array, vertex previous value array, vertex degree, and out-edge pointer. The vertex new value array and vertex previous value data are applied for space by the external business system according to its needs.

[0049] Create a vertex data file; assume that the snapshot union contains N snapshots. If the external business system needs k snapshots from i+1 to i+k, where 1≤i+1≤i+k≤N, the system sequentially accesses the vertices and their bitmap files. Each time a vertex is accessed, the system judges the i+1 to i+k bits of the bitmap corresponding to the vertex, where 1≤i+1≤i+k≤N. If all k bits are "0", skip the vertex and continue to access the next vertex in the file; if at least one of the k bits is "1", add a line of vertex data to the vertex data file, where the vertex ID is the vertex ID of the vertex; the vertex bitmap is the k-bit bitmap mentioned above, and add Add the k+1th bit to indicate whether the vertex belongs to the snapshot intersection; the vertex new value array has k+1 data units, the first k data units are used to save the calculation results of the k snapshots in the current superstep, and the k+1th data unit is used to save the calculation results of the snapshot intersection in the current superstep; the vertex previous value array has k+1 data units, the first k data units are used to save the calculation results of the k snapshots in the previous superstep, and the k+1th data unit is used to save the calculation results of the snapshot intersection in the previous superstep; the vertex degree records the number of outgoing edges that the current vertex has in the snapshot group union, initialized to "0"; the outgoing edge pointer points to an outgoing edge list, initialized to "NULL";

[0050] Repeat the above operation until the end of the vertex and its bitmap file is accessed, and then the operation is terminated. A multi-snapshot point data file based on snapshot union reduction is successfully created and temporarily stored in the memory.

[0051] The implementation method of step C22 is:

[0052] According to the needs of the external business system, determine the snapshot group range that needs to be accessed, select the edge and the corresponding bit in its bitmap, and determine whether to insert edge data into the edge data file based on the bitmap composed of the corresponding bits. The edge data contains the source vertex ID, target vertex ID, and edge bitmap.

[0053] Create an edge data file; assume that the snapshot union contains N snapshots. If the external business system requires a total of k snapshots, from i+1 to i+k, where 1≤i+1≤i+k≤N, the system sequentially accesses the edges and their bitmap files stored in the external memory. Each time an edge is accessed, the system judges the i+1 to i+k bits of the bitmap corresponding to the edge, where 1≤i+1≤i+k≤N. If all k bits are "0", skip the edge and continue to access the next edge in the file; if at least one of the k bits is "1", add a row of edge data to the edge data file, where the source vertex ID is the vertex ID of the source point of the edge; the target vertex ID is the vertex ID of the target vertex of the edge; the edge bitmap is the k-bit bitmap mentioned above, and the k+1th bit is added to it to indicate whether the edge belongs to the snapshot intersection;

[0054] Repeat the above operation until the end of the edge and its bitmap file is accessed, and then the operation is terminated. The multi-snapshot edge data file based on snapshot union deletion is successfully created and temporarily stored in the memory.

[0055] The implementation method of step C23 is:

[0056] Since the edge list file in the external memory is ordered, the edges with the same source vertex in the edge data file are continuous; after writing the edge data file, initialize an access pointer pointing to the first line of the edge data file, and then sequentially access the vertex data file that has been created and already exists in the memory. When accessing each vertex data, start to sequentially access the edge data file in the memory from the position pointed by the access pointer, point the outgoing edge pointer of the vertex data to the first outgoing edge in the edge data file with the vertex as the source vertex, and count the number of adjacent outgoing edges with the same source, which is set as the degree of the vertex in the vertex data in the snapshot group union, until the outgoing edge with another vertex as the source vertex is accessed, use the access pointer to temporarily store the access position, suspend access to the edge data file, and continue to access the next vertex data;

[0057] Repeat the above operation until the end of the edge and its bitmap file is accessed, and then end the operation. The vertex data file and the edge data file in the memory are successfully merged to form a snapshot group union, that is, multiple snapshots based on snapshot union deletion; complete the memory loading of the snapshot group required by the external business system.

[0058] The present invention also provides a large-scale time-varying graph storage system based on snapshot similarity, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0059] Compared with the prior art, the present invention has the following beneficial effects: it provides a large-scale time-varying graph storage method and system based on snapshot similarity. The method uses a time-varying graph representation method based on snapshot union and snapshot bitmap, utilizes the high similarity between the snapshots that make up the time-varying graph, and stores multiple snapshots in the time-varying graph database in the form of snapshot union and multi-snapshot bitmap, thereby reducing the storage overhead of the time-varying graph and improving storage utilization. Based on this method, an efficient time-varying graph storage system is implemented. The system sorts the data sets (snapshot edge list files and snapshot incremental files) provided by the external business system and selects the baseline snapshot from them; establishes a time-varying graph database, loads the baseline snapshot and multiple snapshot incremental data into the system, obtains the snapshot union and multiple snapshot bitmaps after processing, stores the snapshot union and multiple snapshot bitmaps, and completes the establishment of the time-varying graph database; updates the time-varying graph database, the external business system provides snapshot incremental files, the system processes the incremental files accordingly, merges them with the snapshot union in the time-varying graph database, and completes the addition and storage of new snapshots; according to the needs of the external business system, the required snapshot bitmap can be obtained from the time-varying graph database, the snapshot union is operated according to the bitmap, and the required snapshot is quickly loaded into the memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a schematic diagram of the system architecture of an embodiment of the present invention;

[0061] Figure 2 is a flow chart of converting a time-varying graph into a snapshot union in an embodiment of the present invention;

[0062] Figure 3 is a structural diagram of a snapshot union bitmap in an embodiment of the present invention;

[0063] Figure 4 is a flow chart of sorting a snapshot edge list according to a degree file in an embodiment of the present invention;

[0064] Figure 5 is a flow chart of memory loading based on snapshot union deletion in an embodiment of the present invention;

[0065] Figure 6 It is a data structure diagram of the snapshot group loaded into the memory in the embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0067] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0068] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0069] This embodiment provides a method for efficiently storing a large-scale time-varying graph based on snapshot similarity, and the method includes the following steps:

[0070] Step S1, representing a time-varying graph based on a snapshot union and a snapshot bitmap;

[0071] Step S2, establishing a time-varying graph database;

[0072] Step S3, updating the time-varying graph database;

[0073] Step S4: Loading multiple snapshots into memory based on snapshot union reduction.

[0074] Accordingly, this embodiment provides a large-scale time-varying graph efficient storage system based on snapshot similarity (hereinafter referred to as TgStore), and its architecture diagram is as follows: Figure 1 The system includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the above method steps can be implemented.

[0075] like Figure 2 , 3 As shown, the implementation method of step S1 is:

[0076] The time-varying graph consists of multiple snapshots, such as Figure 2 As shown, each snapshot represents a state of the time-varying graph at a certain point in time (equivalent to a static graph). The basic research results of the proposers of the present invention on different types of time-varying graphs show that adjacent static snapshots are highly similar, that is, there are a large number of identical vertices and edges between two adjacent snapshots.

[0077] If a time-varying graph is represented in a traditional way, and if a time-varying graph algorithm is to be executed and analyzed on multiple snapshots in the time-varying graph, a large number of servers (from dozens to tens of thousands) must be used to increase the memory capacity in order to load large-scale multiple snapshots into the memory. The present invention proposes a time-varying graph representation method based on snapshot union and snapshot bitmap, which eliminates redundant vertices and edges between adjacent snapshots in the time-varying graph, while maximally saving computing costs, reducing the waste of storage space and improving storage utilization.

[0078] In TgStore, a time-varying graph representation method based on snapshot union and snapshot bitmap is proposed. Figure 2 As shown, the method first performs a "union" operation on multiple snapshots in the time-varying graph to obtain the union of the time-varying graph snapshots, which includes all vertices and edges that have appeared in the multiple snapshots of the time-varying graph, and each object (vertex or edge) is stored only once and will not be stored repeatedly.

[0079] like Figure 3 As shown in the figure, in the process of creating a snapshot union, TgStore creates a corresponding bitmap consisting of N bits for each object (a point bitmap is created for a vertex, and an edge bitmap is created for an edge), where the number of bits N in the bitmap is equal to the total number of snapshots, and each bit of the object bitmap occupies a storage space of one bit. TgStore uses the method of existence bits to indicate whether an object (vertex or edge, here vertex) belongs to a certain snapshot. For the i-th (1≤i≤N) bit of an object bitmap, "0" means that the object does not belong to the i-th snapshot, and "1" means that the object belongs to the i-th snapshot.

[0080] The step S2 specifically includes the following steps:

[0081] Step A1: Obtain baseline snapshot and incremental snapshot data;

[0082] Step A2, create a corresponding time-varying graph directory in TgStore;

[0083] Step A3, creating vertices and their bitmap files according to the baseline snapshot;

[0084] Step A4: Create edges and their bitmap files according to the baseline snapshot.

[0085] like Figure 4 As shown, the implementation method of step A1 is:

[0086] In the data set provided by the external business system, the first snapshot is stored in the external memory in the form of an edge list, and the remaining snapshots are stored in the form of snapshot incremental data. To facilitate the execution of subsequent steps, TgStore sorts the first snapshot and snapshot incremental data.

[0087] To reduce the sorting cost, Figure 4 As shown in the figure, TgStore uses the auxiliary structure of "snapshot degree file". The snapshot degree file records the degree of each vertex in the current snapshot. First, the source vertices in the first snapshot edge list are sorted, and the edge list space is pre-allocated for each source vertex in the snapshot according to the vertex degree. Then, the target vertices with the same source vertex are sorted, so as to obtain a snapshot edge list file with ordered source vertices and ordered target vertices. TgStore selects it as the baseline snapshot.

[0088] The snapshot incremental file includes the snapshot vertex incremental file and the snapshot edge incremental file. The snapshot vertex / edge incremental file describes the addition and deletion of vertices / edges in the new snapshot edge list with reference to the previous snapshot edge list. The snapshot vertex incremental file directly sorts the vertices. The snapshot edge incremental file also uses the auxiliary structure of the "snapshot degree file" for sorting.

[0089] After sorting is completed, the baseline snapshot and snapshot incremental files are stored in external memory.

[0090] The implementation method of step A3 is:

[0091] TgStore creates vertices and their bitmap files in the corresponding time-varying graph directory, sequentially accesses the baseline snapshot edge list file, creates a vertex data for each vertex in the baseline snapshot, and writes the vertex data to the vertex and its bitmap files. The vertex data contains a unique ID assigned to each vertex and a bitmap (since only one snapshot union is currently stored in TgStore, the number of bits in the vertex bitmap is 1), and sets the first bit to "1", indicating that the vertex belongs to the first snapshot.

[0092] Repeat the above operation until the end of the baseline snapshot edge list file is accessed, and then the operation is terminated. The vertices and bitmap files of the baseline snapshot are successfully created and stored in the external memory.

[0093] The implementation method of step A4 is:

[0094] After TgStore creates the vertices and their bitmaps, it creates edges and their bitmap files in the corresponding time-varying graph directory, sequentially accesses the baseline snapshot edge list file, creates an edge data for each edge in the baseline snapshot, and writes the edge data into the edge and its bitmap files. The edge data includes the source vertex ID, target vertex ID, edge weight, and edge bitmap of the edge (since only one snapshot union is stored in the current TgStore, the number of bits in the edge bitmap is 1), and sets the first bit to "1", indicating that the edge belongs to the first snapshot.

[0095] Repeat the above operation until the end of the baseline snapshot edge list file is accessed, and then end the operation. The baseline snapshot edge and its bitmap file are successfully created and stored in the external memory.

[0096] The step S3 specifically comprises the following steps:

[0097] Step B1, update the vertex and its bitmap;

[0098] Step B2: Update the edge and its bitmap.

[0099] The implementation method of step B1 is:

[0100] Each row of data in the snapshot vertex increment file required to update the vertex and its bitmap consists of a vertex ID and a change type. When the change type is "0", it means deleting the vertex, and when it is "1", it means restoring or adding the vertex.

[0101] Assuming that N snapshots are currently stored in TgStore, due to the addition of the N+1th new snapshot, TgStore opens the vertex increment file of the N+1th snapshot and the vertex and bitmap files of the snapshot union, and adds the N+1th bit to the point bitmap of each vertex in the file. Before TgStore performs an update operation on the vertex and bitmap files of the snapshot union, it needs to determine the legality of the operation based on the vertices and their bitmaps. Access the vertex incremental file sequentially. If the change type in the current data row is "0", the operation is legal only when the vertex in the data row belongs to the previous snapshot. Access the bitmap corresponding to the corresponding vertex and set the N+1th bit of the bitmap to "0", indicating that the vertex is deleted in the N+1th snapshot. If the change type in the current data row is "1" and the vertex corresponding to the vertex ID of the data row does not belong to the previous snapshot but belongs to the vertices and bitmap files of the snapshot union, the operation is legal. Access the bitmap corresponding to the corresponding vertex and set the N+1th bit of the bitmap to "1", indicating that the vertex is restored in the N+1th snapshot. If the change type in the current data row is "1" and the vertex corresponding to the vertex ID of the data row does not belong to the vertices and bitmap files of the snapshot union, TgStore finds a suitable position in the vertex and bitmap files of the snapshot union, inserts a row of vertex data, the vertex ID is the vertex ID, the first N bits of the vertex bitmap are all set to "0", and the N+1th bit is set to "1", indicating that a new vertex that does not belong to the snapshot union is added to the N+1th snapshot. For vertices that have not changed, the N+1th bit in the vertex bitmap is the same as the Nth bit.

[0102] Repeat the above operation until the end of the vertex increment file is accessed, and then end the operation to obtain the vertices and bitmap files of the updated snapshot union.

[0103] The implementation method of step B2 is:

[0104] Each line of data in the snapshot edge incremental file required to update the edge and its bitmap consists of the source vertex ID, the target vertex ID, the edge weight, and the change type. When the change type is "0", it means deleting the edge, "1" means adding the edge, and "2" means restoring the edge.

[0105] Assuming that N snapshots are currently stored in TgStore, due to the addition of the N+1th new snapshot, TgStore opens the edge incremental file of the N+1th snapshot and the edges and their bitmap files of the snapshot union respectively, and adds the N+1th bit to the edge bitmap of each edge in the file. Before TgStore performs an update operation on the edges and their bitmap files of the snapshot union, it needs to determine the legality of the operation based on the edges and their bitmaps and the updated vertices and their bitmaps, sequentially access each row of data in the edge increment file, and take different operations according to the change type of the edge. If the current change type is "0", the operation is legal only if the edges in the data row belong to the previous snapshot. The N+1th bit of the edge bitmap of the edge is set to "0", indicating that an edge belonging to the previous snapshot is deleted; if the change type is "1" and the edge does not belong to the edges and their bitmaps of the snapshot union but the two vertices of the edge belong to the updated vertices and their bitmaps, it means that the operation is legal, the first N bits of the edge bitmap of the edge are all set to "0", and the N+1th bit is set to "1", indicating that an edge that does not exist in the snapshot union is added; if the change type is "2" and the edge does not belong to the previous snapshot but belongs to the edges and their bitmaps of the snapshot union, it means that the operation is legal, and the N+1th bit of the edge bitmap of the edge is set to "1", indicating that an edge that belongs to the snapshot union but not to the previous snapshot is restored. For edges that have not changed, the N+1th bit in the edge bitmap is the same as the Nth bit.

[0106] Repeat the above operation until the end of the edge incremental file is accessed, end the operation, and obtain the edges and their bitmap files of the updated snapshot union.

[0107] The step S4 specifically comprises the following steps:

[0108] Step C1, the external business system selects the required snapshot group;

[0109] Step C2: Generate a snapshot group union according to the requirements of the external business system and load it into the memory.

[0110] Wherein, the step C2 specifically includes the following steps:

[0111] Step C21, creating a vertex data file based on snapshot union deletion;

[0112] Step C22, creating an edge data file based on snapshot union deletion;

[0113] Step C23: Merge the vertex data file and the edge data file.

[0114] The implementation method of step C21 is:

[0115] According to the needs of the external business system, TgStore determines the range of snapshot groups that need to be accessed, selects vertices and their corresponding bits in the bitmap, and determines whether to insert vertex data into the vertex data file based on the bitmap composed of the corresponding bits. The vertex data contains vertex ID, vertex bitmap, vertex new value array, vertex previous value array, vertex degree, and out-edge pointer. The new vertex value array and vertex previous value data are applied for by the external business system according to its own needs.

[0116] TgStore creates a vertex data file. Figure 5 As shown in the figure, assuming that the snapshot union contains N snapshots, if the external business system needs the i+1 to i+k (1≤i+1≤i+k≤N) snapshots, a total of k snapshots, TgStore sequentially accesses the vertex and its bitmap file. Each time a vertex is accessed, the system judges the i+1 to i+k (1≤i+1≤i+k≤N) bits of the bitmap corresponding to the vertex. If all k bits are "0", the vertex is skipped and the next vertex in the file is accessed. If at least one of the k bits is "1", a line of vertex data is added to the vertex data file, such as Figure 6 As shown on the left, the vertex ID is the vertex ID of the vertex; the vertex bitmap is the k-bit bitmap mentioned above, and the k+1th bit is added to it to indicate whether the vertex belongs to the snapshot intersection; the vertex new value array has k+1 data units, the first k data units are used to save the calculation results of the k snapshots in the current superstep, and the k+1th data unit is used to save the calculation results of the snapshot intersection in the current superstep; the vertex previous value array has k+1 data units, the first k data units are used to save the calculation results of the k snapshots in the previous superstep, and the k+1th data unit is used to save the calculation results of the snapshot intersection in the previous superstep; the vertex degree records the number of outgoing edges that the current vertex has in the snapshot group union, initialized to "0"; the outgoing edge pointer points to an outgoing edge list, initialized to "NULL".

[0117] Repeat the above operation until the end of the vertex and its bitmap file is accessed, and then the operation is terminated. A multi-snapshot point data file based on snapshot union reduction is successfully created and temporarily stored in the memory.

[0118] The implementation method of step C22 is:

[0119] According to the needs of the external business system, TgStore determines the scope of the snapshot group that needs to be accessed, selects the edge and the corresponding bit in its bitmap, and determines whether to insert the edge data into the edge data file based on the bitmap composed of the corresponding bits. The edge data contains the source vertex ID, target vertex ID, and edge bitmap.

[0120] TgStore creates a side data file. Figure 5As shown in the figure, assuming that the snapshot union contains N snapshots, if the external business system needs the i+1 to i+k (1≤i+1≤i+k≤N) snapshots, a total of k snapshots, TgStore sequentially accesses the edges and their bitmap files stored in the external memory. Each time an edge is accessed, the system judges the i+1 to i+k (1≤i+1≤i+k≤N) bits of the bitmap corresponding to the edge. If all k bits are "0", skip the edge and continue to access the next edge in the file. If at least one of the k bits is "1", add a row of edge data to the edge data file, such as Figure 6 As shown on the right, the source vertex ID is the vertex ID of the source point of the edge; the target vertex ID is the vertex ID of the target vertex of the edge; the edge bitmap is the k-bit bitmap mentioned above, with the k+1th bit added to it to indicate whether the edge belongs to the snapshot intersection.

[0121] Repeat the above operation until the end of the edge and its bitmap file is accessed, and then end the operation. The multi-snapshot edge data file based on snapshot union deletion is successfully created and temporarily stored in the memory.

[0122] The implementation method of step C23 is:

[0123] Since the edge list file in the external memory is ordered, the edges with the same source vertex in the edge data file are continuous. After TgStore finishes writing the edge data file, it initializes an access pointer pointing to the first line of the edge data file, and then sequentially accesses the vertex data file that has been created and already exists in the memory. Each time a vertex data is accessed, TgStore starts to sequentially access the edge data file in the memory from the position pointed to by the access pointer, such as Figure 6 As shown, the outgoing edge pointer of the vertex data points to the first outgoing edge with the vertex as the source vertex in the edge data file, and the number of adjacent outgoing edges with the same source is counted, which is set as the degree of the vertex in the vertex data in the snapshot group union, until the outgoing edge with another vertex as the source vertex is accessed, the access pointer is used to temporarily store the access position, the access to the edge data file is suspended, and the access to the next vertex data continues.

[0124] Repeat the above operation until the end of the edge and its bitmap file is accessed, and the operation is terminated. The vertex data file in the memory is successfully merged with the edge data file to form a snapshot group union, that is, multiple snapshots based on snapshot union deletion. The memory loading of the snapshot group required by the external business system is completed.

[0125] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0126] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0129] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A large-scale time-varying graph storage method based on snapshot similarity, characterized in that: The following steps are involved: Step S1, representing a time-varying graph based on a snapshot union and a snapshot bitmap; Step S2, establishing a time-varying graph database; Step S3, updating the time-varying graph database; Step S4, multiple snapshots are loaded into memory based on snapshot union deletion; The implementation method of step S1 is: The time-varying graph is composed of multiple snapshots, each of which represents a state of the time-varying graph at a certain point in time. The method first performs a union operation on multiple snapshots in the time-varying graph to obtain a snapshot union of the time-varying graph, which includes all vertices and edges that have appeared in multiple snapshots of the time-varying graph, and each object, that is, each vertex or edge is stored only once and will not be stored repeatedly; in the process of creating the snapshot union, a corresponding bitmap consisting of N bits is created for each object, including creating a point bitmap for vertices and an edge bitmap for edges, wherein the number of bits N of the bitmap is equal to the total number of snapshots, and each bit of the object bitmap occupies a storage space of one bit; the existence bit method is used to indicate whether an object belongs to a certain snapshot, and for the i-th bit of an object bitmap, 1≤i≤N, "0" indicates that the object does not belong to the i-th snapshot, and "1" indicates that the object belongs to the i-th snapshot; The step S2 specifically includes the following steps: Step A1: Obtain baseline snapshot and incremental snapshot data; Step A2, creating a corresponding time-varying graph directory; Step A3, creating vertices and their bitmap files according to the baseline snapshot; Step A4, creating edges and their bitmap files according to the baseline snapshot; The implementation method of step A1 is: In the data set provided by the external business system, the first snapshot is stored in the external memory in the form of an edge list, and the remaining snapshots are stored in the form of snapshot incremental data; the first snapshot and the snapshot incremental data are sorted; to reduce the sorting cost, the snapshot degree file is used as an auxiliary structure. The snapshot degree file records the degree of each vertex in the current snapshot. The source vertices in the edge list of the first snapshot are first sorted, and the edge list space is pre-allocated for each source vertex in the snapshot according to the vertex degree. Then, the target vertices with the same source vertex are sorted, so as to obtain a snapshot edge list file with ordered source vertices and ordered target vertices, which is selected as the baseline snapshot; The snapshot incremental files include snapshot vertex incremental files and snapshot edge incremental files, which are used to describe the addition and deletion of vertices or edges in the new snapshot edge list with reference to the previous snapshot edge list. The snapshot vertex incremental files directly sort the vertices, while the snapshot edge incremental files also use the snapshot degree file as an auxiliary structure for sorting. After sorting is completed, the baseline snapshot and snapshot incremental files are stored in external memory; The implementation method of step A3 is: Create vertex and bitmap files in the corresponding time-varying graph directory, access the baseline snapshot edge list file sequentially, create a vertex data for each vertex in the baseline snapshot, and write the vertex data into the vertex and bitmap files. The vertex data contains a unique ID and a bitmap assigned to each vertex. Since only one snapshot union is stored in the current system, the number of bits in the vertex bitmap is 1, and the first bit is set to "1", indicating that the vertex belongs to the first snapshot. Repeat the above operation until the end of the baseline snapshot edge list file is accessed, and then the operation ends. The vertices and bitmap files of the baseline snapshot are successfully created and stored in the external memory. The implementation method of step A4 is: After creating the vertices and their bitmaps, create edges and their bitmap files in the corresponding time-varying graph directory, access the baseline snapshot edge list file sequentially, create an edge data for each edge in the baseline snapshot, and write the edge data into the edge and its bitmap file. The edge data contains the source vertex ID, target vertex ID, edge weight, and edge bitmap of the edge. Since only one snapshot union is stored in the current system, the number of bits in the edge bitmap is 1, and the first bit is set to "1", indicating that the edge belongs to the first snapshot; Repeat the above operation until the end of the baseline snapshot edge list file is accessed, and then the operation is terminated. The baseline snapshot edge and its bitmap file are successfully created and stored in the external memory. The step S3 specifically comprises the following steps: Step B1, update the vertex and its bitmap; Step B2, update the edge and its bitmap; The implementation method of step B1 is: Each row of data in the snapshot vertex increment file required to update the vertex and its bitmap consists of the vertex ID and the change type. When the change type is "0", it means deleting the vertex, and when it is "1", it means restoring or adding the vertex. Assuming that N snapshots are stored in the current system, due to the addition of the N+1th new snapshot, the system opens the vertex increment file of the N+1th snapshot and the vertex and bitmap files of the snapshot union respectively, and adds the N+1th bit to the point bitmap of each vertex in the file; before performing the update operation on the vertex and bitmap files of the snapshot union, the legality of the operation is judged according to the vertices and their bitmaps; sequentially access the vertex increment files, if the change type in the current data row is "0", the operation is legal only if the vertices in the data row belong to the previous snapshot, access the bitmap corresponding to the corresponding vertex, and set the N+1th bit of the bitmap to "0", indicating that the vertex is deleted in the N+1th snapshot; if the change type in the current data row is "1" and the data in the row is If the vertex corresponding to the vertex ID does not belong to the previous snapshot but belongs to the vertices and bitmap files of the snapshot union, the operation is legal. The bitmap corresponding to the corresponding vertex is accessed and the N+1th bit of the bitmap is set to "1", indicating that the vertex is restored in the N+1th snapshot. If the change type in the current data row is "1" and the vertex corresponding to the vertex ID of the row of data does not belong to the vertices and bitmap files of the snapshot union, a suitable position is found in the vertex and bitmap files of the snapshot union, and a row of vertex data is inserted. The first N bits of the vertex bitmap are all set to "0" and the N+1th bit is set to "1", indicating that a new vertex that does not belong to the snapshot union is added to the N+1th snapshot. For vertices that have not changed, the N+1th bit of their vertex bitmap is the same as the Nth bit. Repeat the above operation until the end of the vertex increment file is accessed, and then end the operation to obtain the vertices and bitmap files of the updated snapshot union; The implementation method of step B2 is: Each row of data in the snapshot edge incremental file required to update the edge and its bitmap consists of the source vertex ID, the target vertex ID, the edge weight, and the change type. When the change type is "0", it means deleting the edge, "1" means adding the edge, and "2" means restoring the edge. Assuming that N snapshots are currently stored, due to the addition of the N+1th new snapshot, the edge incremental file of the N+1th snapshot and the edge and bitmap files of the snapshot union are opened respectively, and the N+1th bit is added to the edge bitmap of each edge in the file; before performing the update operation on the edge and bitmap files of the snapshot union, the legality of the operation is judged according to the edges and their bitmaps and the updated vertices and their bitmaps, and each row of data in the edge incremental file is accessed sequentially. Different operations are taken according to the change type of the edge. If the current change type is "0", the operation is legal only if the edge in the data row belongs to the previous snapshot. The N+1th bit of the edge bitmap of the edge is set to "0", indicating that an edge is deleted. The edge belongs to the previous snapshot; if the change type is "1" and the edge does not belong to the edge and its bitmap of the snapshot union, but the two vertices of the edge belong to the updated vertices and their bitmap, it means that the operation is legal, and the first N bits of the edge bitmap of the edge are set to "0", and the N+1th bit is set to "1", indicating that an edge that does not exist in the snapshot union is added; if the change type is "2" and the edge does not belong to the previous snapshot but belongs to the edge and its bitmap of the snapshot union, it means that the operation is legal, and the N+1th bit of the edge bitmap of the edge is set to "1", indicating that an edge that belongs to the snapshot union but does not belong to the previous snapshot is restored; for edges that have not changed, the N+1th bit of its edge bitmap is the same as the Nth bit; Repeat the above operation until the end of the edge increment file is accessed, and then end the operation to obtain the edge and bitmap file of the updated snapshot union; The step S4 specifically comprises the following steps: Step C1, the external business system selects the required snapshot group; Step C2: Generate a snapshot group and set based on the requirements of the external business system and load it into the memory; The step C2 specifically comprises the following steps: Step C21, creating a vertex data file based on snapshot union deletion; Step C22, creating an edge data file based on snapshot union deletion; Step C23, merging the vertex data file and the edge data file; The implementation method of step C21 is: According to the needs of the external business system, determine the snapshot group range that needs to be accessed, select the vertex and the corresponding bit in its bitmap, and determine whether to insert the vertex data into the vertex data file based on the bitmap composed of the corresponding bits. The vertex data includes the vertex ID, vertex bitmap, vertex new value array, vertex previous value array, vertex degree, and out-edge pointer. The vertex new value array and vertex previous value data are applied for space by the external business system according to its needs. Create a vertex data file; assume that the snapshot union contains N snapshots. If the external business system needs k snapshots from i+1 to i+k, where 1≤i+1≤i+k≤N, the system sequentially accesses the vertices and their bitmap files. Each time a vertex is accessed, the system judges the i+1 to i+k bits of the bitmap corresponding to the vertex, where 1≤i+1≤i+k≤N. If all k bits are "0", skip the vertex and continue to access the next vertex in the file; if at least one of the k bits is "1", add a line of vertex data to the vertex data file, where the vertex ID is the vertex ID of the vertex; the vertex bitmap is the k-bit bitmap corresponding to the vertex, and Add the k+1th bit to indicate whether the vertex belongs to the snapshot intersection; the vertex new value array has k+1 data units, the first k data units are used to save the calculation results of the k snapshots in the current superstep, and the k+1th data unit is used to save the calculation results of the snapshot intersection in the current superstep; the vertex previous value array has k+1 data units, the first k data units are used to save the calculation results of the k snapshots in the previous superstep, and the k+1th data unit is used to save the calculation results of the snapshot intersection in the previous superstep; the vertex degree records the number of outgoing edges that the current vertex has in the snapshot group union, initialized to "0"; the outgoing edge pointer points to an outgoing edge list, initialized to "NULL"; Repeat the above operation until the end of the vertex and its bitmap file is accessed, and then the operation is terminated. A multi-snapshot point data file based on snapshot union reduction is successfully created and temporarily stored in the memory. The implementation method of step C22 is: According to the needs of the external business system, determine the snapshot group range that needs to be accessed, select the edge and the corresponding bit in its bitmap, and determine whether to insert edge data into the edge data file based on the bitmap composed of the corresponding bits. The edge data contains the source vertex ID, target vertex ID, and edge bitmap. Create an edge data file; assume that the snapshot union contains N snapshots. If the external business system requires a total of k snapshots, from i+1 to i+k, where 1≤i+1≤i+k≤N, the system sequentially accesses the edges and their bitmap files stored in the external memory. Each time an edge is accessed, the system judges the i+1 to i+k bits of the bitmap corresponding to the edge, where 1≤i+1≤i+k≤N. If all k bits are "0", skip the edge and continue to access the next edge in the file; if at least one of the k bits is "1", add a row of edge data to the edge data file, where the source vertex ID is the vertex ID of the source point of the edge; the target vertex ID is the vertex ID of the target vertex of the edge; the edge bitmap is the k-bit bitmap corresponding to the edge, and the k+1th bit is added to it to indicate whether the edge belongs to the snapshot intersection; Repeat the above operation until the end of the edge and its bitmap file is accessed, and then the operation is terminated. The multi-snapshot edge data file based on snapshot union deletion is successfully created and temporarily stored in the memory. The implementation method of step C23 is: Since the edge list file in the external memory is ordered, the edges with the same source vertex in the edge data file are continuous; after writing the edge data file, initialize an access pointer pointing to the first line of the edge data file, and then sequentially access the vertex data file that has been created and exists in the memory. When accessing each vertex data, start to sequentially access the edge data file in the memory from the position pointed by the access pointer, point the outgoing edge pointer of the vertex data to the first outgoing edge in the edge data file with the vertex as the source vertex, and count the number of adjacent outgoing edges with the same source, which is set as the degree of the vertex in the vertex data in the snapshot group union, until the outgoing edge with another vertex as the source vertex is accessed, use the access pointer to temporarily store the access position, suspend access to the edge data file, and continue to access the next vertex data; Repeat the above operation until the end of the edge and its bitmap file is accessed, and then end the operation. The vertex data file and the edge data file in the memory are successfully merged to form a snapshot group union, that is, multiple snapshots based on snapshot union deletion; complete the memory loading of the snapshot group required by the external business system.

2. A large-scale time-varying graph storage system based on snapshot similarity, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in claim 1 can be implemented.

Images